Terminal device

Abstract

A terminal device according to the present invention, which is a display control device that causes a head-mounted transmission display device to display an image concerning a virtual space VS, is provided with: an acquisition unit that acquires the number of unread messages addressed to the terminal device when the distance between the display device and the terminal device in a shared space shared by the display control device and the terminal device is within a prescribed value; and a display control unit that causes the display device to display a virtual object corresponding to the number of unread messages.

Description

[Technical Field]

[0001] This invention relates to a terminal device. In particular, this invention relates to a terminal device for determining whether or not there are unread messages addressed to other terminal devices. [Background technology]

[0002] In Augmented Reality (AR) technology, the real-world environment perceived by the user is augmented through computer-aided means. This technology makes it possible, for example, to add information displayed in a virtual space to the real-world space viewed by the user through AR glasses worn on their head.

[0003] For example, Patent Document 1 discloses a technology in which, when a user who has received a business card reads the QR code (registered trademark) printed on the business card using a user terminal, 3D model data of the face of the user who provided the business card and an AR video as a self-introduction message are superimposed on the real image of the business card. Furthermore, in the technology described in Patent Document 1, an icon indicating the contact information of the user who provided the business card is superimposed on the AR video. The user who has received the business card can make a phone call or perform other actions by selecting this icon. [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2021-026285 [Overview of the project] [Problems that the invention aims to solve]

[0005] However, conventional technology only displays self-introduction messages from other users when a user interacts with an object in the real world. Therefore, when multiple users exchange messages using AR technology, only the first user could be aware that a message had arrived addressed to them. In particular, a second user, who is different from the first user, could not realize that the message addressed to the first user remained unread.

[0006] Therefore, the present invention aims to provide a terminal device that, when sending and receiving messages using XR technology, can determine in a virtual space whether or not there are unread messages addressed to other users. [Means for solving the problem]

[0007] A preferred embodiment of the present invention is a terminal device that displays an image relating to a virtual space on a transparent display device worn on the user's head, comprising: a detection unit that detects whether or not there are unread messages addressed to other terminal devices when the distance between the transparent display device and the other terminal device in a shared space shared by the terminal device and the other terminal device is within a predetermined value; and a display control unit that displays a virtual object on a transparent display device corresponding to the presence or absence of the unread messages. [Effects of the Invention]

[0008] According to the present invention, when sending and receiving messages using XR technology, it is possible to determine in the virtual space whether or not there are unread messages addressed to other users. [Brief explanation of the drawing]

[0009] [Figure 1] A diagram showing the overall configuration of the information processing system 1 according to the first embodiment. [Figure 2] A perspective view showing the appearance of the XR glasses 20 according to the first embodiment. [Figure 3] A block diagram showing an example configuration of the XR glasses 20 according to the first embodiment. [Figure 4] Block diagram showing a configuration example of the terminal device 10 according to the first embodiment. [Figure 5] Explanatory diagram showing an example of the operations of the calculation unit 113, the determination unit 114, and the display control unit 115. [Figure 6] Block diagram showing a configuration example of the server 30. [Figure 7] Table showing an example of the location information database LD. [Figure 8] Table showing an example of the message database MD. [Figure 9] Flowchart showing the operation of the terminal device 10 according to the first embodiment. [Figure 10] Block diagram showing a configuration example of the terminal device 10A according to the second embodiment. [Figure 11] Explanatory diagram showing an example of the operations of the reception unit 117 and the display control unit 115A. [Figure 12] Explanatory diagram showing an example of the operations of the reception unit 117 and the display control unit 115A. [Figure 13] Explanatory diagram showing an example of the operations of the reception unit 117-2 and the display control unit 115A-2. [Figure 14] Explanatory diagram showing an example of the operations of the reception unit 117-2 and the display control unit 115A-2. [Figure 15] Block diagram showing a configuration example of the server 30A according to the second embodiment. [Figure 16] Table showing an example of the group database GD. [Figure 17] Flowchart showing the operation of the terminal device 10A according to the second embodiment.

Embodiments for Carrying Out the Invention

[0010] 1: First Embodiment Hereinafter, referring to FIGS. 1 to 9, the configuration of the information processing system 1 including the terminal device 10 according to the first embodiment of the present invention will be described.

[0011] 1-1: Configuration of the First Embodiment 1-1-1: Overall Configuration Figure 1 shows the overall configuration of the information processing system 1 according to the first embodiment of the present invention. The information processing system 1 is a system that provides a virtual space to a user U1 wearing XR glasses 20, which will be described later, using XR technology.

[0012] Information processing system 1 comprises a terminal device 10, XR glasses 20, and a server 30. In information processing system 1, the terminal device 10 and the server 30 are connected to each other via a communication network NET. The terminal device 10 and the XR glasses 20 are also connected to each other in a manner that allows them to communicate. In the following description, when distinguishing between terminal devices 10 and XR glasses 20 used by each user, the subscript "-X" is used. X is any integer greater than or equal to 1. The same applies to the components of each terminal device 10 and XR glasses 20. In Figure 1, terminal device 10-1 and XR glasses 20 are connected to each other in a manner that allows them to communicate. Also, in Figure 1, two terminal devices 10 and one XR glasses 20 are shown, but these numbers are just one example, and information processing system 1 may have any number of terminal devices 10 and XR glasses 20.

[0013] In Figure 1, it is assumed that user U1 is using a set of terminal device 10-1 and XR glasses 20. It is also assumed that user U2 is using terminal device 10-2. In the information processing system 1 shown in Figure 1, a virtual object corresponding to the presence or absence of unread messages addressed to user U2 is displayed on the XR glasses 20. These "unread messages addressed to user U2" may include messages sent from terminal device 10-1 to terminal device 10-2 that have not yet been read by user U2. Furthermore, these "unread messages addressed to user U2" may also include messages sent from other devices not shown in Figure 1 to terminal device 10-2 that have not yet been read by user U2.

[0014] Server 30 provides various data and cloud services to terminal devices 10 via the communication network NET.

[0015] Terminal device 10-1 displays virtual objects placed in a virtual space on XR glasses 20 worn on the head of user U1. This virtual space is, for example, a celestial sphere. The virtual objects include, for example, virtual objects representing data such as still images, videos, 3DCG models, HTML files, and text files, and virtual objects representing applications. Examples of text files include memos, source code, diaries, and recipes. Examples of applications include browsers, applications for using social networking services, and applications for generating document files. Terminal device 10 is preferably a mobile terminal device such as a smartphone or tablet. Terminal device 10-1 is an example of a display control device.

[0016] Terminal device 10-2 is a device for user U2 to receive messages from other users, including user U1. Terminal device 10-2 may display virtual objects placed in a virtual space on the display 14 (described later) or on XR glasses (not shown) connected to terminal device 10-2. In particular, if terminal device 10-2 is a device that displays virtual objects, the configuration of terminal device 10-2 basically includes the same configuration as terminal device 10-1. Also, like terminal device 10-1, terminal device 10-2 is preferably a mobile terminal device such as a smartphone or tablet.

[0017] The XR glasses 20 are a see-through wearable display worn on the head of user U1. The XR glasses 20 are controlled by the terminal device 10-1 to display virtual objects on display panels provided on each of the lenses for both eyes. Note that the XR glasses 20 are just one example of a display device. Furthermore, the following describes an example where the XR glasses 20 are AR glasses. However, the XR glasses 20 being AR glasses is just one example, and the XR glasses 20 may also be VR (Virtual Reality) glasses or MR (Mixed Reality) glasses.

[0018] 1-1-2: Configuration of XR Glasses Figure 2 is a perspective view showing the appearance of the XR glasses 20. As shown in Figure 2, the appearance of the XR glasses 20 is similar to that of ordinary eyeglasses, with temples 91 and 92, a bridge 93, frames 94 and 95, and lenses 41L and 41R.

[0019] An imaging device 26 is provided on the bridge 93. The imaging device 26 captures images of the outside world. The imaging device 26 also outputs imaging information indicating the captured images.

[0020] Lenses 41L and 41R are each equipped with a half-mirror. Frame 94 is provided with a liquid crystal panel or an organic EL panel for the left eye. The liquid crystal panel or organic EL panel will be collectively referred to as the display panel below. Frame 94 is also provided with an optical element that guides light emitted from the display panel for the left eye to lens 41L. The half-mirror provided in lens 41L transmits light from the outside world and guides it to the left eye, and also reflects the light guided by the optical element and causes it to enter the left eye. Frame 95 is provided with a display panel for the right eye and an optical element that guides light emitted from the display panel for the right eye to lens 41R. The half-mirror provided in lens 41R transmits light from the outside world and guides it to the right eye, and also reflects the light guided by the optical element and causes it to enter the right eye.

[0021] The display 28, described later, includes a lens 41L, a display panel for the left eye, and an optical component for the left eye, as well as a lens 41R, a display panel for the right eye, and an optical component for the right eye.

[0022] In the above configuration, user U1 can observe the image displayed by the display panel in a see-through state, superimposed on the external environment. Furthermore, in the XR glasses 20, by displaying the left-eye image on the left-eye display panel and the right-eye image on the right-eye display panel, user U1 can perceive the displayed image as if it had depth and three-dimensionality.

[0023] Figure 3 is a block diagram showing an example configuration of XR glasses 20. The XR glasses 20 comprises a processing unit 21, a storage device 22, a gaze detection device 23, a GPS device 24, a motion detection device 25, an imaging device 26, a communication device 27, and a display 28. Each element of the XR glasses 20 is interconnected by one or more buses for communicating information. In this specification, the term "device" may be replaced with other terms such as circuit, device, or unit.

[0024] The processing unit 21 is a processor that controls the entire XR glasses 20. The processing unit 21 is configured using, for example, one or more chips. The processing unit 21 is also configured using a central processing unit (CPU) that includes, for example, interfaces with peripheral devices, arithmetic units and registers. Some or all of the functions of the processing unit 21 may be implemented by hardware such as a DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), PLD (Programmable Logic Device), and FPGA (Field Programmable Gate Array). The processing unit 21 executes various processes in parallel or sequentially.

[0025] The storage device 22 is a recording medium that can be read from and written to by the processing device 21. The storage device 22 also stores multiple programs, including the control program PR2 executed by the processing device 21.

[0026] The gaze detection device 23 detects the gaze of user U1 and generates gaze information indicating the detection result. Any method may be used for gaze detection by the gaze detection device 23. For example, the gaze detection device 23 may detect gaze information based on the position of the inner corner of the eye and the position of the iris. The gaze information indicates the direction of user U1's gaze. The gaze detection device 23 supplies the gaze information to the processing device 21, which will be described later. The gaze information supplied to the processing device 21 is transmitted to the terminal device 10 via the communication device 27.

[0027] The GPS device 24 receives radio waves from multiple satellites. The GPS device 24 also generates location information from the received radio waves. The location information indicates the position of the XR glasses 20. The location information can be in any format as long as it can identify the location. For example, the location information indicates the latitude and longitude of the XR glasses 20. As an example, the location information is obtained from the GPS device 24. However, the XR glasses 20 may acquire the location information by any method. The acquired location information is supplied to the processing device 21. The location information supplied to the processing device 21 is transmitted to the terminal device 10 via the communication device 27.

[0028] The motion detection device 25 detects the movement of the XR glasses 20. The motion detection device 25 includes inertial sensors such as an acceleration sensor for detecting acceleration and a gyroscope sensor for detecting angular acceleration. The acceleration sensor detects acceleration along the orthogonal X, Y, and Z axes. The gyroscope sensor detects angular acceleration with the X, Y, and Z axes as the central axes of rotation. Based on the output information from the gyroscope sensor, the motion detection device 25 can generate attitude information indicating the orientation of the XR glasses 20. The motion information includes acceleration data indicating the acceleration of each of the three axes and angular acceleration data indicating the angular acceleration of each of the three axes. The motion detection device 25 also supplies attitude information indicating the orientation of the XR glasses 20 and motion information related to the movement of the XR glasses 20 to the processing device 21. The attitude information and motion information output to the processing device 21 are transmitted to the terminal device 10 via the communication device 27.

[0029] The imaging device 26 outputs imaging information obtained by imaging the external environment. The imaging device 26 also includes, for example, a lens, an image sensor, an amplifier, and an AD converter. Light focused through the lens is converted into an analog imaging signal by the image sensor. The amplifier amplifies the imaging signal and outputs it to the AD converter. The AD converter converts the amplified analog imaging signal into digital imaging information. The converted imaging information is supplied to the processing device 21. The imaging information supplied to the processing device 21 is transmitted to the terminal device 10 via the communication device 27.

[0030] The communication device 27 is hardware that acts as a transmitting and receiving device for communicating with other devices. The communication device 27 is also referred to as, for example, a network device, network controller, network card, and communication module. The communication device 27 may include a connector for wired connection and an interface circuit corresponding to the connector. The communication device 27 may also include a wireless communication interface. Examples of wired connection connectors and interface circuits include products compliant with wired LAN, IEEE 1394, and USB. Examples of wireless communication interfaces include products compliant with wireless LAN and Bluetooth®.

[0031] The display 28 is a device for displaying images. The display 28 displays various images under the control of the processing unit 21. As described above, the display 28 includes a lens 41L, a display panel for the left eye, and an optical element for the left eye, as well as a lens 41R, a display panel for the right eye, and an optical element for the right eye. Various display panels such as liquid crystal display panels and organic EL display panels are preferably used as the display panels.

[0032] The processing unit 21 functions as an acquisition unit 211 and a display control unit 212, for example, by reading and executing the control program PR2 from the storage device 22.

[0033] The acquisition unit 211 acquires image information from the terminal device 10-1 that indicates the image displayed on the XR glasses 20.

[0034] Furthermore, the acquisition unit 211 acquires gaze information input from the gaze detection device 23, position information input from the GPS device 24, posture information and motion information input from the motion detection device 25, and imaging information input from the imaging device 26. Then, the acquisition unit 211 outputs the acquired gaze information, position information, posture information, motion information, and imaging information to the communication device 27.

[0035] The display control unit 212 causes the display 28 to display the image indicated by the image information acquired from the terminal device 10-1 by the acquisition unit 211.

[0036] 1-1-3: Terminal device configuration Figure 4 is a block diagram showing an example configuration of terminal device 10. Terminal device 10 comprises a processing unit 11, a storage device 12, a communication device 13, a display 14, an input device 15, and an inertial sensor 16. Each element of terminal device 10 is interconnected by one or more buses for communicating information. Below, we will basically describe the configuration of terminal device 10-1 as the configuration of terminal device 10.

[0037] The processing unit 11 is a processor that controls the entire terminal device 10. The processing unit 11 is configured using, for example, one or more chips. The processing unit 11 is configured using, for example, a central processing unit (CPU) that includes interfaces with peripheral devices, an arithmetic unit, and registers. Some or all of the functions of the processing unit 11 may be implemented by hardware such as a DSP, ASIC, PLD, and FPGA. The processing unit 11 executes various processes in parallel or sequentially.

[0038] The storage device 12 is a recording medium that the processing unit 11 can read and write to. The storage device 12 also stores multiple programs, including the control program PR1 executed by the processing unit 11.

[0039] The communication device 13 is hardware acting as a transmitting and receiving device for communicating with other devices. The communication device 13 is also called, for example, a network device, a network controller, a network card, and a communication module. The communication device 13 may be equipped with a connector for wired connection and an interface circuit corresponding to the connector. The communication device 13 may also be equipped with a wireless communication interface. Examples of connectors and interface circuits for wired connection include products compliant with wired LAN, IEEE1394, and USB. Examples of wireless communication interfaces include products compliant with wireless LAN and Bluetooth®.

[0040] The display 14 is a device that displays images and text information. The display 14 displays various images under the control of the processing unit 11. For example, various display panels such as liquid crystal display panels and organic EL (Electro-Luminescence) display panels are suitably used as the display 14. However, if XR glasses 20 are connected to the terminal device 10, the display 14 does not have to be an essential component. In this case, the XR glasses 20 will have the same functions as the display 14.

[0041] The input device 15 receives input from user U1 wearing the XR glasses 20 on their head. For example, the input device 15 includes a keyboard, touchpad, touch panel, or pointing device such as a mouse. If the input device 15 includes a touch panel, it may also function as the display 14.

[0042] The inertial sensor 16 is a sensor that detects inertial force. The inertial sensor 16 includes, for example, one or more sensors from among an acceleration sensor, an angular velocity sensor, and a gyroscope sensor. The processing unit 11 detects the orientation of the terminal device 10 based on the output information of the inertial sensor 16. Furthermore, based on the orientation of the terminal device 10, the processing unit 11 accepts the selection of a virtual object VO, the input of characters, and the input of instructions in the spherical virtual space VS. For example, when user U1 directs the central axis of the terminal device 10 towards a predetermined area in the virtual space VS, a virtual object VO placed in the predetermined area is selected by operating the input device 15. User U1's operation on the input device 15 is, for example, a double tap. In this way, user U1 can select a virtual object VO by operating the terminal device 10 without looking at the input device 15 provided on the terminal device 10.

[0043] Furthermore, if the XR glasses 20 are not connected to the terminal device 10, it is preferable that the terminal device 10 be equipped with a GPS device similar to the GPS device 24 provided in the XR glasses 20.

[0044] The processing unit 11 functions as an output unit 111, an acquisition unit 112, a calculation unit 113, a determination unit 114, and a display control unit 115 by reading and executing the control program PR1 from the storage device 12.

[0045] The output unit 111 transmits to the server 30 the device ID (identifier) ​​of the terminal device 10, the username of the user using the terminal device 10, and the location information acquired by the terminal device 10 from the XR glasses 20. For example, if the terminal device 10 is terminal device 10-1 used by the first user, user U1, the output unit 111-1 transmits to the server 30 the device ID of the terminal device 10-1, the username of the terminal device 10-1, "U1", and the location information generated by the XR glasses 20-1 worn on the head of user U1.

[0046] Furthermore, the output unit 111-1 sends a message created by user U1 using the input device 15 to the server 30. The message specifies the sender, the recipient, and the content of the message. The content of the message includes at least one of text and an image.

[0047] Furthermore, when a message addressed to user U1, acquired by the acquisition unit 112-1 (described later), is marked as read by user U1, the output unit 111-1 sends information indicating that the message has been read to the server 30.

[0048] The acquisition unit 112 acquires from the server 30 the device ID of the other terminal device 10, the username of the user of the other terminal device 10, and location information generated by the other terminal device 10 or the XR glasses 20 connected to the other terminal device 10, which have been transmitted to the server 30 by the other terminal device 10. For example, if the other terminal device 10 is terminal device 10-2 used by a second user, user U2, the acquisition unit 112 acquires from the server 30 the device ID of terminal device 10-2, the username of the user of terminal device 10-2, "U2", and location information generated by terminal device 10-2.

[0049] Furthermore, the acquisition unit 112 acquires messages addressed to user U from the server 30. If terminal device 10 is terminal device 10-1 used by the first user, user U1, the acquisition unit 112-1 acquires messages addressed to user U1 from the server 30. The "message addressed to user U1" may be a message generated by another user using terminal device 10-2 and sent to terminal device 10-1 via the communication network NET. Alternatively, the "message addressed to user U1" may be a message generated by another user using another device not shown in Figure 1 and sent to terminal device 10-1 via the communication network NET. The "other device" may be, for example, a mobile terminal device such as a smartphone or tablet. Alternatively, the "other device" may be, for example, a stationary PC (Personal Computer). Also, the "other device" does not necessarily have to be a device that displays a virtual space. Similarly, with respect to terminal device 10-2, the acquisition unit 112-2 acquires messages addressed to user U2 from server 30.

[0050] The calculation unit 113 calculates the distance between users U from the location information generated by the XR glasses 20 and the location information generated by other terminal devices 10 or XR glasses connected to other terminal devices 10. If terminal device 10 is terminal device 10-1 used by user U1, who is the first user, the calculation unit 113-1 calculates the distance between user U1 and user U2 from the location information generated by the XR glasses 20 and the location information generated by terminal device 10-2.

[0051] The determination unit 114 determines whether the distance between users U is within a predetermined value. If terminal device 10 is terminal device 10-1 used by user U1 as the first user, the determination unit 114-1 determines whether the distance between user U1 and user U2, calculated by the calculation unit 113-1, is within a predetermined value. If the determination result by the determination unit 114-1 is positive, that is, if the determination unit 114-1 determines that the distance is within a predetermined value, the acquisition unit 112-1 detects whether there are any unread messages addressed to user U2 as the second user. The acquisition unit 112-1 may also detect the number of unread messages addressed to user U2. In this case, the number of unread messages is assumed to be 2 or more. The acquisition unit 112-1 is an example of a "detection unit".

[0052] As described later, the server 30 is equipped with a message database MD. The message database MD stores messages sent and received between multiple users U using terminal devices 10 connected to the server 30 or other terminal devices not shown in Figure 1. The calculation unit 312, also provided in the server 30 and described later, refers to the message database MD. Based on the result of the reference, the calculation unit 312 calculates the number of unread messages addressed to user U2. The acquisition unit 112-1, provided in the terminal device 10-1, uses the communication device 13 and connects to the server 30 via the communication network NET to detect the number of unread messages addressed to user U2.

[0053] The display control unit 115 displays virtual object VO on the XR glasses 20, which acts as a display device, according to the presence or absence of unread messages detected by the acquisition unit 112. In particular, it is preferable for the display control unit 115 to display virtual object VO on the XR glasses 20, which acts as a display device, according to the number of unread messages detected by the acquisition unit 112.

[0054] As a result, user U1 can understand the status of unread messages addressed to user U2 in the virtual space VS. Specifically, a virtual object VO corresponding to the presence or number of unread messages addressed to user U2 is displayed on the XR glasses 20 worn on user U1's head. This display allows user U1 to recognize the status of unread messages addressed to user U2 in the virtual space VS. User U1 can verbally confirm with user U2, for example, the presence or number of unread messages and the course of action for those unread messages. As a result, realistic communication between user U1 and user U2 is promoted.

[0055] Figure 5 is an explanatory diagram showing an example of the operation of the calculation unit 113, the determination unit 114, and the display control unit 115. In the following explanation, we assume that the X, Y, and Z axes are mutually orthogonal in the virtual space VS. As an example, the X axis extends in the front-to-back direction of user U1. Furthermore, from the perspective of user U1, the forward direction along the X axis is designated as the X1 direction, and the backward direction along the X axis is designated as the X2 direction. The Y axis extends in the left-to-right direction of user U1. Furthermore, from the perspective of user U1, the rightward direction along the Y axis is designated as the Y1 direction, and the leftward direction along the Y axis is designated as the X2 direction. These X and Y axes constitute a horizontal plane. The Z axis is orthogonal to the XY plane and extends in the up-to-down direction of user U1. Furthermore, from the perspective of user U1, the downward direction along the Z axis is designated as the Z1 direction, and the upward direction along the Z axis is designated as the Z2 direction.

[0056] Referring to Figure 5, in the virtual space VS, user U1 is at coordinates (x,y,z)=(x u1 ,y u1 ,z u1 It is assumed that user U2 is located at (x,y,z)=(x u2 ,y u2 ,z u2It is assumed that the user U1 is located in the virtual space VS. The coordinates of user U1 in the virtual space VS correspond to the position of user U1 in the real space RS. The coordinates of user U2 in the virtual space VS correspond to the position of user U2 in the real space RS. Furthermore, the position of user U1 in the real space RS is indicated by position information generated by the XR glasses 20 worn on user U1's head. Similarly, the position of user U2 in the real space RS is indicated by position information generated by the terminal device 10-2. Note that the virtual space VS and the real space RS are examples of shared spaces shared by user U1 and user U2. More specifically, based on the fact that user U1 and user U2 share the virtual space VS, the above shared space is the virtual space VS. Based on the fact that user U1 and user U2 share the real space RS, the above shared space is the real space RS. Based on the fact that user U1 and user U2 share a composite space MS which is a combination of the virtual space VS and the real space RS, the above shared space is the composite space MS. Here, we assume that user U1 is carrying terminal device 10-1 and wearing XR glasses 20 on their head. Furthermore, we assume that user U2 is carrying terminal device 10-2.

[0057] The calculation unit 113-1 in terminal device 10-1 calculates the distance D between user U1 and user U2 based on the position information generated by the XR glasses 20 and the position information generated by terminal device 10-2. The determination unit 114-1 in terminal device 10-1 determines whether the distance D is within a predetermined value. If the determination result by the determination unit 114-1 is positive, i.e., the distance D is within a predetermined value, the display control unit 115-1 in terminal device 10-1 causes the XR glasses 20 to display a virtual object VO1 corresponding to the presence or absence of unread messages for user U2 at coordinates (x,y,z)=(x1,y1,z1) in the virtual space VS. It is preferable for the display control unit 115-1 to display a virtual object VO1 corresponding to the number of unread messages for user U2.

[0058] The display of the virtual object VO1 is limited to when the distance between user U1 and user U2 is within a predetermined value. Therefore, terminal device 10-1 can suppress the display of the virtual object VO1 when there is no need to encourage real communication. As a result, user U1 will not be confused by the display of numerous virtual objects VO corresponding to multiple other users U in the virtual space VS. Furthermore, since the acquisition of the presence or number of unread messages is performed only when the distance between user U1 and user U2 is within a predetermined value, which is the condition for displaying the virtual object VO1, the processing load on terminal device 10-1 can be reduced.

[0059] In the example shown in Figure 5, the virtual object VO1 is a collection of individual objects SO1 to SO6, each corresponding one-to-one with an unread message addressed to user U2. That is, the virtual object VO1 contains the same number of individual objects SO1 to SO6 as the number of unread messages addressed to user U2.

[0060] As described above, the message database MD on the server 30 stores multiple messages exchanged between multiple users U. Each of these multiple messages is assigned a message ID to identify it. The acquisition unit 112-1 on the terminal device 10-1 acquires the message IDs attached to unread messages addressed to user U2 from the server 30, for each unread message. The display control unit 115-1 on the terminal device 10-1 associates each message ID acquired by the acquisition unit 112-1 with individual objects SO1 to SO6 on a one-to-one basis. The display control unit 115-1 displays individual objects SO1 to SO6 on the XR glasses 20-1. The image information used by the display control unit 115-1 to display individual objects SO1 to SO6 on the XR glasses 20 may be information stored in the storage device 12-1. Alternatively, the image information may be information acquired by the acquisition unit 112-1 from the server 30.

[0061] Also, in the virtual space VS shown in FIG. 5, the display control unit 115-1 changes the position of the virtual object VO1 according to the movement of the user U2. In particular, it is preferable that the relative positional relationship between the coordinates (x U2 , y u2 , z u2 ) of the user U2 and the coordinates (x1, y1, z1) of the virtual object VO1 does not change. In this case, the display control unit 115-1 provided in the terminal device 10-1 causes the position of the virtual object VO1 in the virtual space VS to follow the movement of the user U2 in the real space, that is, the movement of the terminal device 10-2.

[0062] As a result, the terminal device 10-1 can display the virtual object VO1 within a predetermined range from the user U2 regardless of the movement destination of the user U2.

[0063] Also, the display control unit 115-1 provided in the terminal device 10-1 changes the display mode of the virtual object VO1 according to the number of unread messages addressed to the user U2. For example, as the number of the individual objects SO1 to SO6 included in the virtual object VO1 increases, the display control unit 115-1 provided in the terminal device 10-1 may increase the size of the virtual object VO1. Increasing the size of the virtual object VO1 means increasing the volume of the virtual object VO1. Therefore, if the volume of the virtual object VO1 increases, the shape of the virtual object VO1 does not have to be similar before and after the volume change. Alternatively, as the number of the individual objects SO1 to SO6 included in the virtual object VO1 increases, the display control unit 115-1 provided in the terminal device 10-1 may change at least one of the color of the virtual object VO1 and the colors of the individual objects SO1 to SO6 included in the virtual object VO1.

[0064] As a result, terminal device 10-1 can more effectively communicate to user U1 the increase in unread messages addressed to user U2. As the number of unread messages addressed to user U2 increases, user U1 can more easily perceive the increase in the number of unread messages addressed to user U2 in the virtual space VS.

[0065] Furthermore, the display control unit 115-1 provided in the terminal device 10-1 may display the virtual space VS, which includes the virtual object VO1, on the XR glasses 20 if the number of unread messages addressed to user U2 is greater than or equal to a threshold. On the other hand, the display control unit 115 provided in the terminal device 10-1 may display the virtual space VS, which does not include the virtual object VO1, on the XR glasses 20 if the number of unread messages addressed to user U2 does not reach a threshold.

[0066] As a result, terminal device 10-1 can only display a virtual object VO on the XR glasses 20 worn on user U1's head when the number of unread messages addressed to user U2 exceeds a threshold. Displaying a virtual object VO in a virtual space according to the number of unread messages facilitates communication between users. A typical example of user U1 contacting user U2 regarding the number of unread messages is when user U2 has a larger number of unread messages than other users. Therefore, if the number of unread messages addressed to user U2 does not reach the threshold, there is little need to display the virtual object VO on the XR glasses 20 worn on user U1's head. Consequently, when the number of unread messages addressed to user U2 has not reached the threshold, there is little need for user U1 to specifically check the number of unread messages. Since terminal device 10-1 displays the virtual object VO on the XR glasses 20 when the number of unread messages addressed to user U2 exceeds the threshold, it can display a highly useful virtual object VO in the virtual space. As a result, the processing load on terminal device 10-1 can be reduced, and the virtual space can be used effectively.

[0067] In the above explanation with reference to Figure 5, it was assumed that the virtual object VO1 and individual objects SO1 to SO6 would be displayed on the XR glasses 20 worn on the head of user U1. The virtual object VO1 and individual objects SO1 to SO6 may also be displayed on the display 14 of the terminal device 10-2 carried by user U2. Alternatively, if XR glasses (not shown) are connected to the terminal device 10-2, the virtual object VO1 and individual objects SO1 to SO6 may be displayed on those XR glasses.

[0068] In this case, the acquisition unit 112-2 in the terminal device 10-2 used by user U2 detects the number of unread messages addressed to user U2. The acquisition unit 112-2 in the terminal device 10-2 also acquires message IDs attached to the unread messages addressed to user U2 from the server 30, for each unread message. The display control unit 115-2 in the terminal device 10-2 associates individual objects SO1 to SO6 one-to-one with each message ID acquired by the acquisition unit 112-2, and then displays them on the display 14 or the XR glasses connected to the terminal device 10-2.

[0069] When user U1 and user U2 share the same virtual space VS, the virtual object VO1 and individual objects SO1 to SO6 are displayed at the same coordinates in the same coordinate system in the virtual space VS shown on the XR glasses 20 and the display 14 on the terminal device 10-2 or the XR glasses connected to the terminal device 10-2. For this reason, it is preferable for the server 30 to manage the coordinates in the virtual space VS where the virtual object VO1 and individual objects SO1 to SO6 are displayed by associating them with message IDs.

[0070] As a result, both user U1 and user U2 can view the same virtual object VO1 and engage in realistic communication.

[0071] As another example, as the number of unread messages addressed to user U2 increases, and consequently the number of individual objects SO1 to SO6 contained in virtual object VO1 increases, the display control unit 115-1 in terminal device 10-1 and the display control unit 115-2 in terminal device 10-2 may display virtual object VO1 in the virtual space VS so that it is further away from user U2 and closer to user U1. Furthermore, only when the number of unread messages addressed to user U2 exceeds a threshold, and the distance D between virtual object VO1 and user U1 in the virtual space VS is less than or equal to a predetermined value, the display control unit 115-1 in terminal device 10-1 may display virtual object VO1 on the XR glasses 20.

[0072] As a result, in the virtual space VS, the virtual object VO1 is displayed only within a predetermined distance from user U1. In this case, user U1 does not need to check the virtual object VO displayed far away from themselves in the virtual space VS.

[0073] 1-1-4: Server Configuration Figure 6 is a block diagram showing an example configuration of server 30. Server 30 comprises a processing unit 31, a storage device 32, a communication device 33, a display 34, and an input device 35. Each element of server 30 is interconnected by one or more buses for communicating information.

[0074] The processing unit 31 is a processor that controls the entire server 30. The processing unit 31 is configured using, for example, one or more chips. The processing unit 31 is configured using, for example, a central processing unit (CPU) that includes interfaces with peripheral devices, an arithmetic unit, and registers. Some or all of the functions of the processing unit 31 may be implemented by hardware such as a DSP, ASIC, PLD, and FPGA. The processing unit 31 executes various processes in parallel or sequentially.

[0075] The storage device 32 is a recording medium that the processing device 31 can read and write to. The storage device 32 also stores multiple programs, including the control program PR3 executed by the processing device 31. The storage device 32 also stores a location information database LD in which the location information of each user U is stored. Furthermore, the storage device 32 stores a message database MD in which information related to messages sent and received between multiple users U is stored.

[0076] Figure 7 is a table showing an example of a location information database LD. As described above, the terminal device 10 outputs to the server 30 the device ID of the terminal device 10, the username of the user using the terminal device 10, and the location information acquired by the terminal device 10 from the XR glasses 20 or the location information generated by the terminal device 10. These device ID, username, and location information are stored in the location information database LD. As an example, the location information database LD shown in Figure 7 contains "0001", which is the device ID of terminal device 10-1 used by user U1, "U1", which is the username of user U1, and (x,y,z)=(x u1 ,y u1 ,z u1 The elements are stored in a linked state. In the table shown in Figure 7, "L" is an integer greater than or equal to 2.

[0077] Figure 8 is a table showing an example of the message database MD. As described later, the acquisition unit 311 in the server 30 acquires messages sent and received between the terminal device 10 and user U. More specifically, the acquisition unit 311 acquires information indicating the sender of the message output from the terminal device 10, information indicating the recipient of the message, and information indicating the content of the message. This information is stored in the message database MD. Furthermore, the acquisition unit 311 acquires information from the terminal device 10 indicating that each message has been read by user U. The message management unit 314, described later, adds a flag in the message database MD indicating whether each message has been read or not based on this information. For example, a flag with a value of "0" indicates that the message is unread. On the other hand, a flag with a value of "1" indicates that the message has been read. The message database MD shown in Figure 8 stores, for example, a message ID "1", the sender of the message with message ID=1 "U1", the recipient "U2", the message content, and a flag "0" indicating that the message is unread, all linked together. In the table shown in Figure 8, "m" is an integer greater than or equal to 2.

[0078] Returning to the explanation in Figure 6, the communication device 33 is hardware acting as a transmitting and receiving device for communicating with other devices. The communication device 33 is also called, for example, a network device, a network controller, a network card, and a communication module. The communication device 33 may be equipped with a connector for wired connection and an interface circuit corresponding to the connector. The communication device 33 may also be equipped with a wireless communication interface. Examples of connectors and interface circuits for wired connection include products compliant with wired LAN, IEEE1394, and USB. Examples of wireless communication interfaces include products compliant with wireless LAN and Bluetooth®.

[0079] The display 34 is a device that displays images and text information. The display 34 displays various images under the control of the processing unit 31. For example, various display panels such as liquid crystal display panels and organic EL display panels are preferably used as the display 34.

[0080] The input device 35 is a device that receives operations from the administrator of the information processing system 1. For example, the input device 35 is configured to include a keyboard, touchpad, touch panel, or pointing device such as a mouse. If the input device 35 is configured to include a touch panel, it may also function as the display 34.

[0081] The processing unit 31 functions as an acquisition unit 311, a calculation unit 312, an output unit 313, and a message management unit 314, for example, by reading and executing the control program PR3 from the storage device 32.

[0082] The acquisition unit 311 acquires various types of data from the terminal device 10 using the communication device 33. This data includes, for example, data indicating the operation performed on a virtual object VO, which is input to the terminal device 10 by a user U1 wearing XR glasses 20 on their head.

[0083] In particular, the acquisition unit 311 acquires from the terminal device 10 the device ID of the terminal device 10, the username of the user using the terminal device 10, and location information acquired by the terminal device 10 from the XR glasses 20 or location information generated by the terminal device 10. The acquisition unit 311 stores the acquired device ID, username, and location information in the location information database LD. The acquisition unit 311 also acquires from the terminal device 10 messages sent and received between users U, linking the sender, receiver, and content of the message. The acquisition unit 311 stores the acquired messages in the message database MD. Furthermore, as an example, if a message addressed to user U1 is read by user U1 in terminal device 10-1, the acquisition unit 311 acquires information from terminal device 10-1 indicating that the message has been read.

[0084] The calculation unit 312 refers to the message database MD and calculates, as an example, the number of unread messages addressed to user U2.

[0085] The output unit 313 transmits image information indicating the image displayed on the XR glasses 20 to the terminal device 10. This image information may be stored in the storage device 32. Alternatively, this image information may be generated by a generation unit (not shown).

[0086] Furthermore, the output unit 313, as an example, sends a message addressed to user U1, which is stored in the message database MD, to terminal device 10-1. The same applies to other terminal devices 10.

[0087] Furthermore, the output unit 313 transmits, as an example, to terminal device 10-1 the device ID of the other terminal device 10, the username of the user using the other terminal device 10, and the location information generated by the XR glasses 20 connected to the other terminal device 10, all of which are stored in the location information database LD. The same applies to the other terminal device 10.

[0088] Furthermore, the output unit 313 outputs, as an example, to terminal device 10-1 the presence or absence of unread messages addressed to user U2, calculated by the calculation unit 312, the number of such unread messages, and the message ID corresponding to the unread messages addressed to user U2 stored in the message database MD. The same applies to other terminal devices 10.

[0089] The message management unit 314 manages the message database MD. When the acquisition unit 311 acquires information indicating that a message addressed to user U1 has been read by user U1 in terminal device 10-1, the message management unit 314 changes the flag "0" associated with the message, which indicates that it is unread, to the flag "1" indicating that it has been read.

[0090] 1-2: Operation of the First Embodiment Figure 9 is a flowchart illustrating the operation of terminal device 10 according to the first embodiment, particularly terminal device 10-1 used by user U1 as the first user. The operation of terminal device 10-1 will be described below with reference to Figure 11.

[0091] In step S1, the processing unit 11-1 functions as an acquisition unit 112-1. The processing unit 11-1 acquires the location information of user U1 as the first user and the location information of user U2 as the second user. For example, the processing unit 11-1 acquires the location information of the XR glasses 20 and the location information of the terminal device 10-2. Alternatively, the processing unit 11-1 may acquire the location information of the terminal device 10-1 and the location information of the terminal device 10-2.

[0092] In step S2, the processing unit 11-1 functions as a calculation unit 113-1. The processing unit 11-1 calculates the distance D between user U1, who is the first user, and user U2, who is the second user. For example, the processing unit 11-1 calculates the distance D from the position information of the XR glasses 20 and the terminal device 10-2. Alternatively, the processing unit 11-1 may calculate the distance D from the position information of terminal device 10-1 and terminal device 10-2.

[0093] In step S3, the processing unit 11-1 functions as a determination unit 114-1. The processing unit 11-1 determines whether the distance D calculated by the calculation unit 113-1 is within a predetermined value. If the determination result by the determination unit 114 is positive, i.e., the distance D is within a predetermined value, the processing unit 11-1 executes the process in step S4. If the determination result by the determination unit 114-1 is negative, i.e., the distance D exceeds a predetermined value, the processing unit 11-1 executes the process in step S1.

[0094] In step S4, the processing unit 11-1 functions as an acquisition unit 112-1. The processing unit 11-1 detects whether there are any unread messages addressed to user U2 as a second user. In the operation shown in Figure 9, the processing unit 11-1 acquires not only whether there are any unread messages addressed to user U2, but also the number of such unread messages.

[0095] In step S5, the processing unit 11-1 functions as a display control unit 115-1. The processing unit 11-1 displays virtual objects VO corresponding to the number of unread messages on the XR glasses 20, which are used as a display device. Subsequently, the processing unit 11-1 executes the process of step S1.

[0096] 1-3: Effects of the First Embodiment According to the above description, the terminal device 10, as a display control device, is a display control device that displays images related to the virtual space VS on XR glasses 20, which are transparent display devices worn on the head of user U1, as the first user. The terminal device 10 comprises an acquisition unit 112 and a display control unit 115. The acquisition unit 112 detects the presence or absence of unread messages addressed to user U2, and consequently the number of unread messages, when the distance D from user U1 to user U2 in the real space is within a predetermined value. The display control unit 115 displays virtual objects VO on the XR glasses 20, which are display devices, according to the presence or absence of unread messages and consequently the number of unread messages.

[0097] Since the terminal device 10 has the above configuration, when sending and receiving messages using XR technology, it is possible to know in the virtual space whether there are any unread messages addressed to other users. In particular, user U1 using the terminal device 10 can know the status of unread messages addressed to other user U2 in the virtual space VS. Specifically, a virtual object VO corresponding to whether or not there are any unread messages addressed to user U2 is displayed on the XR glasses 20 worn on the head of user U1. Through this display, user U1 can recognize the status of user U2's unread messages in the virtual space VS. User U1 can, for example, verbally confirm with user U2 the course of action regarding the unread message. In particular, if user U2 has left a message addressed to them unread for a long period of time, user U1 can verbally instruct user U2 on how to handle the message. These confirmations and instructions promote realistic communication between user U1 and user U2. Furthermore, since the display of the virtual object VO is limited to when the distance between user U1 and other user U2 is within a predetermined value, the terminal device 10 can suppress the display of the virtual object VO when there is no need to encourage real communication. As a result, the user U will not be confused by the display of numerous virtual object VOs corresponding to multiple other users U in the virtual space VS. Moreover, since the acquisition of whether or not there are unread messages is performed only when the distance between user U1 and user U2 is within a predetermined value, which is a condition for displaying the virtual object VO, the processing load on the terminal device 10 can be reduced.

[0098] Furthermore, according to the above explanation, the display control unit 115 changes the position of the virtual object VO in accordance with the movement of user U2 as a second user, that is, the movement of the terminal device 10-2.

[0099] Since the terminal device 10 has the above configuration, it can synchronize the movement of user U2 with the movement of the virtual object VO. Consequently, the position of the virtual object VO changes in accordance with the change in the position of user U2. The terminal device 10 can display the virtual object VO within a predetermined range from user U2, regardless of the destination of user U2.

[0100] Furthermore, according to the above explanation, the acquisition unit 112 detects the number of unread messages. The display control unit 115 displays a virtual object VO1 corresponding to the number of unread messages on the XR glasses 20, which are used as a display device.

[0101] Since the terminal device 10 has the above configuration, user U1 can recognize the number of unread messages from user U2 in the virtual space VS. As a result, especially when user U2 has many unread messages pending, user U1 verbally instructs user U2 on how to process those unread messages. This instruction facilitates real-time communication between user U1 and user U2.

[0102] Furthermore, according to the above explanation, the display control unit 115 changes the display mode of the virtual object VO according to the number of unread messages.

[0103] Since the terminal device 10 has the above configuration, it can better communicate to user U1, for example, the increase in the number of unread messages.

[0104] Furthermore, according to the above explanation, the display control unit 115 sets the size of the virtual object VO to a first size when the number of unread messages is a first number, and sets the size of the virtual object VO to a second size which is larger than the first size when the number of unread messages is a second number which is larger than the first number.

[0105] Since terminal device 10 has the above configuration, the size of the virtual object VO can be increased as the number of unread messages addressed to user U2 increases. As a result, user U1 can more easily grasp the increase in the number of unread messages addressed to user U2 in the virtual space VS as the number of unread messages addressed to user U2 increases.

[0106] Furthermore, according to the above explanation, the display control unit 115 displays the virtual object VO in the virtual space VS when the number of unread messages exceeds a threshold.

[0107] Since the terminal device 10 has the above configuration, it can only display the virtual object VO on the XR glasses 20 worn on the head of user U1 when the number of unread messages addressed to user U2 exceeds a threshold. In particular, if the number of unread messages addressed to user U2 does not reach the threshold, there is no need to display the virtual object VO on the XR glasses 20 worn on the head of user U1. Therefore, when the number of unread messages addressed to user U2 has not reached the threshold and user U1 does not need to keep track of the number of unread messages, the processing load on the terminal device 10 can be reduced.

[0108] 2: Second Embodiment The configuration of the information processing system 1A, including the terminal device 10A as a display control device according to the second embodiment of the present invention, will be described below with reference to Figures 10 to 17. In the following description, for the purpose of simplifying the explanation, the same reference numerals will be used for components of the information processing system 1A according to the second embodiment that are the same as those of the information processing system 1 according to the first embodiment, and their descriptions may be omitted.

[0109] 2-1: Configuration of the second embodiment 2-1-1: Overall Structure The information processing system 1A according to the second embodiment of the present invention differs from the information processing system 1 according to the first embodiment in that it includes a terminal device 10A instead of a terminal device 10, and a server 30A instead of a server 30. In all other respects, the overall configuration of the information processing system 1A is the same as the overall configuration of the information processing system 1 according to the first embodiment shown in Figure 1, so its illustration and description are omitted.

[0110] 2-1-2: Terminal device configuration Figure 10 is a block diagram showing an example configuration of terminal device 10A. Unlike terminal device 10, terminal device 10A is equipped with a processing unit 11A instead of processing unit 11, and a storage device 12A instead of storage device 12.

[0111] Unlike the storage device 12, the storage device 12A stores the control program PR2A instead of the control program PR2.

[0112] The processing unit 11A includes an output unit 111A in place of the output unit 111, an acquisition unit 112A in place of the acquisition unit 112, a determination unit 114A in place of the determination unit 114, and a display control unit 115A in place of the display control unit 115, all of which are present in the processing unit 11. In addition to the components present in the processing unit 11, the processing unit 11A also includes a setting unit 116 and a receiving unit 117.

[0113] The configuration unit 116 sets up groups in which multiple users U allow each other to share the number of unread messages. In other words, the configuration unit 116 sets up groups in which multiple devices, including terminal devices 10-1 and 10-2 and other terminal devices 10 not shown in Figure 1, allow each other to share the number of unread messages. More specifically, the configuration unit 116 sets up such groups by, for example, transmitting information indicating the operations performed by user U using the input device 15 to a server 30 equipped with a group management unit 315, which will be described later. Each of the multiple users U belonging to the same group shares the number of their own unread messages with other users U belonging to the same group. On the other hand, multiple users U who do not belong to the same group do not share the number of their own unread messages with each other.

[0114] The determination unit 114A, similar to the determination unit 114 in the first embodiment, determines whether the distance D between user U1 as the first user and user U2 as the second user, calculated by the calculation unit 113, is within a predetermined value. The determination unit 114A further determines whether user U1 as the first user and user U2 as the second user belong to the same group. In other words, the determination unit 114A determines whether terminal device 10-1 and terminal device 10-2 belong to the same group. In terminal device 10-1, if both determination results by the determination unit 114A-1 are positive, that is, if the distance D between user U1 and user U2 is within a predetermined value and user U1 and user U2 belong to the same group, the acquisition unit 112A-1 detects the number of unread messages addressed to user U2 as the second user.

[0115] The reception unit 117 receives an operation in which user U selects a first individual object SO from among two or more individual objects SO that constitute the virtual object VO. If terminal device 10A is terminal device 10A-1 used by user U1, who is the first user, the reception unit 117-1 receives an operation in which user U1 selects a first individual object SO from among two or more individual objects SO that constitute the virtual object VO. Similar to the information processing system 1 according to the first embodiment, the virtual object VO is, for example, a collection of two or more individual objects SO. User U1 selects a first individual object SO from among two or more individual objects SO included in the virtual object VO as a collection. The reception unit 117-1 receives an operation in which user U1 selects a first individual object SO.

[0116] The display control unit 115A causes the content of the unread message corresponding to the selected individual object SO by the operation received by the reception unit 117 to be displayed on the XR glasses 20, which is a display device, or on the display 14 of the terminal device 10A. If the terminal device 10A is terminal device 10A-1 used by the first user, user U1, the display control unit 115A-1 causes the content of the unread message corresponding to the selected individual object SO to be displayed on the XR glasses 20.

[0117] The content of unread messages to be displayed by the display control unit 115A is acquired from the server 30 by the acquisition unit 112A. More specifically, the output unit 111A outputs the message ID corresponding to the individual object SO selected by user U1 to the server 30A. The output unit 313A of the server 30A, described later, outputs the content of the message associated with the message ID in the message database MD to the terminal device 10A. The acquisition unit 112A of the terminal device 10A acquires the content of the message associated with the message ID from the server 30.

[0118] Figures 11 and 12 are explanatory diagrams showing an example of the operation of the reception unit 117 and the display control unit 115A. More specifically, Figures 11 and 12 show an example of the operation of the reception unit 117-1 and the display control unit 115A-1 included in the terminal device 10A-1. Referring to Figure 11, user U1 moves within the virtual space VS, with coordinates (x,y,z)=(x u1 ,y u1 ,z u1 From the position of ), coordinates (x,y,z)=(x u1’ ,y u1’ ,z u1’ Assume that user U1 has moved to the position of ( ). Subsequently, assume that user U1 has made contact with individual object SO6 in the virtual space VS.

[0119] As a result, as illustrated in Figure 12, the display control unit 115A-1 makes the sphere, which is the individual object SO6, disappear. After this disappearance, the display control unit 115A-1 displays the individual object SO7, which represents the content of the message corresponding to the individual object SO6, at coordinates (x,y,z)=(x 13 ,y 13 ,z 13 ) is displayed. Individual object SO7 is, for example, a text file or an image showing text. Note that the method of displaying individual object SO7 is not limited to the method of making the sphere, which is individual object SO6, disappear. For example, the display control unit 115A-1 may apply morphing to transform individual object SO6 into individual object SO7.

[0120] As a result, user U1 can check the content of unread messages addressed to user U2, and based on the content of those unread messages that they have actually checked, they can engage in face-to-face, real-time communication with user U2.

[0121] Furthermore, in the examples shown in Figures 11 and 12, the individual object SO6 disappears when user U1 approaches and then touches the individual object SO6. However, the method by which user U1 touches the individual object SO6 is not limited to this. For example, user U1 may touch the individual object SO6 by pulling on a string (not shown) attached to the virtual object VO1 or the individual object SO6.

[0122] As described above, in the information processing system 1 according to the first embodiment, the virtual object VO1 and individual objects SO1 to SO6 may be displayed on a display 14 provided on a terminal device 10A-2 carried by user U2, or on XR glasses connected to the terminal device 10A-2. In the information processing system 1A according to the second embodiment, the reception unit 117-2 provided on the terminal device 10A-2 may further accept an operation in which user U2 selects one individual object SO from two or more individual objects SO1 to SO6 that constitute the virtual object VO1. Furthermore, in the terminal device 10A-2 used by user U2, the display control unit 115A-2 may cause the content of an unread message corresponding to the selected individual object SO to be displayed on a display 14 provided on the terminal device 10A-2, or on XR glasses connected to the terminal device 10A-2.

[0123] Figures 13 and 14 are explanatory diagrams showing an example of the operation of the reception unit 117-2 and the display control unit 115A-2 included in the terminal device 10A-2. Referring to Figure 13, user U2 moves within the virtual space VS, with coordinates (x,y,z)=(x u2 ,y u2 ,z u2 From the position of ), coordinates (x,y,z)=(x u2’ ,y u2’ ,z u2’ Assume that user U2 has moved to the position of ( ). Subsequently, assume that user U2 has made contact with individual object SO6 in the virtual space VS.

[0124] As a result, as illustrated in Figure 14, the display control unit 115A-2 makes the sphere, which is the individual object SO6, disappear. After this disappearance, the display control unit 115A-2 displays the individual object SO7, which represents the content of the message corresponding to individual object SO6, at coordinates (x,y,z)=(x 13 ,y 13 ,z 13 Display it in ).

[0125] 2-1-3: Server Configuration Figure 15 is a block diagram showing an example configuration of server 30A. Unlike server 30, server 30A is equipped with a processing unit 31A instead of processing unit 31, and a storage device 32A instead of storage device 32.

[0126] Unlike the storage device 32, the storage device 32A stores control program PR3A instead of control program PR3. In addition to the components stored by the storage device 32, the storage device 32A also stores the group database GD.

[0127] Figure 16 is a table showing an example of a group database GD. The group database GD stores the username of user U and the group to which user U belongs, in an associated manner. For example, the username "U1" and the group "A" to which user U1 belongs are stored in a linked manner. In the table shown in Figure 16, "n" is an integer of 2 or greater. In addition, the group database GD may store the ID of the terminal device 10A used by user U instead of the username of user U.

[0128] Returning to the explanation of Figure 15, the processing unit 31A includes an acquisition unit 311A ​​in place of the acquisition unit 311 in the processing unit 31, and an output unit 313A in place of the output unit 313. In addition, the processing unit 31A includes a group management unit 315 in addition to the components of the processing unit 31.

[0129] The acquisition unit 311A, like the acquisition unit 311, acquires various data from the terminal device 10 using the communication device 33. Also, like the acquisition unit 311, the acquisition unit 311A ​​acquires the device ID of the terminal device 10A, the username of the user using the terminal device 10A, and the location information acquired by the terminal device 10A from the XR glasses 20. Furthermore, like the acquisition unit 311, the acquisition unit 311A ​​acquires messages sent and received between users U. Additionally, as an example, if a message addressed to user U1 is read by user U1 on terminal device 10-1, the acquisition unit 311A ​​acquires information indicating that the message has been read.

[0130] Furthermore, the acquisition unit 311A ​​acquires information from the terminal device 10A indicating the operation content performed by user U using the input device 15. The acquisition unit 311A ​​also acquires a message ID from the terminal device 10A corresponding to the individual object SO selected by user U1.

[0131] The group management unit 315 manages the group database GD based on the information indicating the operations performed by user U, which is acquired by the acquisition unit 311A. For example, based on this information, the group management unit 315 associates the username of user U with the group to which user U belongs and stores it in the group database GD. Alternatively, based on this information, the group management unit 315 may associate the ID of the terminal device 10A used by user U with the group to which the terminal device 10A belongs and store it in the group database GD.

[0132] The output unit 313A transmits the number of unread messages addressed to other users U belonging to the same group to terminal devices 10A used by multiple users U belonging to the same group in the group database GD.

[0133] Furthermore, the output unit 313A transmits the message content, which is linked to the message ID obtained from the terminal device 10A in the message database MD, to the terminal device 10A.

[0134] As a result, user U1 can set up groups that share the number of unread messages with each other, allowing them to share the number and content of unread messages depending on their relationship with other users U.

[0135] 2-2: Operation of the second embodiment Figure 17 is a flowchart showing the operation of terminal device 10 according to the second embodiment, particularly terminal device 10A-1 used by user U1 as the first user. The operation of terminal device 10A-1 will be described below with reference to Figure 17.

[0136] In step S11, the processing unit 11A-1 functions as an acquisition unit 112A-1. The processing unit 11A-1 acquires the location information of user U1 as the first user and the location information of user U2 as the second user. For example, the processing unit 11A-1 acquires the location information of the XR glasses 20 and the location information of the terminal device 10A-2 as the location information of user U1 and user U2. Alternatively, the processing unit 11A-1 may acquire the location information of the terminal device 10A-1 and the location information of the terminal device 10A-2 as the location information of user U1 and user U2.

[0137] In step S12, the processing unit 11A-1 functions as a calculation unit 113-1. The processing unit 11A-1 calculates the distance D between user U1 and user U2. For example, the processing unit 11A-1 calculates the distance D from the position information of the XR glasses 20 and the terminal device 10A-2. Alternatively, the processing unit 11A-1 may calculate the distance D from the position information of terminal device 10A-1 and terminal device 10A-2.

[0138] In step S13, the processing unit 11A-1 functions as a determination unit 114A-1. The processing unit 11A-1 determines whether the distance D calculated by the calculation unit 113-1 is within a predetermined value. If the determination result by the determination unit 114A-1 is positive, i.e., the distance D is within a predetermined value, the processing unit 11A-1 executes the process in step S14. If the determination result by the determination unit 114A-1 is negative, i.e., the distance D exceeds a predetermined value, the processing unit 11A-1 executes the process in step S11.

[0139] In step S14, the processing unit 11A-1 functions as a determination unit 114A-1. The processing unit 11A-1 determines whether user U1 and user U2 belong to the same group. In other words, the processing unit 11A-1 determines whether the terminal device 10A-1 used by user U1 and the terminal device 10A-2 used by user U2 belong to the same group. If the determination result by the determination unit 114A-1 is positive, that is, if it is determined that user U1 and user U2 belong to the same group, the processing unit 11A-1 executes the process in step S15. If the determination result by the determination unit 114A-1 is negative, that is, if it is not determined that user U1 and user U2 belong to the same group, the processing unit 11A-1 executes the process in step S11.

[0140] In step S15, the processing unit 11A-1 functions as the acquisition unit 112A-1. The processing unit 11A-1 detects whether there are any unread messages addressed to user U2. In particular, in the operation shown in Figure 17, the processing unit 11A-1 detects not only whether there are any unread messages addressed to user U2, but also the number of such unread messages.

[0141] In step S16, the processing unit 11A-1 functions as a display control unit 115A-1. The processing unit 11A-1 displays virtual objects VO corresponding to the number of unread messages on the XR glasses 20, which are used as a display device. Subsequently, the processing unit 11A-1 executes the process of step S11.

[0142] 2-3: Effects of the second embodiment According to the above description, in the terminal device 10A, which acts as a display control device, the number of unread messages is two or more. Furthermore, the virtual object VO is a collection of two or more individual objects SO that correspond one-to-one with the two or more unread messages. In addition to the components of the terminal device 10, the terminal device 10A further includes a reception unit 117. The reception unit 117 receives an operation from user U1, who acts as the first user, to select one individual object SO from among the two or more individual objects SO that constitute the virtual object VO. The display control unit 115A displays the content of the unread message corresponding to the one individual object SO selected by the above operation on the XR glasses 20, which acts as a display device.

[0143] Since the terminal device 10A has the above configuration, when user U1 selects an individual object SO, it can display the content of the unread message addressed to user U2 that corresponds to that individual object SO. In particular, since user U1 can check the content of the unread message addressed to user U2, user U1 can have real, face-to-face communication with user U2 based on the content of the unread message that has been actually checked.

[0144] Furthermore, according to the above description, the terminal device 10A further includes a setting unit 116. The setting unit 116 sets up groups that allow multiple users U to share the number of unread messages with each other. Also, user U1 as the first user and user U2 as the second user belong to the same group.

[0145] Since terminal device 10A has the above configuration, the number of unread messages can be shared among users U belonging to the same group. In particular, user U1, as the first user, can set up a group that shares the number of unread messages with each other, so the number of unread messages can be shared according to the relationship with other users U.

[0146] 3: Variant This disclosure is not limited to the embodiments illustrated above. Specific variations are illustrated below. Two or more embodiments may be arbitrarily selected from the following examples.

[0147] 3-1: Variation 1 In the information processing system 1 according to the first embodiment and the information processing system 1A according to the second embodiment, individual objects SO1 to SO6 correspond one-to-one with unread messages of user U2. These unread messages are messages that have not been read among messages exchanged between multiple users U, which are stored in the message database MD. However, the messages that individual objects SO1 to SO6 correspond to are not limited to messages exchanged between multiple users U. For example, these messages may be notifications to user U2 generated by an application.

[0148] Furthermore, while the embodiments described above explained, as an example, that the virtual object corresponding to the number of unread messages is a collection containing a number of individual objects corresponding to the number of messages, this disclosure is not limited to this. A single virtual object corresponding to the number of unread messages may be displayed in the virtual space, and the display characteristics, such as the size or color of the virtual object, may be changed according to the number of unread messages.

[0149] 3-2: Variation 2 In the information processing system 1 according to the first embodiment, server 30 outputs messages stored in the message database MD to terminal device 10. In the information processing system 1A according to the second embodiment, server 30A outputs messages stored in the message database MD to terminal device 10A. However, the method of outputting messages from server 30 to terminal device 10 and the method of outputting messages from server 30A to terminal device 10A are not limited to these. For example, the information processing system 1 according to the first embodiment and the information processing system 1A according to the second embodiment may further include a content server. More specifically, server 30 may acquire content including a message for user U from the content server and output the acquired content to terminal device 10. The same applies to server 30A.

[0150] 3-3: Modification 3 In the information processing system 1 according to the first embodiment, the terminal device 10 and the XR glasses 20 are implemented as separate components. However, the implementation method of the terminal device 10 and the XR glasses 20 in the embodiments of the present invention is not limited to this. For example, the XR glasses 20 may have the same functions as the terminal device 10, and the terminal device 10 and the XR glasses 20 may be implemented within a single housing. The same applies to the information processing system 1A according to the second embodiment. In other words, the components of the terminal device 10 may be built into the XR glasses 20.

[0151] 3-4: Modification 4 Information processing system 1 according to the first embodiment and information processing system 1A according to the second embodiment include XR glasses 20. XR glasses 20 are, for example, AR glasses. However, information processing system 1 and information processing system 1A may include either VR (Virtual Reality) glasses or MR (Mixed Reality) glasses as XR glasses 20. Alternatively, information processing system 1 and information processing system 1A may include either an HMD (Head Mounted Display) employing VR technology, AR technology, or MR technology. Alternatively, information processing system 1 may include either a regular smartphone or a tablet equipped with an imaging device instead of XR glasses 20. These HMDs, XR glasses, smartphones, and tablets are examples of display devices. The above explanation assumed that information processing system 1 and information processing system 1A would be used in real space. However, if the XR glasses 20 are VR glasses or an HMD employing VR technology, information processing system 1 and information processing system 1A can be used in a virtual space, such as a virtual office.

[0152] 4: Other (1) In the embodiments described above, the storage devices 12-12A, 22, and 32-32A are exemplified by ROM and RAM, but can also be flexible disks, magneto-optical disks (e.g., compact disks, digital multipurpose disks, Blu-ray® disks), smart cards, flash memory devices (e.g., cards, sticks, key drives), CD-ROMs (Compact Disc-ROMs), registers, removable disks, hard disks, floppy® disks, magnetic strips, databases, servers, and other suitable storage media. The program may also be transmitted from a network via a telecommunications line. The program may also be transmitted from a communication network NET via a telecommunications line.

[0153] (2) In the embodiments described above, the information, signals, etc. may be represented using any of the various different techniques. For example, the data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be mentioned throughout the above description may be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.

[0154] (3) In the embodiments described above, the input and output information may be stored in a specific location (e.g., memory) or managed using a management table. The input and output information may be overwritten, updated, or appended to. The output information may be deleted. The input information may be transmitted to other devices.

[0155] (4) In the embodiments described above, the determination may be made by a value represented using 1 bit (0 or 1), by a boolean value (true or false), or by a numerical comparison (for example, a comparison with a predetermined value).

[0156] (5) The processing procedures, sequences, flowcharts, etc., exemplified in the embodiments described above may be rearranged in order, as long as they do not contradict each other. For example, the methods described in this disclosure present various step elements using an exemplary order and are not limited to the specific order presented.

[0157] (6) Each function illustrated in Figures 1 to 17 is implemented by any combination of at least one of hardware and software. Furthermore, the method of implementing each functional block is not particularly limited. That is, each functional block may be implemented using one device that is physically or logically coupled, or it may be implemented using two or more physically or logically separated devices that are directly or indirectly connected (for example, using wired or wireless connections). A functional block may also be implemented by combining the above one device or the above multiple devices with software.

[0158] (7) The programs illustrated in the embodiments described above should be broadly interpreted to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, execution threads, procedures, functions, etc., whether they are called software, firmware, middleware, microcode, hardware description languages ​​or by other names.

[0159] Furthermore, software, instructions, information, etc., may be transmitted and received via a transmission medium. For example, if software is transmitted from a website, server, or other remote source using at least one of wired technology (such as coaxial cable, fiber optic cable, twisted pair, or digital subscriber line (DSL)) and wireless technology (such as infrared or microwave), then at least one of these wired and wireless technologies is included in the definition of a transmission medium.

[0160] (8) In each of the above-mentioned forms, the terms “system” and “network” shall be used interchangeably.

[0161] (9) The information, parameters, etc. described in this disclosure may be expressed using absolute values, relative values ​​from a given value, or other corresponding information.

[0162] (10) In the embodiments described above, the terminal device 10, terminal device 10A, server 30, and server 30A may be a mobile station (MS). A mobile station may also be referred to by those skilled in the art as a subscriber station, mobile unit, subscriber unit, wireless unit, remote unit, mobile device, wireless device, wireless communication device, remote device, mobile subscriber station, access terminal, mobile terminal, wireless terminal, remote terminal, handset, user agent, mobile client, client, or several other appropriate terms. In this disclosure, terms such as “mobile station,” “user terminal,” “user equipment (UE),” and “terminal” may be used interchangeably.

[0163] (11) In the embodiments described above, the terms “connected,” “coupled,” or any variation thereof means any direct or indirect connection or coupling between two or more elements, and may include the presence of one or more intermediate elements between two elements that are “connected” or “coupled” with each other. The coupling or connection between elements may be a physical coupling or connection, a logical coupling or connection, or a combination thereof. For example, “connection” may be reinterpreted as “access.” As used in this disclosure, two elements are considered to be “connected” or “coupled” with each other using at least one of one or more wires, cables and printed electrical connections, and, in some non-limiting and non-exclusive examples, electromagnetic energy having wavelengths in the radio frequency domain, microwave domain and optical (both visible and invisible) domain.

[0164] (12) In the embodiments described above, the phrase “based on” does not mean “based solely on” unless otherwise specified. In other words, the phrase “based on” means both “based solely on” and “based at least on.”

[0165] (13) The terms “determining” and “determining” as used in this disclosure may encompass a wide variety of actions. “Determining” may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, searching, inquiry (e.g., searching in a table, database or other data structure), and ascertaining. “Determining” may also include, for example, receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, and accessing (e.g., accessing data in memory). Furthermore, "judgment" and "decision" can include considering something as having been "judged" or "decided" after resolving, selecting, choosing, establishing, comparing, etc. In other words, "judgment" and "decision" can include considering something as having been "judged" or "decided" after some action. Also, "judgment (decision)" can be reinterpreted as "assuming," "expecting," "considering," etc.

[0166] (14) Where the terms “include,” “including,” and variations thereof are used in the embodiments described above, these terms are intended to be inclusive, as is the term “comprising.” Furthermore, the term “or” as used in this disclosure is not intended to be exclusive OR.

[0167] (15) In the present disclosure, if articles are added by translation, such as a, an, and the in English, the present disclosure may include the fact that the noun following these articles is plural.

[0168] (16) In this disclosure, the term “A and B are different” may mean “A and B are different from each other.” The term may also mean “A and B are each different from C.” Terms such as “separate” and “combine” may be interpreted in the same way as “different.”

[0169] (17) Each aspect / embodiment described herein may be used individually, in combination, or switched between as needed in practice. Furthermore, notification of certain information (e.g., notification that "X is") is not limited to explicit notification, but may also be implicit (e.g., by not providing such notification).

[0170] Although the present disclosure has been described in detail above, it will be clear to those skilled in the art that the present disclosure is not limited to the embodiments described herein. The present disclosure can be implemented in modified and altered forms without departing from the intent and scope of the present disclosure as defined by the claims. Accordingly, the descriptions in the present disclosure are illustrative and not restrictive in any way. [Explanation of symbols]

[0171] 1, 1A…Information processing system, 10, 10A…Terminal device, 11, 11A…Processing device, 12, 12A…Storage device, 13…Communication device, 14…Display, 15…Input device, 16…Inertial sensor, 20…XR glasses, 21…Processing device, 22…Storage device, 23…Eye-tracking device, 24…GPS device, 25…Motion detection device, 26…Imaging device, 27…Communication device, 28…Display, 30, 30A…Server, 31, 31A…Processing device, 32, 32A…Storage device, 33…Communication device, 34…Display, 35…Input device, 41L, 41R...Lens, 91, 92...Temple, 93...Bridge, 94, 95...Frame, 111, 111A...Output unit, 112, 112A...Acquisition unit, 113...Calculation unit, 114, 114A...Determination unit, 115, 115A...Display control unit, 116...Setting unit, 117...Reception unit, 311, 311A...Acquisition unit, 312...Calculation unit, 313, 313A...Output unit, 314...Message management unit, 315...Group management unit, PR1~PR3...Control program, SO1~SO7...Individual object, U1~U2...User, VO1...Virtual object

Claims

1. A terminal device that displays images related to a virtual space on a transparent display device worn on the user's head, The aforementioned terminal device is the first terminal device, A detection unit that detects whether there are any unread messages addressed to a second terminal device, which is another terminal device, when the distance between the transparent display device and the other terminal device in a shared space shared by the terminal device and the other terminal device is within a predetermined value, A display control unit that displays a virtual object on the transparent display device according to the presence or absence of the aforementioned unread message, Equipped with, The terminal device includes a message sent from the first terminal device to the second terminal device and a message sent from the third terminal device to the second terminal device.

2. The terminal device according to claim 1, wherein the display control unit changes the position of the virtual object in the virtual space in response to the movement of the other terminal device.

3. The detection unit detects the number of unread messages addressed to the other terminal device, The terminal device according to claim 1 or 2, wherein the display control unit causes a virtual object corresponding to the number of unread messages to be displayed on the transparent display device.

4. We also have a reception area, The number of unread messages is two or more. The virtual object is a collection of two or more individual objects that correspond one-to-one to two or more unread messages addressed to the other terminal device. The reception unit receives an operation to select a first individual object from among the two or more individual objects that constitute the virtual object. The terminal device according to claim 3, wherein the display control unit causes the contents of the unread message corresponding to the first individual object selected by the operation to be displayed on the transparent display device.

5. The terminal device according to claim 3, wherein the display control unit changes the display mode of the virtual object according to the number of unread messages.

6. The terminal device according to claim 3, wherein the display control unit sets the size of the virtual object to a first size when the number of unread messages is a first number, and sets the size of the virtual object to a second size which is larger than the first size when the number of unread messages is a second number which is larger than the first number.

7. The terminal device according to claim 3, wherein the display control unit displays the virtual object in the virtual space when the number of unread messages is equal to or greater than a threshold.

8. The system further includes a setting unit for setting up groups that allow multiple devices, including the terminal device and the other terminal devices, to share the number of unread messages with each other. The terminal device according to claim 1 or claim 2, wherein the terminal device and the other terminal device belong to the same group.

Images