Virtual object display control device and display system

Abstract

This display control device controls the display of a virtual object, and comprises a management unit and a display control unit. The management unit manages whether each of a first user and a second user, who share a first virtual object, has logged in as a first state user for which the first virtual object is displayed overlaid on a real space, or has logged in as a second state user for which the first virtual object is displayed overlaid on an image of a virtual space corresponding to a real space. The display control unit switches, for the first user, display control of a second virtual object representing the second user, on the basis of whether the first user has logged in as the first state user or as the second state user.

Description

Technical Field

[0001] The present invention relates to a display control device for virtual objects and a display system.

Background Art

[0002] In AR (Augmented Reality), for example, a virtual object such as a game character is displayed by an HMD (Head Mounted Display) device overlaid on the real space without blocking the user's field of view. AR is a technology that gives the user a feeling as if a virtual object actually exists in the real space. There is a technology similar to AR called VR (Virtual Reality). In VR, an HMD device displays a virtual object overlaid on an image of a virtual space while blocking the user's field of view. VR is a technology that gives the user a feeling as if they are actually present in a virtual space. Patent Document 1 discloses a technology for allowing participants in an event using virtual objects to experience the event by VR or AR.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In VR, the user's field of view is obstructed, so a virtual object representing the user is displayed to allow multiple users sharing the same virtual object to visually recognize each other. An example of a virtual object representing a user is an avatar. On the other hand, in AR, the user's field of view is not obstructed, and the user can see other users around them through the HMD (Head-Mounted Display). Therefore, displaying an avatar is unnecessary in AR. Because the necessity of avatars differs between AR and VR, it has been difficult to combine AR and VR in a single system. [Means for solving the problem]

[0005] A display control device according to a preferred embodiment of the present disclosure comprises a management unit and a display control unit. The management unit manages whether each of the first user and the second user who share the first virtual object has logged in as a first-mode user, in which the first virtual object is displayed superimposed on the real space, or as a second-mode user, in which the first virtual object is displayed superimposed on an image of the virtual space corresponding to the real space. The display control unit switches the display control of the second virtual object representing the second user to the first user based on whether the first user has logged in as a first-mode user or a second-mode user.

[0006] A display system according to a preferred embodiment of the present disclosure comprises a display device and a display control device that controls the display of a first virtual object on the display device. The display control device comprises a management unit and a display control unit. The management unit manages whether each of the first user using the display device and the second user sharing the first virtual object with the first user has logged in as a first-mode user, in which the first virtual object is displayed superimposed on real space, or as a second-mode user, in which the first virtual object is displayed superimposed on an image of virtual space corresponding to real space. The display control unit switches the display control of the second virtual object representing the second user on the display device based on whether the first user has logged in as a first-mode user or a second-mode user. [Effects of the Invention]

[0007] According to this disclosure, AR and VR can be mixed without hindering multiple users from visually perceiving each other. [Brief explanation of the drawing]

[0008] [Figure 1] This is a block diagram showing an example configuration of a display system 1 according to one embodiment of the present disclosure. [Figure 2] This figure shows an example of an image GAR1 for AR users and an example of an image GVR1 for VR users. [Figure 3] This figure shows an example configuration of the display control device 40. [Figure 4] This figure shows an example of a management table (TBL). [Figure 5] This flowchart shows the flow of the control method executed by the processing unit 430 according to program PR1. [Figure 6] This is a sequence diagram showing an example of operation of this embodiment. [Figure 7] This figure shows an example of a management table (TBL). [Figure 8] This figure shows an example of a management table (TBL). [Figure 9] This figure shows an example of an image GAR2 for AR users. [Figure 10] This figure shows an example of a management table (TBL). [Figure 11] This figure shows an example of an image GAR1 for AR users and an example of an image GVR2 for VR users. [Modes for carrying out the invention]

[0009] (A. Embodiment) Figure 1 is a diagram showing an example configuration of a display system 1 according to one embodiment of the present disclosure. As shown in Figure 1, the display system 1 includes portable devices 10A, 10B, and 10C, display devices 20A and 20B, a display device 30, and a display control device 40. The display system 1 is an information system that allows AR users and VR users to share virtual objects. An AR user is a user who experiences virtual objects through AR. A VR user is a user who experiences virtual objects through VR.

[0010] In this embodiment, the virtual object experienced by AR users and VR users is, for example, a game character. In this embodiment, experiencing a virtual object means playing a video game in which the character represented by the virtual object appears. Playing a video game means, if the video game is an action game, performing actions such as fighting against the character represented by the virtual object. The virtual object experienced by AR users and VR users is an example of a first virtual object in this disclosure.

[0011] In this embodiment, user UA, user UB, and user UC are pre-registered in the display control device 40 as users of the video game. Each of user UA, user UB, and user UC can play the game by logging into the display control device 40. In this embodiment, virtual objects are pre-associated with predetermined locations in a real space, such as a commercial facility. AR users can play the video game by going to the real space and logging into the display control device 40. VR users can play the video game by logging into the display control device 40 from, for example, their home.

[0012] Display device 20A is worn on the head of user UA. Display device 20B is worn on the head of user UB. Display device 20A and display device 20B are HMD devices that display virtual objects without obstructing the field of view of the user wearing the display device. In other words, user UA and user UB in this embodiment are AR users. Hereafter, when it is not necessary to distinguish between display device 20A and display device 20B, display device 20A and display device 20B will be referred to as display device 20. Display device 20 is equipped with an imaging function (e.g., a camera). Display device 20 captures an image of the real space corresponding to the field of view of the wearer using its imaging function. The image captured by the imaging function of display device 20 is used to determine the position of display device 20 in real space and the orientation of the optical axis of display device 20. The position of display device 20 in real space corresponds to the position of the user wearing display device 20 in real space, and the orientation of the optical axis of display device 20 corresponds to the direction in which the user wearing display device 20 is looking, i.e., the orientation of the user.

[0013] The display device 30 is worn on the head of the user UC. The display device 30 is an HMD (Head-Mounted Display) device that obscures the field of view of the user UC, who is wearing the display device 30, and displays virtual objects superimposed on an image of a virtual space corresponding to the real space. In other words, the user UC in this embodiment is a VR user.

[0014] Each of the mobile devices 10A, 10B, and 10C is, for example, a smartphone. Hereafter, when it is not necessary to distinguish between each of the mobile devices 10A, 10B, and 10C, each will be referred to as mobile device 10.

[0015] The portable device 10A is worn on the body of the user UA by hanging it from the user UA's neck using a strap or the like. The portable device 10A is connected to the display device 20A by wire. The portable device 10B is worn on the body of the user UB by hanging it from the user UB's neck using a strap or the like. The portable device 10B is connected to the display device 20B by wire. The portable device 10C is worn on the body of the user UC by hanging it from the user UC's neck using a strap or the like. The portable device 10C is connected to the display device 30 by wire. Note that the connection between the display device 20 or the display device 30 and the portable device 10 may be a wireless connection. The portable device 10 is not limited to a smartphone, and may be, for example, a tablet or a notebook personal computer.

[0016] In addition, the portable device 10 communicates with the display control device 40 via the communication network NW. More specifically, the portable device 10 transmits a login message for logging in to the display control device 40 or a logout message for logging out from the display control device 40 in response to a user's operation. The login message includes user identification information, login type information, and position notification information. The user identification information uniquely indicates the user of the portable device 10 that is the source of the login message. The login type information indicates whether the user of the portable device 10 that is the source of the login message logs in as an AR user or a VR user. The logout message includes user identification information that uniquely indicates the user who logs out from the display control device 40. Specific examples of the user identification information include, for example, a character string representing the user's name. The AR user is an example of the user of the first aspect in the present disclosure. The VR user is an example of the user of the second aspect in the present disclosure.

[0017] The location notification information included in the login message transmitted from the mobile device 10 used by the AR user is information for causing the display control device 40 to identify the position and orientation of the AR user in the real space. Specifically, the location notification information is image data representing an image captured by the display device 20. The location notification information included in the login message transmitted from the mobile device 10 used by the VR user is information indicating the position occupied by the avatar corresponding to the VR user and the orientation of the avatar in the virtual space corresponding to the real space. Further, after transmitting the login message, the mobile device 10 transmits notification messages at a predetermined time interval until a logout message is transmitted. This notification message includes user identification information and location notification information.

[0018] The display control device 40 is a server device that provides a content management service for performing display control of virtual objects on the display device 20 or the display device 30. Further, the display control device 40 provides a self-position recognition service for the AR user. The self-position recognition service refers to a service for identifying the position and orientation of the AR user in the global coordinate system based on an image captured by the imaging function of the display device 20. Specific implementation modes of the self-position recognition service include a mode using tags such as AR tags, or a mode using the distribution of feature points extracted from images such as SLAM (Simultaneous Localization and Mapping) or VPS (Visual Positioning Service).

[0019] The display control device 40 has virtual object information pre-stored in association with the position and orientation in the global coordinate system. The virtual object information represents an image of a virtual object to be displayed overlaid on real space, as seen from the corresponding position in the global coordinate system in the corresponding orientation in the global coordinate system. For AR users, the display control device 40 determines the position and orientation of the AR user in real space by performing a self-position recognition service using the position notification information contained in the login message or notification message. Then, the display control device 40 generates AR user image data based on the virtual object information corresponding to the determined position and orientation, and transmits this AR user image data.

[0020] For VR users, the display control device 40 converts the virtual space position and orientation indicated by the location notification information contained in the login message or notification message into a real-world position and orientation. Subsequently, the display control device 40 generates VR user image data based on the virtual object information corresponding to the real-world position and orientation, and transmits this VR user image data. The mobile device 10 displays the image represented by the image data received from the display control device 40 on the connected display device, thereby allowing the user to experience the virtual object.

[0021] Figure 2 shows an example of image GAR1 represented by AR user image data generated by the display control device 40, and an example of image GVR1 represented by VR user image data. Figure 2 illustrates image GAR1 and image GVR1 when user UA and user UC share, for example, a virtual object of a robot. The virtual object of the robot is an example of a first virtual object.

[0022] Image GAR1 is an image in which an avatar AUC is superimposed on a content image GC. The content image GC contains virtual objects. The avatar AUC represents a user UC that shares the virtual objects contained in the content image GC with the user UA. Areas other than the virtual objects in the content image GC are transparent. Therefore, even if image GAR1 is displayed on the display device 20A, the field of view of the user UA wearing the display device 20A on their head will not be completely obstructed. The content image GC may be composed of superimposed images of multiple layers. The multiple layers include a first layer in which virtual objects of game characters are placed and a second layer in which virtual objects of advertisements are placed, etc. Each of the multiple layers corresponds to the type of virtual object placed.

[0023] Image GVR1 includes a content image GC containing virtual objects, an avatar AUA, and a background image GB. The avatar AUA represents a user UA that shares the virtual objects contained in the content image GC with the user UC. The background image GB represents a virtual space that mimics the real space to which the virtual objects contained in the content image GC are associated. In image GVR1, the content image GC is superimposed on the background image GB. The avatar AUA is also superimposed on the content image GC. In areas of image GVR1 that are different from both the virtual objects and the avatar AUA, the background image GB is visible, and therefore these areas are not transparent. Consequently, when the display device 30 displays image GVR1, the user UC's field of view is completely obscured.

[0024] Figure 3 is a block diagram showing an example configuration of the display control device 40. The display control device 40 includes a communication device 410, a storage device 420, a processing device 430, and a bus 440. The communication device 410, the storage device 420, and the processing device 430 are interconnected by a bus 440 for communicating information. The bus 440 may be configured using a single bus, or different buses may be configured for each element of the device, etc.

[0025] The communication device 410 is hardware (transceiver / receiver device) for communicating with the mobile device 10 via the communication network NW. The storage device 420 is a recording medium that the processing unit 430 can read. The storage device 420 may be composed of, for example, ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), RAM (Random Access Memory), or a combination thereof. The storage device 420 pre-stores a management table TBL and program PR1. Although not shown in detail in Figure 3, the storage device 420 also stores coordinate transformation data for converting between position and orientation in real space and position and orientation in virtual space.

[0026] The management table TBL is a table for managing the login status of each user who has been pre-registered as a user of the content management service. Figure 4 shows an example of the management table TBL. As shown in Figure 4, the management table TBL stores login status information, avatar information, and location information, associated with user identification information. In this embodiment, the user identification information for user UA is U_A, the user identification information for user UB is U_B, and the user identification information for user UC is U_C.

[0027] Login status information indicates whether a user is logged in or not. If a user is logged in, the login status information indicates whether the user is logged in as an AR user or a VR user. In this embodiment, the login status information is set to a value of 0, 1, or 2. For users who are not logged in, the login status information is set to 0. For users logged in as an AR user, the login status information is set to 1. For users logged in as a VR user, the login status information is set to 2. The contents stored in the management table TBL shown in Figure 4 represent that user UA is logged in as an AR user, user UB is not logged in, and user UC is logged in as a VR user.

[0028] Avatar information represents the avatar registered by the user indicated by the user identification information associated with that avatar information. A specific example of avatar information is image data representing the avatar's image. Location information indicates the real-world location and orientation of the AR user if the user indicated by the user identification information associated with that location information is logged in as an AR user. If the user indicated by the user identification information associated with the location information is logged in as a VR user, the location information indicates the real-world location and orientation calculated from the virtual-world location and orientation of the VR user (VR user's avatar) based on coordinate transformation data. In Figure 4, NULL means that location information is not registered. The contents of the management table TBL are updated each time a login message, logout message, or notification message is received.

[0029] The processing unit 430 includes one or more CPUs (Central Processing Units). The processing unit 430 reads program PR1 from the storage device 420 when the display control device 40 is powered on. By executing program PR1, the processing unit 430 functions as the management unit 430a and display control unit 430b shown in Figure 3. In other words, the management unit 430a and display control unit 430b shown in Figure 3 are software modules realized by a computer such as a CPU operating according to software such as a program. The functions performed by the management unit 430a and display control unit 430b are as follows:

[0030] The management unit 430a manages the login status of users who have been pre-registered as users of the content management service. More specifically, when the communication device 410 receives a login message, the management unit 430a updates the login status information associated with the user identification information contained in the received login message to 1 or 2, depending on the login type information contained in the login message. The management unit 430a also updates the location information associated with the user identification information contained in the received login message, according to the location notification information contained in the login message. More specifically, for AR users, the management unit 430a first determines the AR user's location and orientation in real space by executing a self-location recognition service using the location notification information contained in the login message. The management unit 430a then updates the location information associated with the user identification information contained in the login message to location information indicating the determined location and orientation. For VR users, the management unit 430a first converts the location and orientation in virtual space indicated by the location notification information contained in the login message into a location and orientation in real space using coordinate transformation data. The management unit 430a then updates the location information associated with the user identification information contained in the login message with location information indicating the position and orientation obtained by the conversion.

[0031] Furthermore, when the management unit 430a receives a notification message, it updates the location information associated with the user identification information contained in the notification message based on the location notification information contained in the notification message. Then, when the communication device 410 receives a logout message, the management unit 430a updates the login status information associated with the user identification information contained in the received logout message to 0.

[0032] The display control unit 430b generates AR user image data for AR users who are logged in and have been pre-registered as users of the content management service. The display control unit 430b transmits the AR user image data to the AR user. The display control unit 430b generates VR user image data for VR users who are logged in. The display control unit 430b transmits the VR user image data to the VR user. More specifically, the display control unit 430b controls the display of virtual objects for logged-in users. The display control unit 430b switches the display control of avatars representing other users who share virtual objects with the logged-in user based on whether the logged-in user is an AR user or a VR user. Being an AR user means that the user has logged in as an AR user. Being a VR user means that the user has logged in as a VR user. Avatars representing other users who share virtual objects with the logged-in user are an example of a second virtual object in this disclosure. Users who are subject to the generation of AR user image data or users who are subject to the generation of VR user image data are an example of a first user in this disclosure. Other users who share virtual objects with the first user are an example of the second user in this disclosure.

[0033] More specifically, the display control unit 430b hides the avatar representing other users if those users are also AR users for AR users. For AR users, the display control unit 430b generates AR user image data that displays the avatar representing other users if those users are also VR users. On the other hand, for VR users, the display control unit 430b generates VR user image data that displays the avatar representing other users who share virtual objects with the user, regardless of whether those users are AR users or VR users.

[0034] Furthermore, the processing unit 430, which operates according to program PR1, executes the control method shown in Figure 5 at predetermined intervals, such as 10 millisecond intervals. As shown in Figure 5, this control method includes a determination process SA110, a management process SA120, and a display control process SA130.

[0035] In the determination process SA110, the processing unit 430 determines whether the communication device 410 has received a login message, notification message, or logout message. If the communication device 410 has received a login message, notification message, or logout message, the determination result of the determination process SA110 is "Yes," i.e., affirmative. If the determination result of the determination process SA110 is "Yes," the processing unit 430 executes the management process SA120 and the display control process SA130. If the communication device 410 has not received any of the login message, notification message, or logout message, the determination result of the determination process SA110 is "No," i.e., negative. If the determination result of the determination process SA110 is "No," the processing unit 430 executes the display control process SA130 without executing the management process SA120.

[0036] In management process SA120, the processing unit 430 functions as a management unit 430a. In management process SA120, the processing unit 430 updates the contents of the management table TBL in response to messages received by the communication device 410.

[0037] In the display control process SA130, the processing unit 430 functions as a display control unit 430b. In the display control process SA130, the processing unit 430 generates image data for AR users who are logged in and are pre-registered as users of the content management service. Subsequently, the processing unit 430 transmits the image data for AR users to the AR users. In the display control process SA130, the processing unit 430 generates image data for VR users who are logged in and are pre-registered as users of the content management service. Subsequently, the processing unit 430 transmits the image data for VR users to the VR users.

[0038] Assuming that none of the users UA, UB, and UC are logged in, a login message MA is sent from mobile device 10A at time T1, as shown in Figure 6. Furthermore, a login message MC is sent from mobile device 10C at time T2, which is later than time T1. And a login message MB is sent from mobile device 10B at time T3, which is even later than time T2. Before time T1, since none of the users UA, UB, and UC are logged in, the login status information stored in the management table TBL is all 0, as shown in Figure 7.

[0039] At time T1, when user UA performs an operation to log in to mobile device 10A as an AR user, a login message MA is sent from mobile device 10A to display control device 40. When login message MA is received by communication device 410, the determination result of determination process SA110 becomes "Yes", and management process SA120 is executed. In this management process SA120, as shown in Figure 8, the processing device 430 updates the login status information associated with user UA's user identification information "U_A" from 0 to 1. At time T1, users UC and UB are not logged in. Therefore, in display control process SA130, the processing device 430 generates AR user image data representing image GAR2 consisting only of content image GC, as shown in Figure 9. Subsequently, the processing device 430 sends this image data to mobile device 10A. As a result, image GAR2 is displayed on display device 20A.

[0040] Next, at time T2, when user UC performs an operation on the mobile device 10C to log in as a VR user, a login message MC is sent from the mobile device 10C to the display control device 40. When the login message MC is received by the communication device 410, the determination result of the determination process SA110 becomes "Yes", and the management process SA120 is executed. In this management process SA120, the processing device 430 updates the login status information associated with user UC's user identification information "U_C" from 0 to 2. As a result, the contents of the management table TBL are in the state shown in Figure 4.

[0041] At time T2, user UA and user UC log in and share a virtual object. Therefore, in the display control process SA130, the processing unit 430 executes display control to display the avatar of a VR user who is one of the other users sharing the virtual object with the AR user. In addition, in the display control process SA130, the processing unit 430 executes display control to display the avatar of a VR user who is sharing the virtual object with the VR user. As a result, AR user image data representing image GAR1 in Figure 2 is sent to user UA. VR user image data representing image GVR1 in Figure 2 is sent to user UC. Image GAR1 is displayed on display device 20A, and image GVR1 is displayed on display device 30. User UA, upon viewing image GAR1, can identify user UC who is sharing the virtual object with user UA through avatar AUC. User UC, upon viewing image GVR1, can identify user UA who is sharing the virtual object with user UC through avatar AUA.

[0042] Next, at time T3, when user UB performs an operation on the mobile device 10B to log in as an AR user, a login message MB is sent from the mobile device 10B to the display control device 40. When the login message MB is received by the communication device 410, the determination result of the determination process SA110 becomes "Yes", and the management process SA120 is executed. In this management process SA120, the processing device 430 updates the login status information associated with user UB's user identification information "U_B" from 0 to 1. As a result, the contents of the management table TBL become as shown in Figure 10.

[0043] At time T3, users UA, UC, and UB log in and share a virtual object. In this case, in the display control process SA130, the processing unit 430 executes display control for the AR user to hide the avatars of other AR users who share the virtual object with the AR user. The processing unit 430 also executes display control for the AR user to display the avatars of VR users who share the virtual object with the AR user. Furthermore, the processing unit 430 executes display control for the VR user to display the avatars of other users who share the virtual object with the VR user, regardless of whether those other users are AR users or VR users. Consequently, AR user image data representing image GAR1 in Figure 11 is transmitted to users UA and UB, and VR user image data representing image GVR2 in Figure 11 is transmitted to user UC. As a result, image GAR1 is displayed on display devices 20A and 20B, and image GVR2 is displayed on display device 30. Image GVR2 includes avatar AUA, which represents the user UA, and avatar AUB, which represents the user UB.

[0044] A user UA who views image GAR1 displayed on display device 20A can identify user UC who shares the virtual object through avatar AUC. Although user UB's avatar is not displayed on display device 20A, user UA can see user UB through display device 20A, thereby identifying user UB. Similarly, a user UB who views image GAR1 displayed on display device 20B can identify user UC who shares the virtual object through avatar AUC. Although user UA's avatar is not displayed on display device 20B, user UB can see user UA through display device 20B, thereby identifying user UA. Furthermore, a user UC who views image GVR2 displayed on display device 30 can identify user UA through avatar AUA, and user UB can identify user UB through avatar AUB.

[0045] As described above, according to this embodiment, virtual objects can be shared by AR users and VR users, that is, AR and VR can be mixed in a single display system 1.

[0046] In addition, according to this embodiment, if another user sharing the virtual object with the AR user is a VR user, an avatar representing that VR user is displayed to the AR user. Therefore, the AR user can identify VR users among the other users sharing the virtual object through the avatar. Furthermore, the AR user can identify AR users among the other users sharing the virtual object through the display device 20. Therefore, the AR user can identify other users sharing the virtual object with the AR user, regardless of whether those users are AR users or VR users.

[0047] On the other hand, for VR users, an avatar representing other users who share the virtual object with the VR user is displayed, regardless of whether those other users are AR users or VR users. Therefore, VR users can recognize other users through the displayed avatar. Accordingly, according to this embodiment, there is no hindrance to multiple users sharing a virtual object visually recognizing each other.

[0048] Thus, according to this embodiment, it becomes possible to combine AR and VR in a single display system without hindering multiple users from visually perceiving each other.

[0049] (B: Transformation) This disclosure is not limited to the embodiments illustrated above. Specific variations are as follows. Two or more embodiments arbitrarily selected from the following examples may be combined.

[0050] (B-1: Torture 1) In the above embodiment, program PR1 was stored in the storage device 420 of the display control device 40, but program PR1 may be manufactured or sold as a standalone product. When selling program PR1, possible ways of providing program PR1 to the buyer include distributing a computer-readable recording medium such as a flash ROM on which program PR1 is written, or distributing program PR1 by download via a telecommunications line.

[0051] (B-2: Variation 2) In the above embodiment, the management unit 430a and the display control unit 430b were software modules. However, either one or both of the management unit 430a and the display control unit 430b may be hardware modules. Specific examples of hardware modules include DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), PLD (Programmable Logic Device), FPGA (Field Programmable Gate Array), etc.

[0052] (C: Other) (1) In the embodiments described above, ROM and RAM were given as examples of the storage device 420, but the storage device 420 may be a flexible disk, magneto-optical disk (e.g., compact disk, digital multipurpose disk, Blu-ray® disk), smart card, flash memory device (e.g., card, stick, key drive), CD-ROM (Compact Disc-ROM), register, removable disk, hard disk, floppy® disk, magnetic strip, database, server, or other suitable storage medium.

[0053] (2) In the embodiments described above, the information, signals, etc. may be represented using any of the various different technologies. For example, the data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be mentioned throughout the above description may be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.

[0054] (3) In the embodiments described above, the input and output information may be stored in a specific location (e.g., memory) or managed using a management table. The input and output information may be overwritten, updated, or appended to. The output information may be deleted. The input information may be transmitted to other devices.

[0055] (4) In the embodiments described above, the determination may be made by a value represented by 1 bit (0 or 1), by a boolean value (true or false), or by a numerical comparison (for example, a comparison with a predetermined value).

[0056] (5) The processing procedures, sequences, flowcharts, etc., exemplified in the embodiments described above may be reordered, as long as they do not contradict each other. For example, in the methods described herein, the elements of various steps are presented using an exemplary order and are not limited to the specific order presented.

[0057] (6) Each function illustrated in Figure 3 is implemented by any combination of at least one of hardware and software. Furthermore, the method of implementing each function block is not particularly limited. That is, each function block may be implemented using one device that is physically or logically coupled, or it may be implemented using two or more physically or logically separated devices that are directly or indirectly connected (for example, using wired, wireless, etc.). A function block may also be implemented by combining the one or more devices with software.

[0058] (7) In the embodiments described above, the program should be interpreted broadly to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, execution threads, procedures, functions, etc., whether the software is called software, firmware, middleware, microcode, hardware description language or by any other name.

[0059] Furthermore, software, instructions, information, etc., may be transmitted and received via a transmission medium. For example, if software is transmitted from a website, server, or other remote source using at least one of wired technology (coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), etc.) and wireless technology (infrared, microwave, etc.), at least one of these wired and wireless technologies is included in the definition of a transmission medium.

[0060] (8) In each of the above-mentioned forms, the terms “system” and “network” shall be used interchangeably.

[0061] (9) The information, parameters, etc. described in this disclosure may be expressed using absolute values, relative values ​​from a given value, or other corresponding information.

[0062] (10) In the embodiments described above, the portable device may be a Mobile Station (MS). A Mobile Station may also be referred to by those skilled in the art as a subscriber station, mobile unit, subscriber unit, wireless unit, remote unit, mobile device, wireless device, wireless communication device, remote device, mobile subscriber station, access terminal, mobile terminal, wireless terminal, remote terminal, handset, user agent, mobile client, client, or several other appropriate terms. In this disclosure, terms such as “Mobile Station,” “User Terminal,” “User Equipment (UE),” and “Terminal” may be used interchangeably.

[0063] (11) In the embodiments described above, the terms “connected,” “coupled,” or any variation thereof, mean any direct or indirect connection or coupling between two or more elements, and may include the presence of one or more intermediate elements between two elements that are “connected” or “coupled” with each other. The coupling or connection between elements may be physical, logical, or a combination thereof. For example, “connection” may be read as “access.” As used in this disclosure, two elements may be considered to be “connected” or “coupled” with each other using at least one of one or more wires, cables, and printed electrical connections, and, in some non-limiting and non-exclusive examples, electromagnetic energy having wavelengths in the radio frequency domain, microwave domain, and optical (both visible and invisible) domain.

[0064] (12) In the embodiments described above, the phrase “based on” does not mean “based solely on” unless otherwise specified. In other words, the phrase “based on” means both “based solely on” and “based at least on.”

[0065] (13) The terms “determining” and “determining” as used in this disclosure may encompass a wide variety of actions. “Determining” may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, searching, inquiry (e.g., searching in a table, database or other data structure), ascertaining, etc. “Determining” may also include receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, accessing (e.g., accessing data in memory), etc. Furthermore, "judgment" and "decision" can include considering something as having been "judged" or "decided" after resolving, selecting, choosing, establishing, comparing, etc. In other words, "judgment" and "decision" can include considering something as having been "judged" or "decided" after some action. Also, "judgment (decision)" can be reinterpreted as "assuming," "expecting," "considering," etc.

[0066] (14) Where the terms “include,” “including,” and variations thereof are used in the embodiments described above, these terms are intended to be inclusive, as is the term “comprising.” Furthermore, the term “or” as used in this disclosure is not intended to be exclusive OR.

[0067] (15) In the present disclosure, if articles are added by translation, such as a, an, and the in English, the present disclosure may include the fact that the noun following these articles is plural.

[0068] (16) In this disclosure, the term “A and B are different” may mean “A and B are different from each other.” The term may also mean “A and B are each different from C.” Terms such as “separate” and “combine” may be interpreted in the same way as “different.”

[0069] (17) Each aspect / embodiment described herein may be used individually, in combination, or switched between as needed in practice. Furthermore, notification of certain information (e.g., notification that "X is") is not limited to explicit notification, but may also be implicit (e.g., by not providing such notification).

[0070] (D: Aspects as understood from the above-described forms or modifications) Although the present disclosure has been described in detail above, it will be clear to those skilled in the art that the present disclosure is not limited to the embodiments described herein. The present disclosure can be implemented in modified and altered forms without departing from the spirit and scope of the present disclosure as defined by the claims. Accordingly, the descriptions in the present disclosure are for illustrative purposes only and are not intended to be restrictive in any way. The following forms can be understood from at least one of the embodiments or modifications described above.

[0071] The display control device according to the first embodiment comprises a management unit and a display control unit. The management unit manages whether each of the first user and the second user who share the first virtual object has logged in as a first-mode user, in which the first virtual object is displayed superimposed on real space, or as a second-mode user, in which the first virtual object is displayed superimposed on an image of virtual space corresponding to real space. An AR user is an example of a first-mode user in this disclosure. A VR user is an example of a second-mode user in this disclosure. The display control unit switches the display control of the second virtual object representing the second user to the first user based on whether the first user has logged in as a first-mode user or a second-mode user. An avatar is an example of a second virtual object in this disclosure. In other words, according to the display control device according to the first embodiment, if the first user is an AR user, the display control of the avatar representing the second user to the first user is switched based on whether the second user is an AR user or a VR user. This makes it possible to mix AR and VR without hindering the first user and the second user from visually perceiving each other.

[0072] In the second embodiment (an example of the first embodiment), the display control unit hides the second virtual object based on the fact that the first user logs in as the first embodiment user when the second user is logged in as the first embodiment user. The display control unit displays the second virtual object based on the fact that the first user logs in as the first embodiment user when the second user is logged in as the second embodiment user. The display control unit also displays the second virtual object based on the fact that the first user logs in as the second embodiment user. According to the display control device in the second embodiment, if the first user is an AR user, an avatar representing the second user is displayed to the first user if the second user is a VR user. If the first user is an AR user, the avatar is hidden if the second user is an AR user. According to the display control device in the second embodiment, if the first user is a VR user, an avatar representing the second user is displayed to the first user regardless of whether the second user is a VR user or an AR user. According to the display control device of the second embodiment, if the first user is an AR user, the display control of the avatar representing the second user to the first user is switched based on whether the second user is an AR user or a VR user. This makes it possible to mix AR and VR without hindering the first and second users from visually recognizing each other.

[0073] The third embodiment of the display system comprises a display device and a display control device that controls the display of a first virtual object on the display device. The display control device comprises a management unit and a display control unit. The management unit manages whether each of the first user using the display device and the second user sharing the first virtual object with the first user logged in as a first-mode user, in which the first virtual object is displayed superimposed on real space, or as a second-mode user, in which the first virtual object is displayed superimposed on an image of virtual space corresponding to real space. The display control unit switches the display control of the second virtual object representing the second user on the display device based on whether the first user logged in as a first-mode user or a second-mode user. According to the third embodiment of the display system, similar to the display control device in the first embodiment, if the first user is an AR user, the display control of the avatar representing the second user to the first user is switched based on whether the second user is an AR user or a VR user. Therefore, it becomes possible to mix AR and VR without hindering the first user and the second user from visually perceiving each other. [Explanation of Symbols]

[0074] 1...Display system, 10, 10A, 10B, 10C...Portable devices, 20, 20A, 20B, 30...Display devices, 40...Display control devices, 410...Communication devices, 420...Storage devices, 430...Processing devices, 430a...Management unit, 430b...Display control unit, 19, 440...Bus, PR1...Program, TBL...Management table.

Claims

1. A management unit manages whether each of the first user and the second user who share the first virtual object logged in as a first-mode user in which the first virtual object is displayed superimposed on the real space, or as a second-mode user in which the first virtual object is displayed superimposed on an image of the virtual space corresponding to the real space. A display control unit that switches the display control of a second virtual object representing the second user to the first user based on whether the first user logged in as a first-mode user or a second-mode user, A display control device for virtual objects, equipped with the following features.

2. The display control unit, In a situation where the second user is logged in as the first user, the second virtual object is hidden based on the first user logging in as the first user. Based on the situation in which the second user is logged in as the second type user, and the first user is logged in as the first type user, the second virtual object is displayed. Based on the first user logging in as the second user, the second virtual object is displayed. A display control device for virtual objects according to claim 1.

3. Display device and The display device comprises a display control device that controls the display of a first virtual object in the display device, The aforementioned display control device is A management unit manages whether each of the first user using the display device and the second user sharing the first virtual object with the first user logged in as a first-mode user in which the first virtual object is displayed superimposed on real space, or as a second-mode user in which the first virtual object is displayed superimposed on an image of the virtual space corresponding to the real space. The system includes a display control unit that switches the display control of the display device for a second virtual object representing the second user based on whether the first user logged in as a first-mode user or a second-mode user. Display system.

Images