Communication System

Abstract

To provide a communication system for allowing participants of a real event and a simulated event to feel a sense of unity.SOLUTION: A communication system 1 comprises: a storage unit for storing display data of a simulated event, corresponding position information between a real event space and a virtual event space, and avatar display data of a first user participating in the real event and a second user participating in the simulated event; a first terminal 30 including an image acquisition unit for acquiring an image of the real event space from a viewpoint corresponding to a position of the first user, and a display unit; a first terminal position acquisition unit; and a second terminal 50 including an input unit and a display unit. The second terminal further includes: a second user position acquisition unit for acquiring position information of an avatar of the second user; and a second display control unit for causing the display unit of the second terminal to display an image of the simulated event in which an avatar of the first user is placed. The first terminal also includes a first display control unit for causing the display unit of the first terminal to display an image in which the avatar of the second user is placed.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a technique used for linking an event taking place in a real space with a simulated event taking place in a virtual space that imitates the event. [Background technology]

[0002] In recent years, with the spread of high-speed Internet networks and advances in digital technology, systems that can hold various types of events in virtual spaces on the Internet are being used. For example, there is a system that can hold simulated events in virtual spaces that imitate events such as music concerts and costume contests held in real spaces. Patent Document 1 also describes holding online classes in a virtual space that has a room that imitates a classroom. In these systems, users participate in the simulated event by making their own avatars that they have set in advance appear in the virtual space. Participants in the simulated event can have the same experience as participating in an event held in real space by moving their avatars in the virtual space. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2013-88878 A Summary of the Invention [Problem to be solved by the invention]

[0004] By using the above-mentioned system, it is possible to hold a simulated event in a virtual space that imitates an event (real event) held in the real space so that the simulated event progresses simultaneously with the real event. However, in the conventional system, the real event held in the real space and the simulated event held in the virtual space progress independently of each other, and therefore there was a problem that participants of these events could not feel a sense of unity as if they were participating in a common event.

[0005] The problem to be solved by the present invention is to provide a technique that enables participants in an event held in a real space and participants in a simulated event held in a virtual space to feel a sense of unity. [Means for solving the problem]

[0006] The communication system according to the present invention, which has been developed to solve the above problems, comprises: a storage unit that stores display data for displaying in a virtual event space a simulated event corresponding to a real event held in a real event space, corresponding position information that associates a position in the real event space with a position in the virtual event space, display data of an avatar of a first user participating in the real event in the real event space, and display data of an avatar of a second user participating in the simulated event in the virtual event space; an image acquisition unit, which is a terminal used by the first user, that acquires an image of the real event space captured from a viewpoint corresponding to a position of the first user; and Displaying the image acquired by the image acquisition unit A first terminal having a display unit; a first terminal position acquisition unit that acquires information about the position of the first terminal in the real event space; a second terminal used by the second user and equipped with an input unit and a display unit; a second user position acquisition unit that acquires information on a position of an avatar of the second user in the virtual event space, the position being set by an operation via the input unit; a first display control unit that converts the position of the avatar of the second user acquired by the second user position acquisition unit into a position in the real event space based on the corresponding position information, and displays an image in which the avatar of the second user is placed at the converted position on a display unit of the first terminal; a second display control unit that converts the position of the first terminal acquired by the first terminal position acquisition unit into a position in the virtual event space based on the corresponding position information, thereby locating an avatar of the first user, and causes a display unit of the second terminal to display an image of the simulated event captured from a viewpoint corresponding to the position of the avatar of the second user; The present invention is characterized by comprising:

[0007] The communication system according to the present invention holds an event (real event) in an event space (real event space) provided in a real space, and Actual Event This system is used when a simulated event corresponding to the above is held in an event space (virtual event space) provided in a virtual space. In this system, a first user who participates in a real event in a real event space uses a first terminal equipped with an image acquisition unit and a display unit that acquire an image of the real event space captured from a viewpoint corresponding to the position of the first user. Also, a second user who participates in a simulated event in a virtual event space uses a second terminal equipped with an input unit and a display unit. Note that the image captured from a viewpoint corresponding to the position of the first user is an image captured from the viewpoint of the first user if the image acquisition unit is provided in a device such as MR glasses worn by the first user on the head, and is an image captured from a shooting unit of the tablet terminal if the image acquisition unit is provided in a portable device such as a tablet terminal used by the first user.

[0008] While the first user is participating in the real event, information on the location of the first terminal used by the first user moving within the real event space is First terminal location acquisition unit The information on the position of the first terminal may include information on the orientation of the first terminal. The second user can make an avatar appear in the virtual event space and move it freely within the virtual event space by operating the input unit of the second terminal. While the second user is participating in the simulated event, information on the position of the second user's avatar is acquired by the second user position acquisition unit.

[0009] The first display control unit converts the position of the second user's avatar into a position in the real event space based on the corresponding position information stored in advance in the storage unit, and displays an image in which the second user's avatar is placed at that position on the display unit of the first terminal. Therefore, the first user can view an image as if the second user's avatar is present in the real event space. Furthermore, the second display control unit converts the position of the first terminal into a position in the virtual space based on the corresponding position information, places the first user's avatar at that position, and displays an image of the simulated event captured from a viewpoint corresponding to the position of the second user's avatar on the display unit of the second terminal. Therefore, the second user can view an image of the virtual event space including the avatar of the first user in the real event space. Note that the viewpoint corresponding to the position of the second user's avatar is not limited to the viewpoint of the avatar itself, but may be a viewpoint that captures the virtual event space including the avatar from behind the avatar. Moreover, such multiple viewpoints may be switchable.

[0010] In this way, in the communication system of the present invention, a first user participating in a real event in a real event space and a second user participating in a simulated event in a virtual event space can display each other's avatars on their respective devices, allowing the participants in the real event taking place in the real space and the participants in the simulated event taking place in the virtual space to feel a sense of unity.

[0011] The communication system according to the present invention comprises: the storage unit further stores reference position information in the real event space, The first terminal position acquisition unit acquires information on the position of the first terminal by comparing an image of the real event space acquired by the image acquisition unit with the reference position information. It can be said that.

[0012] In the communication system of the above aspect, by comparing an image captured by the image acquisition unit of the first terminal with reference position information in the real event space, it is possible to acquire information on the position and orientation of the first terminal with high accuracy.

[0013] The first terminal may be, for example, MR (Mixed Reality) glasses. MR glasses are glasses-type digital devices. There are two types of MR glasses: one that displays an image such as a three-dimensional hologram on the lens of a glasses-type device worn by the user and allows the user to view it together with the real space directly captured by the user's own eyes (optical see-through type), and one that recognizes and images a real event space through a sensor mounted on the device, superimposes an image such as a three-dimensional hologram on the image, and displays it on a display unit (video see-through type). When using video see-through MR glasses as the first terminal, for example, it is sufficient to compare reference position information included in an image of the real event space acquired through a sensor possessed by the MR glasses with an image of the real event space prepared in advance (for example, reference position information of the real event space). Also, when using optical see-through MR glasses as the first terminal, for example, a configuration can be adopted in which short-distance communication is performed between sensors arranged in advance at multiple positions in the real event space, the position of the first terminal is identified from the response time, and the orientation is identified by an acceleration sensor. MR glasses are worn in the same way as general glasses, so both hands are free. Tablet devices are also examples of video see-through type devices.

[0014] In the communication system according to the present invention, the storage unit further stores data of a plurality of emotion images each of which is an image expressing an emotion of a user; moreover, an emotion image selection receiving unit that receives an operation to select one of the emotion images in one of the first terminal and the second terminal; Equipped with an emotion image display unit that displays the selected emotion image together with an avatar of a user of the terminal whose operation has been accepted by the emotion image selection accepting unit, on the other display unit of the first terminal or the second terminal; It is possible to adopt a configuration having the above.

[0015] In addition, in the communication system according to the present invention, the storage unit further stores data of emotional actions, each of which is an avatar's action expressing an emotion of the user; moreover, an emotional action selection receiving unit that receives an operation to select one of the emotional actions in one of the first terminal and the second terminal; an emotional action executing unit that displays, on the other display unit of the first terminal and the second terminal, an image in which an avatar of a user of the terminal whose operation has been accepted by the emotional action selection accepting unit has executed the selected emotional action; It is possible to adopt a configuration having the above.

[0016] In the communication systems of the above two aspects, a wide range of emotional expressions can be carried out between the first and second users. Effect of the Invention

[0017] By using the communication system according to the present invention, participants in an event held in real space and participants in a simulated event held in virtual space can feel a sense of unity. [Brief description of the drawings]

[0018] [Figure 1] 1 is a schematic diagram showing the overall configuration of an embodiment of a communication system according to the present invention; [Diagram 2] FIG. 2 is a diagram showing the main configuration of a server device according to the present embodiment. [Diagram 3] FIG. 2 is a diagram showing the main configuration of a first terminal in the present embodiment. [Figure 4] FIG. 2 is a diagram showing the main configuration of a second terminal in the embodiment; [Diagram 5]4 is an example of a display on a display unit of a first terminal according to the present embodiment. [Figure 6] 13 is an example of a display on a display unit of a second terminal according to the present embodiment. [Figure 7] 6 is a display example showing the movements of an avatar on the display unit of the first terminal of the present embodiment. [Figure 8] 11 is a display example showing the movements of an avatar on the display unit of the second terminal of the present embodiment. [Figure 9] 13 is a display example in which an explanatory text is displayed in an expanded form on the display unit of the first terminal of the present embodiment. [Figure 10] 13 is a display example in which an explanatory text is expanded and displayed on the display unit of the second terminal of the present embodiment. [Figure 11] 13 is a display example in which video is synchronously displayed on the second terminal in this embodiment. [Figure 12] 13 is another display example of the display unit of the first terminal of the present embodiment. [Figure 13] 13 is an example of a chat screen displayed on the display unit of the first terminal of the present embodiment. [Figure 14] 13 is another display example of the display unit of the second terminal of the present embodiment. [Figure 15] 4 is an example of a chat screen displayed on a display unit of the first terminal or the second terminal in the present embodiment. [Figure 16] 13 is an example of a participant list screen displayed on a display unit of the first terminal or the second terminal in the present embodiment. [Figure 17] 4A to 4C are diagrams for explaining a manner in which an image acquired by a first terminal is displayed on a display unit of a second terminal in the present embodiment. [Figure 18] 13 is an example of a state in which an image acquired by a first terminal is displayed full screen on a display unit of a second terminal in this embodiment. [Figure 19] 13 is an example of a state in which an image acquired by a first terminal is displayed on a small screen on a display unit of a second terminal in this embodiment. [Figure 20] 13 is an example of a state in which an image captured by an internal camera of a first terminal is displayed on a display unit of a second terminal in this embodiment. [Figure 21] FIG. 1 is a diagram for explaining a digital twin embodied in the communication system of this embodiment. [Figure 22]13 is an example in which the communication system of this embodiment is applied to an outdoor event. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] An embodiment of a communication system according to the present invention will be described below with reference to the drawings. A communication system 1 of the present embodiment is used when simultaneously holding a real event in a real event space and a simulated event that imitates the real event in a virtual event space.

[0020] FIG. 1 shows a schematic configuration of the entire communication system 1 of this embodiment. The communication system 1 of this embodiment includes a server device 10, a first terminal 30, and a second terminal 50, which are connected to each other via the Internet. The first terminal 30 is used by a user (first user) who participates in a real event in a real event space. The second terminal 50 is used by a user (second user) who participates in a simulated event in a virtual event space. Although FIG. 1 shows two first terminals 30 and three second terminals 50, the number of these can be changed arbitrarily.

[0021] 2 shows the main configuration of the server device 10. In addition to the storage unit 11, the server device 10 includes a user registration reception unit 21, a participation reception unit 22, and a user intermediation unit 23 as functional blocks. The server device 10 of this embodiment is provided as a cloud server. The server device 10 may be any computer device capable of communicating with the first terminal 30 and the second terminal 50, and a personal computer or the like may also be used. The server device 10 may be provided with an input unit and a display unit as necessary.

[0022] The storage unit 11 is provided with a user information storage unit 12 and an event data storage unit 13. User information is stored in the user information storage unit 12. The user information may include identification information (e.g., identification number) of each user, information on the user name and password, contact information (e.g., email address), the type and identification information of the terminal used by each user (e.g., the model number and identification number of the terminal. If the user has multiple terminals, the type and identification information for each terminal), display data of each user's avatar used to display in the virtual event space, and identification information of users who have interactions with each user.

[0023] The event data storage unit 13 stores image data of a space (real event space) where a real event is performed, data of a plurality of reference positions, display data of a space (virtual event space) where a simulated event is performed, and corresponding position information that associates a position in the real event space with a position in the virtual event space. The corresponding position information is stored as information that associates a plurality of reference positions provided in the real event space with a position in the virtual space that corresponds to each of the plurality of reference positions. This information can be created, for example, by using an AR marker or the like at each reference position, or by using a technology called VPS (Visual Positioning Service / System) that analyzes an image from a camera attached to a smartphone or tablet to identify the position of the camera in a three-dimensional map prepared in advance. For example, a position having a visual feature such as a corner of an object existing in the space (a position that is easy to identify) is selected as the reference position in the real event space. The information of the plurality of reference positions provided in the real event space corresponds to the reference position information in the present invention. Note that the plurality of reference positions in the real event space included in the reference position information and the plurality of reference positions in the real event space used in the corresponding position information may be the same or different.

[0024] FIG. 3 shows the main configuration of the first terminal 30. The first terminal 30 comprises a memory unit 31 and an image acquisition unit 32. The first terminal 30 also comprises, as functional blocks, a first terminal position acquisition unit 33, a first display control unit 34, an expanded display unit 35, an emotion image selection receiving unit 36, and an emotion action selection receiving unit 37. These functional blocks are realized by running a dedicated application pre-installed in the first terminal 30, downloading a script (described later) from the server device 10, and executing a program for the first terminal on a processor. The above application contains data that is commonly used regardless of the content of an event (for example, Emotion display button 731 and Action display The function includes a function to read and display the data (data for buttons 732, etc.) from the storage unit 31. On the other hand, a script is a program that is created for each event and performs a process specific to each event.

[0025] The script downloaded to the first terminal 30 includes a program for performing a process of identifying the position and orientation of the first terminal 30 from an image captured in the real event space using reference position information of the real event space stored in the event storage unit 13, and a process of converting a position in the virtual event space into the real event space using corresponding position information. For example, this program may parameterize the reference position information included in the image of the real event space acquired by a sensor at a predetermined time interval, and calibrate the position and orientation of the first terminal 30 by comparing it with reference position information indexed in advance.

[0026] For example, a portable device such as a tablet terminal or a smartphone, or a head-mounted device such as MR glasses is used as the first terminal 30, and a camera of these devices is used as the image acquisition unit 32. The first terminal 30 is also provided with an input unit 41 and a display unit 42. As described above, in this embodiment, each functional block is realized by data and a program contained in a dedicated application and a downloaded script, but these functional blocks may be realized only by a dedicated application or only by a script.

[0027] 4 shows the main configuration of the second terminal 50. The second terminal 50 includes a storage unit 51. The second terminal 50 also includes, as functional blocks, a second user position acquisition unit 52, a second display control unit 53, an expanded display unit 54, an emotion image selection receiving unit 55, and an emotion action selection receiving unit 56. These functional blocks are embodied by running a dedicated application preinstalled in the second terminal 50, downloading a script described below from the server device 10, and executing a program for the second terminal on the processor. This application also includes data that is commonly used regardless of the content of the event (for example, data such as an icon 731 for expressing emotions and an action button 732) and a function for reading and displaying the data from the storage unit 51.

[0028] The script downloaded to the second terminal 50 includes a program that converts a position in the real event space into a virtual event space using the corresponding position information stored in the event storage unit 13. The script is a program that is created for each event and performs processing unique to each event.

[0029] For example, a stationary device such as a personal computer, a portable device such as a tablet terminal or a smartphone, or a head-mounted device such as VR glasses is used as the second terminal 50. The second terminal 50 is also provided with an input unit 61 and a display unit 62. As described above, in this embodiment, each functional block is realized by data and a program contained in a dedicated application and a downloaded script, but these functional blocks may be realized only by a dedicated application or only by a script.

[0030] Next, a flow of events when a real event and a simulated event are held using the communication system 1 of this embodiment will be described.

[0031] When a real event and a simulated event are held, information (such as the event name) that identifies a set of a real event and a simulated event (hereinafter, these are also collectively referred to as an "event set") and information on the date and time of the event are stored in the event data storage unit 13 in advance. In addition, image data of the space where the real event is held (real event space) and display data of the space where the simulated event is held (virtual event space) are prepared and stored in the event data storage unit 13. Furthermore, a plurality of positions in the real event space are set as reference positions, positions in the virtual space corresponding to each of the plurality of reference positions (corresponding reference positions) are determined, corresponding position information that associates the two is created and stored in the event data storage unit 13. Furthermore, data that associates the items, etc. with the text, etc. for each part of the items, such as exhibits, and walls, etc., that are arranged in the real event space and that display text, images, or videos related to the items, etc. (hereinafter, referred to as "extended display targets") is also stored in the event data storage unit 13. Furthermore, the event data storage unit 13 also stores data of images and videos that are planned to be displayed on a screen or the like in the real event space during the real event, as well as information on the display location and timing of the display.

[0032] The event data storage unit 13 also stores scripts (programs) to be executed by the first terminal 30 and the second terminal 50. The program operated by the first terminal 30 performs processing such as identifying the position and orientation of the first terminal 30 by collating an image acquired by the image acquisition unit 32 of the first terminal 30 with data (reference position information) of a plurality of reference positions set in the real event space, and generating display data of the avatar of the second user according to the position and orientation. The program operated by the second terminal 50 performs processing such as generating display data of the avatar of the first user and the avatar of another second user according to the position and orientation of the second user set by an operation via the input unit 61 of the second terminal 50. Details of the operations in the first terminal 30 and the second terminal 50 will be described later.

[0033] When the process of storing these necessary data in the event data storage unit 13 is completed, the system becomes ready to hold real events and simulated events.

[0034] The participation acceptance unit 22 reads out an event set that is in a state where it can be held from among the event sets stored in the storage unit 11 and accepts participation. The participation acceptance unit 22 sends an e-mail containing the address of a website containing the event sets for which participation is accepted to the user's contact information stored in the storage unit 11. The participation acceptance unit 22 also notifies information on the event sets for which participation is accepted through an application pre-installed on a terminal owned by a person who has already been registered as a user. The user who receives the e-mail or notification accesses the website described in the received e-mail or the like, enters his / her user name and password to log in, and then checks information on the event set in which he / she wishes to participate. Then, for the event set in which he / she wishes to participate, he / she selects whether to participate in a real event or a simulated event. If the user selects participation in a real event, a screen for downloading a script to be run on the first terminal 30 is displayed. When the script is downloaded, the participation acceptance unit 22 sets the user as the first user in the event set, sets the terminal of the user stored in the user information storage unit 12 to the first terminal 30, and stores this information in the event data storage unit 13. Furthermore, if the user selects to participate in the simulated event, a screen is displayed for downloading a script to be run on the second terminal 50. When the script is downloaded, the participation acceptance unit 22 sets the user as a second user in the event set, sets the terminal of the user stored in the user information storage unit 12 in the second terminal 50, and stores this information in the event data storage unit 13.

[0035] The site that accepts participation in an event set also has a screen where new user registration can be performed. When an unregistered person selects to display the screen for new user registration, the user registration reception unit 21 causes the user to input contact information (such as an email address). When the contact information is input, the user registration reception unit 21 sends an email to the contact information with a link to a website for user registration, and causes the user to set a user name and password. When the user name and password are set, a new identification number is assigned to the new user, and the number is saved in the user information storage unit 12. Alternatively, new user registration can be performed in the same manner as above by downloading a program for event participants.

[0036] A setting page is provided for each person who is registered as a user, and through this page, the user can change the registration information, create an avatar, input information about the device they are using, and so on.

[0037] Here, we have described the case where users who have registered are allowed to participate in the event, but it is also possible to enable participation in the event without registering as a user, by simply downloading a script and setting a username.

[0038] When the date and time of the event set arrives, the real event starts in the real event space. At the same time, the simulated event also starts on a pre-set website. These real and simulated events are a pair of event sets that proceed synchronously, and it is also possible for a first user participating in the real event and a second user participating in the simulated event to interact with each other. In other words, one event set is executed in the real event space and the digital twin space that is paired with the real event space. A person participating as a second user participates in the simulated event by entering a user name and password to access this website. When participation in the simulated event begins, the avatar of the second user is placed at an initial position (for example, the entrance of the virtual event space). During the simulated event, the second user can move his or her avatar to any position in the virtual event space by operating the input unit.

[0039] In the first terminal 30 used by the first user participating in a real event in the real event space, the image acquisition unit 32 acquires image data of the real event space at a predetermined time interval. The image data acquired at the predetermined time interval may also include video data. The first terminal position acquisition unit 33 compares the image data acquired by the image acquisition unit 32 with data of a plurality of reference positions in the real event space by using the previously downloaded script (program), extracts the reference positions included in the image, and acquires information on the position and orientation in the real event space where the image was acquired (i.e., the position and orientation of the first terminal 30 in the real event space) based on the extracted reference positions. The above-mentioned VPS can be suitably used for this process. The first terminal position acquisition unit 33 transmits the information on the position and orientation of the first terminal 30 to the server device 10 at a predetermined time interval (or upon receiving a request from the server device 10).

[0040] The server device 10 transmits to the first terminal 30 at a predetermined time interval the information on the position and orientation of the avatar of the second user collected at that time and the display data of the avatar. In the first terminal 30, the previously downloaded script (program) converts the position of the avatar of the second user in the virtual event space to a position in the real event space based on the corresponding position information. Next, the first display control unit 34 generates a three-dimensional hologram image of the avatar and generates display data in which each avatar is given a user name. In addition, an image in which display data including the generated three-dimensional hologram image is superimposed on image data of the real event space acquired by the image acquisition unit 32 is displayed on the display unit 42 (if the first terminal 30 is an optical see-through type MR glass, only the three-dimensional hologram image is displayed on the display unit 42). Note that if there are multiple first users and another first user appears in the image of the real event space acquired by the image acquisition unit 32, the image in which the three-dimensional hologram image of the avatar of the second user is superimposed on the image of the real event space including the other first user is displayed on the display unit 42 of the first terminal 30. That is, on the display unit 42 of the first terminal 30, the other first users are displayed as real images, and the second users are displayed as avatars.

[0041] 5 shows an example of a screen displayed on the display unit 42 of the first terminal 30. On the display unit 42 of the first terminal 30 used by the first user, an image in which an avatar 72 of a second user participating in a simulated event and the user name thereof are superimposed on an image of a real event space including real images of other first users participating in the real event is displayed. Note that the example screen shown in FIG. 5 does not include a real image of the first user (i.e., there are no other first users in the direction in which the first user is facing in the real event space), and the avatars 72 of the three second users are displayed superimposed on the background of the real space.

[0042] In the second terminal 50 used by a second user participating in a real event in the virtual event space, every time the second user moves his / her avatar via the input unit 61, the input information is transmitted to the server device 10. In the second terminal 50, every time the avatar moves, the second user position acquisition unit 52 identifies the position and orientation of the second user's avatar in the virtual event space, and sequentially transmits the information to the server device 10.

[0043] In addition, the server device 10 transmits, at a predetermined time interval, the display data of the virtual event space stored in the event data storage unit 13, the position and orientation of the first user, the display data of the avatar of the user, and the position and orientation information of the avatar of the other second user, and the display data of the avatar, to the second terminal 50. In the second terminal 50, the position of the first terminal 30 in the real event space is converted to a position in the virtual event space based on the corresponding position information by the previously downloaded script (program). Next, the second display control unit 53 generates an image in which the avatar of the first user is arranged in the virtual event space as seen from the avatar of the second user who is the user of the second terminal 50 according to the position and orientation of the first terminal 30, and further the avatar of the other second user is arranged according to the position and orientation of the other second user, and displays it on the display unit 42. In this way, the first user and the other second user are both displayed as avatars on the display unit 62 of the second terminal 50. Here, an image of the virtual event space captured from the viewpoint of the second user's avatar is displayed, but an image capturing the virtual event space including the second user's avatar from behind the avatar (an image including the second user's own avatar) may also be displayed. Furthermore, images of the virtual event space captured from multiple viewpoints may be displayed in a switchable manner.

[0044] Fig. 6 shows an example of a screen displayed on the display unit 62 of the second terminal 50. As shown in Fig. 6, the display unit 62 of the second terminal 50 used by the second user displays an avatar 71 of a first user participating in the real event and an avatar 72 of another second user participating in the simulated event together with an image of the virtual event space. Note that by changing the display settings, it is also possible to display the username of the first user and the username of the second user on the screen.

[0045] In the example shown in Fig. 6, an auxiliary screen 73 is displayed at the bottom left of the display unit 62. An emotion display button 731 for switching between displaying and hiding icons for expressing one's own emotion and an action display button 732 for switching between displaying and hiding actions are displayed on this auxiliary screen 73. The display of the auxiliary screen 73 can be turned on / off by a predetermined input operation (for example, double tapping the bottom left area of ​​the screen of the display unit 62). Fig. 6 shows a state in which the emotion display button 731 is clicked, and as a result, icons 7311 corresponding to a plurality of emotional expressions are displayed. The icons 7311 in this embodiment are image data, but are not limited to image data and may be modified character data or the like.

[0046] On the screen shown in FIG. 6, the second user can express his / her own emotion by selecting one of the icons 7311. When the second user performs an operation to select one of the icons 7311, the emotion image selection receiving unit 55 receives the selection, and transmits information on the selected icon 7311 to the server device 10. The server device 10 accumulates information received from the second terminal 50, and when transmitting the position and orientation of the avatar of the second user and display data to the first terminal 30 at the above-mentioned predetermined time interval, transmits the information on the icon 7311 received from the second terminal 50 to the first terminal 30 with the information attached. Similarly, when transmitting the position and orientation of the avatar of another second user and display data to the second terminal 50, transmits the information on the icon 7311 received from the other second terminal 50 with the information attached to the other second terminal 50 (the second terminal 50 used by the above-mentioned other second user) to the other second terminal 50.

[0047] In the first terminal 30, when the information of the icon 7311 is received together with the position information of the second user, the first display control unit 34 generates image data for displaying the icon 7311 near the avatar 72 of the user of the second terminal 50 who selected the icon 7311, and displays the image data on the display unit 42. In the first terminal 30, as shown in FIG. 5, the icon 7311 is displayed above the avatar 72 of the second user. Similarly, in the second display control unit 53, when the information of the icon 7311 is received together with the position information of another second user, the second display control unit 53 generates image data for displaying the icon near the avatar 72 of the user of the second terminal 50 who selected the icon 7311, and displays the image data on the display unit 62. In the second terminal 50, the icon 7311 is displayed as shown in FIG. 6.

[0048] 6, the second user can also express his / her own emotion by clicking the action display button 732 and selecting an action displayed. When the second user performs an operation to select one of the actions, the emotion action selection receiving unit 56 receives the operation, and transmits the selected action information to the server device 10. The server device 10 accumulates information received from the second terminal 50, and when transmitting the position and orientation of the avatar of the second user and the display data to the first terminal 30 at the above-mentioned predetermined time interval, transmits the information of the action received from the second terminal 50 to the first terminal 30 with the information attached. Similarly, when transmitting the position and orientation of the avatar of another second user and the display data to the second terminal 50, transmits the information of the action received from the other second terminal 50 to the other second terminal 50 with the information attached.

[0049] In the first terminal 30, when the information on the action is received together with the position information of the second user, the first display control unit 34 generates image data for making the avatar 72 of the user of the second terminal 50 who selected the action perform an action corresponding to the action, and displays the image data on the display unit 42. In the first terminal 30, as shown in Fig. 7, an image including the action of the avatar 721 is displayed on the display unit 42. Similarly, in the second display control unit 53, when the information on the action is received together with the position information of another second user, image data for making the avatar 721 of the second user perform the action is generated and displayed on the display unit 62, as shown in Fig. 8.

[0050] Here, the second user Emotion display button 731 and Action display Although an example of selecting the button 732 has been described, the first user can also select the icon 731 or the action button 732, and the emotion image selection receiving unit 36 ​​and the emotion action selection receiving unit 37 of the first terminal 30 function in the same manner as described above. When a predetermined operation (for example, double tapping the screen) is performed on the screen of the display example shown in FIG. 5, the auxiliary screen 73 can be displayed on the display unit 42 as in FIG. 6. The auxiliary screen 73 can be operated in the same manner as the second user, and the server device 10 also performs the same process as when the icon 7311 or action is selected by the second user. However, in this embodiment, when the icon 7311 or action is selected by the first user, only the second display control unit 53 generates image data according to the selection and displays it on the display unit 62. In other words, the image data displayed on the display unit 42 of the first terminal 30 viewed by the first user does not change before and after the selection of the action. This is because the other first users displayed on the first terminal 30 are real images, not avatars, and it is impossible to impart an action to the real images of the first users. However, since it is possible to display icon 7311 near the first user displayed as a real image, when the first user selects icon 7311, icon 7311 may be displayed near the real image of the user. Also, when an action is selected, an icon with a movement corresponding to the selected action may be displayed near the real image of the first user.

[0051] During the real event, the expanded display unit 35 identifies the line of sight direction (the direction captured by the image acquisition unit 32 of the first terminal 30) corresponding to the position of the first user based on the information on the position and attitude of the first terminal 30 acquired by the first terminal position acquisition unit 33. Then, when it detects that the line of sight direction of the first user is staying on a preset expanded display target 74 for a certain period of time (e.g., 5 seconds), it reads out expanded display information corresponding to the expanded display target 74 from the event data storage unit 13. When the expanded display information is read out, the first display control unit 34 displays an image in which an expanded display 741 is superimposed near the expanded display target 74 on the display unit 42, as shown in FIG.

[0052] Also, even during the simulated event, the extended display unit 54 identifies the line of sight of the second user's avatar based on the position and posture information of the second user's avatar 72. Then, when it detects that the line of sight of the second user's avatar 72 is staying on a preset extended display target for a certain period of time (e.g., 5 seconds), it reads out extended display information corresponding to the extended display target from the event data storage unit 13. When the extended display information is read out, the second display control unit 53 displays on the display unit 62 an image in which an extended display 741 is superimposed near the extended display target 74, as shown in FIG.

[0053] During the real event, images and the like may be projected on a screen or the like provided in the real event space according to the progress of the event prepared in advance. In this way, when the projection time (time elapsed after the start of the event) and the projection location (screen, etc.) are determined in advance, data of the moving images and audio to be projected are stored in the event data storage unit 13. When the time elapsed after the start of the simulated event reaches the projection time, the second display control unit 53 reads out the projection data of the moving images and audio data stored in the event data storage unit 13, displays an image in which the moving images are projected on the screen in the virtual event space on the display unit 62, and outputs audio based on the audio data. This allows a first user participating in the real event in the real event space (left figure) and a second user participating in the simulated event in the virtual event space (right figure) to simultaneously view the same moving images, such as images, as shown in FIG. 11.

[0054] While a moving image is being projected onto a screen or the like and sound is being output as described above, on the second terminal 50, the second user can increase the volume of the moving image by moving his / her avatar 72 closer to the screen, or decrease the volume of the moving image by moving his / her avatar 72 away from the screen.

[0055] In the communication system 1 of the present embodiment, text chat (open chat and private chat) and voice calls can be performed between the first user and the second user, or between the second user, and images and videos can be delivered from the first user to the second user. These will be described below. In the following operations, similar to the above operations, the first display control unit 34 controls the screen display of the display unit 42 of the first terminal 30, and the second display control unit 53 controls the screen display of the display unit 62 of the second terminal 50, but to avoid redundant description, the first display control unit 34 and the second display control unit 53 will be omitted.

[0056] The open chat is a chat that can be posted and viewed by both the first user who is a participant of the real event and the second user who is a participant of the simulated event. The private chat is divided into TORIMAKI chat and direct chat. The private chat will be described later.

[0057] 12 shows an example of a screen displayed on the display unit 42 of the first terminal 30 used by the first user. At the bottom of the screen, a chat button 81, a distribution start button 82, and a participant button 83 are provided. In addition, in this example screen, an auxiliary screen 73 including an emotion display button 731 (emoticon) for calling up the above-mentioned icon 7311 and an action display button (emo action) 732 is displayed at the upper left of the screen.

[0058] When the chat button 81 is pressed, a chat screen 811 as shown in Fig. 13 is displayed. An OPEN button 812 and a TORIMAKI button 813 are arranged on the chat screen 811, and when the OPEN button 812 is pressed, an open chat screen is displayed, enabling posting and viewing. When the TORIMAKI button 813 is pressed, a TORIMAKI chat screen is displayed. Furthermore, when the chat button 81 is pressed while the chat screen 811 is displayed (Fig. 13), the chat screen is switched to hidden (Fig. 12).

[0059] When the second user performs a predetermined operation to select one of the avatars 71 of the first user displayed on the display unit 62 of the second terminal 50 (e.g., double-tapping the avatar 71), a “Become TORIMAKI” button 92 is displayed near the avatar 71 of the first user (at the top in FIG. 14), as shown in FIG. 14.

[0060] When the second user presses the "Become TORIMAKI" button 92 on the display unit 62, the user intermediation unit 23 in the server device 10 identifies the first user corresponding to the avatar 71 whose button was pressed. Then, the server device 10 notifies the first terminal 30 of the first user that there has been a TORIMAKI request from the second user, and inquires whether to accept or reject the request. If the first user accepts the TORIMAKI request, the information is transmitted to the server device 10. When the server device 10 receives the notice that the first user has accepted the second user to become TORIMAKI, the user intermediation unit 23 links the first terminal 30 with the second terminal 50 owned by the second user who has been accepted as TORIMAKI for the first user who is the owner of the first terminal 30, and registers the second user as TORIMAKI for the first user (setting the connection of the second user to the first user). In addition, a chat room is opened for the two users to have a private chat (TORIMAKI chat). This sets up a link between the first terminal 30 and the second terminal 50 owned by the second user who is authorized as a TORIMAKI of the first user who is the owner of the first terminal 30. In this embodiment, one or more second users can simultaneously become TORIMAKI for one first user.

[0061] When the first user presses the TORIMAKI button 813 arranged on the chat screen 811 shown in Fig. 13, a screen for conducting a private chat with the second user who is the first user's TORIMAKI is displayed. Also, when the second user presses the TORIMAKI button 912 on the chat screen 91 after being permitted as a TORIMAKI (see Fig. 15), a screen for conducting a private chat between the first user, the first user who is the TORIMAKI, and other TORIMAKIs (i.e., the first user and all TORIMAKIs of the first user) is displayed.

[0062] This chat screen 91 is provided with an OPEN button 911, a TORIMAKI button 912, and a DIRECT button 913. FIG. 15 shows an example in which the TORIMAKI button 912 is pressed and the TORIMAKI chat screen is displayed. Only the first user and the second user authorized as the TORIMAKI of the first user can post and view in this TORIMAKI chat. In the TORIMAKI chat, every time text information is input from the first user or the second user, the text information is transmitted to each other's terminal via the user intermediation unit 23 of the server device 10 and displayed on the respective display units 42, 62. Furthermore, when the first user accepts a TORIMAKI request from the second user, in addition to the above private chat, it is also possible to have a voice call between the two. This voice call may be either one between all TORIMAKIs or one between only one of the TORIMAKIs. Moreover, it may be possible to switch between them.

[0063] In addition, the first user participating in the real event and the second user participating in the simulated event can be confirmed by the participant list display 93 (enlarged view is shown again in FIG. 16) displayed in the lower right of the screen shown in FIG. 14. By pressing either of the chat buttons 933 displayed next to the first user name 931 or the second user name 932 in this participant list display 93, a one-to-one direct chat with the user is started. By pressing the DIRECT button 913 on the chat screen 91 shown in FIG. 15, a direct chat screen is displayed. During the direct chat, one-to-one voice communication with the user is also possible. By pressing the button 934 displayed next to the first user, a TORIMAKI request is sent to the first user in the same manner as when the above-mentioned "Become TORIMAKI" button 92 is pressed.

[0064] Although the example screen of the display unit 42 of the first terminal 30 shown in FIG. 13 does not display a direct chat, the first terminal 30 can also perform a direct chat and a voice call by designating either the first user or the second user. When the participant button 83 displayed at the bottom left of the example screens of FIG. 12 and FIG. 13 is pressed, a list of the first user and the second user is displayed as in FIG. 16. Direct chat and a voice call are started by selecting one user from this list and pressing the chat button. Since the first users participating in a real event in the real space can talk to each other on the spot, a configuration may be adopted in which direct chat between the first users is not performed. However, in consideration of a case where a conversation is to be held about matters that should be kept secret from other first users, the present embodiment also enables direct chat between the first users.

[0065] The distribution start button 82 shown in Fig. 12 and Fig. 13 is a button for executing a process in which the first user distributes images (including videos) acquired by the image acquisition unit 32 of the first terminal 50 used by the first user to the second user who is the user's TORIMAKI. For example, the second user who is the TORIMAKI can request the first user to take a picture of a place of interest in real space through chat or voice call, and the first user who receives the request can distribute images and videos taken of the requested place on his / her first terminal 30 to the second user who is the user's TORIMAKI in real time (see Fig. 17). At this time, the first user can orally explain the images and videos being taken, and distribute the audio data to the TORIMAKI at the same time.

[0066] The image, video, and audio data distributed from the first terminal 30 is transmitted to the second terminal 50 owned by the second user who is a TORIMAKI of the first user who is the owner of the first terminal 30, via the user intermediary unit 23 of the server device 10. In the second terminal 50, the images and videos captured by the image acquisition unit 32 of the first terminal 30 are displayed full screen on the display unit 62 (see FIG. 18). This allows the second user to share in real time the images and videos of the real event space captured by the image acquisition unit 32 of the first terminal 30 used by the first user who is the TORIMAKI. The size of the small screen on the display unit 62 of the second terminal 50 can be changed as appropriate.

[0067] However, images and videos distributed by the first user in response to a request from a second user are not necessarily of interest to all TORIMAKI. Therefore, in the full-screen display state shown in Fig. 18, the second terminal 50 can end the display of images and videos distributed from the first terminal 30 by pressing the EXIT button 941, or can display images and videos distributed from the first terminal 30 on the small screen 95 by pressing the reduce button 942 (see Fig. 19).

[0068] 17 to 19 are examples in which the first user distributes images and videos of a real event space taken by the first terminal 30 to his / her TORIMAKI, but by changing the settings of the image acquisition unit 32 of the first terminal 30 (changing to a setting to use the in-camera), the first user can also distribute his / her own image to the TORIMAKI. This allows the first user to distribute his / her own image by talking to his / her TORIMAKI. FIG. 20 is an example of a screen displayed on the display unit 62 of the second terminal 50 when the first user distributes his / her own image using the in-camera. Note that when the first user distributes his / her own image using the in-camera, the image acquired by the in-camera may be displayed full-screen on the display unit 42 of the first terminal 30, or the image acquired by the in-camera may be displayed in a small screen on top of the image acquired by the out-camera.

[0069] The user mediation unit 23 also writes mutual identification information into the user information of the first user and the second user approved by the first user as TORIMAKI, and stores the information in the user information storage unit 12. As a result, the user information of each user is recorded as a user who has an exchange as TORIMAKI in each user's user information. The list of users who have an exchange can be confirmed, for example, in a user information setting unit that each user can access using a username and password. Alternatively, when the real event and the simulated event are ended, or when the first user or the second user leaves the event, the permission to record each other as a user who has an exchange as TORIMAKI is requested, and only when permission is obtained from both users, the user may be recorded as an exchange user. In addition, a configuration may be adopted in which multiple stages are set for the exchange between users, such as disclosing all user information to each other, disclosing only the username and contact information, and disclosing only the username, and the information at the level that each user has approved is disclosed.

[0070] In the communication system 1 of this embodiment, by adopting the above configuration, it is possible to provide an environment (digital twin environment) that is the same as the real space (upper part of FIG. 21) in the virtual space (lower part of FIG. 21). In addition, by utilizing the above TORIMAKI system in such a digital twin environment, images and videos captured in the real space can be shared in real time with a second user participating in the virtual space.

[0071] Conventionally, there are things that allow various experiences in virtual spaces that mimic real spaces, but when reproducing real spaces in virtual spaces, virtual spaces are often formed with some of the information in the real space omitted from the viewpoint of the amount of data and the load on image processing. Therefore, in virtual spaces, it is often not possible to experience the realism of real spaces. In contrast, in the above-mentioned TORIMAKI system, real images and videos taken by a first user in real space can be shared by a second user who becomes the TORIMAKI of the first user. Note that although the above are all indoor real spaces, it is also possible to provide a virtual space that is a digital twin of an outdoor real space, as shown in FIG. 22.

[0072] The communication system 1 of the present embodiment can be used in various situations. For example, when the first user is a celebrity or an influencer, a fan of the first user can become the TORIMAKI of the first user and follow the activities of the first user. Alternatively, for a place where entry is prohibited to those other than those involved, a person involved in the place can become the TORIMAKI and request delivery of images of places (backyards) and scenes (animal birth scenes) that cannot be normally seen and explanations of the place. Examples of such places include amusement parks, zoos, aquariums, factories, and backyards of airports, and places where important cultural properties are installed in temples and art museums. Furthermore, a clerk at a shop such as an apparel brand can become the first user, and a person interested in the shop can become the TORIMAKI and request an introduction of the shop's products, etc. Furthermore, a person with a disability or a recluse who has difficulty going out can become a tour guide or a resident TORIMAKI in a place of interest to them and request delivery of images and explanations of the place of interest. Furthermore, it can also be used to provide an educational opportunity for people living in remote areas.

[0073] As described above, in the communication system 1 of the present embodiment, not only is a real event in the real event space and a simulated event in the virtual event space performed simultaneously in parallel, but the position and orientation of the first terminal 30 are specified based on reference position information stored in advance in the event data storage unit 13, and the position information of the real event space and the virtual event space are mutually converted based on the corresponding position information. This allows the avatar 71 of the first user participating in the event in the real event space to be displayed in the virtual event space and displayed on the display unit 62 of the second terminal 50. In addition, a three-dimensional hologram image of the avatar 72 of the second user can also be displayed on the display unit 42 of the first terminal 30. In addition, since the moving image projected in the real event can be simultaneously projected in the simulated event in the virtual event space, the first user and the second user can simultaneously watch the same moving image.

[0074] In the communication system 1 of the present embodiment, the first user and the second user each express their own emotions. Emotion display button 731 and Action display By selecting the button 732, the user can appeal to other users. Also, a private chat can be held between the first user and the second user, or between second users.

[0075] Furthermore, in the communication system 1 of this embodiment, a second user participating in a simulated event in a virtual space can share realistic images and videos in the real space by becoming a TORIMAKI of a first user participating in a real event taking place in the real space.

[0076] The above embodiment is merely an example and can be modified appropriately in accordance with the spirit of the present invention.

[0077] In the above embodiment, the storage unit 11 of the server device 10 is provided with the event data storage unit 13, and various data required for holding an event is stored therein, but some or all of the data stored in the event data storage unit 13 may be stored in the storage unit 31 of the first terminal 30 or the storage unit 51 of the second terminal 50. In addition, in the above embodiment, the first terminal 30 is provided with the first terminal position acquisition unit 33, the second terminal 50 is provided with the second user position acquisition unit 52, and the first display control unit 34 in the first terminal 30 generates data to be displayed on the display unit 42, and the second display control unit 53 in the second terminal 50 generates data to be displayed on the display unit 62, but a configuration in which some or all of the functional blocks including these are provided in the server device 10 may also be adopted.

[0078] In the above embodiment, the first terminal position acquisition unit 33 identifies the position and orientation of the first terminal 30 by comparing the image data acquired by the image acquisition unit 32 of the first terminal 30 with the data of a plurality of reference positions in the real event space stored in the event data storage unit 13, but other methods may be adopted. For example, sensors may be placed in advance at a plurality of positions in the real event space, and the position of the first terminal 30 may be identified by performing short-range communication with the first terminal 30 to obtain the response time for each sensor, and the orientation of the first terminal may be identified by an acceleration sensor included in the first terminal 30.

[0079] In the above embodiment, video data with a predetermined projection time and projection location is used, but a video that was not planned may be projected before the event. In that case, for example, image data of the real event space may be generated by integrating image data acquired by the image acquisition units 32 of the first terminals 30 at appropriate time intervals, and the image data of the real event space stored in the event data storage unit 13 may be collated to detect differences between the two, and projection data may be generated from image data of the location where the differences were detected. Alternatively, a plurality of image acquisition units may be arranged in the real event space in advance, and when it is detected that a video that was not planned in advance is being projected based on changes in the images acquired by each image acquisition unit, projection data may be generated from data of the image acquisition unit that detected the video.

[0080] In the above embodiment, the avatar 71 of the first user and the avatar 72 of the second user are all displayed in the virtual event space. However, if many second users participate in a simulated event held in the virtual event space, it may be possible that the avatars 72 of the second users cannot all be displayed in the virtual event space. In that case, a plurality of virtual event spaces may be provided, the second users may be divided into a plurality of groups, and the avatars 72 may be displayed in the virtual event spaces provided for each group. In that case, only the avatar 72 of the second user in any one of the virtual event spaces may be displayed on the display unit 42 of the first terminal 30. Also, the first user may be allowed to select which virtual space the avatars of the first user are to be displayed on the display unit 42 of the first terminal 30. Alternatively, only the avatars 72 of the second users who have interactions with the first user may be displayed on the display unit 42 of the first terminal 30.

[0081] Furthermore, when enjoying an event as a TORIMAKI of a first user who is a celebrity, the second user who is the TORIMAKI often does not have a particular interest in other first users or second users, and is likely to feel bothered by the display of avatars 71 of first users other than the first user who is the TORIMAKI and avatars 72 of other second users in the virtual space. In such a case, it is preferable to adopt a configuration in which the display form can be switched to a form in which only the avatar 71 of the first user who is the TORIMAKI is displayed, or a form in which only the avatar 71 of the first user who is the TORIMAKI and the avatar 72 of the second user who is the TORIMAKI are displayed, by the second user performing a predetermined operation.

[0082] In the TORIMAKI system of the above embodiment, one or more second users are TORIMAKIs for one first user, but multiple second users may be TORIMAKIs for multiple first users to form a TORIMAKI group. In that case, the display unit 62 of the second terminal 50 may be provided with separate screens for displaying images and videos simultaneously distributed by multiple first users.

[0083] In the above embodiment, one virtual event space is associated with one real event space, but multiple different virtual event spaces may be associated with one real event space. For example, virtual event spaces provided by multiple different businesses may be associated with one real event, and virtual spaces created by different businesses may be linked via an API (Application Programming Interface) to enable users to move between multiple virtual event spaces provided in correspondence with one real event space. Alternatively, an E-commerce (Electric Commerce) function may be set in a virtual space corresponding to a real event space, so that more experiences can be made in the virtual event space than in the real event space. In the above embodiment, a virtual event space is made to imitate a real event space, but the virtual event space is only required to correspond to the real event space and to have positions in the space associated with each other, and is not necessarily required to imitate the real event space. For example, a virtual event space that is appropriately deformed from a real event space may be used.

[0084] [Aspects] It will be apparent to those skilled in the art that the above-described exemplary embodiments are illustrative of the following aspects.

[0085] (Section 1) A communication system according to one aspect of the present invention comprises: a storage unit that stores display data for displaying in a virtual event space a simulated event corresponding to a real event held in a real event space, corresponding position information that associates a position in the real event space with a position in the virtual event space, display data of an avatar of a first user participating in the real event in the real event space, and display data of an avatar of a second user participating in the simulated event in the virtual event space; an image acquisition unit, which is a terminal used by the first user, that acquires an image of the real event space captured from a viewpoint corresponding to a position of the first user; and Displaying the image acquired by the image acquisition unit A first terminal having a display unit; a first terminal position acquisition unit that acquires information about the position of the first terminal in the real event space; a second terminal used by the second user and equipped with an input unit and a display unit; a second user position acquisition unit that acquires information on a position of an avatar of the second user in the virtual event space, the position being set by an operation via the input unit; a first display control unit that converts the position of the avatar of the second user acquired by the second user position acquisition unit into a position in the real event space based on the corresponding position information, and displays an image in which the avatar of the second user is placed at the converted position on a display unit of the first terminal; a second display control unit that converts the position of the first terminal acquired by the first terminal position acquisition unit into a position in the virtual event space based on the corresponding position information, thereby locating an avatar of the first user, and causes a display unit of the second terminal to display an image of the simulated event captured from a viewpoint corresponding to the position of the avatar of the second user; Equipped with.

[0086] (Section 11) A program for a communication system according to another aspect of the present invention includes: A terminal used by a first user who is a participant of a real event held in a real event space, Depending on location a first terminal including an image acquisition unit that acquires an image of the real event space captured from a viewpoint and a display unit that displays the image acquired by the image acquisition unit; a second terminal used by a second user who is a participant of a simulated event corresponding to the real event held in a virtual event space, the second terminal including an input unit and a display unit; a device that is capable of communicating with the first terminal and the second terminal and that executes the simulated event The above a server device having a storage unit that stores display data for displaying in a virtual event space, corresponding position information that associates a position in the real event space with a position in the virtual event space, and display data of an avatar of a second user who participates in the simulated event in the virtual event space; In a communication system having a second user position acquisition unit that acquires information on a position of an avatar of the second user in the virtual event space, the second terminal or the server device being set by an operation through the input unit, The first terminal, a first terminal position acquisition unit that acquires information about the position of the first terminal in the real event space; a first display control unit that converts the position of the avatar of the second user acquired by the second user position acquisition unit into a position in the real event space based on the corresponding position information, and displays an image in which the avatar of the second user is arranged at the converted position on a display unit of the first terminal. A program characterized by causing the program to operate as follows.

[0087] The communication system according to paragraph 1 and the communication system embodied by executing the program according to paragraph 11 are used when holding an event (real event) in an event space (real event space) set in a real space, and holding a simulated event corresponding to the real event space in an event space (virtual event space) set in a virtual space. In this system, a first user who participates in the real event in the real event space uses a first terminal equipped with an image acquisition unit and a display unit that acquire an image of the real event space captured from a viewpoint corresponding to the position of the first user. Also, a second user who participates in the simulated event in the virtual event space uses a second terminal equipped with an input unit and a display unit. Note that the image captured from a viewpoint corresponding to the position of the first user is an image captured from the viewpoint of the first user if the image acquisition unit is provided in a device such as MR glasses worn by the first user on the head, and is an image captured from a shooting unit of the tablet terminal if the image acquisition unit is provided in a portable device such as a tablet terminal used by the first user.

[0088] While the first user is participating in the real event, information on the position of the first terminal used by the first user moving within the real event space is acquired by a first terminal position information acquisition unit. The information on the position of the first terminal may include information on the orientation of the first terminal. The second user can make an avatar appear in the virtual event space by operating an input unit of the second terminal, and can freely move the avatar within the virtual event space. While the second user is participating in the simulated event, information on the position of the second user's avatar is acquired by a second user position acquisition unit.

[0089] The first display control unit converts the position of the second user's avatar into a position in the real event space based on the corresponding position information stored in advance in the storage unit, and displays an image in which the second user's avatar is placed at that position on the display unit of the first terminal. Therefore, the first user can view an image as if the second user's avatar is present in the real event space. Furthermore, the second display control unit converts the position of the first terminal into a position in the virtual space based on the corresponding position information, places the first user's avatar at that position, and displays an image of the simulated event captured from a viewpoint corresponding to the position of the second user's avatar on the display unit of the second terminal. Therefore, the second user can view an image of the virtual event space including the avatar of the first user in the real event space. Note that the viewpoint corresponding to the position of the second user's avatar is not limited to the viewpoint of the avatar itself, but may be a viewpoint that captures the virtual event space including the avatar from behind the avatar. Moreover, such multiple viewpoints may be switchable.

[0090] In this way, in the above-mentioned communication system, a first user participating in a real event in a real event space and a second user participating in a simulated event in a virtual event space can display each other's avatars on their respective devices, allowing the participants in the real event taking place in the real space and the participants in the simulated event taking place in the virtual space to feel a sense of unity.

[0091] (Section 2) The communication system according to paragraph 2 is a communication system according to paragraph 1, the storage unit further stores image data of the real event space, The first terminal position acquisition unit acquires information on the position of the first terminal by comparing an image of the real event space acquired by the image acquisition unit with the image data.

[0092] In the communication system relating to paragraph 2, information on the position and orientation of the first terminal can be obtained with high accuracy by comparing an image captured by an image acquisition unit of the first terminal with reference position information in the real event space.

[0093] (Section 3) The communication system according to paragraph 3 is a communication system according to paragraph 1 or 2, the storage unit further stores data of a plurality of emotion images each of which is an image expressing an emotion of a user; moreover, an emotion image selection receiving unit that receives an operation to select one of the emotion images in one of the first terminal and the second terminal; an emotion image display unit that displays the selected emotion image together with an avatar of a user of the terminal whose operation has been accepted by the emotion image selection accepting unit, on the other display unit of the first terminal or the second terminal; Equipped with.

[0094] (Section 4) The communication system according to paragraph 4 is a communication system according to any one of paragraphs 1 to 3, the storage unit further stores data of emotional actions, each of which is an avatar's action expressing an emotion of the user; moreover, an emotional action selection receiving unit that receives an operation to select one of the emotional actions in one of the first terminal and the second terminal; an emotional action executing unit that displays, on the other display unit of the first terminal and the second terminal, an image in which an avatar of a user of the terminal whose operation has been accepted by the emotional action selection accepting unit has executed the selected emotional action; Equipped with.

[0095] In the communication systems according to the third and fourth aspects, a wide range of emotional expressions can be carried out between the first and second users.

[0096] (Section 5) A communication system according to paragraph 5 is a communication system according to any one of paragraphs 1 to 4, the storage unit further stores information on augmented display objects to be arranged in each of the real event space and the virtual space, and augmented display content associated with the augmented display objects; moreover, an extended display unit that generates image data for displaying the extended display content together with the extended display target on a display unit of the first terminal when it is determined based on position information acquired by the first terminal position acquisition unit that the line of sight of the user of the first terminal is staying on the extended display target for a predetermined time, and that displays the extended display content together with the extended display target on a display unit of the second terminal when it is determined based on position information acquired by the second user position acquisition unit that the line of sight of the avatar of the second user is staying on the extended display target for a predetermined time. Equipped with.

[0097] In the communication system according to paragraph 5, detailed explanations of exhibits, etc. at real events and simulated events can be provided only to first and second users who are interested in the exhibits, etc. In addition, since the exhibits, etc. are not displayed to users who are not interested in the exhibits, such users do not feel bothered by unnecessary displays.

[0098] (Section 6) A communication system according to paragraph 6 is a communication system according to any one of paragraphs 1 to 5, the storage unit further stores data of a moving image to be projected in the real event space, a projection location of the moving image, and information on a timing of projecting the moving image; The second display control unit causes a display unit of the second terminal to display, at the timing, an image obtained by projecting the moving image at a position in the virtual event space corresponding to the projection location.

[0099] In the communication system according to paragraph 6, moving images can be viewed in real time in the virtual event space simultaneously with the real event space.

[0100] (Section 7) A communication system according to paragraph 7 is a communication system according to any one of paragraphs 1 to 6, When there are a plurality of the second terminals, the plurality of second terminals are divided into a plurality of groups, and a virtual event space is set for each group; The first terminal display control unit displays only avatars of the users of the second terminal who are classified into any one of the plurality of groups on the display unit of the first terminal. 。

[0101] In the communication system of paragraph 7, even if a large number of second users participate in a simulated event, the number of avatars displayed on the display unit of the first terminal and the display unit of the second terminal are limited and arranged in the virtual space, so that the first user and the second user do not feel bothered by the display of a large number of avatars.

[0102] (Section 8) A communication system according to paragraph 8 is a communication system according to any one of paragraphs 1 to 7, moreover, a user intermediation unit that receives a request from the second terminal to specify one of the first users and to establish a connection with the first user, queries a first terminal owned by the first user as to whether or not the first user agrees to establish a connection with a second user who is an owner of a second terminal, and sets up a link between the first terminal and the second terminal when an operation of approval is performed from the first terminal. Equipped with. (Section 8) In the communication system according to paragraph 8, a private connection can be formed between a second user who issues a request and a first user who accepts the request.

[0103] (Section 9) A communication system according to paragraph 9, in which, in the communication system according to paragraph 8, The user intermediation unit further displays a chat screen that can be written on and viewed only from the first terminal and the second terminal for which the linkage is set up, on a display unit of the first terminal used by the first user and on a display unit of the second terminal used by the second user.

[0104] In the communication system according to paragraph 9, only the first user and the second user who have formed a connection can have private interactions through chats that can be written and viewed by the first user and the second user.

[0105] (Section 10) The communication system according to paragraph 10 is a communication system according to paragraph 8 or 9, The user interface unit displays the image acquired by the first terminal on a display unit of a second terminal that is set up to cooperate with the first terminal.

[0106] In the communication system according to paragraph 10, a second user who participates in a simulated event and forms a private connection with a first user can enjoy images captured by the first user in the real event space in real time. [Explanation of symbols]

[0107] 1. Communication system 10. Server equipment 11...Storage section 12...User information storage unit 13...Event data storage section 21...User registration reception section 22…Participant reception desk 23...User Relations Department 30…1st Terminal 31...Storage section 32...Image acquisition section 33...First terminal location acquisition unit 34...First display control unit 35…Expanded display section 36…Emotion image selection reception section 37…Emotion and action selection reception section 41...Input section 42...Display section 50…Second terminal 51...Storage section 52...second user position acquisition unit 53...Second display control unit 54…Expanded display section 55…Emotion image selection reception section 56…Emotion and action selection reception section 61...Input section 62...Display section 71, 711...First user's avatar 72, 721... Avatar of the second user 73...Auxiliary screen 731…Emotion display button 7311...Icon 732...Action display button 74…Expanded display target 741…Extended display 81…Chat button 811…Chat screen 812…OPEN button 813…TORIMAKI button 82…Streaming start button 83…Participant button 91...Chat screen 911…OPEN button 912…TORIMAKI button 913…DIRECT button 92…“Become TORIMAKI” button 93…Participant list display 931...First user name 932...Secondary Username 933…Chat button 934…Button 941…EXIT button 942…Zoom out button 95…Small screen

Claims

1. a storage unit that stores display data for displaying in a virtual event space a simulated event corresponding to a real event held in a real event space, corresponding position information that associates a position in the real event space with a position in the virtual event space, display data of an avatar of a first user participating in the real event in the real event space, and display data of an avatar of a second user participating in the simulated event in the virtual event space; a first terminal used by the first user, the first terminal including an image acquisition unit that acquires an image of the real event space captured from a viewpoint corresponding to a position of the first user, and a display unit that displays the image acquired by the image acquisition unit; a first terminal position acquisition unit that acquires information about a position of the first terminal in the real event space; a second terminal used by the second user and equipped with an input unit and a display unit; a second user position acquisition unit that acquires information on a position of an avatar of the second user in the virtual event space, the position being set by an operation via the input unit; a first display control unit that converts the position of the avatar of the second user acquired by the second user position acquisition unit into a position in the real event space based on the corresponding position information, and displays an image in which the avatar of the second user is placed at the converted position on a display unit of the first terminal; a second display control unit that converts the position of the first terminal acquired by the first terminal position acquisition unit into a position in the virtual event space based on the corresponding position information, thereby locating an avatar of the first user, and causes a display unit of the second terminal to display an image of the simulated event captured from a viewpoint corresponding to the position of the avatar of the second user; Equipped with When there are a plurality of the second terminals, the plurality of second terminals are divided into a plurality of groups, and a virtual event space is set for each group; The first display control unit causes only avatars of the users of the second terminal who are classified into any one of the plurality of groups to be displayed on the display unit of the first terminal. A communication system characterized by:

2. a storage unit that stores display data for displaying in a virtual event space a simulated event corresponding to a real event held in a real event space, corresponding position information that associates a position in the real event space with a position in the virtual event space, display data of an avatar of a first user participating in the real event in the real event space, and display data of an avatar of a second user participating in the simulated event in the virtual event space; a first terminal used by the first user, the first terminal including an image acquisition unit that acquires an image of the real event space captured from a viewpoint corresponding to a position of the first user, and a display unit that displays the image acquired by the image acquisition unit; a first terminal position acquisition unit that acquires information about a position of the first terminal in the real event space; a second terminal used by the second user and equipped with an input unit and a display unit; a second user position acquisition unit that acquires information on a position of an avatar of the second user in the virtual event space, the position being set by an operation via the input unit; a first display control unit that converts the position of the avatar of the second user acquired by the second user position acquisition unit into a position in the real event space based on the corresponding position information, and displays an image in which the avatar of the second user is placed at the converted position on a display unit of the first terminal; a second display control unit that converts the position of the first terminal acquired by the first terminal position acquisition unit into a position in the virtual event space based on the corresponding position information to place an avatar of the first user, and displays an image of the simulated event captured from a viewpoint corresponding to the position of the avatar of the second user on a display unit of the second terminal; a user intermediation unit that receives a request from the second terminal to specify one of the first users and to establish a connection with the first user, queries a first terminal owned by the first user as to whether or not the first user agrees to establish a connection with a second user who is an owner of the second terminal, and sets up a link between the first terminal and the second terminal when an operation of approval is performed from the first terminal; Equipped with When an operation for permitting a connection with a plurality of the second users is performed on the first terminal, the user intermediation unit sets up a link between the first terminal and the plurality of the second terminals corresponding to the plurality of second users, and displays a chat screen that can be written and viewed only from the first terminal and the plurality of second terminals for which the link has been set up on a display unit of the first terminal used by the first user and on display units of the plurality of second terminals. A communication system characterized by:

3. a storage unit that stores display data for displaying in a virtual event space a simulated event corresponding to a real event held in a real event space, corresponding position information that associates a position in the real event space with a position in the virtual event space, display data of an avatar of a first user participating in the real event in the real event space, and display data of an avatar of a second user participating in the simulated event in the virtual event space; a first terminal used by the first user, the first terminal including an image acquisition unit that acquires an image of the first user and the real event space captured from a viewpoint corresponding to a position of the first user, and a display unit that displays the image acquired by the image acquisition unit; a first terminal position acquisition unit that acquires information about a position of the first terminal in the real event space; a second terminal used by the second user and equipped with an input unit and a display unit; a second user position acquisition unit that acquires information on a position of an avatar of the second user in the virtual event space, the position being set by an operation via the input unit; a first display control unit that converts the position of the avatar of the second user acquired by the second user position acquisition unit into a position in the real event space based on the corresponding position information, and displays an image in which the avatar of the second user is placed at the converted position on a display unit of the first terminal; a second display control unit that converts the position of the first terminal acquired by the first terminal position acquisition unit into a position in the virtual event space based on the corresponding position information to place an avatar of the first user, and displays an image of the simulated event captured from a viewpoint corresponding to the position of the avatar of the second user on a display unit of the second terminal; a user intermediation unit that receives a request from the second terminal to specify one of the first users and to establish a connection with the first user, queries a first terminal owned by the first user as to whether or not the first user agrees to establish a connection with a second user who is an owner of the second terminal, and sets up a link between the first terminal and the second terminal when an operation of approval is performed from the first terminal; Equipped with The user intermediation unit displays an image of the first user acquired by the first terminal on a display unit of a second terminal that is set up to cooperate with the first terminal. A communication system characterized by:

4. the storage unit further stores reference position information in the real event space, The first terminal position acquisition unit acquires information on the position of the first terminal by comparing an image of the real event space acquired by the image acquisition unit with the reference position information.

4. A communication system according to claim 1, wherein the communication system is a communication system for communicating with the communication device.

5. the storage unit further stores data of a plurality of emotion images each of which is an image expressing an emotion of a user; moreover, an emotion image selection receiving unit that receives an operation to select one of the emotion images in one of the first terminal and the second terminal; an emotion image display unit that displays the selected emotion image together with an avatar of a user of the terminal whose operation has been accepted by the emotion image selection accepting unit, on the other display unit of the first terminal and the second terminal; 4. The communication system according to claim 1, further comprising:

6. the storage unit further stores data of emotional actions, each of which is an avatar's action expressing an emotion of the user; moreover, an emotional action selection receiving unit that receives an operation to select one of the emotional actions in one of the first terminal and the second terminal; an emotional action executing unit that displays, on the other display unit of the first terminal and the second terminal, an image in which an avatar of a user of the terminal whose operation has been accepted by the emotional action selection accepting unit has executed the selected emotional action; 4. The communication system according to claim 1, further comprising:

7. The storage unit further stores information on augmented display objects to be arranged in each of the real event space and the virtual event space, and augmented display content associated with the augmented display objects; moreover, an extended display unit that generates image data for displaying the extended display content together with the extended display target on a display unit of the first terminal when it is determined based on position information acquired by the first terminal position acquisition unit that the line of sight of the user of the first terminal is staying on the extended display target for a predetermined time, and that displays the extended display content together with the extended display target on a display unit of the second terminal when it is determined based on position information acquired by the second user position acquisition unit that the line of sight of the avatar of the second user is staying on the extended display target for a predetermined time.

4. The communication system according to claim 1, further comprising:

8. the storage unit further stores data of a moving image to be projected in the real event space, a projection location of the moving image, and information on a timing of projecting the moving image; The second display control unit causes a display unit of the second terminal to display an image obtained by projecting the moving image at a position corresponding to the projection location in the virtual event space at the timing.

4. A communication system according to claim 1, wherein the communication system is a communication system for communicating with the communication device.

9. a first terminal used by a first user who is a participant of a real event held in a real event space, the first terminal including an image acquisition unit that acquires an image of the real event space captured from a viewpoint corresponding to a position of the first user, and a display unit that displays the image acquired by the image acquisition unit; a second terminal used by a second user who is a participant of a simulated event corresponding to the real event held in a virtual event space, the second terminal including an input unit and a display unit; a server device provided to be capable of communicating with the first terminal and the second terminal, the server device having a storage unit storing display data for displaying the simulated event in the virtual event space, corresponding position information that associates a position in the real event space with a position in the virtual event space, and display data of an avatar of a second user participating in the simulated event in the virtual event space; In a communication system having a second user position acquisition unit that acquires information on a position of an avatar of the second user in the virtual event space, the second terminal or the server device being set by an operation through the input unit, The first terminal, a first terminal position acquisition unit that acquires information about a position of the first terminal in the real event space; a first display control unit that converts the position of the avatar of the second user acquired by the second user position acquisition unit into a position in the real event space based on the corresponding position information, and causes an image in which the avatar of the second user is arranged at the converted position to be displayed on a display unit of the first terminal. Operate as When there are a plurality of the second terminals, the plurality of second terminals are divided into a plurality of groups, and a virtual event space is set for each group; The first display control unit causes only avatars of the users of the second terminal who are classified into any one of the plurality of groups to be displayed on the display unit of the first terminal. A program characterized by:

10. a first terminal used by a first user who is a participant of a real event held in a real event space, the first terminal including an image acquisition unit that acquires an image of the real event space captured from a viewpoint corresponding to a position of the first user, and a display unit that displays the image acquired by the image acquisition unit; a second terminal used by a second user who is a participant of a simulated event corresponding to the real event held in a virtual event space, the second terminal including an input unit and a display unit; a server device provided to be capable of communicating with the first terminal and the second terminal, the server device having a storage unit storing display data for displaying the simulated event in the virtual event space, corresponding position information that associates a position in the real event space with a position in the virtual event space, and display data of an avatar of a second user participating in the simulated event in the virtual event space; In a communication system having a second user position acquisition unit that acquires information on a position of an avatar of the second user in the virtual event space, the second terminal or the server device being set by an operation through the input unit, The first terminal, a first terminal position acquisition unit that acquires information about a position of the first terminal in the real event space; a first display control unit that converts the position of the avatar of the second user acquired by the second user position acquisition unit into a position in the real event space based on the corresponding position information, and displays an image in which the avatar of the second user is placed at the converted position on a display unit of the first terminal; a user intermediation unit that accepts a request from the second terminal to specify one of the first users and to establish a connection with the first user, inquires of a first terminal owned by the first user as to whether or not the first terminal accepts the connection with a second user who is an owner of the second terminal, and sets up a link between the first terminal and the second terminal when an operation of acceptance is performed from the first terminal. Operate as When an operation for permitting a connection with a plurality of the second users is performed on the first terminal, the user intermediation unit sets up a link between the first terminal and the plurality of the second terminals corresponding to the plurality of second users, and displays a chat screen that can be written and viewed only from the first terminal and the plurality of second terminals for which the link has been set up on a display unit of the first terminal used by the first user and on display units of the plurality of second terminals. A program characterized by:

11. a first terminal used by a first user who is a participant of a real event held in a real event space, the first terminal including an image acquisition unit that acquires an image of the real event space and the first user captured from a viewpoint corresponding to a position of the first user, and a display unit that displays the image acquired by the image acquisition unit; a second terminal used by a second user who is a participant of a simulated event corresponding to the real event held in a virtual event space, the second terminal including an input unit and a display unit; a server device provided to be capable of communicating with the first terminal and the second terminal, the server device having a storage unit storing display data for displaying the simulated event in the virtual event space, corresponding position information that associates a position in the real event space with a position in the virtual event space, and display data of an avatar of a second user participating in the simulated event in the virtual event space; In a communication system having a second user position acquisition unit that acquires information on a position of an avatar of the second user in the virtual event space, the second terminal or the server device being set by an operation through the input unit, The first terminal, a first terminal position acquisition unit that acquires information about a position of the first terminal in the real event space; a first display control unit that converts the position of the avatar of the second user acquired by the second user position acquisition unit into a position in the real event space based on the corresponding position information, and displays an image in which the avatar of the second user is placed at the converted position on a display unit of the first terminal; a user intermediation unit that accepts a request from the second terminal to specify one of the first users and to establish a connection with the first user, inquires of a first terminal owned by the first user as to whether or not the first terminal accepts the connection with a second user who is an owner of the second terminal, and sets up a link between the first terminal and the second terminal when an operation of acceptance is performed from the first terminal. Operate as The user intermediation unit displays an image of the first user acquired by the first terminal on a display unit of a second terminal that is set up to cooperate with the first terminal. A program characterized by:

Images