Experience sharing system and experience sharing method

Abstract

To provide an experience sharing system and an experience sharing method which can be more flexibly utilized according to a user's various convenience.SOLUTION: An experience sharing system comprises: one or more user terminals in which an experience sharing application PN is installed; and an experience sharing server 1 connected with the user terminals through a network as a communication network. Within a session in which one or more users UN constituting a party are included, an experience can be shared on a site through the experience sharing application PN in each user terminal. Session information to manage the session is stored in the experience sharing server 1, and the experience sharing application PN in each user terminal, respectively. A change made in an experience sharing state within the session when the experience is supplied is reflected in each session information.SELECTED DRAWING: Figure 15

Description

[Technical Field]

[0001] The present invention relates to an experience sharing system and an experience sharing method. [Background technology]

[0002] BACKGROUND ART Experience sharing systems that enable remote users to share various experiences using user terminals via a communication network such as the Internet have been proposed and put into practice.

[0003] For example, Patent Document 1 discloses a remote grave visiting system that allows users to visit a grave from a remote location without actually having to go to the grave, but can instead request a service to visit the grave on their behalf via a network, and also allows them to confirm that the visit has been completed.

[0004] Furthermore, due to the impact of the current global COVID-19 pandemic, there is a need for a new lifestyle that allows people to maintain social distance while connecting with each other in all aspects of life, such as education, tourism, medical care, and sports, and allows everyone to have safe and secure shared experiences without leaving anyone behind.

[0005] For this reason, many new experience sharing systems have been proposed that utilize highly integrated spaces (hereafter abbreviated as CP (Cyber ​​Physical) spaces) by incorporating real-world (physical) information into computer-generated virtual spaces (cyber).

[0006] Although many different experience sharing systems have been proposed, given the circumstances described above, there is a need for such experience sharing systems to be able to be used more flexibly in accordance with the various needs of users in the future. However, no such experience sharing system has been seen or heard of at present. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Japanese Patent Application Laid-Open No. 2005-284676 Summary of the Invention [Problem to be solved by the invention]

[0008] The present invention was devised in consideration of the above-mentioned problems, and its purpose is to provide an experience sharing system and an experience sharing method that can be used more flexibly to suit the various conveniences of users. [Means for solving the problem]

[0009] In order to solve the above-mentioned problems, the inventor has provided an experience sharing system comprising one or more user terminals on which an experience sharing app is installed and a server connected to the user terminals via a communication network, wherein experiences can be shared on a site via the experience sharing app on the user terminals within a session including one or more users constituting a party, and the server and the experience sharing app on the user terminals each store session information for managing experience sharing through the session, and when the session information stored on the server or any one of the experience sharing apps is changed during the experience sharing, the change is reflected in the session information stored on the other servers and in the experience sharing apps.

[0010] The experience sharing system according to claim 1 includes one or more user terminals on which an experience sharing app is installed, and a server connected to the user terminals via a communication network, wherein experience sharing is enabled on a site via the experience sharing app on the user terminals within a session including one or more users constituting a party, and the server and the experience sharing app on the user terminals each store session information for managing experience sharing through the session, and when the session information stored on the server or any one of the experience sharing apps is changed during the experience sharing, the change is reflected in the session information stored on the other servers and the experience sharing apps. The one experience sharing application transmits difference data of the session information corresponding to changes that occur during the experience sharing to the server, and the server reflects the received difference data in the session information stored therein, transmits the difference data to the other experience sharing application, accumulates the difference data for a predetermined time, and when transmitting the difference data to the other experience sharing application, if data identical to newly received difference data exists in the already accumulated difference data, deletes the identical data from the accumulated difference data. It is characterized by the following.

[0011] The experience sharing system described in claim 2 is the invention described in claim 1, wherein the server At least to the party of It is characterized by storing party information for user participation authentication.

[0012] The experience sharing system described in claim 3 is the invention described in claim 1 or 2, characterized in that the server manages the site for sharing experiences within the session via a URI.

[0015] Claim 4 The experience sharing method described in is an experience sharing method performed between one or more user terminals on which an experience sharing app is installed and a server connected to the user terminals via a communication network, in which experience sharing is enabled on a site via the experience sharing app on the user terminals within a session including one or more users constituting a party, and session information for managing the session is stored in the server and the experience sharing app on the user terminals, respectively. share In the session Change to When a change occurs, The change in question Reflect The method includes a step of transmitting, by any one of the experience sharing apps, difference data of the session information corresponding to changes that have occurred during the experience sharing to the server, and the server reflecting the received difference data in the session information stored therein and transmitting the difference data to other experience sharing apps, and accumulating the difference data for a predetermined time and then transmitting it to other experience sharing apps, and if data identical to newly received difference data exists in the already accumulated difference data, deleting the identical data from the accumulated difference data. It is characterized by: [Effects of the Invention]

[0016] According to the present invention having the above configuration, when session information stored in an experience sharing server or one of the experience sharing apps is changed, the change is reflected in session information stored in other experience sharing servers and experience sharing apps. This allows various changes that occur when sharing experiences in one experience sharing app to be reflected in other experience sharing apps, thereby facilitating experience sharing between users. [Brief explanation of the drawings]

[0017] [Figure 1] FIG. 1 is a schematic diagram showing a schematic configuration of the entire experience sharing system according to an embodiment of the present invention. [Figure 2] FIG. 2 is a block diagram showing a schematic configuration of an experience sharing system according to an embodiment of the present invention. [Figure 3] FIG. 3(a) is a schematic diagram for explaining a site, FIG. 3(b) is a schematic diagram for explaining a party, and FIG. 3(c) is a schematic diagram for explaining a session. [Figure 4] FIG. 4 is an explanatory diagram showing a specific example 1 of experience sharing in the experience sharing system according to the embodiment of the present invention. [Figure 5] FIG. 5 is an explanatory diagram showing a specific example 2 of experience sharing in the experience sharing system according to the embodiment of the present invention. [Figure 6] FIG. 6 is an explanatory diagram showing a specific example 3 of experience sharing in the experience sharing system according to the embodiment of the present invention. [Figure 7] FIG. 7 is an explanatory diagram showing a specific example 4 of experience sharing in the experience sharing system according to the embodiment of the present invention. [Figure 8] FIG. 8 is an explanatory diagram showing a specific example 5 of experience sharing in the experience sharing system according to the embodiment of the present invention. [Figure 9] FIG. 9 is a plan view showing a state in which a panoramic video camera placed at a certain position in physical space is capturing images in all directions. [Figure 10] FIG. 10 is a plan view showing an example in which an avatar is placed at one of the positions within a CP space recreated based on past video. [Figure 11] FIG. 11 is a diagram showing an example of an experience sharing system to which the present invention is applied, which is configured by an experience sharing application installed in a user terminal used by a user, and an experience sharing server. [Figure 12] FIG. 12 is a diagram showing an example in which one party configures multiple sessions. [Figure 13] FIG. 13 is a diagram showing an example of specifying a site by a URI. [Figure 14] FIG. 14 is a diagram illustrating the main components of the experience sharing system. [Figure 15] FIG. 15 is a diagram for explaining a synchronization method of the experience sharing system according to this embodiment. [Figure 16] FIG. 16 is a diagram showing an example of distributing difference data based on changes made to session information on the experience sharing application side. [Figure 17] FIG. 17 is a diagram illustrating details of session information for managing sessions. [Figure 18] FIG. 18 is a diagram showing an example of exchanging instant messages to immediately notify changes in session information in an experience sharing system to which the present invention is applied. [Figure 19] FIG. 19 shows an example in which each user designates a recipient and sends and receives an instant message to and from the designated recipient without changing the contents of the session information. [Figure 20] FIG. 20 is a diagram showing an example in which a session clock is provided to keep track of session time between the experience sharing server and the experience sharing application. [Figure 21] FIG. 21 is a diagram showing an example of synchronizing event occurrence times in an experience sharing system to which the present invention is applied. [Figure 22] FIG. 22 is an explanatory diagram for explaining details of the account information and party information recorded in the experience sharing server. [Figure 23]FIG. 23 is a diagram for explaining a management technique at the start of a party in session operation management. [Figure 24] FIG. 24 is a diagram for explaining a method for joining and leaving a party in the operation and management of a session. [Figure 25] FIG. 25 is a diagram for explaining a method for moving a session and changing a site in session operation management. [Figure 26] FIG. 26 is a diagram for explaining an example of ending a session and a party in session operation management. [Figure 27] FIG. 27 is a diagram showing an example in which an extended function is provided as a gateway for exchanging content and the like in cooperation with external services and the like. [Figure 28] FIG. 28 is a diagram for explaining a specific example of the extended function in the experience sharing system according to this embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0018] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, exemplary embodiments of the present invention will be described with reference to the accompanying drawings.

[0019] [Embodiment] First, an experience sharing system 100 according to an embodiment of the present invention will be described with reference to Figures 1 and 2. Figure 1 is a schematic diagram showing a general configuration of the entire experience sharing system 100 according to an embodiment of the present invention, and Figure 2 is a block diagram showing a general configuration of the experience sharing system 100 according to an embodiment of the present invention.

[0020] 1, an experience sharing system 100 according to an embodiment of the present invention includes an experience sharing server 1. The experience sharing server 1 is connected to a plurality of user terminals 31 to 3N, on which an experience sharing app is installed, and an administration terminal 5 via a network N, which serves as a communication network.

[0021] The experience sharing server 1, which will be described in detail later, is a host computer that serves as a platform that enables experience sharing on sites 41-4N via experience sharing apps on user terminals 31-3N within a session that includes one or more users making up a party. The experience sharing server 1 also serves as a central control unit that controls the entire experience sharing system 100.

[0022] The user terminals 31 to 3N are terminal devices that allow users to share experiences through an experience-sharing app, and may be, for example, personal computers (PCs), smartphones, tablet devices, or wearable devices. Furthermore, the user terminals 31 to 3N may be configured as head-mounted displays (HMDs) or eyeglass-type devices worn on the user's head. The user terminals 31 to 3N are equipped with at least a display for displaying various content and images captured by a camera. By viewing the content and images displayed on the user terminals 31 to 3N, the user can experience the sensation of actually experiencing the space. The user terminals 31 to 3N can be connected to the network N via wired or wireless connection.

[0023] The management terminal 5 is a terminal device that an administrator operates in the experience sharing system 100 to which the present invention is actually applied, and that allows users to share experiences through an experience sharing app, and is similarly configured as a PC, smartphone, tablet terminal, wearable terminal, etc. The management terminal 5 can be connected to the network N via a wired or wireless connection. The management terminal 5 can become the management terminal 5 by setting one or more of the other user terminals 31 to 3N to be used for management purposes.

[0024] As shown in FIG. 2, the experience sharing server 1 includes an integrated management unit 10 including a CPU (Central Processing Unit) and the like.

[0025] This overall management unit 10 is connected to an interface 11 for transmitting and receiving information to and from multiple user terminals 31 to 3N via a network N, and to a storage unit 2 such as a memory or a database. The overall management unit 10 manages various data stored in the storage unit 2. The overall management unit 10 also records various data in the storage unit 2, and, when required for operating the experience sharing system 100, reads out the various data stored in the storage unit 2 and controls distribution to the outside.

[0026] Next, the operation of the experience sharing system 100 to which the present invention is applied and configured as described above will be described.

[0027] In the experience sharing system 100 to which the present invention is applied, information is shared in a CP space with users who want to share a joint experience in various scenes, such as daily life, work, and entertainment, or with users who want others to share a joint experience. The CP space here refers to a more realistic cyberspace that combines the physical (real world) and the cyber (virtual world). The CP space may be configured not only as a rectangular parallelepiped, but also as a sphere in the case of a 360-degree video, or any other shape. The shared experience here refers to users sharing the same text and audio on their respective user terminals 31-3N, or interacting as avatars in the same video space, so that each user experiences the same experience with the same sense of realism. In other words, by sharing content displayed on one user terminal 31 or video captured by a camera with other user terminals 32-3N, users can share the content they are experiencing in the space where the displayed content or video captured by the camera is displayed. When the content displayed on one user terminal 31 changes, the content can be changed in the other user terminals 32 to 3N in the same way, allowing users to feel as if they are always sharing an experience.

[0028] In the present invention, the following attributes are defined to realize such sharing of experiences between users.

[0029] First, as shown in Figure 3(a), a site means "something or place to be shared," and is composed of text, audio, real video, 3D data, etc., as well as spaces that combine these. This site is a space that users can experience, such as a classroom, conference room, model room, tourist spot, or construction site.

[0030] As shown in Figure 3(b), a party is a group consisting of one or more users U1-UN. Users U1-UN who wish to share their experiences through the site must belong to at least this party; users who do not belong to a party cannot share their experiences through the site. This party is a group formed from one or more people, such as a school class, contest participants, or a group on a group trip, and can be either a group that only those with specific authority can join or a group that those without any authority can enter and leave freely.

[0031] As shown in Figure 3(c), a session is a state in which one or more users U1 to UN share information on a site. In other words, it refers to a state in which users enter a site with the purpose of sharing an experience and experience the content. A session here also includes a state in which one user participates in the site. When multiple users U1 to UN participate in a site and form a session, the same content can be experienced by each of the users U1 to UN participating in the site, which realizes what is known as experience sharing.

[0032] The term "session" as used herein refers to sharing experiences among a group of students in a school class, assuming that the party is a school class. However, the session is not limited to this, and all users in the party can participate. Therefore, if the party is a school class, all students in the class can share experiences through the site.

[0033] Next, specific examples 1 to 5 of experience sharing through a session between users in the experience sharing system 100 according to an embodiment of the present invention will be described with reference to Figures 4 to 8. Figure 4 is an explanatory diagram showing specific example 1 of experience sharing in the experience sharing system 100 according to an embodiment of the present invention, Figure 5 is an explanatory diagram showing specific example 2 of experience sharing in the experience sharing system 100 according to an embodiment of the present invention, Figure 6 is an explanatory diagram showing specific example 3 of experience sharing in the experience sharing system 100 according to an embodiment of the present invention, Figure 7 is an explanatory diagram showing specific example 4 of experience sharing in the experience sharing system 100 according to an embodiment of the present invention, and Figure 8 is an explanatory diagram showing specific example 5 of experience sharing in the experience sharing system 100 according to an embodiment of the present invention.

[0034] First, in specific example 1 of experience sharing system 100 shown in Figure 4, the display image C1, which is the CP space, is an actual image of various still images or videos, and avatars A1 to AN assigned to each user U1 to UN who form a session with each other can walk around within the display image C1, which is the CP space, and can view images of any angle of view within the 360° CP space displayed on each user terminal 31 to 3N, and can interact with other avatars using text, voice, etc., thereby enabling a joint experience.

[0035] In addition to each user independently exploring the CP space and communicating with each other, authorized user U1 among users U1-UN can operate his / her user terminal 31 to view an image of any angle of view within the 360° CP space or perform zooming in or out. In response to this, the other user terminals 32-3N can display the same image or point to a specific point within the image. Thus, each user U1-UN can share an experience by viewing a common image displayed on the user terminals 31-3N and listening to a common audio. This experience-sharing method can be applied, for example, to supplement collaborative experiential activities such as field trips and outdoor learning, or to remote site inspections and room previews.

[0036] Concrete example 2 of the experience sharing system 100 shown in FIG. 5 is an example of remote medical care. Patients (users U2-UN) wear user terminals 32-3N each consisting of an HMD in a CP space, and the images viewed by users U2-UN via their user terminals are displayed on the user terminal 31 of the doctor (user U1) who is participating in the same session. This allows the doctor (user U1) to share the situation viewed by users U2-UN as patients, thereby improving the efficiency of diagnosis. Experience sharing via user terminals 32-3N each consisting of an HMD can be applied to remote medical care via remote medical measurements of multiple subjects, as well as to mindfulness in the relaxation industry, compatibility diagnosis in matchmaking, and even empathy and communication tools for care recipients in nursing homes and care facilities.

[0037] Next, a specific example 3 of the experience sharing system 100 shown in FIG. 6 is an example in which a CP space reproduced with still or moving images is projected into the dome of a planetarium. Multiple avatars A1-AN, each wearing a space suit and assigned to each user U1-UN who is participating in a session, fly freely within the display image C1 in the CP space, which is the planetarium dome, via user terminals 31-3N. By viewing any direction within the 360° CP space, the angle of view of the image corresponding to that viewing direction can be displayed on the user terminals 31-3N. Similarly, when authorized user U1 among users U1-UN operates avatar A1 via his / her user terminal 31, views any direction within the 360° CP space, or performs zooming in or out, similar images can be displayed on the other user terminals 32-3N in response to the operation. This allows each user U1-UN to remotely share their planetarium experiences with one another. In the planetarium dome, any direction in the CP space can be viewed using user terminals 31 to 3N, and the images projected in the dome and the movements of avatars A1 to AN can be viewed from above.

[0038] Next, in a specific example 4 of the experience sharing system 100 shown in FIG. 7 , a display image C1 representing the CP space is composed of a real image captured by a camera mounted on the robot R. The head movements of a user U1 wearing a user terminal 31 consisting of an HMD correspond to the movement of the robot R, and the control devices 7 held in both hands correspond to the movement of the robot R's hands, etc., allowing the user U1 to remotely control the robot R while viewing the display image C1. In this case, the user U1 can also exchange control rights with another user U2 participating in the same session. In such a case, the user U2 shares the same image C1 and operation of the robot R as the user terminal 31 with the user terminal 32 worn by the other user U2, allowing the user U2 to remotely control the robot R in the same way as the user U1. This allows the robot R to be moved by the head movements of the user wearing the user terminal consisting of an HMD, thereby taking advantage of the advantages of hands-free operability and the ability to share control rights, enabling collaborative remote robot work and other tasks that are more convenient than ever before.

[0039] Next, a specific example 5 of the experience sharing system 100 shown in Fig. 8 implements a module that imports 3D data from outside into the CP space. The external 3D data may be static objects such as artwork or exhibited products, or it may be data that dynamically displays a play or the like by importing 3D data of actors. The example shown in Fig. 8 is an example of an online practical haircutting training course, in which a user can observe the practical training by operating a user terminal 32 such as an HMD or tablet device in response to 3D data of a mannequin's head displayed in the CP space.

[0040] To realize such experience sharing, video captured in real time or video captured in the past may be used. These videos are captured using a capture module. Specifically, the capture module captures the video using a video camera, digital camera, 3D scanner, or the like. The plan view in FIG. 9 shows a state in which video is captured in all directions by a panoramic video camera 37 placed at position P in the physical space. The image capture device 321 in the panoramic video camera 37 captures video in the horizontal direction without omission across 360°. This capture is also simultaneously captured in the vertical direction, but the following example will be described using the horizontal capture and a CP space recreated using the video.

[0041] When video is captured in this manner, multiple imaging devices 321 capture video in each direction in chronological order. The captured images are then stored sequentially. When the capturing of past video is completed, still images or moving images captured in all directions horizontally, centered on position P, are stored. At this time, audio may be simultaneously recorded using a microphone (not shown). This type of omnidirectional image capturing is performed similarly for each of the other positions Q to T in the physical space. As a result, 360° omnidirectional images are stored at some or all positions in the physical space.

[0042] When the above-described experience sharing is performed using such past footage, it is assumed that one or more avatars are present at any of positions P to T in the CP space recreated based on the past footage, as shown in Fig. 10. For example, if avatar AN is located at position P, when past footage is to be displayed on user terminals 31 to 3N, user terminals 31 to 3N can read out past footage stored in advance for position P and display it on the display for display.

[0043] At this time, if the line of sight, position, or state of the avatar AN in the CP space is changed via the user terminals 31-3N, the display on the display may be updated accordingly. In this case, an image corresponding to the avatar's line of sight is extracted from previously stored images and displayed on the display of the user terminals 31-3N, or the image is enlarged or reduced depending on the avatar's position. When the avatar AN moves from position P to another position Q-T, a pre-stored image for the new position Q-T is extracted, enlarged, or reduced in response to the avatar AN's operation, and displayed. In this way, by extracting, enlarging, or reducing the image in response to the avatar AN's operation, the user can have a sense of reality as if they were viewing the image with a visual sense corresponding to the avatar AN's line of sight, etc. Furthermore, since pre-stored images can be displayed for each position P-T to which the avatar AN moves, each user U1-UN can experience the same sense of realism as the avatar AN freely wandering around the CP space.

[0044] In an application in which users U1-UN wearing user terminals 31-3N such as HMDs, rather than avatars AN, can move freely within the CP space, it is assumed that users U1-UN are present at any of positions P-T within the CP space where past video was captured. For example, when a user is located at position P, to display past video on user terminals 31-3N, user terminals 31-3N read out past video stored in advance for position P and display it on the display. When user U1-UN moves from position P to another position Q-T, the video stored in advance for the new position Q-T is cropped, enlarged, or reduced in response to an operation by user U1-UN. In such cases, the display on the display reflects the operation of user terminals 31-3N such as HMDs or changes in the line of sight or position of the user wearing user terminals 31-3N, thereby providing a sense of reality in which the user is viewing the video with a visual sense that corresponds to the line of sight, etc. When using the user terminals 31 to 3N each made up of an HMD, the display on the display screen may be based on an operation controller or hand tracking function provided in the HMD, position information of the HMD itself, or the like.

[0045] When sharing experiences in such a CP space, for example, user U1 or his / her avatar A1 is located at position P, and user U2 or his / her avatar A2 is located at position Q. In this case, if user U1 is authorized, user U1 can display the image displayed on the display of user terminal 31 on the display of user terminal 32 operated by user U2. The image displayed on the display of user terminal 31 is the image viewed at position P, but it can be viewed not only by user U1 but also by user U2, who is located at another position Q. Therefore, users U1 and U2 can simultaneously view the image viewed by user U1 or his / her avatar at position P, enabling them to share experiences with each other. At this time, by playing previously recorded audio at each of positions P to T, it is possible to realize experience sharing that includes audio.

[0046] Next, details of the control management etc. of the experience sharing system 100 according to this embodiment will be described with reference to Figures 11 to 28. Figures 11 to 28 are explanatory diagrams for explaining details of the control management etc. of the experience sharing system 100 according to this embodiment.

[0047] First, a general configuration of an experience sharing system 100 according to this embodiment will be described.

[0048] As shown in Fig. 11, the experience sharing system 100 is made up of experience sharing applications P1-PN installed on user terminals 31-3N used by users U1-UN, and an experience sharing server 1 connected to a network N. These experience sharing applications P1-PN provide the same session to users U1-UN. Video and audio captured at each of the above-mentioned positions P-T are transferred to the experience sharing applications P1-PN via the experience sharing server 1, and these videos and audio are read out and displayed on the user terminals 31-3N in response to operations of the experience sharing applications by users U1-UN.

[0049] In this experience sharing system 100, video captured by one user U1 operating the experience sharing app P1 may be distributed to other experience provision apps P2-PN without being stored in the experience sharing server 1. Alternatively, video temporarily stored in the experience sharing server 1 by one user U1 operating the experience sharing app P1 may be streamed to other experience provision apps P2-PN. Alternatively, video temporarily stored in the experience sharing server 1 by one user U1 operating the experience sharing app P1 may be downloaded to other experience provision apps P2-PN.

[0050] As a result, a large number of experience sharing applications P1 to PN can communicate with the experience sharing server 1 via the experience sharing protocol, thereby sharing information for the above-mentioned audiovisual experiences and interactions.

[0051] It should be noted that different experience sharing applications P1 to PN can be mixed and implemented as long as the semantics of using the experience sharing protocol (the content and handling of data exchanged through the experience sharing protocol) are the same.

[0052] As shown in FIG. 12, the above-mentioned party is made up of users U1 to UN and U1' to UN' who can share information through the experience sharing system 100. Among this party, a session for sharing information with one or more users U1 to UN is formed at a certain site via experience sharing applications P1 to PN. Similarly, another session for sharing information with one or more users U1' to UN' is formed at another site via experience sharing applications P1' to PN'. The experience sharing applications P1 to PN and the experience sharing applications P1' to PN' form different sessions from each other.

[0053] In this experience sharing system 100, users U1 to UN and U1' to UN' of the experience sharing app can specify a party, connect to the experience sharing server 1, and optionally specify a site to participate in a session within the party. In addition, in this experience sharing system 100, it is also possible to create a new session within a party or move to a different session midway through.

[0054] Furthermore, as shown in FIG. 13, in this experience sharing system 100, each site, such as content or CP space shared by users U1 to UN or users U1' to UN' participating in the same session, is uniquely linked to the session. Therefore, changing the site switches the session. A site is made up of all kinds of resources that can be specified by a Uniform Resource Identifier (URI). For example, this site also includes content such as online still images and videos, live streaming video, PDF data, 3D data, and CP space data in a unique format that can be specified by a URL.

[0055] As shown in FIG. 14, the main components of the experience sharing system 100 include a client library 52, which is a developer's proprietary program 51 provided to the experience sharing application PN; a server program 53, which is a group of programs on the experience sharing server 1 side; a memory unit 2; and an experience sharing protocol 55, which regulates communication between the client library 52 and the server program 53. In the experience sharing system 100, a developer of the experience sharing application PN can develop an experience sharing application PN such as the above-mentioned application examples by using the client library 52 to write the proprietary program 51. This experience sharing application PN can provide a variety of shared experiences to users UN. In this experience sharing system 100, the experience sharing server 1 may be provided with an extension function 54 that serves as a gateway for extension functions for linking with external servers, etc.

[0056] Next, a method for synchronizing the states of the experience sharing system 100 according to this embodiment will be described.

[0057] In the experience sharing system 100, as shown in FIG. 15, information related to the state of the session, such as audiovisual information, provided by the experience sharing applications P1-PN to users U1-UN is described and stored as session information. The experience sharing server 1 also stores master session information. The experience sharing server 1 provides the latest session information to new experience sharing applications P1-PN that handle the same semantics and that have joined the session. Furthermore, in the experience sharing system 100, changes made to the session information by the experience sharing applications P1-PN are propagated via the experience sharing protocol 55 to the other experience sharing applications P1-PN that handle the same semantics and the experience sharing server 1, thereby maintaining synchronization of the respective session information.

[0058] More specifically, in the experience sharing system 100, the entire latest session information is sent to each of the experience sharing applications P1 to PN using the same semantics when joining a session (step S11). When some change is made to one of the experience sharing applications P1 to PN, for example, to application P1 (step S12), the master session information in the experience sharing server 1 is updated based on the change (step S13). As a result, the difference between the session information before and after the update is distributed as stream data to the other experience sharing applications P2 to P2N on the experience sharing server 1 side (step S14). This makes it possible for changes made to one experience sharing application P1 to be reflected on the other experience sharing applications P2 to P2N sides.

[0059] Furthermore, in the experience sharing system 100, as shown in FIG. 16, the experience sharing applications P1-PN may transmit difference data based on changes made to the session information on the experience sharing applications P1-PN sides to the experience sharing server 1 at regular intervals (step S15). When difference data based on changes made to the session information on each of the experience sharing applications P1-PN sides is transmitted to the experience sharing server 1, the experience sharing server 1 aggregates the transmitted changes and reflects them in the master session information on the experience sharing server 1 at regular intervals (e.g., 0.2 seconds) (step S16). At the same time, the aggregated changes, i.e., difference data, are transmitted to each of the experience sharing applications P1-PN (step S17). This allows difference data based on changes made to each of the experience sharing applications P1-PN to be collectively accumulated on the experience sharing server 1 side, and the accumulated difference data can be reflected on all of the experience sharing applications P1-PN sides. The session information data exchanged at regular intervals is called a session stream. The session streams include a differential stream that transmits only differential data, and a full stream that transmits all data at once.

[0060] 17 shows the configuration of session information for managing each session. The session information is made up of a session management section 61, a party common section 62, a user list section 63, a session specific section 64, and a site specific section 65. The experience sharing apps P1 to PN participating in a session may share any or all of the session management section 61, party common section 62, user list section 63, session specific section 64, and site specific section 65 in the session information. The party common section 62 is shared (synchronized) by all sessions within the party.

[0061] Here, the session management unit 61 includes information such as a session identifier, a site (URI) corresponding to the session, and a time within the session.

[0062] The party common section 62 includes information shared among all sessions within the party, and some of this information may be permanently recorded in the experience sharing server 1 even after the session has ended. That is, the party common section 62 may record information that is updated each time a session is started through its end and deleted when the session ends, as well as information that remains permanent even after the session has ended.

[0063] The user list section 63 includes information about users U1 to UN who are participating in a session. The information about each of users U1 to UN recorded in the user list section 63 is added to the experience sharing applications P1 to PN when the user U1 to UN joins a session, and is deleted when the user leaves the session.

[0064] The session specific part 64 records the temporary state of the experience sharing applications P1 to PN shared within a session. The information about the experience sharing applications P1 to PN recorded in this session specific part 64 is discarded when the session ends.

[0065] The site-specific section 65 records setting information for each site and site-specific information generated by the experience sharing applications P1 to PN. The site-specific information recorded in the site-specific section 65 may be permanently recorded in the experience sharing server 1 even after the session ends. The site-specific section 65 may record information that cannot be fully described by the URI alone, depending on the session, as site information. The site-specific section 65 may also record new information or knowledge generated in the space as notes, and is used to retain necessary knowledge even after the session ends.

[0066] In addition, the meaning (semantics) of the information contained in the four sections of the session information other than the session management section 61 (party common section 62, user list section 63, session specific section 64, and site specific section 65) can also be determined arbitrarily by the developer of the experience sharing apps P1 to PN.

[0067] An example of information actually described as session information in a certain semantics is described below. The semantics exemplified here is an experience sharing application PN in which users U1 to UN appear as avatars in a CP space with 360-degree video, and can freely move within the space, freely change their viewpoint, and zoom in or out by pointing to a location within the video.

[0068] In the session management unit 61, the URL of the online 360° video file is set as the site URL.

[0069] The party common section 62 stores, as permanent information, a list of content that can be displayed by the experience sharing application PN, and information such as operation authority for each user U1 to UN who is participating in the party or has participated in the party in the past. The operation authority here is made up of information such as the playback status of the 360° video, operating the viewpoint of the 360° video, pointing to a certain part of the 360° video, and displaying an avatar within the 360° video.

[0070] In the user list section 63, the user information for each user includes information such as the display name of each user U1 to UN, the position and orientation of their avatar, and the position of the image that the user U1 to UN is pointing at. The information for each user U1 to UN recorded in this user list section 63 may be updated whenever the position or orientation of the avatar operated by the user changes from time to time, for example.

[0071] The session specific portion 64 describes the playback status of the 360° video, whether the user has the authority to operate the viewpoint of the 360° video, point to a specific part of the 360° video, display an avatar within the 360° video, etc.

[0072] The site-specific portion 65 describes information about additional content such as text and reference photos placed within the 360° video.

[0073] That is, according to the experience sharing system 100 to which the present invention is applied, the experience sharing server 1 and the experience sharing applications P1-PN each store session information for managing sessions. Under this premise, if some change occurs in the experience sharing, such as the position or orientation of an avatar, due to operation of the experience sharing application P1, the session information stored in the experience sharing application P1 is changed. This change in session information may also occur on the experience sharing server 1 side. When the session information stored in the experience sharing server 1 or any one of the experience sharing applications P1-PN is changed, the change is reflected in the session information stored in the experience sharing server 1 or the other experience sharing applications P1-PN. This allows various changes that occur in the experience sharing in one of the experience sharing applications P1-PN to be reflected in the other experience sharing applications P1-PN, thereby facilitating experience sharing among users U1-UN.

[0074] Furthermore, the experience sharing system 100 allows instant messaging, as shown in FIG.

[0075] Specifically, the experience sharing applications P1 to PN can send instant messages to the experience sharing server 1 (step S18). These instant messages use the differential data to be synchronized as a parameter in order to synchronize session information immediately without waiting for regular intervals of transmission from the experience sharing server 1. When the experience sharing server 1 receives an instant message from one experience sharing application P1, it immediately distributes the same instant message to the other experience sharing applications P2 to PN (step S19). At the same time, the experience sharing server 1 also overwrites the master session information in the experience sharing server 1 with the contents of this instant message (step S20). When transmitting the newly received differential data to another experience sharing application after storing the differential data for a predetermined period of time, if the newly received differential data contains data identical to the previously stored differential data, the experience sharing server 1 may delete the identical data from the stored differential data (step S21). This prevents duplicated storage of differential data.

[0076] Furthermore, in the experience sharing system 100, as shown in FIG. 19, each of the users U1 to UN may designate a recipient and send and receive instant messages to and from the designated recipient.

[0077] When specifying other users as recipients, specifically, one user U1 specifies other users U2 to UN as recipients and sends an instant message (step S22). Next, the instant message is distributed to the experience sharing applications P2 to PN of the specified recipients (other users U2 to UN) (step S23). Note that the instant message may use the recipient itself as a parameter, or any data as a parameter. Furthermore, the user as a recipient can specify all the members of the party, all the members of the session, a specific user UN, or a specific data object. Furthermore, the method of handling the received parameters (data) (semantics) can be determined arbitrarily by the developer of the experience sharing applications P1 to PN, independently of the session information.

[0078] In the experience sharing system 100, as shown in FIG. 20, the experience sharing server 1 and the experience sharing application PN may have a session clock that keeps track of the session time commonly used within the session.

[0079] Specifically, when the experience sharing application PN joins a session (step S24), the experience sharing server 1 notifies the current session time (step S25), and the session clock on the experience sharing application PN side starts at the notified time (step S26).

[0080] Furthermore, in the experience sharing system 100, the experience sharing application PN continuously monitors the session time of the experience sharing server 1 and synchronizes the session time in the experience sharing application PN with the session time of the experience sharing server 1.

[0081] Specifically, the experience sharing application PN transmits the session time t0 to the experience sharing server 1 (step S27). The experience sharing server 1 returns the received session time t0 and the experience sharing server 1 session time t1 to the experience sharing application PN (step S28). Based on the received session time t2 on the experience sharing application PN side and the received t0 and t1, the communication delay tL and the time difference tD are estimated using the formulas (tL=(t2-t0) / 2 and tD=(t0+t2) / 2-t1), and the difference in the session clock is adjusted (step S29). Note that the above-described method of adjusting the difference in the session clock is merely an example, and it goes without saying that adjustment may be performed based on any other method.

[0082] Furthermore, in the experience sharing system 100, as shown in FIG. 21, the times at which events occur can be synchronized.

[0083] Specifically, the experience sharing server 1 assigns a session time to a message that synchronizes the state of each session. This message that synchronizes the state of a session may be a session stream or an instantaneous message. The experience sharing applications P1 to PN update the representation of the state or event by taking into account the difference between the session time assigned to the message and the current session time.

[0084] Furthermore, as a state synchronization method, for example, when synchronizing video playback, the session specific unit 64 previously stores the session time when the video operation is completed, the new playback state (play / pause / reverse), the video playback position at that time (frame number and time code), etc. Then, when the experience sharing applications P1 to PN perform various video control operations such as stopping playback or seeking, they send this information as parameters in an instant message to the experience sharing server 1. The experience sharing server 1 distributes the playback state of the video to be synchronized to each experience sharing application P1 to PN via a session stream or an instant message. This synchronizes the video playback state in each experience sharing application P1 to PN. An experience provision application PN that newly joins a session can reproduce the video playback state using this information contained in the session stream.

[0085] Next, a method for managing sessions in the experience sharing system 100 according to this embodiment will be described.

[0086] In the experience sharing system 100, as shown in FIG. 22, the storage unit 2 of the experience sharing server 1 stores account information about a plurality of accounts, party information about parties, and the like.

[0087] The account information includes an account identifier, authentication information, type, authority, and the like.

[0088] Party information includes party attribute information, persistent information in the party common section, party participant information, and site-specific information for each site. Party attribute information describes information such as the party's identifier and name, the party creator, various authority information, and initial site. Persistent information in the party common section describes information and knowledge that is used permanently rather than being deleted after the session ends. Party participant information describes the accounts of users who can join the party, the identifiers of each participant that are valid only within the party, and the authority of each participant. Site-specific information describes detailed information for each site. The site-specific information may also describe the URI of the site.

[0089] The experience sharing system 100 uses such account information and party information to manage and operate the session. First, the management method at the start (activation) of the party will be described. As shown in FIG. 23, when the experience sharing application PN connects to the experience sharing server 1 (step S30), the connection is first authenticated based on the account information in the storage unit 2 (step S31). If the connection is authenticated based on this account information, the process proceeds to step S32; if the connection is not authenticated, the activation process ends.

[0090] When proceeding to step S32, it is also possible to restrict participation by comparing the account information with the party participant information in the party information. The party participant information describes the accounts of users U1 to UN who can participate in the party, so by comparing this information, it is possible to authenticate the participation of users who have accessed via the experience sharing apps P1 to PN. If participation is authenticated, the process proceeds to step S33, where the party is activated; if participation is not authenticated, the activation process ends.

[0091] When the party is activated in step S33, a session corresponding to the site specified for the party is established, and session information is created. When creating the session information, the site specified for the party is first established based on the party attribute information in the party information. If there is persistent information in the party common section in the party information, this is shared in the persistent information in the party common section 62 in the session information. Users participating in the session are registered in the user list section 63 in the session information based on the party participant information in the party information. Site-specific information in the party information is written to the site-specific section 65 in the session information. In this way, session information corresponding to the site specified for the party can be generated.

[0092] As a result, the user UN can participate in the session corresponding to this session information via the connected experience sharing application PN (step S34).

[0093] Next, the methods for joining and leaving a party will be described. As shown in Fig. 24, when an experience sharing application PN newly connects to an already active party (step S35), the connection is authenticated based on the account information in the storage unit 2 (step S36). If the connection is authenticated based on this account information, the process proceeds to step S37; if the connection is not authenticated, the connection is simply disconnected.

[0094] When the process proceeds to step S37, the party participant information in the party information is referenced as necessary, and the participation right of user UN who owns the newly connected experience sharing application PN is verified. If the result shows that participation right is not necessary, user UN is added to the session corresponding to the site specified in the party regulations. If participation right is necessary and valid, user UN is added to the session corresponding to the site specified in the party regulations. Specifically, the newly added user UN is added to the user list section 63 in the session information. As a result, user UN can participate in the session corresponding to this session information via the connected experience sharing application PN (step S38). Note that if there is no session corresponding to the specified site, a new session corresponding to the specified site may be created and user UN may participate.

[0095] Furthermore, when user UN who has been participating in the session up until now ends the shared experience using the experience sharing application PN, the experience sharing application PN is disconnected from the experience sharing server 1 (step S39). As a result, the experience sharing application PN leaves the session. At that time, user UN may also leave the party (step S40). Furthermore, when user UN leaves the party, the user UN who has left may be deleted from the user list section 63 in the session information.

[0096] Next, a method for moving a session, i.e., changing sites, will be described. As shown in FIG. 25, the experience-sharing application PN connects to the experience-sharing server 1 and requests a site change (step S41). Next, the connection is moved to a session corresponding to the destination site (step S42). The relevant user UN is also removed from the user list section 63 of the session information before the move (step S43). Furthermore, the party participant information in the party information is referenced, and the relevant user UN is added to the user list section 63 of the destination session information (step S44). Depending on the site change mode, there are three types of target users UN to be moved: only the requested user UN, all users in the same session, or all party participants. If there is no session corresponding to the destination site, a new session may be created based on the party information and the user UN can join the destination session. As a result, the user UN can join the destination session from the session before the move.

[0097] Next, the case of ending a session and a party will be described. In the experience sharing system 100, as shown in FIG. 26, the experience sharing application PN disconnects from the experience sharing server 1 (step S45). Then, the experience sharing application PN itself leaves the session (step S46). The user UN operating the experience sharing application PN is deleted from the user list section 63 in the session information. Next, in step S47, if there is site-specific information in the site-specific section 65 of the session information that should be managed permanently, this is reflected in the party information. The session is also ended, and the session information is deleted (step S48). Then, when all sessions have ended, if there is permanent information in the party common section 62, it is saved as permanent information in the party common section of the party information (step S49), and the party is ended and deactivated (step S50).

[0098] In this way, in the experience sharing system 100 to which the present invention is applied, when user UN starts a party, joins a session, and shares an experience, participation authentication can be performed via party information. Also, when user UN changes the site where he or she shares an experience, or when he or she ends a session or party, the situation can be reflected in the party information, so that joining, leaving, site changes, etc. of the entire experience sharing system 100 can be managed collectively via the party information. This makes it possible to prevent suspicious individuals who do not belong to the party from joining a session without permission and moving freely within the session.

[0099] 27, the experience sharing system 100 may be provided with an extended function 54 that serves as a gateway for exchanging content etc. in cooperation with external services (servers) etc. Finally, a method for extending the functions of the experience sharing system 100 by linking this extended function 54 with another external server 70 will be described.

[0100] FIG. 28 shows an example in which the experience sharing system 100 cooperates with a voice call server. The experience sharing server 1 creates a voice channel corresponding to the session in the voice call server 1' (step S51). When the experience sharing application PN connects to the experience sharing server 1 and participates in the session (step S52), a connection token to the voice call server 1' is issued (step S53) and notified to the experience sharing application PN (step S54). When the experience sharing application PN connects to the voice call server 1' using the token (step S55), it is made to participate in the channel for that session (step S56), and the voice call is initiated (step S57). Note that when the experience sharing application PN moves to a different session, the participating channel of the voice call server 1' is also changed accordingly.

[0101] Although the embodiments of the present invention have been described above, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, and are also included in the inventions and their equivalents as defined in the claims. [Explanation of symbols]

[0102] 1. Experience sharing server 1´ Voice call server 2 Storage section 31~3N User terminal 41~4N site 5. Management terminal 7. Controls U1~UN users U1´~UN´ users U User P1~PN experience sharing app P1´~PN´ experience sharing app R Robot A1~AN Avatar C1 Display image (CP space) N Network (communication network) 10 General Management Department 11 Interface 37 Panoramic Video Camera 41 sites 51 Original Programs 52 Library 53 Server Program 54 Extensions 55 Experience Sharing Protocol 61 Session Management Unit 62 Party Common Area 63 User List Section 64 Session specific part 65 Site-specific part 100 Experience Sharing System

Claims

1. An experience sharing system including one or more user terminals on which an experience sharing application is installed, and a server connected to the user terminals via a communication network, In a session including one or more users constituting a party, experiences can be shared on the site via an experience sharing app on the user terminal, the server and the experience sharing application in the user terminal each store session information for managing the experience sharing through the session; When the session information stored in the server or any one of the experience sharing apps is changed during the experience sharing, the change is reflected in the session information stored in the other servers and the experience sharing apps; the one experience sharing application transmits to the server difference data of the session information corresponding to a change that occurs during the experience sharing; The server reflects the received difference data in session information stored therein, transmits the difference data to another experience-sharing application, accumulates the difference data for a predetermined time, and then transmits the difference data to another experience-sharing application. If the difference data already accumulated contains data identical to the newly received difference data, the server deletes the identical data from the accumulated difference data. An experience sharing system featuring:

2. The server stores at least party information for authenticating users' participation in the party.

2. The experience sharing system according to claim 1, wherein:

3. The server manages the site where the experience is shared within the session via a URI.

3. The experience sharing system according to claim 1 or 2, wherein:

4. An experience sharing method performed between one or more user terminals on which an experience sharing app is installed and a server connected to the user terminals via a communication network, In a session including one or more users constituting a party, experiences can be shared on the site via an experience sharing app on the user terminal, The server and the experience sharing application in the user terminal each store session information for managing the session, and when a change occurs in the session during the experience sharing, the change is reflected in each of the session information; The steps include: transmitting, to the server, difference data of the session information corresponding to a change that has occurred during the experience sharing, by any one of the experience sharing applications; The server reflects the received difference data in the session information stored therein, transmits the difference data to another experience-sharing application, accumulates the difference data for a predetermined time, and then transmits the difference data to another experience-sharing application. If the difference data already accumulated contains data identical to the newly received difference data, the identical data is deleted from the accumulated difference data. An experience sharing method characterized by:

Images