Remote experience system, information processing device, information processing method, and program

Abstract

A remote experience system (100) according to the present disclosure comprises: equipment (1) that has sensors (14-1, 14-2); sensors (3-1 to 3-3) that are placed in a real environment; participant-action detecting devices (5-1, 5-2) that detect the action of a first user participating in an event from a place that is away from the real environment; representation devices (6-1, 6-2) that can provide representations to the first user; representation devices (6-1, 6-2) that can present images to a second user; and an information processing device (2). The information processing device (2) comprises: an equipment control unit (22) that generates equipment control information using action information received from the participant-action detecting devices (5-1, 5-2); and a representation information generating unit (25) that uses first detection information received from the sensors (14-1, 14-2) to generate first representation information to be transmitted to the first user, and that uses second detection information received from the sensors (3-1 to 3-3) to generate second representation information including an image corresponding to a virtual view point of the second user in the real environment.

Description

[Technical field]

[0001] The present disclosure relates to a remote experience system, an information processing device, an information processing method, and a program that provide a user with information about a remote location. [Background technology]

[0002] Systems that allow users to participate in events at remote locations, such as distance education, remote conferences, etc., are being increasingly introduced. For example, there is known a technology in which a teacher is present at an educational site and transmits information in one direction to a user (a student) located at a remote location from the educational site.

[0003] Also known is a technology that generates a virtual space and displays virtual participants representing the participants together with the virtual space to simulate a face-to-face conversation between distant participants. For example, Patent Document 1 discloses a technology related to a remote conference in which in-house workers who are in the office and remote workers who work remotely are mixed. In the technology described in Patent Document 1, a virtual room is displayed on the terminals of the in-house workers and remote workers, and the in-house workers and remote workers are displayed as virtual participants in the virtual room, and the movements of the in-house workers and remote workers are reflected in the movements of the virtual workers, thereby enhancing the sense of realism. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 6888854 Summary of the Invention [Problem to be solved by the invention]

[0005] On the other hand, depending on the content of the event, it may be desirable to experience various conditions in a real environment rather than a virtual space. The above-mentioned Patent Document 1 discloses the participation of virtual participants in a virtual space, but does not disclose a technology for allowing remote participants in an event using a real environment to experience the real environment.

[0006] The present disclosure has been made in consideration of the above, and aims to provide a remote experience system that enables remote participants in an event using a real environment to experience the real environment. [Means for solving the problem]

[0007] In order to solve the above problems and achieve the object, the remote experience system according to the present disclosure includes: The remote experience system allows multiple remote participants to experience an event using a real environment in which real participants are also participating, the system comprising: a real avatar indicating a first remote participant among the multiple remote participants and disposed in the real environment; a detection device for detecting the real avatar and the real participant; a virtual space generation information processing device for generating a virtual space indicating the real environment including the real avatar detected by the detection device and the real participant; a virtual avatar-reflecting virtual space image generating means for generating a virtual avatar-reflecting virtual space image indicating a second remote participant among the multiple remote participants but reflecting a virtual avatar that does not exist in the real environment in the virtual space; and a providing means for providing the virtual avatar-reflecting virtual space image generated by the virtual avatar-reflecting virtual space image generating means to the real participant, the first remote participant, and the second remote participant; Equipped with. Effect of the Invention

[0008] The remote experience system according to the present disclosure has the effect of enabling remote participants in an event that uses a real environment to experience the real environment. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing a configuration example of a remote experience system according to an embodiment; [Diagram 2] FIG. 1 is a diagram showing a schematic diagram of remote learning using a remote experience system according to an embodiment of the present invention; [Diagram 3] A flowchart showing an example of a control procedure of a device in an information processing apparatus according to an embodiment. [Figure 4] A flowchart showing an example of a procedure for generating virtual information related to a virtual avatar in an information processing device according to an embodiment. [Diagram 5] A flowchart showing an example of a procedure for generating expression information in the information processing device according to the embodiment. [Figure 6] FIG. 1 shows an example of a real participant, a real avatar user, and a virtual avatar user participating in an event. [Figure 7] FIG. 1 is a diagram showing an example in which a real avatar user and a virtual avatar user participate in an event. [Figure 8] A diagram showing an example of generating a virtual space corresponding to a virtual avatar. [Figure 9] A diagram showing an example of an image synthesized with a virtual space. [Figure 10] Diagram showing reduction of sensors in equipment [Figure 11] FIG. 1 is a diagram showing an example of the configuration of a computer system that realizes an information processing device according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A remote experience system, an information processing device, an information processing method, and a program according to embodiments will be described in detail below with reference to the accompanying drawings.

[0011] Embodiment 1 is a diagram showing a configuration example of a remote experience system according to an embodiment. The remote experience system 100 according to the present embodiment is used for an event using a real environment 10. The real environment 10 is, for example, a place where the event is held. The event may be, for example, remote education, remote experience, remote skill transfer, remote maintenance, remote training, experiential travel, healing, astronomy observation, on-site games, on-site golf, local sightseeing, boat trips, car trips, exploration, and the like, but is not limited to these.

[0012] The remote experience system 100 of this embodiment can provide three methods of participating in an event: a participation method in which a user participates as a real participant who actually exists in the real environment 10; a participation method in which a user participates using the device 1 as a real avatar that is an avatar of the user; and a participation method in which a user uses a virtual avatar that is an avatar that has no physical presence in the real environment 10. Both a user who participates using a real avatar and a user who participates using a virtual avatar are virtual participants who do not exist in the real environment 10, and are remote participants.

[0013] When a real avatar is used, information obtained by the device 1 from the real environment 10 can be transmitted to the user by the operation of the device 1, such as the device 1 moving in the real environment 10 or touching surrounding objects. On the other hand, when a real avatar is used, it is necessary to prepare the devices 1 according to the number of participating users, and depending on the number of participants in the event, costs may be required. For this reason, the remote experience system 100 of this embodiment allows participation as a virtual avatar, so that the number of participants in the event can be increased while keeping costs down. Details of the real avatar and the virtual avatar will be described later. Below, an example in which real participants, real avatars, and virtual avatars can coexist will be mainly described, but the remote experience system 100 may be constructed so that only real participants and real avatars, or only real avatars, can participate.

[0014] As shown in FIG. 1, the remote experience system 100 includes equipment 1, sensors 3-1 to 3-3, and presentation devices 4-1 and 4-2 installed in a real environment 10, participant motion detection devices 5-1 and 5-2 and presentation devices 6-1 and 6-2 installed in a participation location 7, which is a location where users participating in the event are present, and an information processing device 2.

[0015] Participant movement detection devices 5-1, 5-2 installed at the participation location 7 detect the movements of users participating in the event and transmit the detected information as movement information to the information processing device 2. Participant movement detection devices 5-1, 5-2 are set at the participation location 7 where at least a user (first user) participating using a real avatar is present and detect the movement of that user. Participant movement detection devices 5-1, 5-2 may also be set at the participation location 7 where a user (second user) participating using a virtual avatar is present, as described below, and detect the movement of that user.

[0016] The participant motion detection devices 5-1 and 5-2 are, for example, a positioning sensor, a camera, a microphone, etc. The presentation devices 6-1 and 6-2 receive presentation information indicating the state of the real environment 10 from the information processing device 2, and convey the state of the real environment 10 to the user based on the presentation information. The presentation devices 6-1 and 6-2 are devices capable of presentation that the user can recognize with at least one of the five senses, and can present, for example, an image to the user. The presentation devices 6-1 and 6-2 are, for example, a display, a speaker, a haptic groove, etc. The displays used as the presentation devices 6-1 and 6-2 may be head-mounted displays, VR (Virtual Reality) goggles, etc., or may be terminals such as smartphones and personal computers, or may be cockpit displays that allow the user to move the display together with the user by boarding the display. The presentation devices 6-1 and 6-2 may also be, for example, hologram displays, aerial projection displays, etc. Moreover, the expression devices 6-1 and 6-2 may include an olfactory expression device, a blower, an air conditioner, and the like.

[0017] Hereinafter, when the participant motion detection devices 5-1 and 5-2 are shown without distinction, they will be referred to as participant motion detection devices 5, and when the expression devices 6-1 and 6-2 are shown without distinction, they will be referred to as expression devices 6. In FIG. 1, two participant motion detection devices 5 and two expression devices 6 are shown, but the number of each of the participant motion detection devices 5 and the expression devices 6 is not limited to the example shown in FIG. 1. In addition, in FIG. 1, one participation location 7 is shown, but when users participate from multiple different locations, a similar device is provided at each participation location 7. In addition, when multiple users participate in an event from the same participation location 7, the participant motion detection device 5 and the expression device 6 are basically provided for each user, but some may be shared. For example, when a hologram display, an aerial projection display, or the like is used as the expression device 6, the expression device 6 may transmit information to multiple users. In addition, the participant motion detection device 5 may detect the motions of multiple users.

[0018] The sensors 3-1 to 3-3 are real environment sensors that are installed in the real environment 10 and detect information about objects in the real environment 10. The objects include, for example, at least one of animals, plants, and air that exist in the real environment 10. The sensors 3-1 to 3-3 are sensors that detect the environment in the real environment 10, and are, for example, sensors that detect objects, sounds, temperature, wind speed, wind volume, humidity, illuminance, odors, and types of gas in the real environment 10. That is, the sensors 3-1 to 3-3 detect, for example, at least one of temperature, humidity, wind, and odor in the real environment 10. In addition, the detection of objects may be performed, for example, by capturing images as images, or by collecting sounds. That is, the sensors 3-1 to 3-3 may include a capturing device such as a camera or may include a microphone. Detection information indicating the detection results is transmitted to the information processing device 2. Hereinafter, when the sensors 3-1 to 3-3 are referred to without being individually distinguished, they will be referred to as sensor 3.

[0019] The device 1 is an avatar that represents the user participating in the event, and more specifically, is a real avatar that exists as an object. The device 1 may be, for example, a humanoid robot, a mobile robot having a manipulator and a head that mimics a human face, or a manipulator. As will be described later, the device 1 operates based on control information received from the information processing device 2. The device 1 includes a control information receiving unit 11, a drive control unit 12, a drive unit 13, sensors 14-1 and 14-2, and a detection information transmitting unit 15.

[0020] The control information receiving unit 11 receives control information from the information processing device 2 and outputs the received control information to the drive control unit 12. The drive control unit 12 controls the drive unit 13 based on the control information. The drive unit 13 includes one or more actuators. The actuators are, for example, physical actuators, moving units, speakers, etc. that operate each joint in the device 1, such as a manipulator.

[0021] The sensors 14-1 and 14-2 are sensors mounted on the device and detect information about objects in the real environment 10. The sensors 14-1 and 14-2 include a camera or other imaging device, a speaker, and the like, as with the sensor 3, but may detect at least one of temperature, humidity, wind, and odor in the real environment 10. The sensors 14-1 and 14-2 may also detect force and touch. The sensors 14-1 and 14-2 output detection information indicating the detection result to the detection information transmission unit 15. The detection information transmission unit 15 transmits the detection information to the information processing device 2. The sensors 14-1 and 14-2 are mounted on the device 1, so that they can detect information according to the location of the device 1. Hereinafter, the sensors 14-1 and 14-2 will be referred to as sensor 14 when not individually distinguishing them. For example, if the device 1 is provided with a part simulating a face and a camera is provided as the sensor 14 in a part corresponding to the eyes, the sensor 14 can acquire an image that depends on the movement of the device 1 and the direction of the face. Although two sensors 14 are shown in FIG. 1, the number of sensors 14 is not limited to the example shown in FIG.

[0022] For example, the device 1 may include a manipulator, and the sensor 14 may include a force sensor that detects the force of the hand of the manipulator. If the representation device 6 corresponding to the user using the real avatar includes a force sensor device that transmits the force to the hand of the user using the real avatar, the force detected by the device 1 can be transmitted to the user using the real avatar, as described below.

[0023] The presentation devices 4-1 and 4-2 are devices that perform presentations to communicate the virtual environment superimposed on the real environment 10 to users participating in an event in the real environment 10. The presentation devices 4-1 and 4-2 are, for example, displays, speakers, etc. The displays used as the presentation devices 4-1 and 4-2 are similar to the displays used as the presentation device 6 described above, but only need to be able to present the portion that becomes the virtual environment, as described below. Hereinafter, when the presentation devices 4-1 and 4-2 are referred to without being individually distinguished, they will be referred to as presentation device 4.

[0024] As described later, depending on the configuration of the device 1 and the participation method of the user participating in the event, the representation device 4 may not be provided in the real environment 10. Depending on the configuration of the device 1 and the participation method of the user participating in the event, the sensor 3 may not be provided in the real environment 10. Although three sensors 3 and two representation devices 4 are illustrated in FIG. 1, the number of sensors 3 and the number of representation devices 4 are not limited to the example illustrated in FIG. 1. Although one device 1 is illustrated in FIG. 1, the number of devices 1 is also not limited to the example illustrated in FIG. 1.

[0025] The information processing device 2 includes a control information transmitting unit 21, a device control unit 22, an operation information receiving unit 23, a detection information receiving unit 24, an expression information generating unit 25, and an expression information transmitting unit .

[0026] The motion information receiving unit 23 receives motion information indicating the motions acquired by the participant motion detection device 5 from the participant motion detection device 5 at the participation location 7, and outputs the received motion information to the equipment control unit 22. Identification information capable of identifying the corresponding user is added to the motion information. This identification information may be user identification information determined when the user registers for participation in the event, or may be identification information of the participant motion detection device 5. When the identification information added to the motion information is the identification information of the participant motion detection device 5, it is assumed that the identification information of the participant motion detection device 5 and the user identification information are associated with each other when the user registers for participation in the event.

[0027] The device control unit 22 uses the operation information to generate control information for making the device 1 perform an operation corresponding to the operation information. That is, the device control unit 22 uses the operation information to generate control information for controlling the device 1 so that the device 1, which is the real avatar of the user, performs an operation corresponding to the user's operation, and outputs the generated control information to the control information transmission unit 21. The control information is, for example, information indicating how to drive each actuator of the device 1. Note that the device control unit 22 holds correspondence information indicating the correspondence between the device 1 and the user who uses the device 1 as a real avatar, and identifies the device 1 corresponding to the received operation information using the correspondence information and the identification information added to the operation information. The correspondence information may be determined, for example, when the user registers to participate in an event, or may be selected by the user at the start of the event. The function and position of each device 1 may be confirmed, and the user may select the device 1 to be the real avatar, and notify the information processing device 2 of the selected device 1 via the user's terminal (not shown). The control information transmission unit 21 transmits the control information received from the device control unit 22 to the device 1.

[0028] The detection information receiving unit 24 receives detection information from at least one of the sensor 14 and the sensor 3, and outputs the received detection information to the expression information generating unit 25. The expression information generating unit 25 uses the detection information to generate expression information indicating the content to be expressed by the expression device 6, and outputs the generated expression information to the expression information transmitting unit 26, which transmits the expression information to the expression device 6. Note that, similar to the correspondence between the device 1 and the user described above, the expression information generating unit 25 holds correspondence information indicating the correspondence between the expression device 6 and the user. In addition, the expression information generating unit 25 generates expression information to be transmitted to a user participating in the real environment 10, and outputs the generated expression information to the expression information transmitting unit 26, which transmits the expression information to the expression device 4.

[0029] The communication line between the information processing device 2 and each device may be a wireless line, a wired line, or a combination of a wireless line and a wired line. Any communication method may be used for the communication between the information processing device 2 and each device. For example, by performing communication according to Beyond 5G (5th Generation: 5th generation mobile communication system) that realizes large-capacity and low-latency transmission, information can be transmitted to the user with low latency, and the sense of realism can be enhanced. That is, at least one of the communication line between the information processing device 2 and the participant motion detection device 5, the communication line between the information processing device 2 and the sensor 14, the communication line between the information processing device 2 and the sensor 3, the communication line between the information processing device 2 and the device 1, the communication line between the information processing device 2 and the expression device 6, and the communication line between the information processing device 2 and the expression device 4 may be a Beyond 5G line.

[0030] Next, an application example of the remote experience system 100 of this embodiment will be described. FIG. 2 is a diagram that shows a model of remote education using the remote experience system 100 of this embodiment. In the example shown in FIG. 2, for example, real participants 8-1 to 8-4 actually exist in a forest that is a real environment 10. Also, in the forest that is the real environment 10, there exists a device 1 used as a real avatar. Furthermore, there exists a user who participates in the real environment 10 as a virtual avatar 9 that has no physical body. The real participant 8-1 is, for example, a host such as a teacher, and the real participants 8-2 to 8-4, the user who participates as the virtual avatar 9, and the user who participates using the device 1 as a real avatar are students. Hereinafter, when the real participants 8-1 to 8-4 are referred to without being individually distinguished, they will be described as a real participant 8.

[0031] In the example shown in FIG. 2, the real environment 10 is a forest, and therefore the presence of birds, animals, plants, etc. produces smells, air flows, and changes in sound caused by the movement of birds and animals. In addition, by touching the animals and plants present in the forest, participants can have experiences unique to the real environment 10. By conveying various conditions to the user in addition to images and sounds, for example, it is possible to promote understanding of the user's Umwelt. In this embodiment, a more realistic experience of the real environment 10 is provided even to users who participate in distance education remotely.

[0032] The motion of a user (hereinafter, also referred to as a real avatar user) participating using the device 1 as a real avatar is detected by a participant motion detection device 5 at a participation location 7 shown in FIG. 1, and motion information indicating the motion is transmitted to an information processing device 2. The information processing device 2 controls the motion of the device 1 using the motion information, so that the device 1 performs a motion according to the motion of the real avatar user. In addition, the information processing device 2 causes the representation device 6 to represent information indicating the state of the real environment 10, so that the real avatar user can recognize, for example, plants, animals, and the like in the forest that is the real environment 10 by images, sounds, and the like. Therefore, when the real avatar user performs a motion to touch a plant in the real environment 10, the device 1 that is the real avatar can touch the plant in the real environment 10. The haptic sensation caused by touching the plant is detected by a sensor 14 in the device 1 and transmitted to a representation device 6 such as a haptic glove via the information processing device 2. This allows the representation device 6 to transmit the haptic sensation to the user.

[0033] Furthermore, by using a sensor for detecting air conditions such as temperature, wind speed, air volume, humidity, odor, and type of gas as the sensor 3, the real avatar user can experience not only images and sounds, but also the air conditions in the real environment 10. In this manner, in the present embodiment, the user can participate using the device 1 present in the real environment 10 as a real avatar, so that remote participants in an event using the real environment 10 can more appropriately experience the real environment 10, and interaction with the real environment can be realized. By fusing the real environment 10 with a virtual object, the real avatar user can experience the real environment 10 with a sense of realism.

[0034] In addition, by using an odor reproduction device, an air conditioner, a blower, or the like as the expression device 6 that expresses the air state, the air state may be expressed as something similar to the detection information, but the air state may be converted into at least one of visual information and auditory information. Similarly, the detection information of the haptic sensation may be converted into at least one of visual information and auditory information.

[0035] Furthermore, with regard to device 1, which is a real avatar, the device 1 may be transmitted as an image directly to the real avatar user and the user participating using a virtual avatar (hereinafter also referred to as a virtual avatar user), and the real participants may directly view device 1, or the information processing device 2 may generate an image simulating the real avatar user and replace the device 1 portion with that image.

[0036] Although the virtual avatar does not have a physical presence in the real environment 10, the position of the virtual avatar in the real environment 10 is specified, and information simulating the presence of the virtual avatar user at the specified position is provided to the user. For example, by using a 360-degree free viewpoint camera, a microphone for realizing 360-degree stereophonic sound, or the like as the sensor 3, it is possible to provide the virtual avatar user with images and sounds corresponding to the position and face direction of the virtual avatar. The position and face direction of the virtual avatar may be determined by the information processing device 2 based on the motion information detected by the participant motion detection device 5, as in the case of a real avatar user, or may be specified by the virtual avatar user using a terminal not shown. In addition, when a sensor for detecting the state of air, such as temperature, wind speed, air volume, humidity, odor, type of gas, or the like, is used as the sensor 3, the state of air in the real environment 10 is transmitted to the virtual avatar user, as in the case of a real avatar user, so that the virtual avatar user can also experience the state of air in the real environment 10.

[0037] Although the virtual avatar cannot interact with real objects in the real environment 10, such as by touching or moving the real objects, the information processing device 2 may generate a virtual space around the virtual avatar based on an image of the real environment 10, and allow the virtual avatar to perform actions such as touching or moving the object in the virtual space. In this case, the type of object may be determined from the image of the real environment 10, and the determination result may be used to generate information indicating the haptic sensation obtained by touching the object, and the information may be transmitted to the representation device 6 to convey the sensation of touching the object to the virtual avatar user. The initial values ​​of the position and orientation of the virtual avatar in the real environment 10 may be specified by the virtual avatar user or may be determined in advance.

[0038] Furthermore, the information processing device 2 may generate an image simulating the virtual avatar user or a three-dimensional image of an arbitrary shape based on an image of the virtual avatar user, and may transmit the image to the representation device 6 and the representation device 4 as representation information by superimposing the image on an image of the real environment 10. The virtual avatar may also be provided to the real avatar user and the real participant as an image, or the virtual avatar user may be able to select whether or not to display the virtual avatar. Similarly, for the real avatar, an image simulating the real avatar user or a three-dimensional image of an arbitrary shape may be generated and displayed at a location corresponding to the device 1, and the real avatar user may be able to select whether or not to display the image.

[0039] The above-mentioned virtual space generated for the virtual avatar user may be reflected in the representation information transmitted to the real avatar user so that the real avatar user can share information with the virtual avatar user. Also, the virtual space may be represented by the representation device 4 so that the virtual space is superimposed on the real environment 10 that the real participants directly view so that the real participants can share information with the virtual avatar users.

[0040] Next, the operation of the information processing device 2 of this embodiment will be described. Fig. 3 is a flowchart showing an example of a control procedure of the device 1 in the information processing device 2 of this embodiment. The information processing device 2 acquires motion information (step S1). In detail, the motion information receiving unit 23 acquires motion information by receiving it from the participant motion detection device 5 that detects the motion of a real avatar user, and outputs the acquired motion information to the device control unit 22.

[0041] Next, the information processing device 2 generates control information for reflecting the motion information in the motion of the real avatar (step S2). In detail, the information processing device 2 generates control information for making the device 1, which is the real avatar of the real avatar user corresponding to the motion information, perform the motion corresponding to the motion information, using the motion information, and outputs the generated control information to the control information transmission unit 21.

[0042] Next, the information processing device 2 transmits the control information (step S3). In detail, the control information transmitting unit 21 transmits the control information to the device 1. By performing the above-mentioned process, for example, for each control period, the movement of the real avatar user is reflected in the device 1, which is the real avatar corresponding to the real avatar user.

[0043] 4 is a flowchart showing an example of a procedure for generating virtual information related to a virtual avatar in the information processing device 2 according to the present embodiment. The information processing device 2 performs step S1 in the same manner as in the example shown in FIG. 3. However, the motion information is motion information related to a virtual avatar user, and the motion information receiving unit 23 outputs the motion information to the expression information generating unit 25.

[0044] Next, the information processing device 2 generates virtual information corresponding to the virtual avatar reflecting the motion information (step S4). In detail, the expression information generating unit 25 uses the motion information to generate virtual information, which is expression information such as images and sounds corresponding to a virtual space including a virtual avatar of a virtual avatar user corresponding to the motion information. The above-mentioned process is performed, for example, for each control period, to generate virtual information for expressing a virtual space reflecting the motion of the virtual avatar user. Note that the generation of the virtual avatar and the generation of information related to the virtual space can be generated using technologies such as a space reconstruction method, MR (Mixed Reality), and AR (Augmented Reality), but the generation of the virtual avatar and the generation of information related to the virtual space are not limited to these examples.

[0045] 5 is a flowchart showing an example of a procedure for generating expression information in the information processing device 2 according to the present embodiment. The information processing device 2 acquires detection information by a sensor (step S11). In detail, the detection information receiving unit 24 acquires the detection information by receiving the detection information from at least one of the sensor 14 and the sensor 3, and outputs the acquired detection information to the expression information generating unit 25.

[0046] Next, the information processing device 2 generates first synthesis information by synthesizing virtual information with detection information to be synthesized among the detection information (step S12). In detail, the expression information generating unit 25 generates the first synthesis information by synthesizing the detection information to be synthesized among the detection information with the virtual information generated in step S4 shown in FIG. 4. The detection information to be synthesized is detection information to be synthesized with the virtual information. For example, if the virtual information is an image, the detection information to be synthesized is an image, and if the virtual information is an image and sound, the detection information to be synthesized is an image and sound. Note that here, an example is described in which the image and sound are acquired by the sensor 14 mounted on the device 1, but if the image and sound are acquired by the sensor 3, such as a 360-degree free viewpoint camera, the expression information generating unit 25 uses the detection information to generate information indicating an image and sound corresponding to a virtual avatar or a real avatar, and synthesizes the virtual information with the generated information. If there are multiple remote participants, the virtual information is synthesized with the corresponding detection information or the information generated using the detection information of the sensor 3 for each remote participant. Note that synthesis of the virtual information does not have to be performed.

[0047] Next, the information processing device 2 converts the detection information to be converted from among the detection information into another type of information, and generates second composite information by synthesizing the converted information with the first composite information (step S13). In detail, the expression information generating unit 25 converts the detection information to be converted from among the detection information into another type of information, synthesizes the converted information with the first composite information, and outputs the second composite information to the expression information transmitting unit 26 as the expression information. For example, when converting the detection information of the state of air, such as temperature and humidity, into visual information, the detection information of the state of air becomes the detection information to be converted, and the other type of information is a character or an image. For example, this conversion can obtain a conversion result by learning in advance the contents of the other type of information that allows a person to feel a similar state according to the value of the detection information to be converted, for example, by supervised machine learning using clinical trial results, and inputting the detection information to be converted into the learned model. The conversion method is not limited to this example. Examples of expressing the state of air with visual information, audio information, etc. include, but are not limited to, generating audio information that exaggerates the sound of birds or insects moving, displaying a mirage-like image shaking to represent humidity, and displaying a clear edge to represent coldness. Similarly, information about taste may be converted into visual information, audio information, etc. If there is no detection information to be converted, step S13 is not performed, and the first synthesis information is output to the expression information transmission unit 26 as expression information. If there are multiple remote participants, steps S12 and S13 are performed for each remote participant.

[0048] Next, the information processing device 2 transmits the second synthesis information and the detection information that is not a target for synthesis and conversion to the representation devices 6 of the remote participants (step S14). In detail, the representation information generator 25 transmits the second information and the detection information that is not a target for synthesis and conversion to the representation devices 6 of the remote participants, i.e., the representation devices 6 of the virtual avatar users and the real avatar users.

[0049] Next, the information processing device 2 transmits the virtual information to the representation device 4 in the real environment 10 (step S15). In detail, the representation information generating unit 25 transmits the virtual information to the representation device 4 in the real environment 10, that is, the representation device 4 that transmits information to the real participant. The information converted in step S13 and the virtual information may be combined and transmitted to the representation device 4 of the real participant. Since the real participant can directly feel the state of the air in the real environment 10, the information converted in step S13 does not need to be transmitted to the representation device 4. However, the converted information is also transmitted to the real participant, so that the real participant can share information with the virtual avatar user and the real avatar user. The above-mentioned process is performed, for example, every control period, whereby information indicating the state of the real environment 10 is transmitted to the remote participant and virtual information is transmitted to the real participant.

[0050] In addition, assuming that the representation device 6 corresponding to the real avatar user (first user) is the first representation device and the representation device 6 corresponding to the virtual avatar user (second user) is the second representation device, the first representation device only needs to be capable of representation that the first user can recognize with at least one of the five senses, and the second representation device only needs to be capable of presenting at least an image to the second user. In addition, the sensor 3 includes an imaging device that detects information about objects in the real environment 10 by imaging at least the real environment 10 in order to generate an image to be provided to the virtual avatar user. The representation information generating unit 25 generates first representation information to be transmitted to the first user (real avatar user) using the first detection information, which is the detection information received from the sensor 14, and generates second representation information including an image corresponding to a virtual viewpoint in the real environment 10 of the second user (virtual avatar user) using the second detection information received from the sensor 3. The representation information transmitting unit 26 transmits the first representation information to the first representation device and transmits the second representation information to the second representation device.

[0051] Also, as described above, the expression information generating unit 25 may convert the first detection information indicating at least one of the detection results of temperature, humidity, wind, and odor in the real environment 10 into first expression information that can be recognized by at least one of vision and hearing. Also, the expression information generating unit 25 may convert the second detection information indicating at least one of the detection results of temperature, humidity, wind, and odor in the real environment 10 into second expression information that can be recognized by at least one of vision and hearing. Also, the sensor may include information on taste, and the expression information generating unit 25 may generate information on taste in the real environment 10. In this case, the expression device 4 and the expression device 6 include a taste expression device that reproduces taste. Alternatively, the information on taste may be converted into other types of methods such as visual and auditory.

[0052] Furthermore, the expression information generator 25 may generate the first expression information by using the second detection information. That is, the expression information to be transmitted to the real avatar user may be generated by using the information acquired by the sensor 3.

[0053] Alternatively, the expression information generating unit 25 may generate the second expression information so that the virtual avatar is displayed at the virtual viewpoint, the participant movement detecting device 5 detects the movement of the virtual avatar user, the expression information generating unit 25 virtually generates an image in which the object of the real environment 10 and the virtual avatar change according to the movement of the virtual avatar user in the virtual space that is the range including the virtual viewpoint based on the movement information of the virtual avatar user and the second detection information (image data of the real environment 10), synthesizes the virtual information indicating the generated image with the second expression information, and the expression information transmitting unit 26 transmits the second expression information with the virtual information synthesized to the second expression device. Alternatively, the expression information generating unit 25 may synthesize the virtual information with the first expression information, and the expression information transmitting unit 26 may transmit the first expression information with the virtual information synthesized to the first expression device. Note that the virtual avatar may have only a virtual viewpoint, or may be capable of hearing and speaking in addition to the virtual viewpoint.

[0054] Furthermore, the expression information transmitting unit 26 may transmit the virtual information to the expression device 4, which is a third expression device capable of presenting an image to a real participant (third user). Furthermore, the information processing device 2 may be capable of setting whether information regarding the virtual avatar user and the real avatar user is to be made public or private. The information regarding the virtual avatar user and the real avatar user may be, but is not limited to, at least a part of information that is frequently used for security purposes, such as face information, eye information, and voiceprint, and personal information, such as real name, age, and date of birth. Furthermore, if a user does not wish to make his or her actual face or voice public for privacy reasons, the face or voice may be changed and made public.

[0055] Next, an example of a mode of participation of participants will be described. FIG. 6 is a diagram showing an example in which real participants, real avatar users, and virtual avatar users participate in an event. In the example shown in FIG. 6, real participants 8-1 to 8-3 participate in the event in a real environment 10, and from the participation location 7-1, a user 70-1 who is a real avatar user participates in the event using a device 1 that is a real avatar, and from the participation locations 7-2 and 7-3, users 70-2 and 70-3 who are virtual avatar users participate in the event using virtual avatars 9-1 and 9-2, respectively. Although not shown in the figure, a participant motion detection device 5 and a representation device 6 are provided at each of the participation locations 7-1 to 7-3. Note that, when the virtual avatars 9-1 and 9-2 are fixed without being moved, the participant motion detection device 5 does not need to be provided at the users 70-2 and 70-3 who are virtual avatar users.

[0056] The device 1 includes sensors 14 for detecting information related to each of the senses of sight, hearing, smell, and touch, and includes a physical effector, a moving unit, and a speaker as actuators. The sensors 3-1 and 3-2 installed in the real environment 10 are cameras for acquiring images for generating images from the viewpoints of the virtual avatars 9-1 and 9-2, and microphones for collecting sounds for generating sounds at the positions of the virtual avatars 9-1 and 9-2. Although two sensors 3 are illustrated in FIG. 6 for the sake of simplicity, there are generally multiple cameras for acquiring images for generating images from the viewpoints of the virtual avatars 9-1 and 9-2, and multiple microphones for collecting sounds for generating sounds at the positions of the virtual avatars 9-1 and 9-2. At least one of the sensors 3 and 14 may further include a sensor for detecting odors, temperature, wind speed, wind volume, humidity, and types of gas in the real environment 10.

[0057] In the example shown in FIG. 6, all information to be transmitted to the user 70-1, who is a real avatar user, is detected by the sensor 14 mounted on the device 1. As described above, the device 1 operates in response to the motion of the real avatar user, and, for example, the field of view of the sensor 14, such as a camera that obtains visual information in the device 1, is changed, and an image in the real environment 10 corresponding to the motion of the real avatar user is displayed by the representation device 6. The same is true for sounds, smells, etc., and information corresponding to sounds and smells in the real environment 10 is transmitted to the real avatar user as the device 1 moves. In addition, the device 1 touches an actual object in the real environment 10 in response to the motion of the real avatar user, and information related to haptics is detected by the sensor 14 and transmitted to the real avatar user. This allows the real avatar user to recognize the real environment 10 with a sense of realism.

[0058] Moreover, images from the viewpoints of the virtual avatars 9-1 and 9-2, sounds from the virtual avatars 9-1 and 9-2, etc. are transmitted to the users 70-2 and 70-3, who are virtual avatar users, using the detection information acquired by the sensor 3. Furthermore, if a sensor that detects odors is provided as the sensor 3, the information processing device 2 can transmit information indicating odors to the users 70-2 and 70-3, who are virtual avatar users, via the expression device 6, using the detection information obtained by the sensor 3.

[0059] In the case where the real participants 8-1 to 8-3 can view the device 1 as it is and do not need to view the virtual avatars 9-1 and 9-2, the rendering device 4 does not need to be provided in the real environment 10. As described above, in the case where an image corresponding to a real avatar is projected at the location where the device 1 is present and an image corresponding to a location corresponding to the virtual avatars 9-1 and 9-2 is projected, the rendering device 4 for projecting these is provided. Similarly, in the case where the real participants 8-1 to 8-3 can recognize sounds corresponding to the virtual avatars 9-1 and 9-2, the sounds generated by the users 70-2 and 70-3 who are virtual avatar users are transmitted to the real participants by the rendering device 4 via the rendering device 4.

[0060] Fig. 7 is a diagram showing an example in which real avatar users and virtual avatar users participate in an event. In the example shown in Fig. 7, real avatar users 70-1, 70-4, and 70-5 participate in the event from participation locations 7-1, 7-4, and 7-5 using devices 1-1 to 1-3 that are their corresponding real avatars, and virtual avatar users 70-2 and 70-3 participate in the event from participation locations 7-2 and 7-3 using virtual avatars 9-1 and 9-2, respectively. Although not shown in the figure, a participant motion detection device 5 and a representation device 6 are provided at each of the participation locations 7-1 to 7-5. Each of the devices 1-1 to 1-3 is the device 1 described above.

[0061] The configuration of each of the devices 1-1 to 1-3 is similar to that of the device 1 shown in Fig. 6. The method of transmitting information to the real avatar user and the virtual avatar user in the example shown in Fig. 7 is also similar to the example shown in Fig. 6. In the example shown in Fig. 7, since there is no real participant 8, the representation device 4 does not need to be provided in the real environment 10.

[0062] Fig. 8 is a diagram showing an example of generating a virtual space corresponding to a virtual avatar. In the example shown in Fig. 8, similar to the example shown in Fig. 6, real participants 8-1 to 8-3 participate in an event in a real environment 10, a user 70-1 who is a real avatar user participates in the event using a device 1 which is a real avatar from a participation location 7-1, and users 70-2 and 70-3 who are virtual avatar users participate in the event using virtual avatars 9-1 and 9-2 from participation locations 7-2 and 7-3, respectively. Although not shown in the figure, a participant motion detection device 5 and a representation device 6 are provided at each of the participation locations 7-1 to 7-3.

[0063] 8, the information processing device 2 acquires motion information from a participant motion detection device 5 corresponding to a user 70-3 who is a virtual avatar user, and generates a virtual space 90 around a virtual avatar 9-2 using the acquired information. Then, the information processing device 2 synthesizes the virtual information indicating the virtual space 90 with an image, sound, etc. from the viewpoint of the virtual avatar 9-2 generated using the detection information from the sensor 3, and transmits the synthesized information as expression information to the expression device 6 corresponding to the user 70-3 who is a virtual avatar user. The synthesized information is also transmitted to the user 70-1 who is a real avatar user and the user 70-2 who is a virtual avatar user.

[0064] FIG. 9 is a diagram showing an example of an image synthesized with a virtual space. In the example shown in FIG. 9, for example, an image 201 of a virtual space corresponding to a virtual avatar 9-2 is synthesized with an image 200 of a real environment 10 from the viewpoint of a user 70-2 who is a virtual avatar user. The image 200 is a real image based on information photographed by the sensor 3, while the image 201 is information showing a virtual space generated based on information photographed by the sensor 3. The image 201 reflects the movement of the virtual avatar 9-2 in the virtual space. Therefore, when the virtual avatar 9-2 touches an object in the virtual space, the object also changes in the virtual space. Similarly, the image synthesized with the virtual space is transmitted to a user 70-3 who is a virtual avatar user corresponding to the virtual avatar 9-2, so that the user 70-3 who is a virtual avatar user can experience the same sensation as when touching an object in the real environment 10.

[0065] In addition, by transmitting the virtual information to the representation devices 4 corresponding to the live participants 8-1 to 8-3, the live participants 8-1 to 8-3 can visually recognize the state in which the projected virtual space is superimposed on the actual real environment 10.

[0066] Fig. 10 is a diagram showing a reduction in sensors in devices. In the example shown in Fig. 10, similar to the example shown in Fig. 6, real participants 8-1 to 8-3 participate in an event in a real environment 10, a user 70-1 who is a real avatar user participates in the event using a device 1 that is a real avatar from a participation location 7-1, and users 70-2 and 70-3 who are virtual avatar users participate in the event using virtual avatars 9-1 and 9-2 from participation locations 7-2 and 7-3, respectively. Although not shown in the figure, a participant motion detection device 5 and a representation device 6 are provided at each of the participation locations 7-1 to 7-3.

[0067] In the example shown in FIG. 10, the sensors 3-1 to 3-4 include a camera for acquiring a 360-degree free viewpoint image and a microphone for reproducing a sound according to a position. The sensors 3-1 to 3-4 are used to generate information indicating an image of the real environment 10 from the viewpoint of the users 70-2 and 70-3 who are virtual avatar users and a sound at the positions of the users 70-2 and 70-3 who are virtual avatar users, and are also used to generate information indicating an image of the real environment 10 from the viewpoint of a real avatar and a sound at the position of the real avatar. As a result, the sensor 14 included in the device 1 does not need to include a sensor for detecting visual and auditory information. As shown in FIG. 10, by generating information about a real avatar using the sensor 3, the configuration of the device 1 can be simplified more than the example shown in FIG. 6. In the example shown in FIG. 10, visual and auditory information about a real avatar is generated from detection information of the sensor 3, but the present invention is not limited to this. One of the visual and auditory information may be generated from detection information of the sensor 3, and the other may be generated from detection information of the sensor 14. Furthermore, by using the sensor 3 that detects odors, the number of sensors that detect information related to the sense of smell among the sensors 14 included in the device 1 may be reduced.

[0068] Next, the hardware configuration of the information processing device 2 of this embodiment will be described. In the information processing device 2 of this embodiment, a computer program in which processing in the information processing device 2 is described is executed on the computer system, so that the computer system functions as the information processing device 2. FIG. 11 is a diagram showing an example of the configuration of a computer system that realizes the information processing device 2 of this embodiment. As shown in FIG. 11, this computer system includes a control unit 101, an input unit 102, a storage unit 103, a display unit 104, a communication unit 105, and an output unit 106, which are connected via a system bus 107. The control unit 101 and the storage unit 103 form a processing circuit.

[0069] In FIG. 11, the control unit 101 is a processor such as a CPU (Central Processing Unit), and executes a program in which the processing in the information processing device 2 of the present embodiment is described. Note that a part of the control unit 101 may be realized by dedicated hardware such as a GPU (Graphics Processing Unit) or an FPGA (Field-Programmable Gate Array). The input unit 102 is composed of, for example, a keyboard, a mouse, and the like, and is used by a user of the computer system to input various information. The storage unit 103 includes various memories such as a RAM (Random Access Memory) and a ROM (Read Only Memory), and a storage device such as a hard disk, and stores the program to be executed by the control unit 101, necessary data obtained in the process of processing, and the like. The storage unit 103 is also used as a temporary storage area for the program. The display unit 104 is composed of a display, an LCD (Liquid Crystal Display Panel), and the like, and displays various screens to the user of the computer system. The communication unit 105 is a receiver and a transmitter that perform communication processing. The output unit 106 is a printer, a speaker, etc. Note that Fig. 11 is just an example, and the configuration of the computer system is not limited to the example of Fig. 11.

[0070] Here, an example of the operation of the computer system until the program of this embodiment is in an executable state will be described. In the computer system having the above-mentioned configuration, for example, a computer program is installed in the storage unit 103 from a CD-ROM or DVD-ROM set in a CD (Compact Disc)-ROM drive or DVD (Digital Versatile Disc)-ROM drive (not shown). Then, when the program is executed, the program read from the storage unit 103 is stored in the main storage area of ​​the storage unit 103. In this state, the control unit 101 executes the process as the information processing device 2 of this embodiment according to the program stored in the storage unit 103.

[0071] In the above description, a program describing the processing in the information processing device 2 is provided using a CD-ROM or DVD-ROM as a recording medium. However, this is not limited to this, and depending on the configuration of the computer system, the capacity of the program to be provided, etc., it is also possible to use a program provided over a transmission medium such as the Internet via the communication unit 105.

[0072] The program of this embodiment causes a computer system to execute, for example, the steps of receiving motion information indicating the motion of a first user participating in an event at a location away from the real environment 10 in which the event is held, using the motion information to generate control information for causing an equipment installed in the real environment 10 to perform an operation according to the motion information, transmitting the control information to the equipment, receiving first detection information detected by an equipment-mounted sensor from an equipment-mounted sensor that is mounted on the equipment and detects information about objects in the real environment 10, and receiving second detection information detected by a real environment sensor from a real environment sensor installed in the real environment 10 that detects information about objects in the real environment 10. The program of this embodiment further causes the computer system to execute the following steps: generating first expression information to be transmitted to a first user using the first detection information; generating second expression information using the second detection information, the second expression information including an image corresponding to a virtual viewpoint in the real environment 10 of a second user participating in the event from a location remote from the real environment 10; and transmitting the first expression information to a first expression device capable of producing an expression that the first user can recognize with at least one of the five senses, and transmitting the second expression information to a second expression device capable of presenting an image to the second user.

[0073] The device control unit 22 and the expression information generation unit 25 shown in FIG. 1 are realized by executing a computer program stored in the storage unit 103 shown in FIG. 11 by the control unit 101 shown in FIG. 11. The device control unit 22 and the expression information generation unit 25 shown in FIG. 1 are realized by the storage unit 103 shown in FIG. 11. The control information transmission unit 21, the action information reception unit 23, the detection information reception unit 24, and the expression information transmission unit 26 shown in FIG. 1 are realized by the communication unit 105 shown in FIG. 11. The information processing device 2 may be realized by a plurality of computer systems. For example, the information processing device 2 may be realized by a cloud computer system. A part of the function of the information processing device 2 may be realized by another device provided separately from the information processing device 2. The other device may be provided in the real environment 10, may be provided near the residence of a user participating in the event, or may be provided in another place.

[0074] As described above, in this embodiment, in an event using the real environment 10, the physical device 1 is used as a real avatar to realize remote user participation in the event. This allows remote participants in the event using the real environment 10 to experience the real environment 10 more appropriately. In addition, by using both participation by a real avatar using the device 1 and participation by a virtual avatar, even if the number of participants in the event increases, it is possible to flexibly respond with reduced costs. In addition, by transmitting information indicating the air condition of the real environment 10 to the remote participants, the remote participants can experience a state closer to the real environment 10. In addition, by generating information corresponding to a real avatar using detection information from the sensor 3 used to provide information corresponding to a virtual avatar to a virtual avatar user, the configuration of the device 1 can be simplified and costs can be reduced.

[0075] The configurations shown in the above embodiments are merely examples, and may be combined with other known technologies, or the embodiments may be combined with each other. Also, parts of the configurations may be omitted or modified without departing from the spirit of the invention. [Explanation of symbols]

[0076] 1, 1-1 to 1-3 equipment, 2 information processing device, 3-1 to 3-4, 14-1, 14-2 sensor, 4-1, 4-2, 6-1, 6-2 expression device, 5-1, 5-2 participant motion detection device, 7, 7-1 to 7-5 participation location, 8-1 to 8-4 actual participant, 9, 9-1, 9-2 virtual avatar, 10 actual environment, 11 control information receiving unit, 12 drive control unit, 13 drive unit, 15 detection information transmitting unit, 21 control information transmitting unit, 22 equipment control unit, 23 motion information receiving unit, 24 detection information receiving unit, 25 expression information generating unit, 26 expression information transmitting unit, 100 remote experience system.

Claims

1. A remote experience system that allows multiple remote participants to experience an event using a real environment in which real participants are also participating, a real avatar representing a first remote participant of the plurality of remote participants and disposed in the real environment; a detection device for detecting the real avatar and the real participant; a virtual space generation information processing device that generates a virtual space representing the real environment including the real avatar detected by the detection device and the real participant; and a virtual avatar-reflected virtual space image generating means for generating a virtual avatar-reflected virtual space image in which a virtual avatar that represents a second remote participant among the plurality of remote participants but does not exist in the real environment is reflected in the virtual space; providing means for providing the virtual avatar-reflected virtual space image generated by the virtual avatar-reflected virtual space image generating means to the live participant, the first remote participant, and the second remote participant; A remote experience system comprising:

2. a device that is the real avatar and is installed in the real environment and has an on-device sensor that detects information about an object in the real environment; a real environment sensor that is installed in the real environment and detects information about an object in the real environment; a participant motion detector for detecting a motion of the first remote participant; a first expression device capable of expressing something that the first remote participant can perceive with at least one of the five senses; a second presentation device capable of presenting the virtual avatar-reflecting virtual space image to the second remote participant; Equipped with the real environment sensor includes an imaging device that detects information about an object in the real environment by imaging the real environment; The virtual space generation information processing device comprises: a motion information receiving unit that receives, from the participant motion detection device, motion information indicating the motion acquired by the participant motion detection device; a device control unit that generates control information for causing the device to perform an operation corresponding to the operation information, using the operation information; a control information transmitting unit that transmits the control information to the device; a detection information receiving unit that receives, from the device-mounted sensor, first detection information detected by the device-mounted sensor, and receives, from the real environment sensor, second detection information detected by the real environment sensor; an expression information generation unit that uses the first detection information to generate first expression information to be transmitted to the first remote participant, and uses the second detection information to generate second expression information including a virtual avatar-reflecting virtual space image corresponding to a virtual viewpoint in the real environment of the second remote participant; an expression information transmitting unit that transmits the first expression information to the first expression device and transmits the second expression information to the second expression device; 2. The remote experience system according to claim 1, further comprising:

3. The remote experience system according to claim 2 , wherein the object includes at least one of an animal, a plant, and air present in the real environment.

4. The remote experience system according to claim 3 , wherein the on-board sensor detects at least one of temperature, humidity, wind, and odor in the real environment.

5. The remote experience system described in claim 4, characterized in that the expression information generation unit converts the first detection information indicating at least one detection result of temperature, humidity, wind, and odor in the real environment into the first expression information that can be recognized by at least one of sight and hearing.

6. 5. The remote experience system according to claim 3, wherein the real environment sensor detects at least one of temperature, humidity, wind, and odor in the real environment.

7. The remote experience system described in claim 4, characterized in that the expression information generation unit converts the second detection information indicating at least one detection result of temperature, humidity, wind, and odor in the real environment into the second expression information that can be recognized by at least one of sight and hearing.

8. 7. The remote experience system according to claim 2, wherein the expression information generating unit generates the first expression information by using the second detection information.

9. The instrument includes a manipulator; The device-mounted sensor includes a force tactile sensor that detects the force and tactile sensation of a hand of the manipulator, 5. The remote experience system of claim 2, wherein the first presentation device includes a haptic device for transmitting haptic sensations to a hand of the first remote participant.

10. 5. The remote experience system according to claim 2, wherein the expression information generation unit generates the second expression information so that the virtual avatar corresponding to the second remote participant is displayed at the virtual viewpoint.

11. a participant motion detection device for detecting a motion of the second remote participant; Equipped with the motion information receiving unit receives, from the participant motion detection device that detects the motion of the second remote participant, motion information indicating the motion of the second remote participant acquired by the participant motion detection device; the expression information generation unit virtually generates an image in which the object in the real environment and the virtual avatar in a virtual space that is a range including the virtual viewpoint change in accordance with the movement of the second remote participant based on the movement information of the second remote participant and the second detection information, and combines virtual information indicating the generated image with the second expression information; The remote experience system according to claim 10 , wherein the expression information transmitting unit transmits the second expression information in which the virtual information is synthesized to the second expression device.

12. The expression information generating unit combines the virtual information with the first expression information, The remote experience system according to claim 11 , wherein the expression information transmitting unit transmits the first expression information in which the virtual information is synthesized to the first expression device.

13. a third presentation device capable of presenting the virtual space image reflecting the virtual avatar to the physical participants; Equipped with 13. The remote experience system according to claim 11, wherein the expression information transmitting unit transmits the virtual avatar-reflecting virtual space image including the virtual information to the third expression device.

14. A remote experience system described in any one of claims 2 to 13, characterized in that at least one of the communication lines between the virtual space generation information processing device and the participant motion detection device, the communication line between the virtual space generation information processing device and the equipment-mounted sensor, the communication line between the virtual space generation information processing device and the real environment sensor, the communication line between the virtual space generation information processing device and the equipment, the communication line between the virtual space generation information processing device and the first representation device, and the communication line between the virtual space generation information processing device and the second representation device is a Beyond 5G line.

15. The remote experience system described in any one of claims 1 to 14, characterized in that the virtual space generation information processing device is capable of setting information regarding the first remote participant and the second remote participant to be public or private.

16. An information processing device in a remote experience system that allows a plurality of remote participants to experience an event using a real environment in which real participants are participating, comprising: a virtual space generation information processing device that generates a virtual space representing the real environment including a real avatar representing a first remote participant among the plurality of remote participants and disposed in the real environment, and the real participant; a virtual avatar-reflected virtual space image generating means for generating a virtual avatar-reflected virtual space image in which a virtual avatar that represents a second remote participant among the plurality of remote participants but does not exist in the real environment is reflected in the virtual space; providing means for providing the virtual avatar-reflected virtual space image generated by the virtual avatar-reflected virtual space image generating means to the live participant, the first remote participant, and the second remote participant; An information processing device comprising:

17. An information processing method for processing information using a remote experience system that allows a plurality of remote participants to experience an event using a real environment in which real participants are participating, comprising: a virtual space generating step of generating a virtual space representing the real environment including a real avatar representing a first remote participant among the plurality of remote participants and disposed in the real environment, and the real participant; a virtual avatar-reflecting virtual space image generating step of generating a virtual avatar-reflecting virtual space image in which a virtual avatar representing a second remote participant among the plurality of remote participants but not present in the real environment is reflected in the virtual space generated by the virtual space generating step; a providing step of providing the virtual avatar-reflected virtual space image generated by the virtual avatar-reflected virtual space image generating step to the live participant, the first remote participant, and the second remote participant; An information processing method comprising:

18. A computer system for processing information using a remote experience system that allows multiple remote participants to experience an event using a real environment in which real participants are participating, a virtual space generating step of generating a virtual space representing the real environment including a real avatar representing a first remote participant among the plurality of remote participants and disposed in the real environment, and the real participant; a virtual avatar-reflecting virtual space image generating step of generating a virtual avatar-reflecting virtual space image in which a virtual avatar representing a second remote participant among the plurality of remote participants but not present in the real environment is reflected in the virtual space generated by the virtual space generating step; a providing step of providing the virtual avatar-reflected virtual space image generated by the virtual avatar-reflected virtual space image generating step to the live participant, the first remote participant, and the second remote participant; A program characterized by executing the above.

Images