Spatial connection system and spatial connection method

Abstract

To couple a real space with a virtual space so as to interlock with movement of an actual person in the real space.SOLUTION: A space coupling system couples a real space with a virtual space. An actual position in the real space and a virtual position in the virtual space are associated with each other. A first actual position is an actual position of an actual person in the real space or an actual position of a screen that moves following the actual person. A first virtual position is a virtual position associated with the first actual position and changes interlocking with the first actual position. The space coupling system arranges an access point accessible by a virtual person in the virtual space at the first virtual position in the virtual space. Further, the space coupling system displays or projects information on the virtual person or an image of the virtual space around the access point on the screen in the real space.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present disclosure relates to a technology for linking real space and virtual space. [Background technology]

[0002] Patent Document 1 discloses a communication system in which multiple terminals connected to a network share a virtual space. In the real space, an area terminal is installed for each predetermined area. An image of a virtual character in the virtual area is displayed on the area terminal. Meanwhile, a virtual display is installed in the virtual area. An image of a real-space user is displayed on the virtual display. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2009 / 060880 Summary of the Invention [Problem to be solved by the invention]

[0004] Linking real space and virtual space is important for promoting communication, business, etc. Here, real people in the real space may want to experience the virtual space in a way that links with their own movement while moving freely, rather than using devices installed in a fixed location.

[0005] One object of the present disclosure is to provide a technology that can link a real space with a virtual space so as to be linked to the movement of a real person in the real space. [Means for solving the problem]

[0006] The first aspect relates to the spatial connection system that connects real space and virtual space. A real position in the real space and a virtual position in the virtual space are associated with each other. The spatial linking system comprises one or more processors. The one or more processors acquire a first real position, which is the real position of the real person in the real space or the real position of the screen that moves to follow the real person. The one or more processors acquire a first virtual position, which is a virtual position that is associated with the first real position and changes in conjunction with the first real position. The one or more processors place an access point at a first virtual location within the virtual space that is accessible by a virtual person within the virtual space. The one or more processors display or project information about the virtual person or an image of the virtual space around the access point on a screen in the real space.

[0007] The second aspect relates to a spatial linking method that is executed by a computer and links a real space with a virtual space. A real position in the real space and a virtual position in the virtual space are associated with each other. The spatial connection method is Acquiring a first real position, which is a real position of a real person in a real space or a real position of a screen that moves to follow the real person; acquiring a first virtual position that is associated with the first real position and that changes in conjunction with the first real position; placing an access point at a first virtual location within the virtual space that is accessible by a virtual person within the virtual space; Displaying or projecting information about a virtual person or an image of the virtual space around the access point on a screen in real space. Includes. [Effects of the Invention]

[0008] According to the present disclosure, real positions in the real space and virtual positions in the virtual space are associated with each other. A screen that moves in accordance with the real person is provided in the real space. Meanwhile, an access point accessible by the virtual person is located at a first virtual position associated with the real position of the real person or the screen. Information about the virtual person or an image of the virtual space 20 around the access point is displayed or projected on the screen in the real space. When the real person moves in the real space, the access point also moves in the virtual space in conjunction with the real person's movement. When the access point moves in the virtual space, the information displayed or projected on the screen in the real space also changes in conjunction with the real person's movement. In this way, the real space and the virtual space are linked in conjunction with the movement of the real person in the real space. This allows the real person to feel as if they are actually moving through the virtual space. In other words, the real person can freely move around and experience the virtual space in conjunction with their own movement, without using a device installed in a predetermined location. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a conceptual diagram illustrating an example of a real space and a virtual space. [Figure 2] FIG. 1 is a conceptual diagram illustrating an example of a connection between a real space and a virtual space. [Figure 3] FIG. 1 is a conceptual diagram illustrating an example of a connection between a real space and a virtual space. [Figure 4] FIG. 1 is a schematic diagram illustrating a configuration example of a spatial connection system. [Figure 5] FIG. 2 is a schematic diagram showing a first example of the configuration of the surroundings of a real person in real space. [Figure 6] FIG. 10 is a schematic diagram showing a second example of the configuration of the surroundings of a real person in real space. [Figure 7] FIG. 10 is a schematic diagram showing a third example of the configuration of the surroundings of a real person in real space. [Figure 8] FIG. 1 is a block diagram showing an example of the configuration of an information processing system included in a spatial connection system. [Figure 9]FIG. 1 is a conceptual diagram illustrating a first example of an access point placed in a virtual space. [Figure 10] FIG. 10 is a conceptual diagram illustrating a second example of an access point placed in a virtual space. [Figure 11] 10 is a flowchart illustrating an example of processing by the information processing system. DETAILED DESCRIPTION OF THE INVENTION

[0010] Embodiments of the present disclosure will be described with reference to the accompanying drawings.

[0011] 1. Example of connecting real space and virtual space FIG. 1 is a conceptual diagram illustrating an example of a real space 10 and a virtual space 20. A certain area AR0 exists in the real space 10. For example, the area AR0 may be a city, a building, etc. The area AR0 is reproduced in the virtual space 20 using DigitalTwin technology or the like. Meanwhile, an area AR1 exists in the real space 10 at a position distant from the area AR0. The area AR1 may be outdoors or indoors. A real person 11 in the area AR1 can indirectly experience the distant area AR0 by experiencing the virtual space 20.

[0012] Here, the real person 11 (user) may wish to feel as if he or she is actually moving within the virtual space 20. In other words, the real person 11 may wish to experience the virtual space 20 while moving freely, in conjunction with his or her own movement, rather than using a device installed in a predetermined position. In order to meet such needs, this embodiment provides a technology that can link the real space 10 and the virtual space 20 in conjunction with the movement of the real person 11 within the real space 10.

[0013] FIG. 2 is a conceptual diagram for explaining an example of connection between the real space 10 and the virtual space 20. As shown in FIG.

[0014] In a real space 10, a drone 12A flies near a real person 11 and moves while following the real person 11. The drone 12A is equipped with a water tank and forms a mist screen 13A by spraying mist. Mist screens are a well-known technology and are disclosed, for example, at "https: / / www.seiko-giken.jp / solution / screen." The drone 12A can stably form a conical mist screen 13A by rotating. In particular, the drone 12A forms a conical mist screen 13A so as to surround the real person 11. This mist screen 13A forms a space exclusively for the real person 11. As the real person 11 moves, the drone 12A moves while following the real person 11, and therefore the mist screen 13A also moves while following the real person 11.

[0015] In the following description, "real position" means a position in real space 10, and "virtual position" means a position in virtual space 20. The real positions in real space 10 and the virtual positions in virtual space 20 are associated with each other in advance. The "first real position" is the real position of real person 11. The "first virtual position" is a virtual position associated with the first real position. Therefore, when the first real position changes, the first virtual position also changes in conjunction with the first real position.

[0016] An avatar 23A of the real person 11 is placed at a first virtual position within the virtual space 20. The facial expression and posture of the real person 11 may be reflected in the avatar 23A. Since the first real position and the first virtual position are associated with each other, when the real person 11 moves within the real space 10, the avatar 23A of the real person 11 also moves in the virtual space 20 in conjunction with the movement of the real person 11.

[0017] The drone 12A is equipped with a projector 15A. The projector 15A can project an image (video) from inside the mist screen 13A towards the mist screen 13A. The image projected onto the mist screen 13A is an image of the surroundings of the avatar 23A in the virtual space 20. Preferably, the image projected onto the mist screen 13A is an image of the virtual space 20 seen from the line of sight of the avatar 23A.

[0018] When the real person 11 moves within the real space 10, the avatar 23A of the real person 11 also moves in conjunction with that in the virtual space 20. When the avatar 23A moves within the virtual space 20, the image of the virtual space 20 projected on the mist screen 13A around the real person 11 also changes in conjunction with that. This allows the real person 11 to get the feeling that he or she is actually moving within the virtual space 20. In other words, the real person 11 can move freely and experience the virtual space 20 in conjunction with his or her own movement without using a device installed in a predetermined position.

[0019] 3 is a conceptual diagram for explaining communication between a real person 11 in a real space 10 and a virtual person 21 in a virtual space 20. In addition to the avatar 23A of the real person 11, many other virtual people 21 (avatars) exist in the virtual space 20. The avatar 23A of the real person 11 in the virtual space 20 can be the "starting point" of communication.

[0020] For example, the virtual person 21 initiates communication. Specifically, in the virtual space 20, the virtual person 21 accesses the avatar 23A of the real person 11. In response to this access, information about the virtual person 21 who has accessed the avatar 23A is projected onto the mist screen 13A around the real person 11 in the real space 10. Typically, an image of the virtual person 21 is projected onto the mist screen 13A. Then, the real person 11 and the virtual person 21 begin communication. The communication may be carried out by voice or by text.

[0021] As another example, the real person 11 may initiate communication. Specifically, the image projected on the mist screen 13A shows one or more virtual people 21 around the avatar 23A in the virtual space 20. The real person 11 designates one of the virtual people 21 shown in the image on the mist screen 13A as the communication partner. When designating a communication partner, the real person 11 may touch or point at the virtual person 21 on the mist screen 13A. The designated virtual person 21 is notified that they have been designated. Then, the real person 11 and the virtual person 21 begin communication. The communication may be carried out by voice or text.

[0022] As described above, according to this embodiment, the real space 10 and the virtual space 20 are linked together so as to be linked to the movement of the real person 11 in the real space 10. This allows the real person 11 to experience the virtual space 20 in conjunction with his or her own movement while moving freely, without using a device installed in a predetermined position.

[0023] 2 and 3, the real person 11 is reproduced by an avatar 23A in the virtual space 20, while the virtual space 20 or the virtual person 21 is reproduced on a mist screen 13A in the real space 10. In that sense, it can be said that the real space 10 and the virtual space 20 are partially overlapped. The overlapping portion of the real space 10 and the virtual space 20 serves as a window for communication between the real person 11 and the virtual person 21. In other words, the real person 11 and the virtual person 21 can communicate with each other via the overlapping portion of the real space 10 and the virtual space 20.

[0024] 2 and 3 are merely examples of the connection between the real space 10 and the virtual space 20. Various examples of the connection between the real space 10 and the virtual space 20 will be described later. Furthermore, the virtual space 20 does not necessarily have to represent the area AR0 in the real world. The virtual space 20 may represent a fictional world.

[0025] The "space linking system 1" that links the real space 10 and the virtual space 20 will be described in detail below.

[0026] 2. Spatial connection system 4 is a schematic diagram showing an example of the configuration of a spatial connection system 1 according to this embodiment. The spatial connection system 1 connects a real space 10 and a virtual space 20. Connecting the real space 10 and the virtual space 20 contributes to promoting communication, business, and the like.

[0027] 2-1. Surroundings of real people The spatial connection system 1 includes a followable object 12, a screen 13, a camera 14, a projector 15, a speaker 16, and a microphone 17 in a real space 10. The followable object 12 is configured to follow a real person 11. The screen 13 is provided to follow the followable object 12. As a result, the screen 13 moves to follow the real person 11. The camera 14 captures images of the real person 11 and his / her surroundings. The projector 15 is provided so as to be able to project various images onto the screen 13. The speaker 16 outputs various sounds toward the real person 11. The microphone 17 detects the sounds of the real person 11.

[0028] Various specific examples of the configuration of the surroundings of the real person 11 will be described below.

[0029] 2-1-1. First example FIG. 5 is a schematic diagram showing a first example of the configuration of the surroundings of a real person 11. The object to be followed 12 is a drone 12A. The screen 13 is a mist screen 13A. The camera 14 includes a camera 14A mounted on the drone 12A to capture the surroundings of the drone 12A. The camera 14 may also include an infrastructure camera 14D. The projector 15 is a projector 15A mounted on the drone 12A. The speaker 16 includes a directional speaker 16A mounted on the drone 12A. The microphone 17 includes a directional microphone 17A mounted on the drone 12A. The real person 11 may also be wearing a headset 18 including the speaker 16 and the microphone 17.

[0030] The drone 12A has a self-position estimation function. The drone 12A may estimate its own position by using a Global Navigation Satellite System (GNSS). The drone 12A may be configured to be capable of autonomous flight.

[0031] The drone 12A moves while following the real person 11. For example, the drone 12A acquires a camera image captured by the camera 14A. A target image, which is an image of the real person 11 to be followed, is registered in advance. The drone 12A recognizes the real person 11 appearing in the camera image based on the target image. A machine learning model is used for person recognition. The drone 12A can follow the real person 11 by tracking the real person 11 recognized in the camera image.

[0032] As another example, the drone 12A may acquire information on the absolute position of the real person 11. The information on the absolute position of the real person 11 may be provided from an external device outside the drone 12A. The drone 12A can track the real person 11 based on its own position obtained by the self-position estimation function and the absolute position of the real person 11.

[0033] The mist screen 13A is formed by mist sprayed from the drone 12A (see Figures 2 and 3). Therefore, the mist screen 13A moves following the drone 12A. The drone 12A can stably form the conical mist screen 13A by using rotation. In particular, the drone 12A forms the conical mist screen 13A so as to surround the real person 11. This mist screen 13A forms a space exclusively for the real person 11.

[0034] The projector 15A projects an image (video) from inside the mist screen 13A toward the mist screen 13A.

[0035] The directional speaker 16A is realized by, for example, a very small area audio spot technology (http: / / www.activeforall.jp / project / project02 / ) that uses ultrasonic waves.

[0036] 2-1-2. Second example 6 is a schematic diagram showing a second example of the configuration around a real person 11. Other than the screen 13 and the projector 15, the configuration is the same as in the first example described above. Explanations that overlap with the first example described above will be omitted as appropriate.

[0037] In the second example, the screen 13 is a screen 13B hanging from the drone 12A. Because the screen 13B is hanging from the drone 12A, the screen 13B moves following the drone 12A.

[0038] For example, the screen 13B is a cloth screen that hangs from the drone 12A. In this case, a projector 15B mounted on the drone 12A projects an image (video) onto the screen 13B.

[0039] As another example, the screen 13B may be the screen of a display device. Examples of the display device include a liquid crystal display and an organic EL display. The display device may be a touch panel. The display device displays an image (video) on the screen 13B (screen). In this case, the projector 15B is not required.

[0040] 2-1-3. Third example FIG. 7 is a schematic diagram showing a third example of the configuration of the surroundings of the real person 11. The object to be followed 12 is a robot 12C. The screen 13 is a screen 13C installed on the robot 12C. The camera 14 includes a camera 14C mounted on the robot 12C to capture the surroundings of the robot 12C. The camera 14 may also include an infrastructure camera 14D. The projector 15 is a projector 15C mounted on the robot 12C. The speaker 16 includes a directional speaker 16C mounted on the robot 12C. The microphone 17 includes a directional microphone 17C mounted on the robot 12C. The real person 11 may also be wearing a headset 18 including the speaker 16 and the microphone 17.

[0041] The robot 12C has a self-position estimation function. The robot 12C may estimate its own position by using GNSS. The robot 12C may be configured to be capable of autonomous travel.

[0042] The robot 12C moves by following the real person 11. For example, the robot 12C acquires a camera image captured by the camera 14C. A target image, which is an image of the real person 11 to be followed, is registered in advance. The robot 12C recognizes the real person 11 appearing in the camera image based on the target image. A machine learning model is used for person recognition. The robot 12C can follow the real person 11 by tracking the real person 11 recognized in the camera image.

[0043] As another example, the robot 12C may acquire information on the absolute position of the real person 11. The information on the absolute position of the real person 11 may be provided from an external device outside the robot 12C. The robot 12C can follow the real person 11 based on its own position obtained by the self-position estimation function and the absolute position of the real person 11.

[0044] The screen 13C is attached to the robot 12C and moves along with the robot 12C. For example, the screen 13C is a screen of a display device. Examples of the display device include a liquid crystal display and an organic EL display. The display device may be a touch panel. The display device displays an image (video) on the screen 13C (screen).

[0045] As another example, the screen 13C is a screen erected above the robot 12C. In this case, a projector 15C mounted on the robot 12C projects an image (video) onto the screen 13C.

[0046] 2-1-4. Fourth Example In the fourth example, the object to be followed 12 is a terminal (e.g., a smartphone) carried by the real person 11. The terminal carried by the real person 11 automatically moves to follow the real person 11. The screen 13 is the screen of the display device of the terminal. The camera 14, speaker 16, and microphone 17 are mounted on the terminal. The terminal also acquires its own location information using GNSS or the like.

[0047] 2-2. Information Processing System As shown in FIG. 4, the spatial connection system 1 further includes an information processing system 100 that executes various information processes.

[0048] 2-2-1. Overview The information processing system 100 directly or indirectly communicates with the object 12, the screen 13, the camera 14, the projector 15, the speaker 16, and the microphone 17 in the real space 10. Furthermore, the information processing system 100 directly or indirectly controls the object 12, the screen 13, the camera 14, the projector 15, the speaker 16, and the microphone 17 in the real space 10.

[0049] For example, the information processing system 100 controls the object to be followed 12. The information processing system 100 may include a control device mounted on the object to be followed 12. That is, at least a part of the information processing system 100 may be included in the object to be followed 12. If a screen 13 is installed on the object to be followed 12, the information processing system 100 may indirectly control the screen 13 by controlling the object to be followed 12. If a camera 14 is mounted on the object to be followed 12, the information processing system 100 may indirectly control the camera 14 by controlling the object to be followed 12, and may also acquire camera images captured by the camera 14 via the object to be followed 12. If a projector 15 is mounted on the object to be followed 12, the information processing system 100 may indirectly control the projector 15 by controlling the object to be followed 12. If a speaker 16 is mounted on the object to be followed 12, the information processing system 100 may indirectly control the speaker 16 by controlling the object to be followed 12. If the microphone 17 is mounted on the followable object 12, the information processing system 100 may indirectly control the microphone 17 by controlling the followable object 12, and may also acquire audio information detected by the microphone 17 via the followable object 12.

[0050] The information processing system 100 also includes a virtual space simulation system 200. The virtual space simulation system 200 performs a simulation of a virtual space 20.

[0051] The virtual space 20 is configured to reproduce, for example, an area AR0 in the real space 10 (see FIG. 1). For example, the area AR0 is a city, a building, etc. A sensor group 300 is arranged in the area AR0 to detect the situation in the area AR0. The situation in the area AR0 includes the situation of people and moving objects present in the area AR0. The virtual space simulation system 200 communicates with the sensor group 300 and acquires information about the situation in the area AR0 detected by the sensor group 300. Then, the virtual space simulation system 200 reproduces the area AR0 in the virtual space 20 using DigitalTwin technology.

[0052] The virtual space 20 does not necessarily have to represent the area AR0 in the real world, but may represent a fictional world.

[0053] The user device 400 is a device used by a user other than the real person 11 to participate in the virtual space 20. Examples of the user device 400 include user terminals such as PCs and smartphones. The user device 400 may be a wearable device. The user device 400 includes an input device and an output device. Examples of the input device include a keyboard, a touch panel, a mouse, and a microphone. Examples of the output device include a display device and a speaker. The user sets a virtual person 21, which is the user's alter ego, in the virtual space 20 by operating the input device of the user device 400. The virtual person 21 may be the user's avatar. The user also operates the input device of the user device 400 to move the virtual person 21 in the virtual space 20. The microphone detects the user's voice. The display device presents various information to the user. The speaker outputs various sounds to the user.

[0054] The virtual space simulation system 200 communicates with the user device 400. The virtual space simulation system 200 reflects information input by the user to an input device of the user device 400 in the virtual space 20. The virtual space simulation system 200 also outputs various information related to the virtual space 20 from an output device of the user device 400.

[0055] 2-2-2.Configuration example FIG. 8 is a block diagram showing an example configuration of an information processing system 100. The information processing system 100 includes one or more processors 101 (hereinafter simply referred to as processors 101), one or more storage devices 102 (hereinafter simply referred to as storage devices 102), and an interface 103. The processor 101 executes various processes. For example, the processor 101 includes a CPU (Central Processing Unit). The storage device 102 stores various information. Examples of the storage device 102 include a volatile memory, a non-volatile memory, an HDD (Hard Disk Drive), and an SSD (Solid State Drive). The interface 103 is an interface for communicating with various components (such as a tracking object 12, a screen 13, a camera 14, a projector 15, a speaker 16, a microphone 17, a sensor group 300, and a user device 400). The processor 101 can communicate with the various components via the interface 103.

[0056] The control program 104 is a computer program executed by the processor 101. The processor 101 executing the control program 104 works in cooperation with the storage device 102 to realize the functions of the information processing system 100. The control program 104 is stored in the storage device 102. Alternatively, the control program 104 may be recorded on a computer-readable recording medium. The control program 104 may be provided via a network.

[0057] The storage device 102 stores real space information 110 relating to the real space 10, virtual space information 120 relating to the virtual space 20, and position correspondence information 130. The position correspondence information 130 indicates a predetermined correspondence between a real position in the real space 10 and a virtual position in the virtual space 20.

[0058] The real space information 110 includes real person information 111, object to be followed information 112, screen information 113, an image 114, and projector information 115. The image 114 is captured by a camera 14. The camera 14 includes a camera 14 (14A, 14C: see FIGS. 5 to 7) mounted on the object to be followed 12, and an infrastructure camera 14D.

[0059] The real person information 111 is information relating to the real person 11. For example, the real person information 111 includes registration information of the real person 11. For example, the registration information includes ID information of the real person 11, a target image which is an image of the real person 11, information on the avatar 23A of the real person 11, etc. The real person 11 may register the registration information in advance in the information processing system 100 by operating a terminal (not shown).

[0060] The real person information 111 also includes status information indicating the position (absolute position), moving direction, posture, etc. of the real person 11. For example, a camera 14 mounted on the object to be followed 12 captures an image 114 of an area including the real person 11. The object to be followed 12 also estimates its own position using a self-position estimation function. The processor 101 acquires the image 114 captured by the camera 14 and the position information of the object to be followed 12. The processor 101 recognizes the real person 11 appearing in the image 114 based on a pre-registered target image. A machine learning model is used for person recognition. The processor 101 calculates the position (absolute position) of the real person 11 by combining the position (absolute position) of the object to be followed 12 and the recognition result of the real person 11. The processor 101 also calculates the moving direction of the real person 11 from the position history of the real person 11. The processor 101 also recognizes the posture of the real person 11 based on the recognition result of the real person 11 in the image 114.

[0061] As another example, an image 114 of an area including the real person 11 is captured by the infrastructure camera 14D. The infrastructure camera 14D also provides installation information indicating its own installation position and installation orientation. The processor 101 acquires the image 114 captured by the infrastructure camera 14D and the installation information of the infrastructure camera 14D. The processor 101 recognizes the real person 11 appearing in the image 114 based on a pre-registered target image. A machine learning model is used for person recognition. The processor 101 calculates the position (absolute position) of the real person 11 by combining the installation information of the infrastructure camera 14D and the recognition result of the real person 11. The processor 101 also calculates the traveling direction of the real person 11 from the position history of the real person 11. Furthermore, the processor 101 recognizes the posture of the real person 11 based on the recognition result of the real person 11 in the image 114.

[0062] As yet another example, since the object to be followed 12 moves while following the real person 11, the position and movement direction of the object to be followed 12 may be used approximately as the position and direction of travel of the real person 11. In particular, if the object to be followed 12 is a terminal carried by the real person 11, the position and movement direction of the terminal are used as the position and direction of travel of the real person 11.

[0063] The object information 112 to be tracked is information about the object 12 to be tracked. The object information 112 to be tracked includes ID information of the object 12 to be tracked. The object information 112 to be tracked also includes ID information of the real person 11 assigned to the object 12 to be tracked, i.e., the real person 11 to be tracked by the object 12 to be tracked. In other words, the object information 112 to be tracked includes information on the assignment relationship between the real person 11 and the object 12 to be tracked. The assignment relationship between the real person 11 and the object 12 to be tracked is determined in advance by the processor 101. Furthermore, the object information 112 to be tracked includes position information of the object 12 to be tracked. As described above, the position information of the object 12 to be tracked is obtained by the self-position estimation function of the object 12 to be tracked. The processor 101 acquires the position information from the object 12 to be tracked.

[0064] The object to be followed information 112 may further include control information for controlling the object to be followed 12. The processor 101 acquires the position (absolute position) and traveling direction of the real person 11 from the real person information 111. The processor 101 calculates the amount of control required for the object to be followed 12 to follow the real person 11 based on the position of the object to be followed 12 and the position and traveling direction of the real person 11. The processor 101 then provides the control information including the calculated amount of control to the object to be followed 12, thereby controlling the object to be followed 12.

[0065] The screen information 113 is information related to the screen 13. The screen information 113 indicates the position, size, shape, etc. of the screen 13. The position of the screen 13 may be the absolute position of the screen 13 or the relative position of the screen 13 with respect to the object 12 to be followed. If the screen 13 is a mist screen 13A (see FIG. 5), the relative position, size, and shape of the mist screen 13A can be estimated based on the mist spray direction and spray pressure. If the screen 13 is a screen 13B (see FIG. 6) or a screen 13C (see FIG. 7), the relative position, size, and shape of the screen 13 are predetermined. The absolute position of the screen 13 is obtained by combining the absolute position of the object 12 to be followed and the relative position of the screen 13. If the screen 13 is the screen of a terminal carried by the real person 11, the terminal position is used as the position of the screen 13.

[0066] The projector information 115 is information relating to the projector 15. The projector information 115 indicates the installation position and installation orientation of the projector 15 on the object 12 to be followed.

[0067] The virtual space information 120 is information relating to the virtual space 20, and is used particularly by the virtual space simulation system 200. The virtual space information 120 includes virtual person information 121, virtual space configuration information 122, and access point information 123.

[0068] The virtual person information 121 is information related to the virtual person 21. For example, the virtual person information 121 includes registration information of a user who participates in the virtual space 20 via the virtual person 21. For example, the registration information includes ID information of the user, an image of the user, an image of the virtual person 21 (avatar), etc. The user may operate the user device 400 to register the registration information in advance in the information processing system 100.

[0069] The virtual person information 121 also includes position information of the virtual person 21 in the virtual space 20. The user operates the input device of the user device 400 to move the virtual person 21 in the virtual space 20. The processor 101 calculates the position of the virtual person 21 in the virtual space 20 based on the operation of the input device by the user.

[0070] Virtual space configuration information 122 indicates the configuration of virtual space 20. For example, virtual space configuration information 122 indicates the three-dimensional arrangement of structures (e.g., roads, road structures, buildings, etc.) within virtual space 20. Virtual space configuration information 122 also indicates the positions of moving objects (e.g., vehicles, robots, etc.) within virtual space 20.

[0071] The access point information 123 is information about the "access points 23" placed in the virtual space 20. The access points 23 will be described in detail below.

[0072] Access Points The processor 101 places an access point 23 accessible by the virtual person 21 in the virtual space 20.

[0073] More specifically, the processor 101 places the access point 23 at a first virtual position in the virtual space 20. The first virtual position is a virtual position that is associated with a first real position in the real space 10. The first real position is the real position of the real person 11, or the real position of the screen 13 that moves following the real person 11. The real position of the real person 11 is obtained from the real person information 111. The real position of the screen 13 is obtained from the screen information 113. The correspondence between the real positions in the real space 10 and the virtual positions in the virtual space 20 is given by the position correspondence information 130. Therefore, the processor 101 can convert the first real position into a first virtual position based on the position correspondence information 130. Then, the processor 101 places the access point 23 at the first virtual position.

[0074] When the real person 11 moves and the first real position changes, the first virtual position also changes in conjunction with this. In other words, when the real person 11 moves in the real space 10, the access point 23 also moves in the virtual space 20 in conjunction with this.

[0075] The access point 23 indicates information about the real person 11 or an image of the real space 10 around the first real position. Some examples of the access point 23 will be described below.

[0076] 2-3-1. First example of access point 9 is a conceptual diagram for explaining a first example of the access point 23. In the first example, the access point 23 is an avatar 23A of the real person 11. The avatar 23A of the real person 11 corresponds to the information of the real person 11 described above. The first real position is the real position of the real person 11 in the real space 10. The first virtual position is a virtual position associated with the real position of the real person 11.

[0077] The processor 101 obtains information about the avatar 23A of the real person 11 from the real person information 111. The processor 101 places the avatar 23A of the real person 11 at a first virtual position in the virtual space 20. When the real person 11 moves in the real space 10, the avatar 23A of the real person 11 also moves in the virtual space 20 in conjunction with the movement of the real person 11. The processor 101 may also apply the posture of the real person 11 to the avatar 23A. The posture of the real person 11 is obtained from the real person information 111.

[0078] The processor 101 displays or projects various types of information on a screen 13 in the real space 10. For example, the processor 101 projects various types of information on the screen 13 by controlling a projector 15. When the screen 13 is a screen of a display device, the processor 101 displays various types of information on the screen 13 (screen) by controlling the display device.

[0079] For example, processor 101 displays or projects an image of virtual space 20 around avatar 23A onto screen 13. Preferably, processor 101 displays or projects an image of virtual space 20 seen from the line of sight of avatar 23A onto screen 13. Processor 101 renders the image of virtual space 20 seen from the line of sight of avatar 23A based on the position (first virtual position) of avatar 23A and virtual space configuration information 122.

[0080] In addition to the avatar 23A of the real person 11, many other virtual persons 21 (avatars) exist in the virtual space 20. The avatar 23A of the real person 11 in the virtual space 20 can be the "starting point" of communication.

[0081] For example, the virtual person 21 operated by the user provides an opportunity for communication. Specifically, the user operates the input device of the user device 400 to cause the virtual person 21 to access the avatar 23A. For example, when the user clicks on the avatar 23A, the virtual person 21 accesses the avatar 23A.

[0082] In response to the virtual person 21 accessing the avatar 23A, the processor 101 displays or projects information about the virtual person 21 on the screen 13 in the real space 10. For example, the processor 101 displays or projects an image of the virtual person 21 or an image of the user operating the virtual person 21 on the screen 13. Such information about the virtual person 21 is obtained from the virtual person information 121.

[0083] Then, the real person 11 and the virtual person 21 start communicating. The communication may be performed by voice or text. For example, the processor 101 acquires the voice of the real person 11 from the microphone 17 and outputs the voice of the real person 11 from the speaker of the user device 400. The processor 101 also acquires the user's voice from the microphone of the user device 400 and outputs the user's voice from the speaker 16.

[0084] In this way, by placing the avatar 23A of the real person 11 in the virtual space 20, communication is promoted.

[0085] As another example, the real person 11 may initiate communication. As described above, an image of the virtual space 20 around the avatar 23A is displayed or projected on the screen 13. One or more virtual persons 21 around the avatar 23A are shown in the image. The real person 11 designates one of the virtual persons 21 shown in the image on the screen 13 as a communication partner. When designating a communication partner, the real person 11 may touch or point at the virtual person 21 on the screen 13. Such a designation action by the real person 11 is detected based on the image 114 captured by the camera 14. Alternatively, if the screen 13 is a touch panel, the designation action by the real person 11 is detected by the touch panel. The processor 101 displays or projects information about the virtual person 21 designated by the real person 11 on the screen 13 in the real space 10. For example, the processor 101 displays or projects an image of the designated virtual person 21 or an image of a user operating the designated virtual person 21 on the screen 13. Such information about the virtual person 21 is obtained from the virtual person information 121. Then, the real person 11 and the virtual person 21 start to communicate.

[0086] 2-3-2.Second example of access point 10 is a conceptual diagram for explaining a second example of the access point 23. Explanations that overlap with the first example described above will be omitted as appropriate.

[0087] In a second example, the access point 23 is a virtual screen 23B. The first real position may be the real position of the real person 11 or the real position of the screen 13. The size and shape of the virtual screen 23B may be set to be the same as the size and shape of the screen 13 in the real space 10. The size and shape of the screen 13 are obtained from the screen information 113. The processor 101 places the virtual screen 23B at the first virtual position in the virtual space 20. When the real person 11 moves in the real space 10, the virtual screen 23B also moves in the virtual space 20 in conjunction with the movement of the real person 11.

[0088] The processor 101 displays or projects an image of the real space 10 around the first real position on a virtual screen 23B in the virtual space 20. The image of the real space 10 around the first real position is obtained from an image 114 captured by the camera 14.

[0089] Meanwhile, the processor 101 displays or projects an image of the virtual space 20 around the virtual screen 23B onto the screen 13 in the real space 10. The processor 101 draws the image of the virtual space 20 around the virtual screen 23B based on the position of the virtual screen 23B (first virtual position) and the virtual space configuration information 122.

[0090] As a result, the screen 13 in the real space 10 and the virtual screen 23B in the virtual space 20 overlap, creating a "transparent window." A real person 11 in the real space 10 can see the state of the virtual space 20 through the screen 13. A virtual person 21 in the virtual space 20 can see the state of the real space 10 through the virtual screen 23B. This type of configuration also contributes to promoting communication.

[0091] The method of communication between the real person 11 and the virtual person 21 is the same as in the first example described in section 2-3-1 above.

[0092] 2-4.Processing flow FIG. 11 is a flowchart showing an example of processing by the information processing system 100 (processor 101).

[0093] In step S110, the information processing system 100 acquires real person information 111 related to a real person 11 in the real space 10.

[0094] In step S120, the information processing system 100 acquires a first real position in the real space 10. The first real position is the real position of the real person 11, or the real position of the screen 13 that moves following the real person 11. The real position of the real person 11 is obtained from the real person information 111. The real position of the screen 13 is obtained from the screen information 113.

[0095] In step S130, the information processing system 100 acquires a first virtual position associated with the first real position. The correspondence between the real position in the real space 10 and the virtual position in the virtual space 20 is given by the position correspondence information 130. Therefore, the information processing system 100 converts the first real position into a first virtual position based on the position correspondence information 130. When the first real position changes, the first virtual position also changes in conjunction with the first real position.

[0096] In step S140, the information processing system 100 places an access point 23 at the first virtual location that can be accessed by the virtual person 21 in the virtual space 20. The access point 23 indicates information about the real person 11 or an image of the real space 10 around the first real location.

[0097] For example, the access point 23 is an avatar 23A of the real person 11 (see FIG. 9). The avatar 23A of the real person 11 is obtained from the real person information 111. The posture of the real person 11 may be applied to the avatar 23A. The posture of the real person 11 is obtained from the real person information 111.

[0098] As another example, the access point 23 may be a virtual screen 23B (see FIG. 10 ). The information processing system 100 displays or projects an image of the real space 10 around the first real position on the virtual screen 23B. The image of the real space 10 around the first real position is obtained from an image 114 captured by the camera 14.

[0099] In step S150, the information processing system 100 displays or projects various pieces of information on the screen 13 in the real space 10. For example, the information processing system 100 projects various pieces of information on the screen 13 by controlling the projector 15. If the screen 13 is a screen of a display device, the information processing system 100 displays various pieces of information on the screen 13 (screen) by controlling the display device.

[0100] The information displayed or projected on the screen 13 is, for example, information about the virtual person 21 who is the communication partner. Examples of the information about the virtual person 21 include an image (avatar) of the virtual person 21, an image of the user, etc. Such information about the virtual person 21 is obtained from the virtual person information 121.

[0101] As another example, the information displayed or projected on the screen 13 may be an image of the virtual space 20 around the access point 23. The image of the virtual space 20 around the access point 23 is an image of the virtual space 20 as seen from the first virtual position (the position of the access point 23). The processor 101 renders the image of the virtual space 20 as seen from the first virtual position based on the first virtual position and the virtual space configuration information 122.

[0102] The processor 101 may determine the size of the image to be displayed or projected on the screen 13, taking into account the distance between the real person 11 and the screen 13, so that the sense of distance when viewed from the real person 11 is appropriate. The position of the real person 11 is obtained from the real person information 111. The position of the screen 13 is obtained from the screen information 113. The processor 101 can calculate the distance between the real person 11 and the screen 13 based on the positions of the real person 11 and the screen 13, and determine an appropriate image size.

[0103] 3.Effects According to this embodiment, real positions in real space 10 and virtual positions in virtual space 20 are associated with each other. A screen 13 that moves following a real person 11 is provided in real space 10. Meanwhile, an access point 23 accessible by a virtual person 21 is placed at a first virtual position associated with the real position of the real person 11 or the screen 13. Information about the virtual person 21 or an image of the virtual space 20 around the access point 23 is displayed or projected on screen 13 in real space 10. When the real person 11 moves in real space 10, the access point 23 also moves in virtual space 20 in conjunction with the movement. When the access point 23 moves in virtual space 20, the information displayed or projected on screen 13 in real space 10 also changes in conjunction with the movement of the access point 23.

[0104] In this way, the real space 10 and the virtual space 20 are linked together so as to be linked to the movement of the real person 11 in the real space 10. This allows the real person 11 to have the sensation of actually moving within the virtual space 20. In other words, the real person 11 can move freely and experience the virtual space 20 in a way that is linked to his or her own movement, without using a device installed in a predetermined position.

[0105] According to this embodiment, the real space 10 or real person 11 is reproduced at the access point 23 in the virtual space 20, while the virtual space 20 or virtual person 21 is reproduced on the screen 13 in the real space 10. In that sense, it can be said that the real space 10 and the virtual space 20 are partially overlapped.

[0106] The overlapping portion of the real space 10 and the virtual space 20 serves as a window for communication between the real person 11 and the virtual person 21. In other words, the real person 11 and the virtual person 21 can communicate with each other through the overlapping portion of the real space 10 and the virtual space 20. [Explanation of symbols]

[0107] 1...spatial connection system, 10...real space, 11...real person, 12...following object, 13...screen, 14...camera, 20...virtual space, 21...virtual person, 23...access point, 23A...avatar, 23B...virtual screen, 100...information processing system, 200...virtual space simulation system

Claims

1. A spatial connection system that connects real space and virtual space, a real position in the real space and a virtual position in the virtual space are associated with each other; the spatial linking system comprises one or more processors; the one or more processors: Controlling a drone that moves following a real person in the real space; acquiring a first real position that is the real position of the real person in the real space or the real position of a screen that is provided to follow the drone; acquiring a first virtual position, which is the virtual position associated with the first real position and which changes in conjunction with the first real position; placing an access point at the first virtual location in the virtual space that is accessible by a virtual person in the virtual space; Displaying or projecting information about the virtual person or an image of the virtual space around the access point on the screen in the real space. It was configured as Spatial connection system.

2. 2. The spatial connection system of claim 1, The access point indicates information about the real person or an image of the real space around the first real location. Spatial connection system.

3. 3. The spatial connection system according to claim 2, In response to the virtual person accessing the access point, the one or more processors display or project the information of the virtual person on the screen so that the real person and the virtual person can communicate. Spatial connection system.

4. 3. The spatial connection system according to claim 2, the one or more processors display or project the image of the virtual space around the access point on the screen; When the real person designates the virtual person appearing in the image on the screen, the one or more processors display or project the information of the designated virtual person on the screen so that the real person and the virtual person can communicate with each other. Spatial connection system.

5. 3. The spatial connection system according to claim 2, the first real position is the real position of the real person in the real space; the first virtual position is the virtual position associated with the real position of the real person in the real space, the access point is an avatar of the real person; The one or more processors place the avatar of the real person at the first virtual location within the virtual space. Spatial connection system.

6. 6. The spatial connection system according to claim 5, The one or more processors further acquiring information about the posture of the real person in the real space; Applying the pose to the avatar of the real person Spatial connection system.

7. 3. The spatial connection system according to claim 2, the access point is a virtual screen; the one or more processors: displaying or projecting the image of the real space around the first real position onto the virtual screen in the virtual space; The image of the virtual space around the virtual screen is displayed or projected onto the screen in the real space. Spatial connection system.

8. 8. A spatial connection system according to any one of claims 1 to 7, The screen is a mist screen formed by mist sprayed from the drone, The mist screen is formed to surround the real person. Spatial connection system.

9. A spatial connection method that is executed by a computer and connects a real space and a virtual space, comprising: a real position in the real space and a virtual position in the virtual space are associated with each other; The spatial connection method includes: Controlling a drone that moves to follow a real person in the real space; acquiring a first real position, which is the real position of the real person in the real space or the real position of a screen that is provided to follow the drone; acquiring a first virtual position, which is the virtual position associated with the first real position and which changes in conjunction with the first real position; placing an access point at the first virtual location within the virtual space that is accessible by a virtual person within the virtual space; displaying or projecting information about the virtual person or an image of the virtual space around the access point on the screen in the real space; Contains Spatial connection method.

Images