Space coupling system and space coupling method

JP2024158889AActive Publication Date: 2024-11-08TOYOTA JIDOSHA KK
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
JP2023074506
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-04-28
Publication Date
2024-11-08
Estimated Expiration
2043-04-28

AI Technical Summary

Benefits of technology

【0008】 本開示によれば、実空間の実位置と仮想空間内の仮想位置が互いに対応付けられる。また、実空間には実人物に追従して移動するスクリーンが設けられる。一方、実人物あるいはスクリーンの実位置に対応付けられた第1仮想位置には、仮想人物によってアクセス可能なアクセスポイントが配置される。実空間内のスクリーンには、仮想人物の情報、あるいは、アクセスポイントの周辺の仮想空間20の画像が表示あるいは投影される。実空間内で実人物が移動すると、それに連動して仮想空間内でアクセスポイントも移動する。そして、仮想空間内でアクセスポイントが移動すると、それに連動して、実空間内のスクリーンに表示あるいは投影される情報も変化する。このように、実空間内の実人物の移動に連動するように実空間と仮想空間とが結びつけられる。これにより、実人物は、自分自身があたかも仮想空間内を実際に移動しているような感覚を得ることができる。すなわち、実人物は、所定位置に設置されたデバイスを用いることなく、自由に移動しながら自身の移動と連動するように仮想空間を体験することが可能となる。

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024158889000001_ABST
    Figure 2024158889000001_ABST
Patent Text Reader

Abstract

To couple a real space with a virtual space so as to interlock with movement of an actual person in the real space.SOLUTION: A space coupling system couples a real space with a virtual space. An actual position in the real space and a virtual position in the virtual space are associated with each other. A first actual position is an actual position of an actual person in the real space or an actual position of a screen that moves following the actual person. A first virtual position is a virtual position associated with the first actual position and changes interlocking with the first actual position. The space coupling system arranges an access point accessible by a virtual person in the virtual space at the first virtual position in the virtual space. Further, the space coupling system displays or projects information on the virtual person or an image of the virtual space around the access point on the screen in the real space.SELECTED DRAWING: Figure 4
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure relates to a technology for linking real space and virtual space. [Background technology]

[0002] Patent Document 1 discloses a communication system in which multiple terminals connected to a network share a virtual space. In a real space, an area terminal is installed for each predetermined area. An image of a virtual character in the virtual area is displayed on the area terminal. Meanwhile, a virtual display is installed in the virtual area. An image of a real space user is displayed on the virtual display. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2009 / 060880 Summary of the Invention [Problem to be solved by the invention]

[0004] Linking real space and virtual space is important for promoting communication, business, etc. Here, a real person in the real space may wish to experience the virtual space while moving freely and in conjunction with his / her own movement, rather than using a device installed in a fixed position.

[0005] One object of the present disclosure is to provide a technology that can link a real space and a virtual space so as to be linked with the movement of a real person in the real space. [Means for solving the problem]

[0006] The first aspect is related to the spatial connection system that connects the real space and the virtual space. A real position in the real space and a virtual position in the virtual space are associated with each other. The spatial linking system comprises one or more processors. The one or more processors obtain a first real position, which is a real position of the real person in real space, or a real position of a screen that moves to follow the real person. The one or more processors acquire a first virtual position, which is a virtual position that is associated with the first real position and changes in conjunction with the first real position. The one or more processors place an access point at a first virtual location within the virtual space that is accessible by a virtual person within the virtual space. The one or more processors display or project information about the virtual person or an image of the virtual space around the access point on a screen in the real space.

[0007] The second aspect relates to a spatial linking method that is executed by a computer and links a real space with a virtual space. A real position in the real space and a virtual position in the virtual space are associated with each other. The spatial linking method is Obtaining a first real position which is a real position of a real person in a real space or a real position of a screen which moves following the real person; acquiring a first virtual position which is a virtual position associated with the first real position and which changes in conjunction with the first real position; disposing an access point at a first virtual location within the virtual space, the access point being accessible by a virtual person within the virtual space; Displaying or projecting information about a virtual person or an image of a virtual space around an access point on a screen in a real space; Includes. Effect of the Invention

[0008] According to the present disclosure, a real position in the real space and a virtual position in the virtual space are associated with each other. A screen that moves following the real person is provided in the real space. Meanwhile, an access point accessible by the virtual person is disposed at a first virtual position associated with the real person or the real position of the screen. Information about the virtual person or an image of the virtual space 20 around the access point is displayed or projected on the screen in the real space. When the real person moves in the real space, the access point also moves in the virtual space in conjunction with the movement of the real person. And when the access point moves in the virtual space, the information displayed or projected on the screen in the real space also changes in conjunction with the movement of the real person. In this way, the real space and the virtual space are linked to each other so as to be associated with the movement of the real person in the real space. This allows the real person to get the feeling that he or she is actually moving in the virtual space. That is, the real person can experience the virtual space in conjunction with his or her own movement while moving freely without using a device installed at a predetermined position. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a conceptual diagram for explaining an example of a real space and a virtual space. [Diagram 2] FIG. 1 is a conceptual diagram for explaining an example of a connection between a real space and a virtual space. [Diagram 3] FIG. 1 is a conceptual diagram for explaining an example of a connection between a real space and a virtual space. [Figure 4] FIG. 1 is a schematic diagram showing a configuration example of a spatial connection system. [Diagram 5] FIG. 2 is a schematic diagram showing a first example of a configuration of the surroundings of a real person in real space. [Figure 6] FIG. 2 is a schematic diagram showing a second example of a configuration of the surroundings of a real person in real space. [Figure 7] FIG. 13 is a schematic diagram showing a third example of a configuration of the surroundings of a real person in real space. [Figure 8] FIG. 2 is a block diagram showing a configuration example of an information processing system included in the spatial connection system. [Figure 9]FIG. 11 is a conceptual diagram illustrating a first example of an access point placed in a virtual space. [Figure 10] FIG. 11 is a conceptual diagram illustrating a second example of an access point placed in a virtual space. [Figure 11] 11 is a flowchart illustrating an example of processing by the information processing system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Embodiments of the present disclosure will be described with reference to the accompanying drawings.

[0011] 1. Example of linking real and virtual spaces FIG. 1 is a conceptual diagram for explaining an example of a real space 10 and a virtual space 20. A certain area AR0 exists in the real space 10. For example, the area AR0 is a city, a building, etc. The area AR0 is reproduced in the virtual space 20 by DigitalTwin technology or the like. Meanwhile, an area AR1 exists at a position distant from the area AR0 in the real space 10. The area AR1 may be outdoors or indoors. A real person 11 in the area AR1 can indirectly experience the distant area AR0 by experiencing the virtual space 20.

[0012] Here, the real person 11 (user) may wish to feel as if he or she is actually moving within the virtual space 20. In other words, the real person 11 may wish to experience the virtual space 20 while moving freely and in conjunction with his or her own movement, rather than using a device installed at a predetermined position. In order to meet such needs, this embodiment provides a technique that can link the real space 10 and the virtual space 20 in conjunction with the movement of the real person 11 in the real space 10.

[0013] FIG. 2 is a conceptual diagram for explaining an example of a connection between the real space 10 and the virtual space 20. As shown in FIG.

[0014] In the real space 10, a drone 12A flies near a real person 11 and moves following the real person 11. The drone 12A is equipped with a water tank and forms a mist screen 13A by spraying mist. The mist screen is a well-known technology and is disclosed, for example, at "https: / / www.seiko-giken.jp / solution / screen". The drone 12A can stably form a cone-shaped mist screen 13A by using rotation. In particular, the drone 12A forms a cone-shaped mist screen 13A so as to cover the periphery of the real person 11. Such a mist screen 13A forms a space dedicated to the real person 11. When the real person 11 moves, the drone 12A moves following the real person 11, and therefore the mist screen 13A also moves following the real person 11.

[0015] In the following description, "real position" means a position in real space 10, and "virtual position" means a position in virtual space 20. The real positions in real space 10 and the virtual positions in virtual space 20 are associated with each other in advance. The "first real position" is the real position of real person 11. The "first virtual position" is a virtual position associated with the first real position. Therefore, when the first real position changes, the first virtual position also changes in conjunction with the first real position.

[0016] An avatar 23A of the real person 11 is placed at a first virtual position in the virtual space 20. The facial expression and posture of the real person 11 may be reflected in the avatar 23A. Since the first real position and the first virtual position are associated with each other, when the real person 11 moves in the real space 10, the avatar 23A of the real person 11 also moves in the virtual space 20 in conjunction with the movement of the real person 11.

[0017] The drone 12A is equipped with a projector 15A. The projector 15A can project an image (video) from inside the mist screen 13A toward the mist screen 13A. The image projected onto the mist screen 13A is an image of the surroundings of the avatar 23A in the virtual space 20. Preferably, the image projected onto the mist screen 13A is an image of the virtual space 20 seen from the line of sight of the avatar 23A.

[0018] When the real person 11 moves in the real space 10, the avatar 23A of the real person 11 also moves in the virtual space 20 in conjunction with the movement of the avatar 23A in the virtual space 20. When the avatar 23A moves in the virtual space 20, the image of the virtual space 20 projected on the mist screen 13A around the real person 11 also changes in conjunction with the movement of the avatar 23A. This allows the real person 11 to get the feeling as if he or she is actually moving in the virtual space 20. In other words, the real person 11 can experience the virtual space 20 while moving freely in conjunction with his or her own movement without using a device installed in a predetermined position.

[0019] 3 is a conceptual diagram for explaining communication between a real person 11 in the real space 10 and a virtual person 21 in the virtual space 20. In the virtual space 20, there exist many virtual persons 21 (avatars) in addition to the avatar 23A of the real person 11. The avatar 23A of the real person 11 in the virtual space 20 can be the "starting point" of communication.

[0020] For example, the virtual person 21 gives a cue to communication. Specifically, in the virtual space 20, the virtual person 21 accesses the avatar 23A of the real person 11. In response to this access, information about the virtual person 21 who has accessed the avatar 23A is projected onto the mist screen 13A around the real person 11 in the real space 10. Typically, an image of the virtual person 21 is projected onto the mist screen 13A. Then, the real person 11 and the virtual person 21 start communicating with each other. The communication may be performed by voice or by text.

[0021] As another example, the real person 11 may provide an opportunity for communication. Specifically, one or more virtual persons 21 around the avatar 23A in the virtual space 20 are shown in the image projected on the mist screen 13A. The real person 11 designates one of the virtual persons 21 shown in the image on the mist screen 13A as a communication partner. When designating a communication partner, the real person 11 may touch or point at the virtual person 21 on the mist screen 13A. The designated virtual person 21 is notified of the designation. Then, the real person 11 and the virtual person 21 start communication. The communication may be performed by voice or by text.

[0022] As described above, according to this embodiment, the real space 10 and the virtual space 20 are linked together so as to be linked to the movement of the real person 11 in the real space 10. This enables the real person 11 to experience the virtual space 20 while moving freely so as to be linked to his or her own movement, without using a device installed at a predetermined position.

[0023] 2 and 3, the real person 11 is reproduced by the avatar 23A in the virtual space 20, while the virtual space 20 or the virtual person 21 is reproduced on the mist screen 13A in the real space 10. In that sense, it can be said that the real space 10 and the virtual space 20 are partially overlapped. The overlapping portion of the real space 10 and the virtual space 20 serves as a window for communication between the real person 11 and the virtual person 21. In other words, the real person 11 and the virtual person 21 can communicate with each other via the overlapping portion of the real space 10 and the virtual space 20.

[0024] 2 and 3 are merely examples of the connection between the real space 10 and the virtual space 20. Various examples of the connection between the real space 10 and the virtual space 20 will be described later. Furthermore, the virtual space 20 does not necessarily have to represent the area AR0 in the real world. The virtual space 20 may represent a fictional world.

[0025] The "spatial linking system 1" that links the real space 10 and the virtual space 20 will be described in detail below.

[0026] 2. Spatial connection system 4 is a schematic diagram showing a configuration example of the spatial connection system 1 according to the present embodiment. The spatial connection system 1 connects a real space 10 and a virtual space 20. Connecting the real space 10 and the virtual space 20 contributes to the promotion of communication, business, and the like.

[0027] 2-1. Composition of the real person's surroundings The spatial connection system 1 includes a follow object 12, a screen 13, a camera 14, a projector 15, a speaker 16, and a microphone 17 in a real space 10. The follow object 12 is configured to follow the real person 11. The screen 13 is provided to follow the follow object 12. As a result, the screen 13 moves to follow the real person 11. The camera 14 captures images of the real person 11 and its surroundings. The projector 15 is provided so as to be able to project various images onto the screen 13. The speaker 16 outputs various sounds toward the real person 11. The microphone 17 detects the sounds of the real person 11.

[0028] Various specific examples of configurations of the surroundings of the real person 11 will be described below.

[0029] 2-1-1. First example FIG. 5 is a schematic diagram showing a first example of the configuration of the surroundings of the real person 11. The tracking object 12 is a drone 12A. The screen 13 is a mist screen 13A. The camera 14 includes a camera 14A mounted on the drone 12A and capturing the surroundings of the drone 12A. The camera 14 may also include an infrastructure camera 14D. The projector 15 is a projector 15A mounted on the drone 12A. The speaker 16 includes a directional speaker 16A mounted on the drone 12A. The microphone 17 includes a directional microphone 17A mounted on the drone 12A. The real person 11 may also wear a headset 18 including the speaker 16 and the microphone 17.

[0030] The drone 12A has a self-location estimation function. The drone 12A may estimate its own location by using a Global Navigation Satellite System (GNSS). The drone 12A may be configured to be capable of autonomous flight.

[0031] The drone 12A moves while following the real person 11. For example, the drone 12A acquires a camera image captured by the camera 14A. A target image, which is an image of the real person 11 to be followed, is registered in advance. The drone 12A recognizes the real person 11 appearing in the camera image based on the target image. A machine learning model is used for person recognition. The drone 12A can follow the real person 11 by tracking the real person 11 recognized in the camera image.

[0032] As another example, the drone 12A may obtain information on the absolute position of the real person 11. The information on the absolute position of the real person 11 may be provided from an external device outside the drone 12A. The drone 12A can track the real person 11 based on its own position obtained by the self-position estimation function and the absolute position of the real person 11.

[0033] The mist screen 13A is formed by mist sprayed from the drone 12A (see Figs. 2 and 3). Therefore, the mist screen 13A moves following the drone 12A. The drone 12A can stably form the conical mist screen 13A by using rotation. In particular, the drone 12A forms the conical mist screen 13A so as to cover the periphery of the real person 11. Such a mist screen 13A forms a space exclusively for the real person 11.

[0034] The projector 15A projects an image (video) from inside the mist screen 13A towards the mist screen 13A.

[0035] The directional speaker 16A is realized, for example, by a very small area audio spot technology (http: / / www.activeforall.jp / project / project02 / ) that uses ultrasonic waves.

[0036] 2-1-2. Second example 6 is a schematic diagram showing a second example of the configuration around a real person 11. Other than the screen 13 and the projector 15, it is the same as the first example described above. Descriptions that overlap with the first example described above will be omitted as appropriate.

[0037] In the second example, the screen 13 is a screen 13B that is suspended from the drone 12A. Because the screen 13B is suspended from the drone 12A, the screen 13B moves following the drone 12A.

[0038] For example, the screen 13B is a cloth screen that hangs from the drone 12A. In this case, the projector 15B mounted on the drone 12A projects an image (video) onto the screen 13B.

[0039] As another example, the screen 13B may be a screen of a display device. Examples of the display device include a liquid crystal display and an organic EL display. The display device may be a touch panel. The display device displays an image (video) on the screen 13B (screen). In this case, the projector 15B is not required.

[0040] 2-1-3.Third example FIG. 7 is a schematic diagram showing a third example of the configuration of the surroundings of the real person 11. The object to be followed 12 is a robot 12C. The screen 13 is a screen 13C installed on the robot 12C. The camera 14 includes a camera 14C mounted on the robot 12C and capturing the surroundings of the robot 12C. The camera 14 may also include an infrastructure camera 14D. The projector 15 is a projector 15C mounted on the robot 12C. The speaker 16 includes a directional speaker 16C mounted on the robot 12C. The microphone 17 includes a directional microphone 17C mounted on the robot 12C. The real person 11 may also wear a headset 18 including the speaker 16 and the microphone 17.

[0041] The robot 12C has a self-location estimation function. The robot 12C may estimate its own location by using GNSS. The robot 12C may be configured to be capable of autonomous travel.

[0042] The robot 12C moves by following the real person 11. For example, the robot 12C acquires a camera image captured by the camera 14C. A target image, which is an image of the real person 11 to be followed, is registered in advance. The robot 12C recognizes the real person 11 appearing in the camera image based on the target image. A machine learning model is used for person recognition. The robot 12C can follow the real person 11 by tracking the real person 11 recognized in the camera image.

[0043] As another example, the robot 12C may obtain information on the absolute position of the real person 11. The information on the absolute position of the real person 11 may be provided from an external device outside the robot 12C. The robot 12C can follow the real person 11 based on its own position obtained by a self-position estimation function and the absolute position of the real person 11.

[0044] The screen 13C is attached to the robot 12C and moves following the robot 12C. For example, the screen 13C is a screen of a display device. Examples of the display device include a liquid crystal display and an organic EL display. The display device may be a touch panel. The display device displays an image (video) on the screen 13C (screen).

[0045] As another example, the screen 13C is a screen that is erected on the upper part of the robot 12C. In this case, a projector 15C mounted on the robot 12C projects an image (video) onto the screen 13C.

[0046] 2-1-4. Fourth Example In the fourth example, the object to be tracked 12 is a terminal (e.g., a smartphone) carried by the real person 11. The terminal carried by the real person 11 automatically moves to follow the real person 11. The screen 13 is a display screen of the terminal. The camera 14, speaker 16, and microphone 17 are mounted on the terminal. In addition, the terminal acquires its own position information by using GNSS or the like.

[0047] 2-2. Information Processing System As shown in FIG. 4, the spatial connection system 1 further includes an information processing system 100 that executes various types of information processing.

[0048] 2-2-1. Overview The information processing system 100 directly or indirectly communicates with the object 12, the screen 13, the camera 14, the projector 15, the speaker 16, and the microphone 17 in the real space 10. Furthermore, the information processing system 100 directly or indirectly controls the object 12, the screen 13, the camera 14, the projector 15, the speaker 16, and the microphone 17 in the real space 10.

[0049] For example, the information processing system 100 controls the object 12 to be followed. The information processing system 100 may include a control device mounted on the object 12 to be followed. That is, at least a part of the information processing system 100 may be included in the object 12 to be followed. When the screen 13 is installed on the object 12 to be followed, the information processing system 100 may indirectly control the screen 13 by controlling the object 12 to be followed. When the camera 14 is mounted on the object 12 to be followed, the information processing system 100 may indirectly control the camera 14 by controlling the object 12 to be followed, and may also acquire a camera image acquired by the camera 14 via the object 12 to be followed. When the projector 15 is mounted on the object 12 to be followed, the information processing system 100 may indirectly control the projector 15 by controlling the object 12 to be followed. When the speaker 16 is mounted on the object 12 to be followed, the information processing system 100 may indirectly control the speaker 16 by controlling the object 12 to be followed. When the microphone 17 is mounted on the following object 12, the information processing system 100 may indirectly control the microphone 17 by controlling the following object 12, and may also acquire audio information detected by the microphone 17 via the following object 12.

[0050] The information processing system 100 also includes a virtual space simulation system 200. The virtual space simulation system 200 performs a simulation of a virtual space 20.

[0051] The virtual space 20 is configured to reproduce, for example, an area AR0 in the real space 10 (see FIG. 1). For example, the area AR0 is a city, a building, etc. A sensor group 300 for detecting the situation of the area AR0 is arranged in the area AR0. The situation of the area AR0 includes the situation of people and moving objects existing in the area AR0. The virtual space simulation system 200 communicates with the sensor group 300 and acquires information regarding the situation of the area AR0 detected by the sensor group 300. Then, the virtual space simulation system 200 reproduces the area AR0 in the virtual space 20 by DigitalTwin technology.

[0052] The virtual space 20 does not necessarily have to represent the area AR0 in the real world. The virtual space 20 may represent a fictional world.

[0053] The user device 400 is a device used by a user other than the real person 11 to participate in the virtual space 20. Examples of the user device 400 include user terminals such as a PC and a smartphone. The user device 400 may be a wearable device. The user device 400 includes an input device and an output device. Examples of the input device include a keyboard, a touch panel, a mouse, a microphone, and the like. Examples of the output device include a display device, a speaker, and the like. The user sets a virtual person 21, which is an alter ego of the user, in the virtual space 20 by operating the input device of the user device 400. The virtual person 21 may be the user's avatar. The user also operates the input device of the user device 400 to move the virtual person 21 in the virtual space 20. The microphone detects the user's voice. The display device presents various information to the user. The speaker outputs various sounds to the user.

[0054] The virtual space simulation system 200 communicates with the user device 400. The virtual space simulation system 200 reflects information input by a user to an input device of the user device 400 in the virtual space 20. The virtual space simulation system 200 also outputs various types of information related to the virtual space 20 from an output device of the user device 400.

[0055] 2-2-2. Configuration example FIG. 8 is a block diagram showing a configuration example of an information processing system 100. The information processing system 100 includes one or more processors 101 (hereinafter simply referred to as the processor 101), one or more storage devices 102 (hereinafter simply referred to as the storage device 102), and an interface 103. The processor 101 executes various processes. For example, the processor 101 includes a CPU (Central Processing Unit). The storage device 102 stores various information. Examples of the storage device 102 include a volatile memory, a non-volatile memory, a HDD (Hard Disk Drive), and an SSD (Solid State Drive). The interface 103 is an interface for communicating with various configurations (a tracking object 12, a screen 13, a camera 14, a projector 15, a speaker 16, a microphone 17, a sensor group 300, a user device 400, and the like). The processor 101 can communicate with the various configurations via the interface 103.

[0056] The control program 104 is a computer program executed by the processor 101. The functions of the information processing system 100 are realized by cooperation between the processor 101, which executes the control program 104, and the storage device 102. The control program 104 is stored in the storage device 102. Alternatively, the control program 104 may be recorded in a computer-readable recording medium. The control program 104 may be provided via a network.

[0057] The storage device 102 stores real space information 110 related to the real space 10, virtual space information 120 related to the virtual space 20, and position correspondence information 130. The position correspondence information 130 indicates a predetermined correspondence relationship between a real position in the real space 10 and a virtual position in the virtual space 20.

[0058] The real space information 110 includes real person information 111, following object information 112, screen information 113, image 114, and projector information 115. The image 114 is captured by the camera 14. The camera 14 includes a camera 14 (14A, 14C: see Figs. 5 to 7) mounted on the following object 12, and an infrastructure camera 14D.

[0059] The real person information 111 is information related to the real person 11. For example, the real person information 111 includes registration information of the real person 11. For example, the registration information includes ID information of the real person 11, a target image which is an image of the real person 11, information on the avatar 23A of the real person 11, etc. The real person 11 may register the registration information in advance in the information processing system 100 by operating a terminal (not shown).

[0060] Moreover, the real person information 111 includes state information indicating the position (absolute position), moving direction, posture, etc. of the real person 11. For example, an image 114 of a range including the real person 11 is captured by a camera 14 mounted on the following object 12. Moreover, the following object 12 estimates its own position by a self-position estimation function. The processor 101 acquires the image 114 captured by the camera 14 and the position information of the following object 12. The processor 101 recognizes the real person 11 appearing in the image 114 based on a target image registered in advance. A machine learning model is used for person recognition. The processor 101 calculates the position (absolute position) of the real person 11 by combining the position (absolute position) of the following object 12 and the recognition result of the real person 11. Moreover, the processor 101 calculates the moving direction of the real person 11 from the position history of the real person 11. Furthermore, the processor 101 recognizes the posture of the real person 11 based on the recognition result of the real person 11 in the image 114.

[0061] As another example, an image 114 of a range including the real person 11 is captured by the infrastructure camera 14D. The infrastructure camera 14D also provides installation information indicating its own installation position and installation orientation. The processor 101 acquires the image 114 captured by the infrastructure camera 14D and the installation information of the infrastructure camera 14D. The processor 101 recognizes the real person 11 appearing in the image 114 based on a target image registered in advance. A machine learning model is used for person recognition. The processor 101 calculates the position (absolute position) of the real person 11 by combining the installation information of the infrastructure camera 14D and the recognition result of the real person 11. The processor 101 also calculates the traveling direction of the real person 11 from the position history of the real person 11. Furthermore, the processor 101 recognizes the posture of the real person 11 based on the recognition result of the real person 11 in the image 114.

[0062] As yet another example, since the object to be followed 12 moves while following the real person 11, the position and the moving direction of the object to be followed 12 may be used approximately as the position and the moving direction of the real person 11. In particular, when the object to be followed 12 is a terminal carried by the real person 11, the position and the moving direction of the terminal are used as the position and the moving direction of the real person 11.

[0063] The object information 112 to be tracked is information related to the object 12 to be tracked. The object information 112 to be tracked includes ID information of the object 12 to be tracked. The object information 112 to be tracked also includes ID information of the real person 11 assigned to the object 12 to be tracked, i.e., the real person 11 to be tracked by the object 12 to be tracked. In other words, the object information 112 to be tracked includes information on the assignment relationship between the real person 11 and the object 12 to be tracked. The assignment relationship between the real person 11 and the object 12 to be tracked is determined in advance by the processor 101. Furthermore, the object information 112 to be tracked includes position information of the object 12 to be tracked. As described above, the position information of the object 12 to be tracked is obtained by the self-location estimation function of the object 12 to be tracked. The processor 101 acquires the position information from the object 12 to be tracked.

[0064] The object to be followed information 112 may further include control information for controlling the object to be followed 12. The processor 101 acquires the position (absolute position) and traveling direction of the real person 11 from the real person information 111. The processor 101 calculates a control amount necessary for the object to be followed 12 to follow the real person 11, based on the position of the object to be followed 12 and the position and traveling direction of the real person 11. The processor 101 then provides the control information including the calculated control amount to the object to be followed 12, thereby controlling the object to be followed 12.

[0065] The screen information 113 is information related to the screen 13. The screen information 113 indicates the position, size, shape, etc. of the screen 13. The position of the screen 13 may be the absolute position of the screen 13, or the relative position of the screen 13 with respect to the object 12 to be followed. When the screen 13 is a mist screen 13A (see FIG. 5), the relative position, size, and shape of the mist screen 13A can be estimated based on the mist spray direction and spray pressure. When the screen 13 is a screen 13B (see FIG. 6) or a screen 13C (see FIG. 7), the relative position, size, and shape of the screen 13 are determined in advance. The absolute position of the screen 13 is obtained by combining the absolute position of the object 12 to be followed and the relative position of the screen 13. When the screen 13 is a screen of a terminal carried by the real person 11, the terminal position is used as the position of the screen 13.

[0066] The projector information 115 is information related to the projector 15. The projector information 115 indicates the installation position and installation orientation of the projector 15 on the object 12 to be tracked.

[0067] The virtual space information 120 is information related to the virtual space 20, and is used particularly by the virtual space simulation system 200. The virtual space information 120 includes virtual person information 121, virtual space configuration information 122, and access point information 123.

[0068] The virtual person information 121 is information related to the virtual person 21. For example, the virtual person information 121 includes registration information of a user who participates in the virtual space 20 via the virtual person 21. For example, the registration information includes ID information of the user, an image of the user, an image of the virtual person 21 (avatar), and the like. The user may operate the user device 400 to register the registration information in advance in the information processing system 100.

[0069] The virtual person information 121 also includes position information of the virtual person 21 in the virtual space 20. The user operates the input device of the user device 400 to move the virtual person 21 in the virtual space 20. The processor 101 calculates the position of the virtual person 21 in the virtual space 20 based on the content of the user's operation of the input device.

[0070] Virtual space configuration information 122 indicates the configuration of virtual space 20. For example, virtual space configuration information 122 indicates the three-dimensional arrangement of structures (e.g., roads, road structures, buildings, etc.) in virtual space 20. In addition, virtual space configuration information 122 indicates the positions of moving objects (e.g., vehicles, robots, etc.) in virtual space 20.

[0071] The access point information 123 is information relating to the "access point 23" placed in the virtual space 20. The access point 23 will be described in detail below.

[0072] 2-3.Access points The processor 101 places an access point 23 accessible by the virtual person 21 within the virtual space 20 .

[0073] More specifically, the processor 101 places the access point 23 at a first virtual position in the virtual space 20. The first virtual position is a virtual position that is associated with a first real position in the real space 10. The first real position is a real position of the real person 11, or a real position of the screen 13 that moves following the real person 11. The real position of the real person 11 is obtained from the real person information 111. The real position of the screen 13 is obtained from the screen information 113. The correspondence between the real positions in the real space 10 and the virtual positions in the virtual space 20 is given by the position correspondence information 130. Thus, the processor 101 can convert the first real position into a first virtual position based on the position correspondence information 130. Then, the processor 101 places the access point 23 at the first virtual position.

[0074] When the real person 11 moves and the first real position changes, the first virtual position also changes in conjunction with the movement of the real person 11. That is, when the real person 11 moves in the real space 10, the access point 23 also moves in the virtual space 20 in conjunction with the movement of the real person 11.

[0075] The access point 23 indicates information about the real person 11 or an image of the real space 10 around the first real position. Some examples of the access point 23 will be described below.

[0076] 2-3-1. First example of access point 9 is a conceptual diagram for explaining a first example of the access point 23. In the first example, the access point 23 is an avatar 23A of the real person 11. The avatar 23A of the real person 11 corresponds to the above-mentioned information of the real person 11. The first real position is the real position of the real person 11 in the real space 10. The first virtual position is a virtual position associated with the real position of the real person 11.

[0077] The processor 101 obtains information on the avatar 23A of the real person 11 from the real person information 111. The processor 101 places the avatar 23A of the real person 11 at a first virtual position in the virtual space 20. When the real person 11 moves in the real space 10, the avatar 23A of the real person 11 also moves in conjunction with the movement of the real person 11 in the virtual space 20. The processor 101 may also apply the posture of the real person 11 to the avatar 23A. The posture of the real person 11 is obtained from the real person information 111.

[0078] The processor 101 displays or projects various information on a screen 13 in the real space 10. For example, the processor 101 projects various information on the screen 13 by controlling a projector 15. When the screen 13 is a screen of a display device, the processor 101 displays various information on the screen 13 (screen) by controlling the display device.

[0079] For example, the processor 101 displays or projects an image of the virtual space 20 around the avatar 23A onto the screen 13. Preferably, the processor 101 displays or projects an image of the virtual space 20 seen from the line of sight of the avatar 23A onto the screen 13. The processor 101 renders the image of the virtual space 20 seen from the line of sight of the avatar 23A based on the position (first virtual position) of the avatar 23A and the virtual space configuration information 122.

[0080] In the virtual space 20, there exist many virtual persons 21 (avatars) in addition to the avatar 23A of the real person 11. The avatar 23A of the real person 11 in the virtual space 20 can be a "starting point" of communication.

[0081] For example, the virtual person 21 operated by the user provides a cue for communication. Specifically, the user operates the input device of the user device 400 to cause the virtual person 21 to access the avatar 23A. For example, when the user clicks on the avatar 23A, the virtual person 21 accesses the avatar 23A.

[0082] In response to the virtual person 21 accessing the avatar 23A, the processor 101 displays or projects information about the virtual person 21 on the screen 13 in the real space 10. For example, the processor 101 displays or projects an image of the virtual person 21 or an image of the user operating the virtual person 21 on the screen 13. Such information about the virtual person 21 is obtained from the virtual person information 121.

[0083] Then, the real person 11 and the virtual person 21 start communication. The communication may be performed by voice or text. For example, the processor 101 acquires the voice of the real person 11 from the microphone 17 and outputs the voice of the real person 11 from the speaker of the user device 400. The processor 101 also acquires the user's voice from the microphone of the user device 400 and outputs the user's voice from the speaker 16.

[0084] In this way, by placing the avatar 23A of the real person 11 in the virtual space 20, communication is promoted.

[0085] As another example, the real person 11 may give a chance to communicate. As described above, an image of the virtual space 20 around the avatar 23A is displayed or projected on the screen 13. One or more virtual persons 21 around the avatar 23A are shown in the image. The real person 11 designates one of the virtual persons 21 shown in the image on the screen 13 as a communication partner. When designating a communication partner, the real person 11 may touch or point at the virtual person 21 on the screen 13. Such a designation action by the real person 11 is detected based on the image 114 captured by the camera 14. Alternatively, if the screen 13 is a touch panel, the designation action by the real person 11 is detected by the touch panel. The processor 101 displays or projects information on the virtual person 21 designated by the real person 11 on the screen 13 in the real space 10. For example, the processor 101 displays or projects an image of the designated virtual person 21 or an image of a user operating the designated virtual person 21 on the screen 13. Such information about the virtual person 21 is obtained from the virtual person information 121. Then, the real person 11 and the virtual person 21 start communicating with each other.

[0086] 2-3-2.Second example of access point 10 is a conceptual diagram for explaining a second example of the access point 23. Descriptions that overlap with the above-mentioned first example will be omitted as appropriate.

[0087] In a second example, the access point 23 is a virtual screen 23B. The first real position may be the real position of the real person 11 or the real position of the screen 13. The size and shape of the virtual screen 23B may be set to be the same as the size and shape of the screen 13 in the real space 10. The size and shape of the screen 13 are obtained from the screen information 113. The processor 101 places the virtual screen 23B at the first virtual position in the virtual space 20. When the real person 11 moves in the real space 10, the virtual screen 23B also moves in the virtual space 20 in conjunction with the movement of the real person 11.

[0088] The processor 101 displays or projects an image of the real space 10 around the first real position on a virtual screen 23B in the virtual space 20. The image of the real space 10 around the first real position is obtained from an image 114 captured by the camera 14.

[0089] Meanwhile, the processor 101 displays or projects an image of the virtual space 20 around the virtual screen 23B onto the screen 13 in the real space 10. The processor 101 renders the image of the virtual space 20 around the virtual screen 23B based on the position of the virtual screen 23B (first virtual position) and the virtual space configuration information 122.

[0090] As a result, the screen 13 in the real space 10 and the virtual screen 23B in the virtual space 20 overlap, like a "transparent window." The real person 11 in the real space 10 can see the state of the virtual space 20 through the screen 13. The virtual person 21 in the virtual space 20 can see the state of the real space 10 through the virtual screen 23B. This form also contributes to promoting communication.

[0091] The method of communication between the real person 11 and the virtual person 21 is the same as in the first example described in section 2-3-1 above.

[0092] 2-4. Processing flow FIG. 11 is a flowchart showing an example of processing by the information processing system 100 (processor 101).

[0093] In step S110, the information processing system 100 acquires real person information 111 related to a real person 11 in the real space 10.

[0094] In step S120, the information processing system 100 acquires a first real position in the real space 10. The first real position is the real position of the real person 11, or the real position of the screen 13 that moves following the real person 11. The real position of the real person 11 is obtained from the real person information 111. The real position of the screen 13 is obtained from the screen information 113.

[0095] In step S130, the information processing system 100 acquires a first virtual position associated with the first real position. The correspondence between the real positions in the real space 10 and the virtual positions in the virtual space 20 is given by position correspondence information 130. Therefore, the information processing system 100 converts the first real position into the first virtual position based on the position correspondence information 130. When the first real position changes, the first virtual position also changes in conjunction with the first real position.

[0096] In step S140, the information processing system 100 places an access point 23 accessible by the virtual person 21 in the virtual space 20 at a first virtual position. The access point 23 indicates information about the real person 11 or an image of the real space 10 around the first real position.

[0097] For example, the access point 23 is an avatar 23A of the real person 11 (see FIG. 9). The avatar 23A of the real person 11 is obtained from the real person information 111. The posture of the real person 11 may be applied to the avatar 23A. The posture of the real person 11 is obtained from the real person information 111.

[0098] As another example, the access point 23 may be a virtual screen 23B (see FIG. 10). The information processing system 100 displays or projects an image of the real space 10 around the first real position on the virtual screen 23B. The image of the real space 10 around the first real position is obtained from an image 114 captured by the camera 14.

[0099] In step S150, the information processing system 100 displays or projects various pieces of information on the screen 13 in the real space 10. For example, the information processing system 100 projects various pieces of information on the screen 13 by controlling the projector 15. When the screen 13 is a screen of a display device, the information processing system 100 displays various pieces of information on the screen 13 (screen) by controlling the display device.

[0100] The information displayed or projected on the screen 13 is, for example, information on a virtual person 21 who is a communication partner. Examples of the information on the virtual person 21 include an image (avatar) of the virtual person 21, an image of the user, etc. Such information on the virtual person 21 is obtained from virtual person information 121.

[0101] As another example, the information displayed or projected on the screen 13 may be an image of the virtual space 20 around the access point 23. The image of the virtual space 20 around the access point 23 is an image of the virtual space 20 seen from a first virtual position (the position of the access point 23). The processor 101 renders the image of the virtual space 20 seen from the first virtual position, based on the first virtual position and the virtual space configuration information 122.

[0102] The processor 101 may determine the size of an image to be displayed or projected on the screen 13, taking into account the distance between the real person 11 and the screen 13, so that the sense of distance when viewed from the real person 11 is appropriate. The position of the real person 11 is obtained from the real person information 111. The position of the screen 13 is obtained from the screen information 113. The processor 101 can calculate the distance between the real person 11 and the screen 13 based on the positions of the real person 11 and the screen 13, and determine an appropriate image size.

[0103] 3.Effects According to this embodiment, a real position in the real space 10 and a virtual position in the virtual space 20 are associated with each other. A screen 13 that moves following the real person 11 is provided in the real space 10. Meanwhile, an access point 23 accessible by the virtual person 21 is disposed at a first virtual position associated with the real position of the real person 11 or the screen 13. Information on the virtual person 21 or an image of the virtual space 20 around the access point 23 is displayed or projected on the screen 13 in the real space 10. When the real person 11 moves in the real space 10, the access point 23 also moves in the virtual space 20 in conjunction with the movement. And when the access point 23 moves in the virtual space 20, the information displayed or projected on the screen 13 in the real space 10 also changes in conjunction with the movement.

[0104] In this way, the real space 10 and the virtual space 20 are linked together so as to be linked to the movement of the real person 11 in the real space 10. This allows the real person 11 to have the sensation as if he or she were actually moving within the virtual space 20. In other words, the real person 11 can experience the virtual space 20 while moving freely so as to be linked to his or her own movement, without using a device installed at a predetermined position.

[0105] According to this embodiment, the real space 10 or the real person 11 is reproduced at the access point 23 in the virtual space 20, while the virtual space 20 or the virtual person 21 is reproduced on the screen 13 in the real space 10. In that sense, it can be said that the real space 10 and the virtual space 20 are partially overlapped.

[0106] The overlapping portion between the real space 10 and the virtual space 20 serves as a window for communication between the real person 11 and the virtual person 21. In other words, the real person 11 and the virtual person 21 can communicate with each other through the overlapping portion between the real space 10 and the virtual space 20. [Explanation of symbols]

[0107] 1...spatial connection system, 10...real space, 11...real person, 12...following object, 13...screen, 14...camera, 20...virtual space, 21...virtual person, 23...access point, 23A...avatar, 23B...virtual screen, 100...information processing system, 200...virtual space simulation system

Claims

1. A spatial connection system that connects a real space and a virtual space, a real position in the real space and a virtual position in the virtual space are associated with each other; the spatial linking system comprising one or more processors; The one or more processors: acquiring a first real position which is the real position of the real person in the real space or the real position of a screen which moves following the real person; acquiring a first virtual position, the first virtual position being associated with the first real position and changing in conjunction with the first real position; disposing an access point at the first virtual location in the virtual space that is accessible by a virtual person in the virtual space; Information about the virtual person or an image of the virtual space around the access point is displayed or projected on the screen in the real space. It was configured as Spatial connection system.

2. 2. The spatial connection system of claim 1, The access point indicates information about the real person or an image of the real space around the first real location. Spatial connection system.

3. 3. The spatial connection system according to claim 2, In response to the virtual person accessing the access point, the one or more processors display or project the information of the virtual person on the screen so that the real person and the virtual person can communicate with each other. Spatial connection system.

4. 3. The spatial connection system according to claim 2, The one or more processors display or project the image of the virtual space around the access point on the screen; When the real person designates the virtual person appearing in the image on the screen, the one or more processors display or project the information of the designated virtual person on the screen so that the real person and the virtual person can communicate with each other. Spatial connection system.

5. 3. The spatial connection system according to claim 2, the first real position is the real position of the real person in the real space; the first virtual position is a virtual position associated with the real position of the real person in the real space, the access point is an avatar of the real person; The one or more processors place the avatar of the real person at the first virtual location within the virtual space. Spatial connection system.

6. 6. The spatial connection system of claim 5, The one or more processors further include: acquiring information about a posture of the real person in the real space; Applying the pose to the avatar of the real person Spatial connection system.

7. 3. The spatial connection system according to claim 2, the access point is a virtual screen; The one or more processors: displaying or projecting the image of the real space around the first real position onto the virtual screen in the virtual space; The image of the virtual space surrounding the virtual screen is displayed or projected onto the screen in the real space. Spatial connection system.

8. A spatial connection system according to any one of claims 1 to 7, The one or more processors further control a following object that moves to follow the real person in the real space; The screen is arranged to follow the follow object. Spatial connection system.

9. 9. The spatial connection system of claim 8, the tracking object comprises a projector; The one or more processors control the projector to project the information of the virtual person or the image of the virtual space around the access point onto the screen. Spatial connection system.

10. 9. The spatial connection system of claim 8, The following object is a drone Spatial connection system.

11. 11. The spatial connection system of claim 10, The screen is a mist screen formed by mist sprayed from the drone, The mist screen is formed to surround the real person. Spatial connection system.

12. A spatial connection method for connecting a real space and a virtual space, which is executed by a computer, comprising: a real position in the real space and a virtual position in the virtual space are associated with each other; The spatial concatenation method includes: acquiring a first real position which is the real position of a real person in the real space or the real position of a screen which moves following the real person; acquiring a first virtual position, the virtual position being associated with the first real position and changing in conjunction with the first real position; placing an access point at the first virtual location within the virtual space, the access point being accessible by a virtual person within the virtual space; displaying or projecting information about the virtual person or an image of the virtual space around the access point on the screen in the real space; Includes Spatial connection method.

Citation Information

Patent Citations

  • System, program, method, and information processing device

    JP2020004284A

  • Information processing system, information processing program, information processor, and method for processing information

    JP2022079362A

  • Composite reality providing system and composite reality providing method

    JP2022173028A

  • A program and method executed by a computer that provides a virtual space, and an information processing device that executes the program

    JP6392945B1

  • Communication system, method, and program

    WO2009060880A1