Method for updating virtual space, server device, and system

Abstract

This server device updates spatial data 25 about a virtual space having coordinates corresponding to the position of a real space in which time information 7b and position information 7a are acquired, the spatial data 25 being updated on the basis of image information 8a obtained from a user terminal 27 of a visitor of the real space. The server device comprises: a site information acquisition unit 3 for acquiring site information 3a composed of space-time information 7c, image information 8a corresponding to the space-time information, and distance information about the distance to an object corresponding to the image information; and an update unit for updating the spatial data 25 on the basis of the site information. The site information may be of an extremely large amount. The immediacy of the generation / update of the virtual space is achieved by controlling the depth of imaging data under limitations due to the computational capability of the server device and the communication amount from a site to the server device.

Description

Virtual space update method, server device, and system

[0001] The present invention relates to spatial data of a virtual space that allows visitors who visit a real space and virtual visitors who visit a virtual space to interact through the virtual space.

[0002] In recent years, many technologies using virtual spaces have been developed. These technologies provide users with new experiences through virtual spaces. For example, a technology called digital twin acquires information from the real world, reproduces the environment of the real world, and creates a virtual space that is paired with the real world.

[0003] Patent Literature 1 discloses a server device used in a communication method via a virtual farm and a real farm. The server device includes: a personal information acquisition means for acquiring pre-registered personal information based on identification information for identifying a visitor to the farm or a virtual visitor of the virtual farm, to be transmitted to a user terminal of a virtual visitor visiting the virtual farm; a virtual space processing means for transmitting a viewpoint image of the virtual visitor moving around the virtual farm based on the operation of the virtual visitor to the user terminal of the virtual visitor; and a coordinate acquisition means for acquiring position information of the visitor. The virtual space processing means displays a pre-set avatar display element of the visitor at a coordinate position in the virtual farm corresponding to the position information.

[0004] Patent No. 7300220

[0005] In a digital twin, a virtual space exists that is paired with the real space, so it is advantageous to incorporate changes in the real space over time into the virtual space. For example, in a virtual farm / virtual event space, if visitors in the real space and virtual visitors in the virtual space can simultaneously share the same or similar experiences in the same or similar scenery, it can encourage empathy between them. However, incorporating all changes in the real space into the virtual space is cumbersome, and there are limits to the processing speed of the server device used for calculations, as well as the amount of data that can be transmitted per hour.

[0006] Therefore, an object of the present invention is to provide a method for updating spatial data of a virtual space in which visitors in the real space and virtual visitors in the virtual space interact through the virtual space, a server device for updating spatial data of a virtual space, and a system for updating spatial data of a virtual space.

[0007] (1) The method for updating a virtual space of the present invention is executed by a server device having an update unit and a visitor terminal having a location information acquisition unit, an image information acquisition unit, and a distance information acquisition unit and held by a visitor visiting an event area in real space, and is a method for updating spatial data of a virtual space having coordinates corresponding to a position in the real space, wherein the updated event area of ​​the virtual space is displayed on a display unit of the virtual visitor terminal that is communicatively connected to the visitor terminal and held by the virtual visitor visiting the virtual space, and the method comprises the steps of: the visitor terminal acquiring local information consisting of time-space information and location information acquired by the location information acquisition unit, image information corresponding to the time-space information acquired by the image information acquisition unit, and distance information from the visitor terminal to an object corresponding to the image information acquired by the distance information acquisition unit; and the update unit updating the spatial data corresponding to the local information among the spatial data based on the local information.

[0008] (2) In such a method for updating a virtual space, it is preferable that the distance information acquisition unit acquires the distance information using a Lidar sensor. (3) It is also preferable that the server device includes a determination unit, and the determination unit includes a step of determining whether or not there is a difference between the spatial data corresponding to the local information and existing spatial data, and if there is a difference, updating the part of the existing spatial data where the difference exists.

[0009] (4) Another aspect of the virtual space updating method of the present invention is a method for updating spatial data of a virtual space having coordinates corresponding to a position in the real space, which is executed by a server device having an update unit and an adjustment unit, and a visitor terminal having a position information acquisition unit, an image information acquisition unit, and a distance information acquisition unit and held by a visitor in the real space, and is characterized in that the method includes the steps of: the visitor terminal acquiring on-site information consisting of spatiotemporal information consisting of time information and position information acquired by the position information acquisition unit, image information corresponding to the spatiotemporal information acquired by the image information acquisition unit, and distance information from the visitor terminal to an object corresponding to the image information acquired by the distance information acquisition unit; the update unit updating the spatial data based on the on-site information; and the adjustment unit adjusting, before the on-site information acquisition step, to reduce the distance to be acquired by the distance information acquisition unit and / or the range for acquiring the distance information when processing information regarding the processing speed of the server device and / or communication information regarding the communication speed between the server device and the visitor terminal decreases.

[0010] (5) Preferably, the spatial data is updated at predetermined intervals by a peripheral camera installed in the real space. (6) Preferably, the peripheral camera updates the spatial data using a Lidar sensor.

[0011] (7) A server device according to yet another aspect of the present invention is a server device that updates spatial data in a virtual space having coordinates corresponding to positions in real space, and is characterized in that it comprises: a local information acquisition unit that acquires local information obtained from a visitor terminal held by a visitor to an event area in the real space, the local information being time-space information consisting of time information and position information, image information corresponding to the time-space information, and distance information from the visitor terminal to an object corresponding to the image information; and an update unit that updates the spatial data corresponding to the local information based on the local information, and is communicatively connected to the visitor terminal, and displays the updated event area of ​​the virtual space on the display unit of the virtual visitor terminal held by a virtual visitor visiting the virtual space.

[0012] (8) In such a server device, it is preferable that the distance information is acquired using a Lidar sensor. (9) It is also preferable that the server device includes a determination unit that determines whether there is a difference between the spatial data corresponding to the local information and existing spatial data, and if there is a difference, updates the part of the existing spatial data that has the difference.

[0013] (10) A server device according to yet another aspect of the present invention is a server device that updates spatial data in a virtual space having coordinates corresponding to positions in the real space, and is characterized by comprising: a local information acquisition unit that acquires local information that is obtained from a visitor terminal held by a visitor in the real space and that includes spatiotemporal information consisting of time information and position information, image information corresponding to the spatiotemporal information, and distance information from the visitor terminal to an object corresponding to the image information; an update unit that updates the spatial data based on the local information; and an adjustment unit that adjusts the distance to be acquired and / or the range within which the distance information is acquired when processing information regarding its own processing speed and / or communication information regarding the communication speed with the visitor terminal decreases.

[0014] (11) Preferably, the spatial data is updated at predetermined time intervals by a peripheral camera installed in the real space. (12) Preferably, the peripheral camera updates the spatial data using a Lidar sensor.

[0015] (13) A further aspect of the system of the present invention is a system that updates spatial data of the virtual space having coordinates corresponding to positions in the real space, the system comprising a server device, a visitor terminal held by a visitor visiting an event area in real space, and a virtual visitor terminal held by a virtual visitor visiting a virtual space, wherein the visitor terminal comprises a location information acquisition unit that acquires spatiotemporal information consisting of time information and location information, an image information acquisition unit that acquires image information corresponding to the spatiotemporal information, and a distance information acquisition unit that acquires distance information from the visitor terminal of the visitor to an object corresponding to the image information, the server device comprises an update unit that updates, based on the local information, the spatial data corresponding to local information consisting of the spatiotemporal information, image information, and distance information, the spatial data, and the virtual visitor terminal comprises a display unit that is communicatively connected to the visitor terminal and displays the updated event area of ​​the virtual space.

[0016] "Location information" refers to information about the location of a visitor within the actual event area. This concept includes, for example, coordinates representing a two-dimensional or three-dimensional point, coordinates representing a predetermined area such as a plane or space, and quantities that can be converted into these. For example, it may be information expressed as latitude and longitude, or information expressed with altitude added. Furthermore, "location information" may be information expressed as other index values ​​that can be converted into latitude, longitude, and altitude. If the location being visited is a farm, this is information about the location of a visitor within the actual farm. This concept includes, for example, coordinates representing a two-dimensional or three-dimensional point, coordinates representing a predetermined area such as a plane or space, and quantities that can be converted into these. "Time information" refers to information indicating the time at which the location information was identified. Furthermore, "time information" may be Japanese time, overseas time, or the elapsed time relative to a reference time. Furthermore, "time information" may be information expressed as, for example, time or other index values ​​that can be converted into the elapsed time relative to a reference time. "Image information" refers to image and video data captured by a user device, and includes data that can be converted into such data. "Distance information" refers to information about the distance from the user device to the location captured in the image information, and includes quantities that can be converted into such data. "Coordinate location information" refers to coordinates that indicate two-dimensional or three-dimensional points in virtual space, or coordinates that indicate a specific range, such as a plane or space, and includes quantities that can be converted into such coordinates. "Processing information" refers to processing information related to the processing speed of a server device. Processing information is affected by factors such as the performance of the server device's CPU, available memory space, hard disk read / write speed, the degree of congestion (capacity and load) of the server device, and the PHP (Hypertext Preprocessor) processing method. "Communication information" refers to communication information related to the communication speed between the server device and the user device.

[0017] The spatial data in the virtual space can be easily updated.

[0018] FIG. 1 is a schematic diagram showing an overview of a virtual space update system. FIG. 2 is a functional block diagram showing a virtual space update system. FIG. 3 is a schematic diagram showing the appearance of the display screens of the user terminals of a visitor and a virtual visitor. FIG. 4 is a schematic diagram showing an example of the hardware configuration of a server device. FIG. 5 is a schematic diagram showing an example of the hardware configuration of a user terminal. FIG. 6 is a schematic diagram showing an example of the hardware configuration of a head-mounted display unit. FIG. 7 is a flowchart showing an embodiment showing the flow of an update process. FIG. 8 is a schematic diagram showing the difference in spatial data indicating the data structure of an adjustment amount database.

[0019] [1. Overview] The virtual space updating system of this embodiment reflects (updates) changes in the real space in the virtual space in a digital twin environment. Rather than using pre-recorded spatial data, the system allows virtual visitors to experience phenomena occurring on-site with the help of real visitors. Specifically, the system updates the spatial data based on on-site information obtained from the user device of the real-space visitor, i.e., spatiotemporal information (time and location information), image information, and distance information from the user device to the object in the image information. Updating everything using the visitor's on-site information would increase the amount of data and computation. Therefore, the depth and range of the information to be updated are narrowed. The degree of narrowing is adjusted depending on the computing power and communication speed of the server device / system. Even if the amount of information is narrowed, a practical virtual space in the digital twin can be constructed by combining real-time information on the required parts of the space by multiple visitors in the real space, while also providing a general image of the entire space.

[0020] Furthermore, local information that visitors in the real world pay attention to tends to be concentrated in specific locations and times. This allows for efficient updating of spatial data without updating all of the data.

[0021] A virtual space update system would be good for things like morning markets in tourist spots, fish landed from boats returning to the fishing port, open-air markets, antique and second-hand clothing stores, general merchandise stores, and other things that are sold on the spot, or things where the negotiation between seller and buyer is valuable as an experience, such as bargain sales of bananas.Other things that virtual visitors can experience as part of their sightseeing are things like the aurora borealis and blizzards.

[0022] [2. Overview] (Outline of Virtual Space Update System 10) First, an overview of the virtual space update system will be described using Fig. 1. The virtual space update system (hereinafter referred to as the system) 10 shown in Fig. 1 is primarily used between visitors 12 visiting a market / morning market or the like (event area) 11 and remote persons 13 who are not visiting the event area 11 and who are, for example, at home. The remote persons 13 include virtual visitors 13 who visit an event area 14 (see Fig. 3) in a virtual space made up of three-dimensional spatial data of an imaged space formed from an image of the event area 11 or a pseudo-space formed to simulate the image.

[0023] A virtual visitor 13 visits an event area 14 in the virtual space to, for example, participate in an event, exhibit at the event, view / purchase event-related merchandise 18, or interact with other participants / exhibitors. A virtual visitor 13 can also interact with other virtual visitors 13 who are visiting the event area 14 in the virtual space.

[0024] Retailers selling event-related products 18 include, for example, retailers within the event area 11 and the event area 14 in the virtual space, as well as roadside stations, local shops, department stores, supermarkets, and convenience stores, and are concepts that can be converted into these.

[0025] (Event Area 11) The event area 11 is not particularly limited, but includes, for example, an event venue where a specific event is held, and is an area for events held outdoors and / or indoors. Examples of event venues include street corners, martial arts halls, exhibition halls, art museums, temples, churches, amusement parks, concert halls, gymnasiums, schools, auditoriums, baseball fields, athletic stadiums, parks, farms, ranches, nursing homes, care facilities, and hospitals. Examples of events include expositions such as world expositions, exhibitions, sales events, product fairs, conferences, concerts, plays, music events, sports days, competitions, festivals, fireworks displays, and tourist events. Events also include sightseeing. By linking with event areas in other virtual spaces, it is possible to travel between the different virtual spaces.

[0026] (Event area 14 in virtual space) The event area 14 in virtual space (see Figure 3) is spatial data consisting of three-dimensional spatial data of an imaged space formed from an image of the event area 11 or a pseudo-space formed by calculation using the image and depth (distance information 9a).

[0027] [3. Components] (System 10) Next, the system 10 according to one embodiment of the present invention will be described with reference to the functional block diagram of FIG.

[0028] The system 10 comprises a server device 1, a user terminal 26 (hereinafter referred to as a remote party terminal / virtual visitor terminal) held by a virtual visitor 13, and a user terminal 27 (hereinafter referred to as a visitor terminal) held by a visitor 12. The server device 1, the virtual visitor terminal 26, and the visitor terminal 27 are communicatively connected to each other via a communication network 17 (see FIG. 1) such as the Internet (intranet). As will be described later, reference numeral 20 denotes a head-mounted display system (see FIG. 6). The remote party 13 can also interact with the visitor 12 via text information without visiting the event area 14 in the virtual space.

[0029] (Server Device 1) In this embodiment, for example, a personal computer is used as the server device 1. The server device 1 mainly includes an adjustment unit 2, a local information acquisition unit 3, a determination unit 4, and an update unit 5. The server device 1 further includes a virtual space processing unit 6.

[0030] (Virtual Visitor Terminal 26) In this embodiment, the virtual visitor terminal 26 is, for example, a smartphone. It may also be a wearable computer, a head-mounted display system, a mobile phone, a tablet terminal, or a personal computer (PC), which can be attached or worn. The virtual visitor terminal 26 (20) primarily includes a position information acquisition unit 7 and an image information acquisition unit 8. The virtual visitor terminal 26 (20) may also include a distance information acquisition unit 9. The virtual visitor terminal 26 (20) includes a display unit 26a (20a). Reference numeral 20a denotes the display unit of the head-mounted display system (see FIG. 6). The virtual visitor terminal 26 (20) displays avatar display elements of the visitor 12 in a viewpoint image of a virtual space as virtual reality (VR). The virtual visitor terminal 26 also identifies its current position and orientation using, for example, a gyro sensor.

[0031] (Visitor Terminal 27) In this embodiment, the visitor terminal 27 is almost the same as the virtual visitor terminal 26 (20), so the same parts are denoted by the same reference numerals and their description will be omitted. In this embodiment, the visitor 12 wears a visitor terminal 27 such as smart glasses or AR glasses. The visitor terminal 27 projects pre-registered avatar display elements as digital images onto the field of view of the scenery in the real event area 11 as augmented reality (AR). The display unit 27a of the smart glasses or AR glasses allows the scenery to be viewed through the display and displays a digital image superimposed on it.

[0032] (Location Information Acquisition Unit 7) The location information acquisition unit 7 has a function of acquiring location information 7a and time information 7b of the visitor terminal 27. For example, the location information acquisition unit 7 acquires location information using a GPS (Global Positioning System) sensor. Positioning methods other than GPS sensors, such as Wi-Fi positioning and beacon positioning, may also be used. The location within a room can be identified using Wi-Fi positioning and beacon positioning. The location information 7a and time information 7b constitute spatiotemporal information 7c.

[0033] (Image Information Acquisition Unit 8) The image information acquisition unit 8 is an imaging sensor such as an image sensor or an RGB sensor. For example, it is a conventionally known device equipped with a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor. The imaging sensor receives ambient light with a photodiode, detects RGB values, and obtains a color image. The image information acquisition unit 8 generates image information 8a. The image information acquisition unit 8 may be incorporated into the visitor terminal 27 and integrated therewith, or may be separate from the visitor terminal 27. The image information acquisition unit 8 may also be equipped with a sound collection device (not shown), such as a microphone.

[0034] (Distance information acquisition unit 9) The distance information acquisition unit 9 acquires distance information 9a relating to the distance (depth) to the object in the generated image information 8a. Furthermore, upon receiving the amount of information (adjustment amount) 2a from the adjustment unit 2, the distance information acquisition unit 9 shortens the search depth and reduces (reduces) the amount of information 2a in the distance information 9a. Note that the reduced amount of information 2a may also be restored to its original state. Next, the reduced distance information 9a is transmitted to the server device 1. At that time, the image information 8a corresponding to the reduced distance information 9a is also reduced. As a result, the image information 8a and distance information 9a with a short search depth, for example, of the hand side, with a short search depth, are transmitted to the server device 1. Note that the image information 8a of a portion not linked to the distance information 9a may be transmitted to the server device 1. Distance can be acquired using known techniques, such as a Lidar (Light Detection and Ranging) sensor, an iToF (Indirect Time of Flight) sensor, or a triangulation method using infrared light. Alternatively, a stereo camera may be used. When Visual SLAM is used to acquire spatial data 25 (described later), distance information 9a is acquired by the distance information acquisition unit 9 from two or more pieces of image information 8a. The multiple pieces of image information 8a may be acquired from multiple visitor terminals 27, or may be acquired as image information 8a acquired by a single visitor terminal 27 at different locations. The multiple pieces of image information 8a to be selected are preferably those acquired within a predetermined time range 7b.

[0035] (Spatial Data 25) A known technology can be used to acquire the spatial data 25 (see FIG. 9) for updating. One example is a technology called SLAM (Simultaneous Localization and Mapping). SLAM is a general term for technologies that simultaneously perform "self-location estimation" to estimate one's current location and "environmental mapping" to understand the situation in the surrounding area. SLAM can be categorized into Depth SLAM, Visual SLAM, LiDAR SLAM, etc., depending on the sensors installed. Existing spatial data before updating is generated by a peripheral camera 24 (see FIG. 1) that collects two-dimensional and three-dimensional data.

[0036] (Peripheral camera 24) The peripheral camera 24 generates an image of three-dimensional data, creates a virtual space to be displayed, and stores the image as spatial data in the storage unit 31. In this embodiment, the peripheral camera 24 is fixedly installed in the event area 11. A plurality of peripheral cameras 24 may be installed. In this embodiment, the peripheral camera 24 takes an image of the event area 11 at predetermined times, for example, once a month, and updates the spatial data 25.

[0037] (Adjustment Unit 2) The adjustment unit 2 adjusts the amount of information 2a (see FIG. 2 ) of distance information 9a to be acquired in accordance with processing information related to the processing speed of the server device 1 and / or communication information related to the communication speed between the server device 1 and the visitor terminal 27. The adjustment amount of the distance information acquisition unit 9 may be an amount that can be converted into the amount of information 2a. The adjustment amount is, for example, the distance (depth) to be acquired by the distance information acquisition unit 9 and / or the range of distance information to be acquired. The adjustment unit 2 transmits the amount of information 2a to the visitor terminal 27. Note that the local information acquisition unit 3 may delete unnecessary information to achieve the adjusted amount of information 2a, thereby reducing the amount of calculation by the server device 1.

[0038] (Local Information Acquisition Unit 3) The local information acquisition unit 3 acquires local information 3a including spatiotemporal information 7c, image information 8a corresponding to the spatiotemporal information 7c, and distance information 9a to an object corresponding to the image information 8a. Next, the spatial data based on the local information 3a is aligned with the existing spatial data 25. A known technique can be used for the alignment. For example, feature points may be extracted from the spatial data based on the local information 3a and the existing spatial data 25, and the coordinates of the spatial data may be aligned based on the feature points. As an example, the feature points may be extracted by extracting vertices of a rectangle using the image information 8a.

[0039] The local information acquisition unit 3 also acquires coordinate positions (viewpoint coordinates) in three-dimensional space corresponding to the viewpoint image displayed on the virtual visitor terminal 26 (20). For example, in this embodiment, avatar display elements representing virtual visitors 13 other than the user are displayed on the display unit 26a (20a) based on the viewpoint coordinates. Note that an avatar display element representing the user may also be displayed on the display unit 26a (20a) of the user's own terminal 26 (20).

[0040] (Determination Unit 4) The determination unit 4 determines whether there is a predetermined difference between the spatial data 25 of coordinates corresponding to the local information 3a. If there is a difference, the determination unit 4 updates the different portion 25a of the spatial data 25 (see FIG. 9 ). On the other hand, if there is no difference, the determination unit 4 does not update the spatial data 25. The determination is made using a known technique. As an example, based on the three-dimensional coordinate values ​​(X, Y, Z) and / or color information (R, G, B) as point cloud data of the obtained local information 3a and the existing spatial data 25, the determination unit 4 determines whether there is a difference of a predetermined value or more in the three-dimensional coordinates and / or color information. For example, it determines that there is a difference when there is a difference of a predetermined value or more in multiple values. A predetermined range including the differing coordinates is set as the different portion 25a. Alternatively, the predetermined different portion 25a may be extracted from the spatial data generated from the local information 3a and the existing spatial data 25 using a machine learning learning model.

[0041] (Update Unit 5) The update unit 5 replaces / overwrites the different portion 25a with the corresponding existing spatial data 25 to update it.

[0042] (Virtual Space Processing Unit 6) The virtual space processing unit 6 generates a virtual image to be displayed on the display unit 26a (20a) of the virtual visitor terminal 26 (20) based on operation information 6a relating to the amount of operation output from the input / output device 32 (operation unit 32a) of the virtual visitor terminal 26 (20) (see FIG. 3). For example, in cooperation with the control device 22 of the head-mounted display system 20, it generates images to be displayed on the display unit 20a and performs various calculation processes.

[0043] (Display Units 26a, 27a, 20a) As the display units 26a, 27a, 20a, for example, devices for displaying images such as a liquid crystal display (LCD), a plasma display panel (PDP), or an organic electroluminescence (EL) display may be used. A touch panel may also be provided on the display unit. The touch panel outputs an output signal in response to a user operation. By operating the touch panel, the user can perform operations such as selecting information displayed on the display units 26a, 27a, 20a.

[0044] 3 is a schematic diagram showing the display screens of the user terminals 27, 26 of the visitor 12 and the virtual visitor 13. The avatar display elements 12a, 13a are, for example, two-dimensional or three-dimensional spatial data of captured images formed by capturing images of the visitor 12 and the virtual visitor 13, or pseudo images formed by simulating the captured images. Note that, regardless of the captured images, any desired images may be used as the avatar display elements.

[0045] (Avatar display element 12a of visitor 12) Specifically, the visitor terminal 27 acquires location information 7a within the event area 11 based on GPS or indoor positioning technology, and transmits the location information 7a to the local information acquisition unit 3 of the server device 1. The virtual space processing unit 6 of the server device 1 converts the location information 7a into a coordinate position in the virtual space corresponding to the location information 7a, and transmits the avatar display element 12a of the visitor 12 to be displayed at that coordinate position to the virtual visitor terminal 26 (20).

[0046] (Avatar display element 13a of virtual visitor (remote person) 13) Meanwhile, the server device 1 acquires coordinate position information relating to the virtual space coordinate position of the virtual visitor 13. The virtual space processing unit 6 transmits to the visitor terminal 27 a preset avatar display element 13a of the virtual visitor to be displayed superimposed on the field of view of the visitor 12 through the visitor terminal 27 at a position in the event area 11 corresponding to the acquired coordinate position information.

[0047] 3, we will explain the display screens of the virtual visitor terminal 26 (20) and the visitor terminal 27. The upper part of the display screen shown in the figure is the display screen of the display unit 26a of the virtual visitor terminal 26 (20), and the lower part is the display screen of the display unit 27a of the visitor terminal 27.

[0048] (Display screen of display unit 26a of virtual visitor terminal) The display screen of virtual visitor terminal 26 (20) displays event area 14, a virtual space consisting of two-dimensional or three-dimensional spatial data of an imaged space formed from an image of event area 11 or a pseudo-space formed to simulate the image. In this embodiment, a virtual space consisting of three-dimensional spatial data formed from an image is displayed.

[0049] (Display screen of display unit 27a of visitor terminal) The display screen of display unit 27a of visitor terminal 27 corresponds to the field of view of visitor 12 seen through an image of the event area 11 or smart glasses or the like. Avatar display element 13a of virtual visitor 13 is displayed on the display screen of display unit 27a. Reference numeral 18 denotes agricultural products (event-related products).

[0050] The user can select the avatar display elements 12a and 13a using the input / output device 32 or touch the corresponding part of the display unit with a finger, thereby selecting the corresponding person or product.

[0051] 3. Hardware Configuration Next, the hardware configurations of the server device 1, the user terminals 26 and 27, and the head-mounted display system 20 will be described with reference to FIGS. 4, 5, and 6, respectively.

[0052] (Hardware Configuration of Server Device 1) As shown in Fig. 4, the server device 1 of this embodiment uses, for example, a computer. The computer is equipped with a CPU 30. The CPU 30 is connected via a bus line 35 to a memory (hereinafter referred to as a storage unit) 31, an input / output device 32 including a keyboard, a mouse, etc., a connection port 33 for connecting / reading a storage device 33a, etc., and a communication circuit 34 for communicating with the outside via a network. The storage unit 31 stores an adjustment amount database (adjustment amount DB) 28, local information 23, a server program 36 for processing the system 10, a browser program 37, and an OS 38 (operating system).

[0053] In this embodiment, the server program 36 operates in cooperation with the OS 38 and the browser program 37. Note that the server program 36 may also operate independently without using the browser program 37 or the OS 38. The server program 36 is stored, for example, in the storage device 33a. The server program 36 is installed in the server device 1 by reading the storage device 33a from the connection port 33.

[0054] (Hardware Configuration of User Terminals 26, 27) Next, the hardware configuration of the user terminals will be described. The user terminals are used as the virtual visitor terminal 26 and the visitor terminal 27. Here, the hardware configuration of the virtual visitor terminal 26 will be described, and the description of the visitor terminal 27 will be omitted. The hardware configuration of the virtual visitor terminal 26 is substantially the same as the hardware configuration of the server device 1 described above, so the same components are denoted by the same reference numerals and their description will be omitted. As shown in FIG. 5 , the virtual visitor terminal 26 of this embodiment may be implemented using, for example, a smartphone, a laptop computer, or a head-mounted display system. The virtual visitor terminal 26 includes a CPU 30. The CPU 30 is connected via a bus line 35 to a memory unit 31, a connection port 33 for connecting / reading a memory device 33a, and a communication circuit 34 for communicating with the outside world via a network. The input / output device 32 includes, for example, a microphone, a speaker, and an operation unit 32a. Note that the microphone, speaker, and operation unit 32a are the same as the corresponding components of the head-mounted display 21 described below. The storage unit 31 stores a user terminal program 39, a browser program 37, and an OS (operating system) 38 for processing the system 10. The user terminal program 39 is stored in, for example, a storage device 33a. By reading the storage device 33a from the connection port 33, the user terminal program 39 is installed on the user terminals 26 and 27.

[0055] In the hardware configuration of the server device 1, virtual visitor terminal 26, and visitor terminal 27 described above, the functions of the system shown in Figure 2 are realized using, for example, a CPU 30, a server program 36, and a virtual visitor terminal (visitor terminal) program 39, but some or all of these may be sequence-controlled using a logic circuit such as a microcomputer or a PLC (programmable logic controller).

[0056] (Hardware Configuration of Head-Mounted Display System 20) Next, we will explain the hardware configuration of the head-mounted display system 20. In the hardware configuration of the head-mounted display system 20, parts that are almost the same as those in the hardware configuration of the virtual visitor terminal 26 described above are assigned the same reference numerals, and their explanation will be omitted.

[0057] (Head-mounted display system 20) Next, a head-mounted display system serving as a virtual visitor terminal 26 will be described with reference to Figure 6. The head-mounted display system 20 shown in the figure includes a head-mounted display 21 worn on the user's head, and a control device 22. Note that the head-mounted display system 20 described below is an example and is not limited to this.

[0058] (Head-mounted display 21) The head-mounted display 21 mainly comprises a display unit 21a that displays an image of the virtual space, and a head position detection sensor 21b. In this embodiment, the head-mounted display 21 further comprises a speaker 21c, a microphone 21d, and an operation unit 21e. Note that the speaker 21c, the microphone 21d, and the operation unit 21e do not have to be integrated with the head-mounted display 21, and any one of them may be integrated with the head-mounted display 21.

[0059] (Display unit 21 a) The display unit 21 a has left and right image display surfaces disposed at positions corresponding to the left and right eyes of the user so as to cover the left and right eyes, respectively. A three-dimensional image may be displayed to the user by displaying images with parallax on the left and right image display surfaces, respectively.

[0060] (Head position detection sensor 21b) The head position detection sensor 21b detects posture information such as the orientation and tilt of the head-mounted display 21. The sensor for detecting posture information is realized by appropriately combining a gyro sensor, an acceleration sensor, an angular acceleration sensor, etc. The head position detection sensor 21b detects the position and posture of the user's head, and reflects the detection results in a three-dimensional image of the virtual space displayed by the display unit 21a, thereby presenting a three-dimensional image (viewpoint image) seen from a viewpoint position corresponding to the movement of the user's head.

[0061] (Speaker 21c, Microphone 21d) The speaker 21c outputs audio. It outputs real audio in the virtual space, audio and music that take into account the situation in the virtual space, and audio from conversations with the visitor 12 and virtual visitor 13. The microphone 21d inputs the user's voice when interacting with the visitor 12 and virtual visitor 13.

[0062] (Operation Unit 21e) The operation unit 21e (32a) includes, for example, an operation lever and an operation button. By operating these, the user can select an option displayed on the display unit 21a and confirm the selected option. For example, in this embodiment, a plurality of agricultural products available for purchase in the virtual space are displayed, and the user can select the agricultural product to purchase from these and confirm the purchase. Furthermore, by operating the operation unit 21e, the user can move the three-dimensional image viewed from the viewpoint position, allowing the user to experience as if they were moving within the virtual space. The operation unit 21e is communicably connected to the control device 22 via a communication circuit.

[0063] (Control device 22) The hardware configuration of the control device 22 is almost the same as the hardware configuration of the server device 1 or virtual visitor terminal 26 described above, so the same parts are given the same reference numerals and their description will be omitted. The control device 22 generates images to be displayed on the display unit 21a of the head-mounted display 21 and performs various arithmetic processing. The memory unit 31 stores processing programs for generating images to be displayed on the display unit 21a and performing various arithmetic processing.

[0064] 7 is a flowchart showing an embodiment of the processing of the user terminal program 39 used in the visitor terminal 27 of the system 10 and the server program 36 used in the server device 1 (see steps S1 to S4 in FIG. 7).

[0065] (S1: Acquire and transmit adjustment amount) The server device 1 acquires the adjustment amount 2a based on a predetermined calculation formula. The acquired adjustment amount 2a is transmitted to the visitor terminal 27. The image information 8a and distance information 9a are integrated into 3D data (local information 3a). If the distance information 9a is deep, the amount of 3D data becomes extremely large. This exceeds the available communication volume between the visitor terminal 27 and the server device 1. Furthermore, if the transmitted local information 3a is large, the amount of calculation within the server device 1 will overflow. Therefore, the server device 1 determines its own communication volume capacity and computing power and requests the visitor terminal 27 to reduce the amount of 3D data (consisting of image information and distance information) transmitted. The processing information 28a and communication information 28b may be acquired by executing measurement software installed on the server device 1. This execution may be performed at predetermined intervals. Other methods for acquiring the adjustment amount 2a based on the processing speed 28a and communication speed 28b may also be used.

[0066] (Data Structure of Adjustment Amount DB 28) In addition to obtaining the adjustment amount 2a using an arithmetic expression, the adjustment amount 2a may also be obtained using a predetermined table. FIG. 8 shows the data structure of the adjustment amount DB. In the adjustment amount DB 28 shown in the figure, for example, the upper table shows the relationship between the processing information 28a and the adjustment amount 2a, and the lower table shows the relationship between the communication information 28b and the adjustment amount 2a. If the processing information 28a and the communication information 28b decrease in either the upper or lower table, the adjustment amount 2a (search depth) becomes shorter. The adjustment amount 2a is obtained from the processing information 28a and the communication information 28b, and the smaller value is set as the adjustment amount 2a to be adopted. Note that the adjustment amount 2a in the adjustment amount DB 28 may be the amount of information.

[0067] (S2: Visitor terminal acquires distance information) When the visitor terminal 27 acquires the adjustment amount 2a, the distance information acquisition unit 9, for example, retains distance information 9a that is equal to or smaller than the adjustment amount 2a and deletes larger distance information 9a. At this time, it may retain the location information 4a or image information 8a that corresponds to the distance information 9a that should be retained and delete the others. The visitor terminal 27 transmits the local information 3a with the reduced amount of information to the server device 1.

[0068] (S3: Server device acquires local information) Returning to FIG. 7, the server device 1 acquires local information 3a. The location information 4a is converted into coordinate position information. Feature points are extracted from the spatial data based on the local information 3a and the existing spatial data 25. The coordinates of the respective spatial data are aligned based on the feature points.

[0069] (S4: Determine Differences) It is determined whether there are differences between the spatial data based on the local information 3a and the existing spatial data 25. If there are differences, the parts 25a with differences are extracted. On the other hand, if there are no differences, the process ends.

[0070] Figure 9 is a schematic diagram showing the differences in spatial data. The figure shows a grape farm as the event area 11. The top diagram shows existing spatial data 25. The bottom diagram shows spatial data based on local information 3a. The bottom diagram shows a different part 25a (see the two-dot chain line). In the bottom diagram, grapes are growing.

[0071] (S5: Update) Returning to FIG. 7, the different portion 25a is replaced / overwritten with the corresponding coordinate position of the existing spatial data 25.

[0072] [5. Other Embodiments] (Modification 40) Next, a description will be given of a modification or another embodiment of the system 10. The system 40 is substantially similar to the system 10 described above, and therefore the same parts are denoted by the same reference numerals and their description will be omitted.

[0073] The system 40 does not include the adjustment unit 2 and the determination unit 4, as compared to the system 10. For example, the system 40 can acquire local information 3a from a plurality of visitor terminals 27, and thereby generate new spatial data 25 with the help of the visitors 12.

[0074] Unlike the system 10, the system 41 does not include the adjustment unit 2. Since the system 41 includes the determination unit 4, it is possible to update only the necessary parts, which is efficient.

[0075] [6. Other] In the head-mounted display system 20 described above, the head-mounted display 21 and the control device 22 may be separate or integrated. The head-mounted display system 20 does not have to be a dedicated system; for example, it may be used by attaching a smartphone to a goggle-type headset. Head-mounted displays include, for example, virtual reality (VR) type head-mounted displays that display only a virtual space created using computer graphics, and mixed reality (MR) type head-mounted displays that display real space (e.g., real space displayed optically through the real space) and virtual space in real time. The present invention can be applied to either type of head-mounted display. The head-mounted display system 20 may also be used as the visitor terminal 27. In this case, the head-mounted display system 20 as the visitor terminal 27 includes a distance information acquisition unit 9. Furthermore, the display unit of the head-mounted display system 20 as the visitor terminal 27 may project pre-registered avatar display elements as digital images into the field of view of the scenery of the real event area 11 as augmented reality (AR). The visitor terminal 27 and the virtual visitor terminals 20, 26 may be used as a virtual visitor terminal and a visitor terminal, respectively. The distance information acquisition unit 9 may adjust the search distance (depth) by adjusting the output of laser light or radio waves and the sensitivity of the receiver. The adjustment amount 2a may be, for example, the distance (depth) to be acquired by the distance information acquisition unit 9 and / or the range for acquiring distance information. For example, when the processing information 28a regarding the processing speed of the server device 1 and / or the communication information 28b regarding the communication speed between the server device 1 and the visitor terminal 27 decrease, the distance (depth) to be acquired by the distance information acquisition unit 9 and / or the range for acquiring distance information may be narrowed. The storage unit 31 of the server device 1 may store parts (components) for generating a spatial image for the spatial data 25. Furthermore, some or all of the parts may be stored in an external server separate from the server device 1.The virtual space processing unit 6 and / or update unit 5 may acquire the parts and update or generate the spatial data 25. In the above-described embodiment, a server device 1 was used, but some or all of the adjustment unit 2, local information acquisition unit 3, determination unit 4, update unit 5, virtual space processing unit 6, and memory unit 31 of the server device 1 may be provided in the virtual visitor terminal 20, 26. As an example, if the virtual visitor terminal 20, 26 has the functions of the server device 1, the spatial data 25 stored in the memory unit 31 of the virtual visitor terminal 20, 26 is updated or generated based on the local information 3a transmitted from the visitor terminal 27. The items described as variations, other embodiments, and others in the above-described embodiment can be used in appropriate combinations.

[0076] [7. Summary] The server device 1 updates spatial data 25 in a virtual space having coordinates corresponding to a position in the real space based on image information 8a obtained from a user terminal 27 of a visitor in the real space, which acquires time information 7b and location information 7a. The server device 1 includes a local information acquisition unit 3 that acquires local information 3a, including spatiotemporal information 7c, image information 8a corresponding to the spatiotemporal information, and distance information 9a to an object corresponding to the image information, and an update unit 5 that generates / updates the spatial data 25 based on the local information. Therefore, by acquiring local information 3a from multiple visitor terminals 27, the server device can generate or update the spatial data 25 with the help of the visitors 12. Furthermore, the closer the virtual visitor 13 is to the visitor 12 in the virtual space, the more the virtual visitor 13 can see the spatial data 25 updated by the visitor terminal 27. Therefore, the virtual visitor 13 can experience a phenomenon currently occurring at the actual location with the help of the real visitor 12.

[0077] Such a server device 1 includes a determination unit 4 that determines whether or not there is a predetermined difference between the local information 3a and the spatial data 25 corresponding to the local information 3a. If there is a difference, the server device 1 updates the portion 25a of the spatial data 25 that has the difference. This makes it possible to update only the necessary portion, which is efficient.

[0078] Furthermore, if there is no difference, no update is performed, which is more efficient.

[0079] In addition, the system is equipped with an adjustment unit 2 that adjusts the amount of information 2a of distance information 9a to be acquired according to processing information 28a related to the processing speed of the server device 1 and / or communication information 28b related to the communication speed between the server device 1 and the user terminals 26, 27.As a result, by partial and continuous updates of the range to be updated by multiple visitors 12 in the real space, the necessary parts at hand of the virtual visitor 13 are synthesized in real time / almost real time as a general image of the entire space, thereby making it possible to create a practical virtual space in a digital twin.

[0080] 1 Server device, 2 Adjustment unit, 2a Information amount (adjustment amount), 3 Local information acquisition unit, 3a Local information, 4 Determination unit, 5 Update unit, 6 Virtual space processing unit, 6a Operation information, 7 Position information acquisition unit, 7a Position information, 7b Time information, 7c Spatiotemporal information, 8 Image information acquisition unit, 8a Image information, 9 Distance information acquisition unit, 9a Distance information, 10 Virtual space update system, 11 Event area, 12 Visitor, 12a Avatar display element, 13 Remote person (virtual visitor), 13a Avatar display element, 14 Virtual space event area, 17 Communication network, 18 Event-related products, 20 Head-mounted display system, 21 Head-mounted display, 22 Control device, 24 Peripheral camera, 25 Spatial data, 26 User terminal (remote person / virtual visitor terminal), 26a Virtual visitor display screen, 27 User terminal (visitor terminal), 27a Visitor terminal display screen, 28 Adjustment amount database (adjustment amount DB), 28a processing information, 28b communication information, 30 CPU, 31 memory, 32 input / output device, 32a operation unit, 33 connection port, 33a storage device, 34 communication circuit, 35 bus line, 36 server program, 37 browser program, 38 OS, 39 terminal program, 40 modified example, 41 modified example

Claims

1. A method for updating spatial data of a virtual space having coordinates corresponding to a position in the real space, which is executed by a server device having an update unit and a visitor terminal having a position information acquisition unit, an image information acquisition unit, and a distance information acquisition unit and held by a visitor visiting an event area in real space, wherein the updated event area of ​​the virtual space is displayed on a display unit of a virtual visitor terminal held by a virtual visitor visiting the virtual space, which is communicatively connected to the visitor terminal, and comprises the steps of: the visitor terminal acquiring on-site information consisting of time-space information consisting of time information and position information acquired by the position information acquisition unit, image information corresponding to the time-space information acquired by the image information acquisition unit, and distance information from the visitor terminal to an object corresponding to the image information acquired by the distance information acquisition unit; and the update unit updating the spatial data corresponding to the on-site information among the spatial data based on the on-site information.

2. The method for updating a virtual space according to claim 1, wherein the distance information acquisition unit acquires the distance information using a Lidar sensor.

3. A method for updating a virtual space as claimed in claim 1 or 2, wherein the server device is provided with a determination unit, and the determination unit is provided with a step of determining whether there is a difference between the spatial data corresponding to the on-site information and existing spatial data, and if there is a difference, updating the portion of the existing spatial data that is different.

4. A method for updating spatial data of a virtual space having coordinates corresponding to a position in the real space, which is executed by a server device having an update unit and an adjustment unit, and a visitor terminal having a position information acquisition unit, an image information acquisition unit, and a distance information acquisition unit and held by a visitor in the real space, comprising the steps of: the visitor terminal acquiring on-site information consisting of spatio-temporal information consisting of time information and position information acquired by the position information acquisition unit, image information corresponding to the spatio-temporal information acquired by the image information acquisition unit, and distance information from the visitor terminal to an object corresponding to the image information acquired by the distance information acquisition unit; the update unit updating the spatial data based on the on-site information; and the adjustment unit making an adjustment to reduce the distance to be acquired by the distance information acquisition unit and / or the range for acquiring the distance information when processing information regarding the processing speed of the server device and / or communication information regarding the communication speed between the server device and the visitor terminal decreases before the on-site information acquisition step.

5. A method for updating a virtual space according to claim 1 or 2, wherein the space data is updated at predetermined time intervals by a peripheral camera installed in the real space.

6. The method for updating a virtual space according to claim 5, wherein the peripheral camera updates the spatial data using a Lidar sensor.

7. A server device that updates spatial data of a virtual space having coordinates corresponding to positions in real space, comprising: a local information acquisition unit that acquires local information obtained from a visitor terminal held by a visitor to an event area in the real space, the local information being composed of time-space information consisting of time information and position information, image information corresponding to the time-space information, and distance information from the visitor terminal to an object corresponding to the image information; and an update unit that updates the spatial data corresponding to the local information based on the local information, the server device being communicatively connected to the visitor terminal, and displaying the updated event area of ​​the virtual space on the display unit of the virtual visitor terminal held by a virtual visitor visiting the virtual space.

8. The server device according to claim 7, wherein the distance information is acquired using a Lidar sensor.

9. A server device as claimed in claim 7 or 8, further comprising a determination unit which determines whether there is a difference between the spatial data corresponding to the on-site information and existing spatial data, and which updates the portion of the existing spatial data in which there is a difference if there is a difference.

10. A server device that updates spatial data of a virtual space having coordinates corresponding to positions in real space, comprising: a local information acquisition unit that acquires local information consisting of spatiotemporal information consisting of time information and position information, image information corresponding to the spatiotemporal information, and distance information from the visitor terminal to an object corresponding to the image information, obtained from a visitor terminal held by a visitor in the real space; an update unit that updates the spatial data based on the local information; and an adjustment unit that makes an adjustment to reduce the distance to be acquired and / or the range for acquiring the distance information when processing information relating to its own processing speed and / or communication information relating to the communication speed with the visitor terminal decreases.

11. A server device according to claim 7 or 8, wherein the spatial data is updated at predetermined times by peripheral cameras installed in the real space.

12. The server device according to claim 11, wherein the peripheral camera updates the spatial data using a Lidar sensor.

13. A system for updating spatial data of a virtual space having coordinates corresponding to a position in the real space, comprising a server device, a visitor terminal held by a visitor visiting an event area in real space, and a virtual visitor terminal held by a virtual visitor visiting a virtual space, wherein the visitor terminal comprises a position information acquisition unit for acquiring spatiotemporal information consisting of time information and position information, an image information acquisition unit for acquiring image information corresponding to the spatiotemporal information, and a distance information acquisition unit for acquiring distance information from the visitor terminal of the visitor to an object corresponding to the image information, the server device comprises an update unit for updating spatial data corresponding to local information consisting of the spatiotemporal information, image information and distance information based on the local information, and the virtual visitor terminal comprises a display unit communicatively connected to the visitor terminal and displaying the updated event area of ​​the virtual space.

Images