Virtual space update method and server device

Abstract

To provide a method and a server device for updating spatial data of a virtual space where visitors in a real space and virtual visitors who visit the virtual space imitating the real space interact with each other.SOLUTION: In a virtual space update system, a server device updates spatial data of a virtual space having coordinates corresponding to a position in a real space based on image information 8a obtained from a visitor terminal 27 of a visitor in the real space that acquires position information 7a and time information 7b. The server device includes an actual site information acquisition unit 3 that acquires actual site information 3a consisting of spatiotemporal information 7c, image information 8a corresponding to the spatiotemporal information, and distance information to an object corresponding to the image information, and an update unit 5 that updates the spatial data based on the actual site information. Since the actual site information can be extremely large, the server device ensures immediacy in generating / updating the virtual space, by controlling the depth of imaged data, regarding limitations imposed by the computing power of the server device and the amount of communication from an actual site to the server device.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to spatial data of a virtual space for a visitor who visits the real space and a virtual visitor who visits the virtual space to communicate with each other through the virtual space.

Background Art

[0002] In recent years, many technologies using virtual spaces have been developed. These technologies provide new experiences to users through virtual spaces. Also, for example, in a technology called digital twin, information on the real space is acquired, the environment of the real space is reproduced, and a virtual space that corresponds to the real world is constructed.

[0003] Patent Document 1 discloses a server device used for a communication method via a virtual space farm and a real farm. The server device includes: a person information acquisition means for acquiring pre-registered person information based on specific information for identifying a visitor to the farm or a virtual visitor to the virtual space farm, to be transmitted to a user terminal of the virtual visitor visiting the virtual space farm; a virtual space processing means for transmitting a perspective image of moving the virtual space farm based on an operation of the virtual visitor to the user terminal of the virtual visitor; and a coordinate acquisition means for acquiring position information of the visitor. The virtual space processing means displays an avatar display element of the visitor set in advance at a coordinate position in the virtual space farm corresponding to the position information.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In a digital twin, since there is a virtual space that corresponds to the real space, it is advisable to incorporate the time-dependent changes in the real space in the virtual space. For example, in a virtual farm / virtual event space, if a visitor in the real space and a virtual visitor in the virtual space can simultaneously share the same or similar experiences in the same or similar scenery, it can promote empathy between the two. However, it is cumbersome to incorporate all the changes in the real space into the virtual space, as there are limitations to the processing speed of the server device for calculations and there are limits to the amount of data that can be communicated per unit time.

[0006] Therefore, an object of the present invention is to provide a method for updating spatial data of a virtual space in which a visitor in the real space and a virtual visitor in the virtual space communicate via the virtual space, and a server device for updating the spatial data of the virtual space.

Means for Solving the Problems

[0007] (1) The method for updating a virtual space according to the present invention is executed by a server device and a user terminal, and based on image information obtained from a user terminal of a visitor in the real space that acquires time information and position information, it is a method for updating spatial data of a virtual space having coordinates corresponding to the position in the real space, comprising the step of acquiring on-site information consisting of the spatio-temporal information, the image information corresponding to the spatio-temporal information, and distance information to an object corresponding to the image information, and the step of generating / updating the spatial data based on the on-site information.

[0008] (2) Such a method for updating a virtual space includes a step of determining whether there is a predetermined difference from the spatial data corresponding to the on-site information, and when there is a difference, it is preferable to update the different part of the spatial data.

[0009] (3) Also, when there is no difference, it is preferable not to update.

[0010] (4) Also, preferably, before the acquisition step, an adjustment step is provided for adjusting the amount of the distance information to be acquired according to the processing information regarding the processing speed of the server device and / or the communication information regarding the communication speed between the server device and the user terminal.

[0011] (5) A server device according to another aspect of the present invention is a server device that updates spatial data of a virtual space having coordinates corresponding to the position in the real space based on image information obtained from a user terminal of a visitor in the real space who acquires time information and position information, the server device comprising: an acquisition unit that acquires on-site information including the spatio-temporal information, the image information corresponding to the spatio-temporal information, and the distance information to an object corresponding to the image information; and an update unit that generates / updates the spatial data based on the on-site information.

[0012] (6) Such a server device preferably includes a determination unit that determines whether there is a predetermined difference from the spatial data corresponding to the on-site information, and updates the different part of the spatial data when there is a difference.

[0013] (7) Also, preferably, no update is performed when there is no difference.

[0014] (8) Also, preferably, an adjustment unit is provided for adjusting the amount of the distance information to be acquired according to the processing information regarding the processing speed of the server device and / or the communication information regarding the communication speed between the server device and the user terminal.

[0015] - The "position information" is information regarding the position within the event area of a visitor who is visiting an actual event area, and includes, for example, coordinates indicating a two-dimensional or three-dimensional point, coordinates indicating a predetermined range such as a plane or a space, and concepts including quantities convertible thereto. For example, it may be information represented by latitude and longitude, or further information represented by adding altitude. Furthermore, the "position information" may be information represented by another index value convertible to latitude, longitude, and altitude, for example. When the place to be visited is a farm, it is information about the position of the visitor in the farm who is actually visiting the farm. For example, it is a concept including coordinates indicating two-dimensional or three-dimensional points, coordinates indicating a predetermined range such as a plane or space, and quantities convertible thereto. · "Time information" is information indicating the time when the position information is specified. Also, "time information" may be the time in Japan, the time overseas, or the elapsed time with respect to the reference time. Furthermore, "time information" may be information represented by, for example, time or another index value convertible to the elapsed time with respect to the reference time. · "Image information" is data of images or videos taken by the user terminal, and is a concept including data convertible thereto. · "Distance information" is information regarding the distance from the user terminal to the location photographed by the image information, and is a concept including quantities convertible thereto. · "Coordinate position information" is coordinates indicating two-dimensional or three-dimensional points in a virtual space, or coordinates indicating a predetermined range such as a plane or space, and is a concept including quantities convertible thereto. · "Processing information" is processing information regarding the processing speed of the server device. The processing information is affected by the performance of the CPU of the server device, the free capacity of the memory, the read / write speed of the hard disk, the congestion degree (accommodation number·load) of the server device, the processing method of PHP (Hypertext Preprocessor), etc. · "Communication information" is communication information regarding the communication speed between the server device and the user terminal.

Advantages of the Invention

[0016] The spatial data of the virtual space can be easily updated.

Brief Description of the Drawings

[0017]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

[0018] [1. Overview] The virtual space update system according to this embodiment reflects (updates) changes in the real space in the spatial data of the virtual space in a digital twin environment. Instead of pre-recorded spatial data, it allows virtual visitors to experience the phenomena occurring in real time with the help of real visitors. Specifically, the spatial data is updated based on the local information obtained from the user terminal of the visitor in the real space, that is, the spatio-temporal information (time information and location information), the image information, and the distance information from the user terminal to the object of the image information. At that time, if all are updated with the local information of the visitor, the data volume and the calculation volume will increase. Therefore, the depth and range as the amount of information to be updated are narrowed. The degree of narrowing is adjusted according to the respective calculation capabilities and communication speeds of the server device and the system. Even if the amount of information is narrowed, due to the partial and continuous updates of the range to be updated by multiple visitors in the real space, the real-time nature of the necessary parts in the hands of the virtual visitor and the synthesis of the general image of the entire space, a practical virtual space in the digital twin can be constructed.

[0019] In addition, the local information that visitors in the real space should pay attention to tends to concentrate on specific locations and times. Therefore, it is possible to efficiently update the spatial data without updating all of the spatial data.

[0020] The virtual space update system is preferably targeted at, for example, the morning markets in tourist areas, the fish landed from the boats returning to the fishing port, the blue sky market, bones and antiques, second-hand clothing, general merchandise stores, etc., which are only available on-site, or those where the negotiation between the seller and the buyer, such as the pounding and selling of bananas, is valuable as an experience. In addition, things such as auroras and blizzards can be experienced by virtual visitors as tourism.

[0021] [2. General Description] (Overview of the virtual space update system 10) First, the outline of the virtual space update system will be described with reference to FIG. 1. The virtual space update system (hereinafter referred to as the system) 10 shown in FIG. 1 is mainly used between visitors 12 who visit the market / morning market, etc. (event area) 11 and remote users 13 who are not visiting the event area 11, for example, those who are at home. The remote users 13 include virtual visitors 13 who visit the event area 14 (see FIG. 3) of the virtual space composed of three-dimensional spatial data of the imaging space formed from the imaging of the event area 11 or the pseudo space formed by simulating the imaging.

[0022] The virtual visitor 13 visits the event area 14 of the virtual space, for example, to participate in the event, to exhibit at the event, to check / purchase event-related products 18, or to communicate with other participants / exhibitors. The virtual visitor 13 can also communicate with other virtual visitors 13 who are visiting the event area 14 of the virtual space.

[0023] The stores that sell event-related products 18 include, for example, the stores in the event area 11 and the event area 14 of the virtual space, and include roadside stations, local stores, department stores, supermarkets, convenience stores, and concepts that can be converted into them.

[0024] (Event Area 11) The event area 11 is not particularly limited, but includes, for example, an event venue where a particular event is held, and is an area for events held outdoors and / or indoors. Examples of event venues include street corners, martial arts halls, exhibition halls, art museums, temples, churches, amusement parks, concert halls, gymnasiums, schools, auditoriums, baseball fields, athletic stadiums, parks, farms, ranches, nursing homes, care facilities, hospitals, etc. Examples of events include expositions such as the World Expo, exhibitions, sales events, product fairs, conferences, concerts, plays, music recitals, athletic meets, competitions, festivals, fireworks displays, tourism events, etc. Tourism is also included in the category of events. Furthermore, by linking with event areas in other virtual spaces, it will be possible to travel between these virtual spaces.

[0025] (Virtual space event area 14) An event area 14 (see FIG. 3) in a virtual space is spatial data consisting of three-dimensional spatial data of an imaged space formed by imaging the event area 11 or a pseudo space formed by calculation based on the imaging and depth (distance information 9a).

[0026] [3.Each configuration] (System 10) Next, using the functional block diagram in Figure 2 A system 10 according to one embodiment of the present invention will now be described.

[0027] The system 10 comprises a server device 1, a user terminal 26 (hereinafter referred to as a remote party terminal / virtual visitor terminal) held by the virtual visitor 13, and a user terminal 27 (hereinafter referred to as a visitor terminal) held by the visitor 12. The server device 1, the virtual visitor terminal 26, and the visitor terminal 27 are communicatively connected to each other via a communication network 17 (see FIG. 1) such as the Internet (intranet). As will be described later, reference numeral 20 denotes a head mounted display system (see FIG. 6). Note that the remote person 13 can also communicate with the visitor 12 in text information without visiting the event area 14 in the virtual space.

[0028] (Server device 1) In this embodiment, the server device 1 uses, for example, a personal computer. The server device 1 mainly includes an adjustment unit 2, a local information acquisition unit 3, a determination unit 4, and an update unit 5. The server device 1 further includes a virtual space processing unit 6.

[0029] (Virtual visitor terminal 26) In this embodiment, the virtual visitor terminal 26 is, for example, a smartphone. Note that it may also be a wearable computer as a computer that can be worn or carried, a head-mounted display system, a mobile phone, a tablet terminal, a personal computer, or the like. The virtual visitor terminal 26 (20) mainly includes a position information acquisition unit 7 and an image information acquisition unit 8. The virtual visitor terminal 26 (20) may further include a distance information acquisition unit 9. The virtual visitor terminal 26 (20) includes a display unit 26a (20a). The reference numeral 20a is the display unit of the head-mounted display system (see FIG. 6). The virtual visitor terminal 26 (20) displays the avatar display element of the visitor 12 in the perspective image of the virtual space as virtual reality (VR). The virtual visitor terminal 26 also identifies the current position and orientation, for example, using a gyro sensor or the like.

[0030] (Visitor terminal 27) In this embodiment, the visitor terminal 27 is substantially the same as the virtual visitor terminal 26(20), so the same parts are denoted by the same reference numerals and their description is omitted. In this embodiment, the visitor 12 wears the visitor terminal 27 such as smart glasses or AR glasses. The visitor terminal 27 projects a pre-registered avatar display element as a digital image onto the view of the scenery of the actual event area 11 as augmented reality (AR). The display unit 27a of the smart glasses and AR glasses is a part that allows the scenery to be visually perceived through it and also allows the digital image to be superimposed and displayed.

[0031] (Position information acquisition unit 7) The position information acquisition unit 7 has a function of acquiring the position information 7a and time information 7b of the visitor terminal 27. For example, the position information acquisition unit 7 acquires the position information using a GPS (Global Positioning System) sensor. Also, positioning methods other than the GPS sensor, for example, WiFi positioning or beacon positioning, may be used. The indoor position can be identified using WiFi positioning or beacon positioning. Note that the position information 7a and time information 7a constitute the spatio-temporal information 7c.

[0032] (Image information acquisition unit 8) The image information acquisition unit 5 is an imaging sensor such as an image sensor or an RGB sensor. For example, it is a conventionally known device equipped with a CCD (Charge Coupled Device) image sensor, a CMOS (Complementary Metal Oxide Semiconductor) image sensor, etc. The imaging sensor, for example, receives ambient light with a photodiode to detect RGB values and obtains a color image. The image information acquisition unit 8 generates the image information 8a. The image information acquisition unit 8 may be incorporated into the visitor terminal 27 and integrated, or may be separate from the visitor terminal 27. Note that the image information acquisition unit 8 may be provided with a sound collection device such as a microphone (not shown).

[0033] (Distance information acquisition unit 9) The distance information acquisition unit 9 acquires distance information 9a regarding the distance (depth) to the object in the generated image information 8a. When receiving the information amount (adjustment amount) 2a from the adjustment unit 2, it shortens the search depth and narrows down (reduces) the information amount 2a of the distance information 9a. Note that the narrowed information amount 2a may be restored. Next, the reduced distance information 9a is transmitted to the server device 1. At this time, the image information 8a corresponding to the reduced distance information 9a is also reduced. As a result, the image information 8a and distance information 9a with a short search depth, for example, on the user side, are transmitted to the server device 1. Note that the image information 8a of the part not associated with the distance information 9a may be transmitted to the server device 1. Known techniques can be used to obtain the distance, for example, a Lidar (Light Detection and Ranging) sensor, an iToF (Indirect Time of Flight) sensor, a triangulation method using infrared light, etc. can be used. In addition, a stereo camera may be used. Also, when using Visual SLAM for acquiring the spatial data 25 described later, the distance information acquisition unit 9 acquires the distance information 9a from two or more pieces of image information 8a. The plurality of pieces of image information 8a may be acquired from a plurality of visitor terminals 27, or may be acquired as image information 8a acquired by one visitor terminal 27 at different positions. It is preferable to use the plurality of selected pieces of image information 8a acquired within the range of the predetermined time information 7b.

[0034] (Spatial data 25) For acquiring the spatial data 25 (see FIG. 9) for update, known techniques can be adopted. As an example, a technique called SLAM (Simultaneous Localization and Mapping) can be used. SLAM is a general term for techniques that simultaneously perform "self-position estimation" for estimating where one is currently located and "environmental map creation" for grasping the surrounding situation. Depending on the sensors installed, there are Depth SLAM, Visual SLAM, LiDAR SLAM, etc. The existing spatial data before being updated is generated by the surrounding camera 24 (see Fig. 1) that collects 2D / 3D data.

[0035] (Surrounding camera 24) The surrounding camera 24 generates an image of 3D data, creates a virtual space to be displayed, and stores it in the storage unit 31 as spatial data. In this embodiment, the surrounding camera 24 is fixedly installed in the event area 11. A plurality of surrounding cameras 24 may be installed. In this embodiment, the surrounding camera 24 photographs the event area 11 at predetermined times, for example, every month, and updates the spatial data 25.

[0036] (Adjustment unit 2) The adjustment unit 2 adjusts the amount of information 2a (see Fig. 2) of the distance information 9a to be acquired according to the processing information regarding the processing speed of the server device 1 and / or the communication information regarding the communication speed between the server device 1 and the visitor terminal 27. The amount of adjustment may be the adjustment amount of the distance information acquisition unit 9 in terms of the amount convertible to the amount of information 2a. The adjustment amount is, for example, the distance (depth) to be acquired by the distance information acquisition unit 9 and / or the range for acquiring the distance information. The adjustment unit 2 transmits the amount of information 2a to the visitor terminal 27. In addition, the on-site information acquisition unit 3 may delete extra information so that the adjusted amount of information 2a is obtained, thereby reducing the calculation amount of the server device 1.

[0037] (On-site information acquisition unit 3) The on-site information acquisition unit 3 acquires on-site information 3a including the spatio-temporal information 7c, the image information 8a corresponding to the spatio-temporal information 7c, and the distance information 9a to the object corresponding to the image information 8a. Next, alignment is performed between the spatial data based on the on-site information 3a and the existing spatial data 25. Known techniques can be used for the alignment. For example, feature points may be extracted from the spatial data based on the on-site information 3a and the existing spatial data 25, and the coordinates of the mutual spatial data may be aligned based on the feature points. As an example, the extraction of the feature points is performed, for example, by using the image information 8a to extract the vertices of a rectangle.

[0038] In addition, the local information acquisition unit 3 acquires the coordinate position (viewpoint coordinates) in the three-dimensional space corresponding to the viewpoint image displayed on the virtual visitor terminal 26(20). For example, in the present embodiment, based on the viewpoint coordinates, an avatar display element representing a virtual visitor 13 other than oneself is displayed on the display unit 26a(20a). Note that the avatar display element representing oneself may be displayed on the display unit 26a(20a) of one's own terminal 26(20).

[0039] (Determination unit 4) The determination unit 4 determines whether there is a predetermined difference from the spatial data 25 of the coordinates corresponding to the local information 3a. When there is a difference, the different part 25a (see FIG. 9) of the spatial data 25 is updated. On the other hand, when there is no difference, it is not updated. The determination is performed using known techniques. As an example, based on the three-dimensional coordinate values (X, Y, Z) and / or color information (R, G, B) as the point cloud data of the obtained local information 3a and the existing spatial data 25, it is determined whether there is a difference greater than a predetermined value in the three-dimensional coordinates and / or color information. For example, when there is a difference greater than a predetermined value in a plurality of values, it is determined that there is a difference. A predetermined range including the different coordinates is set as the different part 25a. In addition, a part 25a with a predetermined difference may be extracted from the spatial data generated from the local information 3a and the existing spatial data 25 using a learning model of machine learning.

[0040] (Update unit 5) The update unit 7 replaces / overwrites the different part 25a with the corresponding existing spatial data 25 for updating.

[0041] (Virtual space processing unit 6) The virtual space processing unit 6 generates a virtual image to be displayed on the display unit 27a(20a) of the virtual visitor terminal 26(20) based on the operation information 6a regarding the operation amount output from the input / output device 32 (operation unit 32a) of the virtual visitor terminal 26(20) (see FIG. 3). For example, in cooperation with the control device 21 of the head-mounted system 20, it generates an image for display on the display unit 20a and performs various arithmetic processes.

[0042] (Display units 26a, 27a, 20a) As the display units 26a, 27a, 20a, for example, devices for displaying images such as a liquid crystal display (LCD), a plasma display panel (PDP), and an organic EL (EL: Electroluminescence) are used. A touch panel may be provided on the display unit. The touch panel outputs an output signal corresponding to an operation by the user. The user can perform operations such as selection of information displayed on the display units 26a, 27a, 20a by operating the touch panel.

[0043] FIG. 3 is a schematic diagram showing a state of screen displays of the user terminals 27, 26 of the visitor 12 and the virtual visitor 13. The avatar display elements 12a, 13a are composed of, for example, two-dimensional or three-dimensional spatial data of a captured image formed from the imaging of the visitor 12, the virtual visitor 13 or a pseudo-image formed imitating the imaging. Note that, regardless of imaging, a favorite image may be used as the avatar display element.

[0044] (Avatar display element 12a of the visitor 12) Specifically, the visitor terminal 27 acquires position information 7a within the event area 11 based on GPS or indoor positioning technology, and transmits the position information 7a to the on-site information acquisition unit 3 of the server device 1. The virtual space processing unit 6 of the server device 2 converts it into the coordinate position of the virtual space corresponding to the position information 4a, and transmits the avatar display element 12a of the visitor 12 to be displayed at the coordinate position to the virtual visitor terminal 26 (20).

[0045] (Avatar display element 13a of the virtual visitor (remote person) 13) On the other hand, the server device 1 acquires coordinate position information regarding the coordinate position of the virtual space of the virtual visitor 13. The virtual space processing unit 6 transmits the preset avatar display element 13a of the virtual visitor, which should be displayed at the position of the event area 11 corresponding to the acquired coordinate position information and overlapping the field of view through the visitor terminal 27 of the visitor 12, to the visitor terminal 27.

[0046] (Presentation screen) Next, with reference to FIG. 3, the screen displays of the virtual visitor terminal 26(20) and the visitor terminal 27 will be described. The screen display shown in the figure has the display screen 26a of the display unit of the virtual visitor terminal 26(20) at the upper stage and the display screen 27a of the display unit of the visitor terminal 27 at the lower stage.

[0047] (Display screen 26a of the virtual visitor terminal) On the display screen 26a of the virtual visitor terminal 26(20), an event area 14 of a virtual space composed of two-dimensional or three-dimensional space data of an imaging space formed from the imaging of the event area 11 or a pseudo space formed by simulating the imaging is displayed. In the present embodiment, a virtual space composed of three-dimensional space data formed by imaging is displayed.

[0048] (Display screen 27a of the visitor terminal) The display screen 27a of the visitor terminal 27 corresponds to the field of view of the visitor 12 transmitted through imaging of the event area 11 or via a smart glass or the like. An avatar display element 13a of the virtual visitor 13 is displayed on the display screen 27a. Reference numeral 18 is a crop (event-related product).

[0049] By selecting the avatar display elements 12a and 13a with the input / output device 32 by a user operation or by touching the corresponding part of the display unit with a finger, the corresponding person or product can be selected respectively.

[0050] [3. Hardware configuration] Next, with reference to FIGS. 4, 5 and 6, the hardware configurations of the server device 1, the user terminals 26 and 27, and the head-mounted display system 20 will be described respectively.

[0051] (Hardware configuration of the server device 1) As shown in FIG. 4, in the server device 2 of the present embodiment, for example, a computer is used. The computer is equipped with a CPU 30. Connected to the CPU 30 via a bus line 35 are a memory (hereinafter referred to as a storage unit) 31, an input / output device 32 including a keyboard, a mouse, etc., a connection port 33 for connecting / loading a storage device 33a, etc., and a communication circuit 34 for communicating with the outside via a network. The storage unit 31 stores an adjustment amount database (adjustment amount DB) 28, local information 23, a server program 36 for processing the system 10, a browser program 37, and an OS 38 (operating system).

[0052] In this embodiment, the server program 36 operates in cooperation using the functions of the OS 38 and the browser program 37. Note that the server program 36 may operate independently without using the browser program 37 and the OS 38. The server program 36 is stored, for example, in the storage device 33a. By reading the storage device 33a from the connection port 33, the server program 36 is installed in the server device 1.

[0053] (Hardware configuration of user terminals 26 and 27) Next, the hardware configuration of the user terminal will be described. The user terminal is used as a virtual visitor terminal 26 and a visitor terminal 27. Here, the hardware configuration of the virtual visitor terminal 26 will be described, and the description of the visitor terminal 27 will be omitted. The hardware configuration of the virtual visitor terminal 26 is substantially the same as the hardware configuration of the aforementioned server device 1, so the same parts are denoted by the same reference numerals and the description thereof is omitted. As shown in FIG. 5, in the virtual visitor terminal 26 of this embodiment, for example, a smartphone, a notebook computer, a head-mounted display, etc. may be used. The virtual visitor terminal 26 is equipped with a CPU 30. Connected to the CPU 30 via a bus line 35 are a storage unit 31, a connection port 33 for connecting / reading a storage device 33a, etc., and a communication circuit 34 for communicating with the outside via a network. As the input / output device 32, for example, it includes a microphone, a speaker, and an operation unit 32a. Note that the microphone, the speaker, and the operation unit 32a are the same as the corresponding parts of the head-mounted display 20 described later. Stored in the storage unit 31 are a user terminal program 39 for processing the system 10, a browser program 37, and further an OS 38 (operating system). The user terminal program 39 is stored, for example, in the storage device 33a. By reading the storage device 33a from the connection port 33, the user terminal program 39 is installed in the user terminals 26 and 27.

[0054] In the hardware configurations of the server device 1, the virtual visitor terminal 26, and the visitor terminal 27 described above, the functions of the system shown in FIG. 2 are realized, for example, using the CPU 30, the server program 36, and the virtual visitor terminal (visitor terminal) program 39. However, part or all of them may be sequence-controlled using a logic circuit such as a microcomputer or a PLC (programmable logic controller).

[0055] (Hardware Configuration of the Head-Mounted Display System 20) Next, the hardware configuration of the head-mounted display system 20 will be described. The hardware configuration of the head-mounted display system 20 is given the same reference numerals as the substantially same parts of the hardware configuration of the virtual visitor terminal 26 described above, and the description thereof is omitted.

[0056] (Head-Mounted Display System 20) Next, the head-mounted display system as the virtual visitor terminal 26 will be described with reference to FIG. 6. The head-mounted display system 20 shown in the figure includes a head-mounted display 21 worn on the user's head and a control device 22. The head-mounted display system 20 described below is an example and is not limited thereto.

[0057] (Head-mounted display 21) The head-mounted display 21 mainly consists of a display unit 21a on which an image of a virtual space is displayed and a head position detection sensor 21b. In this embodiment, it further includes a speaker 21c, a microphone 21d, and an operation unit 21e. Note that the speaker 21c, the microphone 21d, and the operation unit 21e do not have to be integrated with the head-mounted display 21, and any one of them may be integrated with the head-mounted display 21.

[0058] (Display unit 21a) The display unit 21a arranges left and right image display surfaces at positions corresponding to the left and right eyes so as to cover the user's left and right eyes. A three-dimensional image may be displayed to the user by displaying images with parallax on the left and right image display surfaces respectively.

[0059] (Head position detection sensor 21b) The head position detection sensor 21b detects posture information such as the orientation and tilt of the head-mounted display 21. The sensor for detecting the posture information is realized by appropriately combining a gyro sensor, an acceleration sensor, an angular acceleration sensor, etc. The position and posture of the user's head are detected, and the detection result is reflected in the three-dimensional image of the virtual space displayed by the display unit 21a, presenting a three-dimensional image (viewpoint image) seen from a viewpoint position corresponding to the movement of the user's head.

[0060] (Speaker 21c, Microphone 21d) The speaker 21c outputs sound. It outputs the actual sound in the virtual space, the sound and music taking into account the situation of the virtual space, the visitor 12, and the sound from the dialogue with the virtual visitor 13. The user's voice is input to the microphone 21d when interacting with the visitor 12 and the virtual visitor 13.

[0061] (Operation unit 21e) The operation unit 21e (32a) includes, for example, an operation lever and operation buttons. By operating them, it is possible to select an option displayed on the display unit 21a and perform an operation to determine the selected option. For example, in the present embodiment, a plurality of agricultural products that can be purchased in the virtual space are displayed, and the agricultural products to be purchased are selected from them and used to determine the purchase. Also, by operating the operation unit 21e, the three-dimensional image viewed from the viewpoint position can be moved, giving the user a feeling as if they themselves are moving within the virtual space. The operation unit 21e is communicably connected to the control device 22 through a communication circuit.

[0062] (Control device 22) Since the hardware configuration of the control device 22 is substantially the same as that of the aforementioned server device 1 or the virtual visitor terminal 26, the same parts are denoted by the same reference numerals and their description is omitted. The control device 22 performs generation of images and the like to be displayed on the display unit 21a of the head-mounted display 20, various arithmetic processes, and the like. The storage unit 31 stores a processing program and the like for generating images and the like to be displayed on the display unit 21a and performing various arithmetic processes.

[0063] [4. Program] (Flowchart showing update processing) FIG. 7 is a flowchart showing an embodiment of the processing of the user terminal program 39 used in the visitor terminal 27 of the system 10 and the server program 36 used in the server device 1.

[0064] (S1: Obtain and transmit adjustment amount) The server device 1 obtains the adjustment amount 2a based on a predetermined arithmetic expression. The obtained adjustment amount 2a is transmitted to the visitor terminal 27. The image information 8a and the distance information 9a are integrated 3D data (local information 3a). If the distance information 9a extends to a great depth, the amount of 3D data will become extremely large. As a result, it will exceed the possible communication volume between the visitor terminal 27 and the server device 1. Also, if the transmitted local information 3a is large, the amount of computation within the server device 1 will overflow. Therefore, the server device 1 determines by itself the communication volume capacity and the computing ability of the server device, and requests the visitor terminal 27 to suppress the transmission amount (of the 3D data (consisting of image information and distance information)). Also, the processing information 28a and the communication information 28b may be obtained by executing measurement software installed in the server device 1. The execution may be performed at predetermined times. Note that other methods may be used to obtain the adjustment amount 2a based on the processing speed 28a and the communication speed 28b.

[0065] (Data Structure of the Adjustment Amount DB 28) In addition to obtaining the adjustment amount 2a by an arithmetic expression, the adjustment amount 2a may be obtained from a predetermined table. Fig. 8 shows the data structure of the adjustment amount DB. The adjustment amount DB 28 shown in the figure, for example, the upper table shows the relationship between the processing information 28a and the adjustment amount 2a, and the lower table shows the relationship between the communication information 28b and the adjustment amount 2a. In both the upper and lower tables, when the processing information 28a and the communication information 28b decrease, the adjustment amount 2a (search depth) becomes shorter. The adjustment amount 2a is obtained from the processing information 28a and the communication information 28b respectively, and the smaller value should be adopted as the adjustment amount 2a. Note that the adjustment amount 2a of the adjustment amount DB 28 may be the amount of information.

[0066] (S2: The Visitor Terminal Obtains Distance Information) When the visitor terminal 27 obtains the adjustment amount 2a, for example, the distance information acquisition unit 9 retains the distance information 9a that is the same as or smaller than the adjustment amount 2a, and deletes the large distance information 9a. At this time, the position information 4a or the image information 8a corresponding to the distance information 9a to be retained may be retained, and the others may be deleted. The visitor terminal 27 transmits the local information 3a with the reduced amount of information to the server device 1.

[0067] (S3: The server device acquires local information) Returning to Fig. 7, the server device 1 acquires local information 3a. It converts the position information 4a into coordinate position information. It extracts feature points from the spatial data based on the local information 3a and the existing spatial data 25. Based on the feature points, it aligns the coordinates of the mutual spatial data.

[0068] (S4: Determine the difference) It determines whether there is a difference between the spatial data based on the local information 3a and the existing spatial data 25. If there is a difference, it extracts the different part 25a. On the other hand, if there is no difference, it ends.

[0069] Fig. 9 is a schematic diagram showing the difference in spatial data. In the figure, a grape farm is shown as the event area 11. The upper figure is the existing spatial data 25. The lower figure is the spatial data based on the local information 3a. The different part 25a (see the two-dot chain line) is described in the lower figure. In the lower figure, the grape fruits are growing.

[0070] (S5: Update) Returning to Fig. 7, it replaces / overwrites the different part 25a at the corresponding coordinate position of the existing spatial data 25.

[0071] [5. Other embodiments] (Modification example 40) Next, a modification example or other embodiments of the system 10 will be described. Since the system 40 is substantially the same as the system 10 described above, the same parts are denoted by the same reference numerals and the description thereof is omitted.

[0072] The system 40 does not include the adjustment unit 2 and the determination unit 3 with respect to the system 10. The system 40 can newly generate the spatial data 25 by borrowing the power of the visitor 12, for example, by acquiring the local information 3a from a plurality of visitor terminals 27.

[0073] The system 41 does not include the adjustment unit 2 with respect to the system 10. Since it includes the determination unit 2, it can update only the necessary parts, which is efficient.

[0074] [6. Others] In the above-described head-mounted display system 20, the head-mounted display 21 and the control device 22 may be separate or integrated. The head-mounted display system 20 does not have to be a dedicated system. For example, it may be a system in which a smartphone is attached to a goggle-type headset and used. Examples of head-mounted displays include a Virtual Reality (VR) type head-mounted display that displays only a virtual space created by computer graphics or the like, and a Mixed Reality (MR) type head-mounted display that displays the real space (for example, the real space optically transmitted and displayed) and the virtual space in real time. The present invention can be applied to any head-mounted display. Regarding the distance information acquisition unit 9, the output of laser light or radio waves and the sensitivity of the receiving unit may be adjusted to adjust the distance (depth) to be searched. The modified examples, other embodiments, and matters described as others in the foregoing embodiments can be used in appropriate combinations respectively.

[0075] [7. Summary] The server device 1 is a server device that updates the spatial data 25 of the virtual space having coordinates corresponding to the position of the real space based on the image information 8a obtained from the user terminal 27 of the visitor in the real space that acquires the time information 7b and the position information 7a, and includes a local information acquisition unit 3 that acquires local information 3a including the spatio-temporal information 7c, the image information 8a corresponding to the spatio-temporal information, and the distance information 9a to the object corresponding to the image information, and an update unit 5 that generates / updates the spatial data 25 based on the local information. Therefore, by acquiring the local information 3a from a plurality of visitor terminals 27, it is possible to newly generate or update the spatial data 25 by borrowing the power of the visitor 12.

[0076] Such a server device 1 is provided with a determination unit 3 that determines whether there is a predetermined difference from the spatial data 25 corresponding to the local information 3a. When there is such a difference, the different part 25a of the spatial data 25 is updated, so that only the necessary part can be updated, which is efficient.

[0077] Also, when there is no difference, it is not updated, which is even more efficient.

[0078] Further, since it is provided with an adjustment unit 2 that adjusts the amount of information 2a of the distance information 9a to be acquired according to the processing information 28a regarding the processing speed of the server device 1 and / or the communication information 28b regarding the communication speed between the server device 1 and the user terminals 26, 27, the necessary part in the hands of the virtual visitor 13 can be synthesized as a general image of the entire space in real time / almost real time by partial and continuous updates of the range to be updated by a plurality of visitors 12 in the real space, so that a practical virtual space in the digital twin can be configured.

Explanation of Reference Numerals

[0079] 1 Server device 2 Adjustment unit 2a Amount of information (adjustment amount) 3 Local information acquisition unit 3a Local information 4 Determination unit 5 Update unit 6 Virtual space processing unit 6a Operation information 7 Position information acquisition unit 7a Position information 7b Time information 7c Spatiotemporal information 8 Image information acquisition unit 8a Image information 9 Distance information acquisition unit 9a Distance information 10 Virtual space update system 11 Event area 12 Visitor 12a Avatar display element 13 Remote person (virtual visitor) 13a Avatar display element 14 Event area in the virtual space 15 Display screen of the virtual visitor terminal 16 Display screen of the visitor terminal 17 Communication network 18 Event-related products 20 Head-mounted display system 21 Head-mounted display 22 Control device 24 Peripheral camera 25 Spatial data 26 User terminal (remote / virtual visitor terminal) 26a Display screen of the virtual visitor 27 User terminal (visitor terminal) 27a Display screen of the visitor terminal 28 Adjustment amount database (Adjustment amount DB) 28a Processing information 28b Communication information 30 CPU 31 Memory 32 Input / output device 32a Operation unit 33 Connection port 33a Storage device 34 Communication circuit 35 Bus line 36 Server program 37 Browser program 38 OS 39 Terminal program 40 Variation 41 Variation

Claims

1. A method for updating spatial data of a virtual space having coordinates corresponding to a position in the real space, the method being executed by a server device and a user terminal, based on image information obtained from a user terminal of a visitor to the real space that acquires time information and position information, the method comprising: acquiring on-site information including the spatiotemporal information, the image information corresponding to the spatiotemporal information, and distance information to an object corresponding to the image information; generating / updating the spatial data based on the time and location information.

2. determining whether or not there is a predetermined difference between the site information and the spatial data corresponding to the site information; 2. The method for updating a virtual space according to claim 1, further comprising the step of: updating, when said difference exists, said part of said space data where said difference exists.

3. The method for updating a virtual space according to claim 2 , wherein no update is performed if there is no difference.

4. A method for updating a virtual space as described in claim 1, 2 or 3, further comprising, before the acquisition step, an adjustment step for adjusting the amount of distance information to be acquired in accordance with processing information regarding the processing speed of the server device and / or communication information regarding the communication speed between the server device and the user terminal.

5. A server device that updates spatial data of a virtual space having coordinates corresponding to a position in the real space based on image information obtained from a user terminal of a visitor in the real space that acquires time information and position information, an acquisition unit that acquires local information including the time-space information, the image information corresponding to the time-space information, and distance information to an object corresponding to the image information; and an update unit that generates / updates the spatial data based on the time and location information.

6. a determination unit that determines whether or not there is a predetermined difference between the site information and the spatial data corresponding to the site information, 6. The server device according to claim 5, wherein, if there is a difference, the part of the spatial data in which the difference exists is updated.

7. The server device according to claim 6, wherein if there is no difference, no update is performed.

8. A server device as described in claim 5, 6 or 7, further comprising an adjustment unit that adjusts the amount of distance information to be acquired in accordance with processing information regarding the processing speed of the server device and / or communication information regarding the communication speed between the server device and a user terminal.

Images