Virtual space management device

Abstract

According to the present invention, an image acquisition unit acquires a designated image containing a person and at least a portion of a facility. A facility model acquisition unit acquires, on the basis of position information added to the designated image and indicating the position of the imaging of the designated image, a designated facility model for reproducing the designated facility in a virtual space. A person model generation unit generates a person model in which the person included in the designated image is imitated. An arrangement unit arranges the person model in the designated facility model on the basis of the designated image. A transmission unit transmits, to a user terminal, a first image for displaying the designated facility model in which the person model is arranged.

Description

Virtual space management device

[0001] The present invention relates to a virtual space management device.

[0002] In recent years, progress has been made in the construction of digital twins that recreate real spaces in virtual spaces. For example, the following Patent Document 1 discloses a digital twin system that includes an acquisition unit, a data creation unit, and a rendering processing unit. The acquisition unit acquires two-dimensional image data capturing an object or a landscape. The data creation unit creates three-dimensional virtual space data including the object or the landscape based on the two-dimensional image data. The rendering processing unit renders a virtual space image viewed from a virtual camera based on the three-dimensional virtual space data. The data creation unit assigns attributes to elements included in the two-dimensional image data and reflects the attributes in the three-dimensional virtual space data. The attributes indicate whether or not an element is to be included in the three-dimensional virtual space data.

[0003] Patent No. 7218979

[0004] Virtual spaces can be used for a wide variety of purposes, but one example is reminiscing about memories. Users can reminisce about the past by visiting memorable places (such as their alma mater) created in the virtual space. However, simply visiting a memorable place limits the things that users can reminisce about.

[0005] An object of the present invention is to broaden the scope of use of virtual spaces.

[0006] A virtual space management device according to one aspect of the present invention includes an image acquisition unit that acquires a designated image including at least a portion of a facility and a person; a facility model acquisition unit that acquires a first facility model for reproducing the facility in a virtual space based on location information added to the designated image that indicates the capture location of the designated image; a person model generation unit that generates a person model that resembles the person included in the designated image; an arrangement unit that arranges the person model in the first facility model based on the designated image; and a transmission unit that transmits a first image displaying the first facility model with the person model arranged to a user terminal.

[0007] According to one aspect of the present invention, the range of uses of virtual spaces can be expanded.

[0008] FIG. 1 is a block diagram showing the configuration of a system 1 according to an embodiment. FIG. 2 is a block diagram showing the configuration of a user terminal 20. FIG. 3 is a block diagram showing the configuration of a server 10. FIG. 4 is a schematic diagram showing an example of a designated image SG. FIG. 5 is a schematic diagram showing an example of a designated facility model SM. FIG. 6 is a schematic diagram showing an example of a person model JM. FIG. 7 is a schematic diagram showing an example of a first image HG. FIG. 8 is a flowchart showing the operation of a processing device 103 of the server 10. FIG. 9 is a block diagram showing the configuration of the server 10 according to a second embodiment.

[0009] A. First Embodiment A-1. System Configuration Fig. 1 is a block diagram showing the configuration of a system 1 according to an embodiment. The system 1 includes a server 10 and a user terminal 20. The user terminal 20 is used by a user U. The server 10 and the user terminal 20 are connected via a network N. Although Fig. 1 shows one user terminal 20, the system 1 may include multiple user terminals 20.

[0010] The system 1 realizes access to a virtual space using a user terminal 20. In this embodiment, the virtual space reproduces a real space. In other words, in this embodiment, the virtual space is a "digital twin" that imitates a real space. A user U can access the virtual space using the user terminal 20 and have various experiences in the virtual space by moving an avatar that imitates the user. For example, the user U can travel in the virtual space as the avatar or preview places that he or she plans to visit in the real space.

[0011] The server 10 is an example of a virtual space management device that manages a virtual space that mimics real space. The server 10 is installed by a platform provider (a platform provider) for providing the virtual space, such as a telecommunications carrier. The server 10 manages virtual space data VD (see FIG. 3 ) for outputting images showing the virtual space and sounds generated in the virtual space to a user terminal 20.

[0012] The virtual space data VD includes multiple facility model data FD. Each facility model data FD is data for displaying a facility model that reproduces a facility that exists in real space. The facility may be, for example, an indoor facility such as a building, a house, or a museum, or an outdoor facility such as a road, a bridge, a park, or a theme park.

[0013] A facility model is created, for example, by a facility owner requesting the creation of a creator who is well versed in virtual space model creation technology. The creator creates a facility model that precisely reproduces the facility, for example, by performing a 3D scan of the facility or using design data for the facility. In this embodiment, facility models have not yet been created for some of the facilities that exist in real space. Hereinafter, facilities for which facility models have not yet been generated may be referred to as "uncreated facilities."

[0014] The facility model data FD includes location information indicating the location of the facility in real space. The location information is, for example, latitude and longitude. The location information may also be a parcel number (or house number). Latitude and longitude and parcel number (or house number) can be converted into each other.

[0015] The facility model data FD may also include sound data indicating sounds generated in the facility. For example, if the facility is a restaurant, the facility model data FD may include sound data indicating at least one of the following: the voices of customers, the sounds of dishes clashing together, and background music (BGM).

[0016] A-2. User Terminal 20 Fig. 2 is a block diagram showing the configuration of the user terminal 20. The user terminal 20 is a terminal used by the user U when accessing the virtual space. The user terminal 20 is, for example, an information processing terminal such as a smartphone, a tablet terminal, a personal computer, or a head-mounted display.

[0017] The user terminal 20 includes a display device 201, an input device 202, a communication device 203, a microphone 204, a speaker 205, a camera 206, a GPS device 207, a memory device 208, a processing device 209, and a bus 220 that interconnects these devices.

[0018] The display device 201 is a display device (for example, various display panels such as a liquid crystal display panel or an organic EL display panel) that displays information to the outside. The input device 202 is an input device (for example, a keyboard, a mouse, a microphone, a switch, a button, or a sensor) that accepts input from the outside. The display device 201 and the input device 202 may be integrated into one device (for example, a touch panel). The communication device 203 has an interface connectable to the network N and communicates with other devices connected to the network N using wireless or wired communication.

[0019] The microphone 204 collects sounds emitted by the user U and sounds generated around the user U and generates audio data. The speaker 205 generates sounds based on audio data acquired from devices other than the user terminal 20. The user U can use the microphone 204 and the speaker 205 to converse with avatars of other users U in the virtual space and to listen to sounds generated in the virtual space.

[0020] The camera 206 captures an image of a subject and outputs image data PD corresponding to the captured image. The camera 206 has, for example, an imaging optical system and an imaging element. The imaging optical system is an optical system including at least one imaging lens. The imaging element is, for example, a charge coupled device (CCD) image sensor or a complementary metal oxide semiconductor (CMOS) image sensor.

[0021] The image captured by camera 206 may be a still image or a video. Camera 206 may be, for example, a stereo camera that uses two or more lenses to capture an image of a subject from different optical axes. When capturing a video while moving camera 206 or when camera 206 is a stereo camera, the captured image is a plurality of images of a subject (e.g., at least a portion of a facility and a person) captured from different optical axes. Camera 206 may be, for example, an omnidirectional camera that uses two or more ultra-wide-angle lenses to capture an image in all directions (360°). When camera 206 is an omnidirectional camera, the captured image is an omnidirectional image. When a designated image SG (described later) is a plurality of images captured from different optical axes or an omnidirectional image, the accuracy of reproduction of the facility is improved when a second facility model is generated in facility model generation unit 113 (described later).

[0022] The GPS device 207 receives radio waves from multiple satellites and generates location information indicating the location of the user terminal 20 from the received radio waves. The location information may be in any format as long as it can identify a location in real space. In this embodiment, latitude and longitude are used as the location information. Note that the location information may be obtained using means other than the GPS device 207. For example, information identifying the name of the facility where the user terminal 20 is located and its location within the facility may be received from a beacon or the like installed in the facility. In this embodiment, the location information is added to the image data PD output from the camera 206. The location where the image indicated by the image data PD was captured is identified based on the location information added to the image data PD.

[0023] The storage device 208 is a recording medium readable by the processing device 209. The storage device 208 includes, for example, a nonvolatile memory and a volatile memory. The nonvolatile memory is, for example, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), and an electrically erasable programmable read-only memory (EEPROM). The volatile memory is, for example, a random access memory (RAM). The storage device 208 stores a program PG2 and image data PD. The program PG2 is a program for operating the user terminal 20. The image data PD is, for example, data representing an image captured by the camera 206. The image data PD may also be data representing an image transmitted from another user terminal 20. A plurality of pieces of image data PD may be stored in the storage device 208.

[0024] The processing device 209 includes one or more central processing units (CPUs). The one or more CPUs are examples of one or more processors. Each of the processor and the CPU is an example of a computer.

[0025] The processing device 209 reads the program PG2 from the storage device 208. By executing the program PG2, the processing device 209 functions as a virtual space access unit 211 and a designation receiving unit 212. These functional units may be configured by circuits such as a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), and an FPGA (Field Programmable Gate Array). Details of these functional units will be described later.

[0026] A-3 Server 10 Fig. 3 is a block diagram showing the configuration of the server 10. The server 10 comprises a communication device 101, a storage device 102, a processing device 103, and a bus 109 that interconnects these devices.

[0027] The communication device 101 has an interface that can be connected to the network N, and communicates with other terminals connected to the network N using wireless or wired communication.

[0028] The storage device 102 is a recording medium readable by the processing device 103. The storage device 102 includes, for example, a nonvolatile memory and a volatile memory. The nonvolatile memory is, for example, a ROM, an EPROM, and an EEPROM. The volatile memory is, for example, a RAM. The storage device 102 stores a program PG1. The program PG1 is a program for operating the server 10.

[0029] The storage device 102 also stores virtual space data VD. The virtual space data VD includes multiple facility model data FD. Each facility model data FD is data indicating a facility model corresponding to a different facility. The virtual space data VD may also include avatar data indicating an "avatar" that represents the user U in the virtual space.

[0030] The processing unit 103 includes one or more CPUs. The one or more CPUs are examples of one or more processors. Each of the processors and CPUs is an example of a computer.

[0031] By executing the program PG1, the processing device 103 functions as an image acquisition unit 111, a facility model acquisition unit 112, a facility model generation unit 113, a person model generation unit 114, a placement unit 115, a transmission unit 116, a change acceptance unit 117, and an imaging range change unit 118. At least some of these functions may be configured by circuits such as a DSP, an ASIC, a PLD, and an FPGA. Details of these functional units will be described later.

[0032] A-4. Details of the Processing Devices 103 and 208 Next, the details of each functional unit realized by the processing device 103 of the server 10 and the processing device 209 of the user terminal 20 will be described.

[0033] The virtual space access unit 211 of the user terminal 20 controls access to the virtual space. For example, when the user U launches an application for accessing the virtual space, the virtual space access unit 211 transmits a request to the server 10 to transmit virtual space data VD. The virtual space access unit 211 causes the display device 201 to display an image showing the virtual space based on the virtual space data VD received from the server 10. The virtual space access unit 211 also causes the speaker 205 to output sound in the virtual space based on the virtual space data VD received from the server 10. The virtual space access unit 211 also accepts input to the input device 202 (e.g., input instructing movement of an avatar), generates a request to transmit new virtual space data VD based on the content of the input, and transmits the request to the server 10.

[0034] The designation receiving unit 212 receives a designation of an image from the user U. The image designated by the user U will be referred to hereinafter as a "designated image SG." The designated image SG may be, for example, an image represented by image data PD stored in the storage device 208 of the user terminal 20. The designated image SG may also be an image represented by image data stored in a device other than the user terminal 20 (for example, a storage server of a storage service used by the user U).

[0035] In this embodiment, the designated image SG is an image that includes at least a portion of a facility and a person. The designated image SG may also be referred to as an image that includes at least a portion of a facility and a person. A facility included in the designated image SG is hereinafter referred to as a "designated facility SF." The person included in the designated image SG may be the user U, or may be a person other than the user U. The person included in the designated image SG may be one person or multiple people.

[0036] Fig. 4 is a schematic diagram showing an example of a designated image SG. The designated image SG shown in Fig. 4 includes a restaurant as a designated facility SF. The designated image SG also includes a person J1 and a person J2. The person J1 is a user U.

[0037] The image acquisition unit 111 of the server 10 acquires the designated image SG. In other words, the image acquisition unit 111 acquires an image including at least a part of the designated facility SF and a person. If the designated image SG is stored in the user terminal 20, the image acquisition unit 111 acquires image data PD representing the designated image SG from the user terminal 20. If the designated image SG is stored in a device other than the user terminal 20, the image acquisition unit 111 acquires image data PD representing the designated image SG from the device that stores the designated image SG.

[0038] The facility model acquisition unit 112 acquires a facility model for recreating the designated facility SF in a virtual space based on location information added to the designated image SG indicating the capture location of the designated image SG. The facility model for recreating the designated facility SF in a virtual space is hereinafter referred to as the "designated facility model SM." The designated facility model SM is an example of a first facility model. The facility model acquisition unit 112 verifies, for example, facility model data FD stored in the storage device 102 to determine whether there is any facility model data FD whose location information matches the location information added to the designated image SG. If there is facility model data FD whose location information matches the location information added to the designated image SG, the facility model data FD is data for displaying the designated facility model SM. On the other hand, if there is no facility model data FD whose location information matches the location information added to the designated image SG, the designated facility model SM has not yet been created. In other words, the designated facility SF is an uncreated facility.

[0039] In this embodiment, the facility model data FD is stored in the storage device 102 of the server 10, but the facility model data FD may be stored in a device other than the server 10. In this case, the facility model acquisition unit 112 transmits location information to the other device that stores the facility model data FD. The other device transmits facility model data FD whose location information matches the location information to the server 10 (facility model acquisition unit 112).

[0040] Furthermore, the facility model acquisition unit 112 may acquire the designated facility model SM based on the shape of the designated facility SF included in the designated image SG in addition to the location information. The shape of the designated facility SF may be the exterior shape of the designated facility SF or the interior shape of the designated facility SF. For example, if the designated facility SF is a store located in a shopping mall or a tenant building, there may be multiple stores in the same building. These multiple stores may each be created as independent facility models. These multiple facility models may have the same location information, and the designated facility model SM may not be determined based on the location information alone. In such cases, the facility model acquisition unit 112 acquires, as the designated facility model SM, a facility model whose shape matches the shape of the designated facility SF included in the designated image SG. More specifically, for example, suppose a restaurant and a beauty salon are located in a single building, and there are two facility models with the same location information. Of the two facility models, the facility model acquisition unit 112 acquires, as the designated facility model SM, the facility model whose shape most closely matches the shapes of the tables and chairs included in the designated image SG.

[0041] Fig. 5 is a schematic diagram showing an example of a designated facility model SM. Fig. 5 shows only the same area of ​​the designated facility model SM as the designated image SG, but in reality, the designated facility model SM is a three-dimensional model that reproduces the entire designated facility SF.

[0042] When the facility model acquisition unit 112 is unable to acquire a designated facility model SM, the facility model generation unit 113 generates a facility model that imitates the designated facility SF based on the designated image SG. Hereinafter, the facility model generated by the facility model generation unit 113 will be referred to as a "simple facility model." The simple facility model is an example of a second facility model. A simple facility model is created, for example, when the designated facility SF is an uncreated facility. The facility model generation unit 113 generates a three-dimensional model for reproducing the designated facility SF included in the designated image SG from the designated image SG, which is a two-dimensional image. The simple facility model reproduces only the portion of the designated facility SF that is included in the designated image SG. When the designated image SG is a set of multiple images captured from different optical axes, the accuracy of the three-dimensional reproduction (e.g., the foreground and background of objects included in the designated image SG) is expected to be improved compared to when the designated image SG is a single still image. Furthermore, when the designated image SG is an omnidirectional image, it is expected that a wider area of ​​the designated facility SF will be reproduced compared to when the designated image SG has a normal angle of view.

[0043] The character model generation unit 114 generates character models JM (JM1, JM2) that imitate characters J (J1, J2) included in the specified image SG. The character model generation unit 114 generates character models JM that reproduce the clothing, hairstyle, face (shape and arrangement of facial features), and pose of characters J1, J2 included in the specified image SG. The character model generation unit 114 may generate character models JM from the specified image SG using a tool such as PiFuHD.

[0044] 6 is a schematic diagram showing an example of a character model JM. Character model JM1 corresponds to character J1, and character model JM2 corresponds to character J2. While Fig. 6 illustrates only the areas of character models JM1 and JM2 that are the same as the designated image SG, in reality, character models JM1 and JM2 are three-dimensional models that reproduce the entire characters J1 and J2.

[0045] The placement unit 115 places the character models JM1 and JM2 on the designated facility model SM based on the designated image SG. As described above, the character models JM1 and JM2 also reproduce the poses of the characters J1 and J2 included in the designated image SG. Therefore, the placement unit 115 simply places the character models JM1 and JM2 in the locations of the designated facility model SM that correspond to the positions of the characters J1 and J2 in the designated image SG. Note that when a simple facility model is generated by the facility model generation unit 113, the placement unit 115 places the character models JM1 and JM2 on the simple facility model.

[0046] The transmission unit 116 transmits a first image HG displaying the designated facility model SM in which the person models JM1 and JM2 are arranged to the user terminal 20. Furthermore, when a simple facility model is generated by the facility model generation unit 113, the transmission unit 116 transmits a second image displaying the simple facility model in which the person models JM1 and JM2 are arranged to the user terminal 20.

[0047] The virtual space access unit 211 of the user terminal 20 causes the display device 201 to display the first image HG transmitted from the server 10. Alternatively, the virtual space access unit 211 causes the display device 201 to display the second image transmitted from the server 10.

[0048] Fig. 7 is a schematic diagram showing an example of a first image HG. If the designated facility model SM includes audio, the audio may be output in addition to the first image HG. The first image HG shown in Fig. 7 is an image in which the designated image SG shown in Fig. 4 is reproduced in a virtual space. In other words, the first image HG shown in Fig. 7 is an image of the designated facility model SM, in which the person models JM1 and JM2 are arranged, viewed from the imaging position of the designated image SG.

[0049] The designated facility model SM is a three-dimensional model for reproducing the entire designated facility SF. Therefore, it is possible to display areas of the designated facility SF other than those included in the designated image SG. The virtual space access unit 211 may receive an instruction from the user U to change the range of the designated facility model SM to be cropped as an image to be transmitted to the user terminal 20 (the virtual camera's imaging range). For example, the first image HG is an image showing the designated facility model SM within a first imaging range viewed from a first location, which is the imaging position of a virtual camera placed in virtual space, in a first direction, which is the imaging direction of the virtual camera. Changing at least one of the imaging position and imaging direction of the virtual camera is referred to as "changing the imaging range." Note that "changing the imaging range" may also include changing the imaging magnification, i.e., zooming in or out. The user U instructs the user U to change the imaging range using the input device 202, for example, by moving an avatar corresponding to the user in the virtual space. Information indicating the user U's instruction to change the imaging range is referred to as "instruction information."

[0050] As described above, when the first image HG is an image showing the designated facility model SM within a first imaging range captured using a virtual camera from a first location in a virtual space in a first direction, the change receiving unit 117 of the server 10 receives, from the user terminal 20, at least one of instruction information (an example of first instruction information) instructing the user to change the imaging position of the virtual camera from the first location to a second location and instruction information (an example of second instruction information) instructing the user to change the imaging direction of the virtual camera from the first direction to a second direction. That is, the change receiving unit 117 receives, from the user terminal 20, instruction information instructing the user to change the imaging range of the designated facility model SM. The imaging range changing unit 118 moves the virtual camera and changes the imaging range within the designated facility model SM based on the instruction information. Specifically, the imaging range changing unit 118, for example, executes at least one of changing the imaging position of the virtual camera to a second location and changing the imaging direction of the virtual camera to a second direction, thereby changing the imaging range of the virtual camera to a second imaging range different from the first imaging range. The transmitter 116 transmits a third image including the designated facility model SM within the second imaging range to the user terminal 20. In other words, the transmitter 116 transmits to the user terminal 20 a third image showing the designated facility model SM within the second imaging range in which the imaging position or imaging direction of the virtual camera has been changed from the first imaging range, based on the instruction information (at least one of the first instruction information and the second instruction information) received by the change receiver 117. The user terminal 20 displays the third image showing the designated facility model SM within the second imaging range.

[0051] For example, if the user U wanted to see an area of ​​the designated facility SF that was not accessible when the designated image SG was captured, the user U can search for an area of ​​the designated facility SF that is not included in the designated image SG by changing the imaging range of the virtual camera. Also, if the user U wants to visit a location in the designated facility SF again, the user U can search for that location regardless of whether it is included in the designated image SG by changing the imaging range of the virtual camera.

[0052] As an example of changing the imaging range of the virtual camera, the imaging position may be moved to the position of the eyes of a specific person model JM. For example, since person J1 corresponds to user U, an image of the designated facility model SM with the imaging position set at the position of the eyes of person model JM1 corresponds to the scenery that user U was looking at when the designated image SG was captured. By viewing such an image, user U may be able to more clearly recall the time when the designated image SG was captured. Furthermore, for example, an image of the designated facility model SM with the imaging position set at the position of the eyes of person model JM2 corresponds to the scenery that person J2, who was present with user U at the time the designated image SG was captured, was looking at. By viewing such an image, user U may be able to notice things that they were not aware of when the designated image SG was captured.

[0053] If the facility model is a simple facility model, only the area included in the designated image SG is reproduced in the virtual space. Therefore, the imaging range cannot be changed, or the change is limited to a range compared to the designated facility model SM.

[0054] 8 is a flowchart showing the operation of the processing device 103 of the server 10. When the designation receiving unit 212 of the user terminal 20 receives a designation of one image (designated image SG) from the user U, the processing device 103 of the server 10 functions as the image acquisition unit 111 and acquires, for example, the designated image SG from the user terminal 20 (step S100).

[0055] The processing device 103 functions as the facility model acquisition unit 112 and acquires a designated facility model SM whose address information matches the location information added to the designated image SG (step S102). If the designated facility model SM is acquired (step S104: YES), the processing device 103 proceeds to step S108. If the designated facility model SM is not acquired (step S104: NO), the processing device 103 functions as the facility model generation unit 113 and generates a simple facility model from the designated image SG (step S106).

[0056] The processing device 103 functions as a character model generation unit 114 and generates character models JM1 and JM2 from the designated image SG (step S108). The processing device 103 functions as an arrangement unit 115 and arranges character models JM1 and JM2 in a facility model (a designated facility model SM if step S104: YES, or a simple facility model if step S104: NO) (step S110). The processing device 103 functions as a transmission unit 116 and transmits an image of the facility model in which character models JM1 and JM2 are arranged (a first image HG if step S104: YES, or a second image if step S104: NO) to the user terminal 20 (step S112). The user terminal 20 displays the image of the facility model in which character models JM1 and JM2 are arranged.

[0057] The processing device 103 functions as a change receiving unit 117 and receives instruction information from the user terminal 20 instructing the user to change the imaging range of the virtual camera in the facility model (step S114). If the instruction information is not transmitted from the user terminal 20 (step S114: NO), the processing device 103 proceeds to step S118. If the instruction information is transmitted from the user terminal 20 (step S114: YES), the processing device 103 functions as an imaging range changing unit 118 and changes the imaging range of the virtual camera based on the instruction information (step S116). Thereafter, the processing device 103 functions as a transmitting unit 116 and transmits an image (third image) of the portion of the facility model included in the imaging range after the movement to the user terminal 20 (step S112).

[0058] The processing device 103 returns to step S114 and continues processing until an instruction to end the display of the virtual space is received from the user U (step S118: NO). When an instruction to end the display of the virtual space is received from the user U (step S118: YES), the processing device 103 ends the processing of this flowchart.

[0059] A-5. Summary of the Embodiment As described above, the server 10 according to the embodiment acquires a designated facility model SM for recreating a designated facility in a virtual space based on a designated image SG that includes at least a portion of the facility and a person. The server 10 transmits to the user terminal 20 a first image HG that displays the designated facility model SM in which person models JM1 and JM2 that resemble the people J1 and J2 included in the designated image SG are arranged. Thus, the user U can experience the scene included in the designated image SG in a virtual space, which can evoke more memories than simply viewing the designated image SG.

[0060] Furthermore, the server 10 acquires a designated facility model SM based on the shape of the designated facility SF included in the designated image SG, in addition to the location information indicating the capture location of the designated image SG. Therefore, even if, for example, multiple facilities are located in the same building and the same location information is associated with multiple facility models, an appropriate designated facility model SM can be acquired.

[0061] Furthermore, if the server 10 does not acquire a designated facility model SM, it generates a simple facility model that imitates the designated facility SF based on the designated image SG. Therefore, even if the designated facility SF is an uncreated facility, the user U can experience the scene depicted in the designated image SG in a virtual space, which can evoke more memories than simply viewing the designated image SG. In particular, if the designated image SG is an omnidirectional image captured by an omnidirectional camera or multiple images captured from different optical axes, it is expected that a more accurate simple facility model will be generated.

[0062] Furthermore, the server 10 changes the imaging range of the virtual camera within the designated facility model SM based on an instruction to change the imaging range received from the user U, and transmits a third image displaying the designated facility model SM within the changed imaging range to the user terminal 20. The user U can visit areas of the designated facility SF other than those included in the designated image SG. This not only allows the user U to recall memories, but also allows for new experiences at the designated facility SF, further expanding the scope of activities in the virtual space.

[0063] B: Second Embodiment A second embodiment of the present invention will be described below. In the following description, for the sake of simplicity, the same components as those in the first embodiment will be designated by the same reference numerals, and a description of their functions may be omitted. In addition, in the following description, for the sake of simplicity, differences between the second embodiment and the first embodiment will be mainly described.

[0064] 9 is a block diagram showing the configuration of the server 10 according to the second embodiment. In the second embodiment, the facility model data FD includes current model data GD and past model data KD. The current model data GD is data for displaying a current facility model for recreating a current facility. The past model data KD is data for displaying a past facility model for recreating a past facility. In other words, the designated facility model SM includes a current facility model for recreating a current facility and a past facility model for recreating a past facility.

[0065] 9 illustrates a plurality of past model data KD, the number of past model data KD may be one. Also, the facility model data FD of a facility that has just been installed may consist of only the current model data GD.

[0066] The current model data GD and the past model data KD are added with date and time information indicating the period of time during which the facilities indicated by the data are intended to be reproduced. For example, the current model data GD is added with date and time information "April 1, 2022 or later." This indicates that the current model data GD is data indicating a facility model "April 1, 2022 or later." Furthermore, one of the past model data KD is added with date and time information "June 1, 2000 to March 31, 2022." This indicates that the past model data KD is data indicating a facility model "April 1, 2022 or later."

[0067] A facility may undergo changes (renovations) to its interior or exterior. Even if no changes are made to the interior or exterior, the appearance of the facility may change over time due to wear and tear. Therefore, the facility model data FD created at a certain point in time may no longer accurately reflect the current facility. In such cases, a new facility model is created or an existing facility model is updated to create a current facility model that accurately reflects the current facility.

[0068] On the other hand, facility models that have been used up until now can also be used as past facility models for recreating past facilities. In particular, when facility models are displayed based on a designated image SG as in this embodiment, a past facility model may be more appropriate than a current facility model. Therefore, when a new current facility model is created, the facility model that had been used as the current facility model up until then is stored as a past facility model. Furthermore, the current facility model that had been used up until then is not necessarily used as the past facility model; a past facility model may also be generated based on, for example, an image of the facility in the past.

[0069] Generally, the data of the designated image SG is accompanied by the date and time of capture. The facility model acquisition unit 112 acquires either a current facility model or a past facility model as the designated facility model SM based on the date and time of capture of the designated image SG. For example, if the date and time of capture of the designated image SG is "June 3, 2023," the facility model acquisition unit 112 acquires the current facility model as the designated facility model SM. Furthermore, if the date and time of capture of the designated image SG is "September 1, 2007," the facility model acquisition unit 112 acquires the past facility model as the designated facility model SM.

[0070] In this way, the user U can experience in the virtual space the designated facility SF that was present at the time the designated image SG was captured. For example, even if the designated facility SF has been renovated and no longer resembles the facility that was present at the time the designated image SG was captured, the user U can still experience in the virtual space the designated facility SF that was present at the time the designated image SG was captured. Furthermore, for example, the user U can experience things that were not possible at the time the designated image SG was captured (such as riding an attraction at a theme park that could not be accessed due to height restrictions), thereby improving convenience.

[0071] C: Others (1) In the above-described embodiment, ROM and RAM are given as examples of storage devices 102 and 208, but storage devices 102 and 208 may also be flexible disks, magneto-optical disks (e.g., compact disks, digital versatile disks, Blu-ray (registered trademark) disks), smart cards, flash memory devices (e.g., cards, sticks, key drives), CD-ROMs (Compact Disc-ROMs), registers, removable disks, hard disks, floppy (registered trademark) disks, magnetic strips, databases, servers, or other suitable storage media.

[0072] (2) In the above-described embodiments, the described information, signals, etc. may be represented using any of a variety of different technologies. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.

[0073] (3) In the above-described embodiment, input and output information may be stored in a specific location (for example, a memory) or may be managed using a management table. Input and output information may be overwritten, updated, or added to. Output information may be deleted. Input information may be transmitted to another device.

[0074] (4) In the above-described embodiment, the determination may be made based on a value represented by one bit (0 or 1), a Boolean value (true or false), or a comparison of numerical values ​​(e.g., a comparison with a predetermined value).

[0075] (5) The order of the process procedures, sequences, flowcharts, etc. illustrated in the above-described embodiments may be rearranged unless it is consistent. For example, the methods described in this disclosure present elements of various steps using an example order, and are not limited to the particular order presented.

[0076] (6) Each function illustrated in Figures 2 and 3 is realized by any combination of hardware and / or software. Furthermore, the method for realizing each functional block is not particularly limited. That is, each functional block may be realized using a single device that is physically or logically coupled, or may be realized using two or more physically or logically separated devices that are directly or indirectly connected (e.g., wired, wireless, etc.) and these multiple devices. A functional block may be realized by combining software with the single device or the multiple devices.

[0077] (7) The programs exemplified in the above-described embodiments should be broadly construed to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, procedures, functions, etc., regardless of whether they are called software, firmware, middleware, microcode, hardware description language, or by other names.

[0078] Software, instructions, information, etc. may also be transmitted or received over a transmission medium. For example, if software is transmitted from a website, server, or other remote source using wired technologies (such as coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL)), and / or wireless technologies (such as infrared, microwave), then these wired and / or wireless technologies are included within the definition of transmission media.

[0079] (8) In each of the foregoing embodiments, the terms "system" and "network" are used interchangeably.

[0080] (9) The information, parameters, etc. described in this disclosure may be expressed using absolute values, relative values ​​from a predetermined value, or corresponding other information.

[0081] (10) In the above-described embodiments, the portable device may be a mobile station (MS). Those skilled in the art may also refer to a mobile station as a subscriber station, mobile unit, subscriber unit, wireless unit, remote unit, mobile device, wireless device, wireless communication device, remote device, mobile subscriber station, access terminal, mobile terminal, wireless terminal, remote terminal, handset, user agent, mobile client, client, or some other appropriate terminology. In this disclosure, the terms "mobile station," "user terminal," "user equipment (UE)," "terminal," etc. may be used interchangeably.

[0082] (11) In the above-described embodiments, the terms "connected," "coupled," or any variation thereof refers to any direct or indirect connection or coupling between two or more elements, and may include the presence of one or more intermediate elements between two elements that are "connected" or "coupled" to each other. The coupling or connection between elements may be physical, logical, or a combination thereof. For example, "connected" may be read as "access." As used in this disclosure, two elements may be considered to be "connected" or "coupled" to each other using at least one of one or more wires, cables, and printed electrical connections, as well as electromagnetic energy having wavelengths in the radio frequency range, microwave range, and optical (both visible and invisible) range, as some non-limiting and non-exhaustive examples.

[0083] (12) In the above embodiments, the phrase "based on" does not mean "based only on," unless otherwise specified. In other words, the phrase "based on" means both "based only on" and "based at least on."

[0084] (13) As used in this disclosure, the terms "determining" and "determining" may encompass a wide variety of actions. "Determining" and "determining" may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, searching, inquiring (e.g., searching in a table, database, or other data structure), ascertaining something as "determining" or "determining," and the like. Furthermore, "judgment" and "decision" may include regarding receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, and accessing (e.g., accessing data in memory) as having been "judgment" or "decision." Furthermore, "judgment" and "decision" may include regarding resolving, selecting, choosing, establishing, comparing, etc. as having been "judgment" or "decision." In other words, "judgment" and "decision" may include regarding some action as having been "judgment" or "decision." Furthermore, "judgment" may be interpreted as "assuming," "expecting," "considering," etc.

[0085] (14) In the above embodiments, when "include," "including," and variations thereof are used, these terms are intended to be inclusive, similar to the term "comprising." Furthermore, the term "or," as used in this disclosure, is not intended to be an exclusive or.

[0086] (15) In this disclosure, where articles are added by translation, such as a, an, and the in English, this disclosure may include the noun following these articles being plural.

[0087] (16) In this disclosure, the term "A and B are different" may mean "A and B are different from each other." The term may also mean "A and B are each different from C." Terms such as "separate" and "combined" may also be interpreted in the same way as "different."

[0088] (17) The aspects / embodiments described in this disclosure may be used alone, in combination, or switched depending on the implementation. Notification of predetermined information (e.g., notification that "X is true") is not limited to being explicit, but may be implicit (e.g., not notifying the predetermined information).

[0089] 1...system, 10...server, 20...user terminal, 103...processing device, 111...image acquisition unit, 112...facility model acquisition unit, 113...facility model generation unit, 114...person model generation unit, 115...placement unit, 116...transmission unit, 117...change reception unit, 118...imaging range change unit, 209...processing device, 211...virtual space access unit, 212...designation reception unit, FD...facility model data, GD...current model data, KD...past model data, J1, J2...person, JM1, JM2...person model, N...network, PD...image data, SF...designated facility, SG...designated image, SM...designated facility model, U...user, VD...virtual space data.

Claims

1. A virtual space management device comprising: an image acquisition unit that acquires a designated image including at least a portion of a facility and a person; a facility model acquisition unit that acquires a first facility model for reproducing the facility in a virtual space based on location information indicating the capture location of the designated image added to the designated image; a person model generation unit that generates a person model imitating the person included in the designated image; a placement unit that places the person model in the first facility model based on the designated image; and a transmission unit that transmits a first image displaying the first facility model with the person model placed thereon to a user terminal.

2. The virtual space management device according to claim 1, wherein the facility model acquisition unit acquires the first facility model based on the shape of the facility included in the image in addition to the location information.

3. A virtual space management device as described in claim 1, further comprising a facility model generation unit that generates a second facility model that imitates the facility based on the specified image when the first facility model is not acquired by the facility model acquisition unit, wherein the placement unit places the person model in the second facility model, and the transmission unit transmits a second image displaying the second facility model in which the person model is placed to the user terminal.

4. A virtual space management device as described in claim 1, wherein the first image is an image showing the first facility model within a first imaging range when viewed in a first direction, which is the imaging direction of a virtual camera, from a first location, which is the imaging position of the virtual camera placed in the virtual space, and the device further comprises a change receiving unit that receives from the user terminal at least one of first instruction information instructing a change of the imaging position of the virtual camera from the first location to a second location and second instruction information instructing a change of the imaging direction of the virtual camera from the first direction to a second direction, and the transmission unit transmits to the user terminal a third image showing the first facility model within a second imaging range in which at least one of the imaging position of the virtual camera and the imaging direction of the virtual camera has been changed from the first imaging range based on at least one of the first instruction information and the second instruction information received by the change receiving unit.

5. A virtual space management device as described in claim 1, wherein the first facility model includes a current facility model for recreating the current facility and a past facility model for recreating the facility in the past, and the facility model acquisition unit acquires either the current facility model or the past facility model as the first facility model based on the image capture date and time of the specified image.

6. The virtual space management device according to claim 1, wherein the specified image is an omnidirectional image captured by an omnidirectional camera.

7. A virtual space management device according to claim 1, wherein the specified images are a plurality of images of at least a portion of the facility and the person captured from different optical axes.

Images