Virtual space generation device
The virtual space generation device allows users to interact with and move objects within the virtual space, enhancing the immersive experience by enabling object manipulation and rearrangement.
Patent Information
- Application Number
- JP2023070674
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-04-24
- Publication Date
- 2024-11-06
AI Technical Summary
Existing virtual space models generated from real space images lack the ability to move objects or utilize them, limiting user interaction and experience.
A virtual space generation device that includes an image acquisition unit, detection unit, and generation units to create and arrange three-dimensional models of objects within a virtual space, allowing users to move and interact with these objects.
Enables a more immersive experience by allowing users to move and rearrange objects within the virtual space, mimicking real-world interactions.
Smart Images

Figure 2024156314000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to a virtual space generating device. [Background technology]
[0002] Conventionally, a technique is known in which a 3D model (virtual space model) of a building, structure, natural object, etc. (hereinafter referred to as "building, etc.") is generated using a 3D scanner, etc., and the virtual space model is projected onto a monitor or VR (Virtual Reality) glasses, etc. This allows a user to have the experience of being in the building, etc. by viewing the virtual space model, even after the building, etc. in the real space has been demolished.
[0003] For example, the following Patent Document 1 includes an acquisition unit that acquires two-dimensional image data capturing an object or a landscape, a data creation unit that creates three-dimensional virtual space data including the object or the landscape based on the two-dimensional image data, and a drawing processing unit that draws a virtual space image viewed from a virtual camera based on the three-dimensional virtual space data. The data creation unit assigns attributes to elements included in the two-dimensional image data and reflects the attributes in the three-dimensional virtual space data. The attributes indicate whether or not an element is to be included in the three-dimensional virtual space data. [Prior art documents] [Patent documents]
[0004] [Patent Document 1] Patent No. 7218979 Summary of the Invention [Problem to be solved by the invention]
[0005] Generally, an object in a virtual space model generated based on an image captured in a real space exists as part of the background. Therefore, for example, the object cannot be moved to another location, and an avatar in the virtual space cannot use the object. A user can only view the scenery of the virtual space model, and there is a problem that the way to use the virtual space model is limited.
[0006] An object of the present invention is to use a virtual space model to enable an experience closer to that of a real space. [Means for solving the problem]
[0007] A virtual space generation device according to one embodiment of the present invention is a virtual space generation device that generates a virtual space model that simulates a real space, and includes an image acquisition unit that acquires a captured image of the real space, a detection unit that detects a plurality of objects included in the captured image, a first generation unit that generates a plurality of three-dimensional models corresponding to each of the plurality of objects, and a second generation unit that arranges the plurality of three-dimensional models based on the captured image to generate the virtual space model. Effect of the Invention
[0008] According to one aspect of the present invention, a virtual space model can be used to enable an experience closer to the real space. [Brief description of the drawings]
[0009] [Figure 1] 1 is a block diagram showing a configuration of a virtual space display system 1 according to an embodiment. [Diagram 2] FIG. 2 is a block diagram showing a configuration of the imaging terminal 10. [Diagram 3] FIG. 2 is a block diagram showing a configuration of an access terminal 30. [Figure 4] FIG. 2 is a block diagram showing the configuration of a server 20. [Diagram 5] FIG. 2 is a schematic diagram showing an example of a captured image PC. [Figure 6]FIG. 13 is a schematic diagram showing an estimation result of a boundary between wall surfaces. [Figure 7] FIG. 2 is a schematic diagram showing a virtual space model MV1 of an indoor space. [Figure 8] FIG. 13 is a schematic diagram showing a virtual space model MV2 to which an object OR has been added. [Figure 9] FIG. 13 is a schematic diagram showing the virtual space model MV3 to which the painting model OV6 has been moved. [Figure 10] FIG. 13 is a schematic diagram showing a virtual space model MV4 according to a comparative example. [Figure 11] 3 is a flow diagram showing the operation of the virtual space display system 1. FIG. [Figure 12] 3 is a flow diagram showing the operation of the virtual space display system 1. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0010] A. First embodiment A-1. System configuration 1 is a block diagram showing a configuration of a virtual space display system 1 according to an embodiment. The virtual space display system 1 includes an imaging terminal 10, a server 20, and an access terminal 30. The imaging terminal 10, the server 20, and the access terminal 30 are connected via a network N.
[0011] The imaging terminal 10 captures an image of the real space SR, and generates imaging data indicating a captured image PC (see FIG. 5). The imaging data generated by the imaging terminal 10 is transmitted to the server 20 and used to generate a virtual space model MV (MV1 to MV3, see FIG. 8 as an example). The imaging terminal 10 is, for example, a smartphone or tablet terminal equipped with a camera, a digital camera, or a dedicated imaging device for capturing images of three-dimensional shapes. In this embodiment, the imaging terminal 10 is a smartphone. A person who captures images using the imaging terminal 10 is referred to as a "photographer."
[0012] The access terminal 30 is used to access the virtual space model MV. The access terminal 30 is, for example, an information processing terminal such as a smartphone, a tablet terminal, a personal computer, or a head-mounted display. The access terminal 30 is a terminal used by a user when accessing the virtual space model MV. In this embodiment, the access terminal 30 is a smartphone. A person who accesses the virtual space model MV using the access terminal 30 is called a "user."
[0013] The imaging terminal 10 and the access terminal 30 may be the same terminal. In this case, the functional unit (imaging data transmission unit 111) of the imaging terminal 10 shown in Fig. 2 is realized by the access terminal 30. In this case, the image capturer and the user are the same person.
[0014] Although only one imaging terminal 10 and one access terminal 30 are illustrated in FIG. 1, the virtual space display system 1 may include a plurality of imaging terminals 10 and a plurality of access terminals 30.
[0015] The server 20 is an example of a virtual space generating device that generates a virtual space model MV that imitates the real space SR. In this embodiment, the server 20 generates and manages the virtual space model MV. More specifically, the server 20 acquires imaging data captured by the imaging terminal 10, and generates model data DM that indicates the virtual space model MV. As described above, the imaging data is data that captures the real space SR. Therefore, the virtual space model MV is a model that imitates the real space SR. Furthermore, when requested by the access terminal 30, the server 20 transmits the model data DM to the access terminal 30.
[0016] In this embodiment, the virtual space model MV reproduces a building or the like in the real space SR. For example, by generating a virtual space model MV that imitates a building or the like that is scheduled to be demolished, the user can tour the building or the like in the virtual space even after the building or the like is demolished. The building or the like does not have to be the entire building or structure, but may be a part of the building or structure, such as a room in a house.
[0017] Here, an overview of a method for generating a virtual space model MV will be described. As a method for generating a virtual space model MV using imaging data, [1] a method using photogrammetry and [2] a method using a LiDAR (Light Detection and Ranging) sensor are known.
[0018] [1] Photogrammetry Photogrammetry is a technology for generating a three-dimensional model from a plurality of two-dimensional images of a subject taken from different angles. In more detail, in photogrammetry, for example, feature points are extracted from a plurality of two-dimensional images taken continuously, and the imaging position of the imaging terminal 10 at the time of shooting, the distance and angle to the subject, etc. are estimated based on the amount of movement of the feature points between the plurality of two-dimensional images. The spatial coordinates of the feature points are calculated from the position of the imaging terminal 10, etc., and a three-dimensional model is generated. Therefore, in photogrammetry, a plurality of two-dimensional images taken of the same area from different positions, that is, a plurality of two-dimensional images with parallax, are required. Such images are obtained, for example, by an imager holding the imaging terminal 10 in his / her hand continuously capturing two-dimensional images while walking through the target space to be modeled.
[0019] [2] Using a 3D scanning application When using this method, the imaging terminal 10 includes an infrared sensor such as a LiDAR (Light Detection and Ranging) sensor. The imaging terminal 10 simultaneously captures a two-dimensional image of the subject and scans the subject using the LiDAR sensor. The LiDAR sensor projects infrared rays onto the subject and receives the infrared rays reflected by the surface of the subject. The LiDAR sensor also calculates the distance to each point on the surface of the subject based on the time from projecting the infrared rays to receiving them. The imaging terminal 10 generates imaging data by associating each point on the two-dimensional image of the subject with the distance to each point on the surface of the subject calculated by the LiDAR sensor. In this method as well, for example, an imager holds the imaging terminal 10 in his / her hand and walks through the modeling target space while continuously capturing and scanning two-dimensional images.
[0020] Hereinafter, in this embodiment, the case where the method using photogrammetry ([1]) is used will be described as an example.
[0021] A-2. Imaging terminal 10 2 is a block diagram showing the configuration of the imaging terminal 10. The imaging terminal 10 includes a display device 101, an input device 102, a communication device 103, an imaging device 104, a storage device 105, a processing device 106, and a bus 109 that interconnects these devices.
[0022] The display device 101 is a display device (e.g., various display panels such as a liquid crystal display panel or an organic EL display panel) that displays information to the outside. The input device 102 is an input device (e.g., a keyboard, a mouse, a microphone, a switch, a button, a sensor, etc.) that accepts input from the outside. The display device 101 and the input device 102 may be integrated into one device (e.g., a touch panel).
[0023] In this embodiment, the display device 101 is mainly used by the image capturer to check the imaging range of the image capture device 104. The input device 102 is mainly used by the image capturer to instruct the image capture device 104 to start and end imaging.
[0024] The communication device 103 communicates with the server 20 using wireless communication or wired communication. In this embodiment, the communication device 103 has an interface connectable to a network N, and communicates with a communication device 201 (see FIG. 4) of the server 20 via the network N.
[0025] The imaging device 104 generates imaging data indicating an imaged image PC obtained by capturing an image of a subject. In this embodiment, the subject is a real space SR that is the source of a virtual space model MV. The imaging device 104 has, for example, an imaging optical system and an imaging element. The imaging optical system is an optical system including at least one imaging lens. For example, the imaging optical system may have various optical elements such as a prism, or may have a zoom lens or a focus lens. The imaging element is, for example, a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary MOS) image sensor.
[0026] The storage device 105 is a recording medium readable by the processing device 106. The storage device 105 includes, for example, a nonvolatile memory and a volatile memory. The nonvolatile memory is, for example, a Read Only Memory (ROM), an Erasable Programmable Read Only Memory (EPROM), and an Electrically Erasable Programmable Read Only Memory (EEPROM). The volatile memory is, for example, a Random Access Memory (RAM). The storage device 105 stores a program PG1. The program PG1 is a program for operating the imaging terminal 10.
[0027] The processing device 106 includes one or more central processing units (CPUs). The one or more CPUs are an example of one or more processors. Each of the processor and the CPU is an example of a computer.
[0028] The processing device 106 reads the program PG1 from the storage device 105. The processing device 106 executes the program PG1 to function as an imaging data transmission unit 111. The function of the imaging data transmission unit 111 may be configured by circuits such as a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), and an FPGA (Field Programmable Gate Array).
[0029] The imaging data transmission unit 111 transmits the imaging data generated by the imaging device 104 to the server 20 using the communication device 103. The imaging data transmission unit 111 may perform predetermined processing on the imaging data (for example, data format conversion, data compression, addition of location information, etc.) as necessary.
[0030] The imaging terminal 10 may capture an image of the real space SR and collect sounds generated in the real space SR. In this case, the imaging terminal 10 is provided with a microphone (not shown). Sound data indicating the collected sounds is transmitted to the server 20 together with the imaging data and is included in the virtual space model MV. For example, if a river flows near the real space SR that is the subject, the sound of the flowing water of the river can be heard constantly. Also, if a school is near the real space SR that is the subject, the voices of children and the sound of a bell can be heard. By including such sounds in the virtual space model MV, an experience closer to the real space SR can be achieved.
[0031] A-3. Access Terminal 30 3 is a block diagram showing the configuration of the access terminal 30. The access terminal 30 includes a display device 301, an input device 302, a communication device 303, a microphone 304, a speaker 305, a storage device 306, a processing device 307, and a bus 309 that interconnects these devices.
[0032] The display device 301, the input device 302, and the communication device 303 are similar to the display device 101, the input device 102, and the communication device 103, respectively, and therefore detailed description thereof will be omitted.
[0033] The microphone 304 picks up sounds made by the user and sounds generated around the user to generate voice data. The speaker 305 generates sounds based on voice data acquired from devices other than the access terminal 30. The user can use the microphone 304 and the speaker 305 to converse with avatars of other users in the virtual space and to listen to sounds generated in the virtual space.
[0034] The storage device 306 is a recording medium that can be read by the processing device 307. The storage device 306 includes, for example, a nonvolatile memory and a volatile memory. The nonvolatile memory is, for example, a ROM, an EPROM, and an EEPROM. The volatile memory is, for example, a RAM. The storage device 306 stores a program PG3. The program PG3 is a program for operating the access terminal 30.
[0035] The processing unit 307 includes one or more CPUs. The one or more CPUs are an example of one or more processors. Each of the processor and the CPU is an example of a computer.
[0036] The processing device 307 reads the program PG3 from the storage device 306. The processing device 307 executes the program PG3 to function as a model data acquisition unit 311, a display control unit 312, and a motion detection unit 313. At least some of the functions of the model data acquisition unit 311, the display control unit 312, and the motion detection unit 313 may be configured by circuits such as a DSP, an ASIC, a PLD, and an FPGA.
[0037] The model data acquisition unit 311, the display control unit 312, and the motion detection unit 313 are functions of a "virtual space access application" installed in the access terminal 30. The user can access the virtual space model MV by using the virtual space access application.
[0038] The model data acquisition unit 311 acquires model data DM for displaying the virtual space model MV from the server 20. More specifically, the model data acquisition unit 311 transmits a transmission request for the model data DM to the server 20. The server 20 (model data transmission unit 216) transmits the model data DM to the access terminal 30 in response to the transmission request. A plurality of model data DM are stored in the server 20. Thus, the model data acquisition unit 311 accepts designation of identification information (such as a name) for identifying the model data DM (virtual space model MV) that the user wishes to access, and transmits a transmission request including the identification information to the server 20.
[0039] The display control unit 312 displays the virtual space model MV on the display device 301 based on the model data DM. The user visually recognizes the virtual space model MV displayed on the display device 301. Note that if the virtual space model MV includes audio data, audio based on the audio data may be output from the speaker 305 together with the image display of the virtual space model MV.
[0040] The motion detection unit 313 detects the motion of the user viewing the virtual space model MV, and generates motion information indicating the user's motion. The motion information generated by the motion detection unit 313 is transmitted to the server 20.
[0041] In this embodiment, the user's movement is a movement that moves a three-dimensional model of an object OR (hereinafter referred to as an "object model OV") that constitutes the virtual space model MV. For example, the access terminal 30 is assumed to have a touch panel in which the display device 101 and the input device 102 are integrated. When the user presses and holds an object model OV on the virtual space model MV displayed on the touch panel, the object model OV is selected. When the object model OV is selected, the display mode is different from that of other object models OV, for example, the outer edge of the object model OV is emphasized. When the user moves his / her finger while touching the object model OV, the object model OV moves in the same direction as the user's finger on the virtual space model MV. In this case, the movement detection unit 313 detects the touch position, touch duration, movement direction, etc. of the user's finger on the touch panel, and transmits them to the server 20 as movement information. In addition, when the access terminal 30 is a head-up display, the movement detection unit 313 detects the movement of the user's hands and feet, and detects, for example, the movement of holding the object model OV, the movement of walking while holding the object model OV, etc.
[0042] A-4. Server 20 4 is a block diagram showing the configuration of the server 20. The server 20 includes a communication device 201, a storage device 202, a processing device 203, and a bus 209 that interconnects these devices.
[0043] The communication device 201 communicates with the imaging terminal 10 and the access terminal 30 using wireless communication or wired communication. In this embodiment, the communication device 201 has an interface connectable to a network N, and communicates with the imaging terminal 10 and the access terminal 30 via the network N.
[0044] The storage device 202 is a recording medium readable by the processing device 203. The storage device 202 includes, for example, a nonvolatile memory and a volatile memory. The nonvolatile memory is, for example, a ROM, an EPROM, and an EEPROM. The volatile memory is, for example, a RAM. The storage device 202 stores a program PG2. The program PG2 is a program for operating the server 20.
[0045] The storage device 202 also stores virtual space data DV. The virtual space data DV includes model data DM. A plurality of model data DM are illustrated in FIG. 3. Each model data DM is data corresponding to a different building or the like. Each model data DM is identifiable by identification information. The virtual space data DV may include avatar data indicating an "avatar" that is the appearance of the user in the virtual space.
[0046] The storage device 202 also stores standard shape data DS. The standard shape data DS is data in which the type of object OR is associated with a standard shape. For example, a writing implement such as a pen has an axisymmetric shape of a roughly cylindrical or prismatic shape. Also, furniture such as a bookshelf has a roughly rectangular parallelepiped shape, a shelf board on the front side (a cupboard or the like has a sliding door or hinged door in addition to the shelf board), and a roughly flat back side. A dining table generally has a top board and four legs arranged at the four corners of the top board. The standard shape data DS is shape data that indicates the standard shape of each type of such object OR.
[0047] The processing unit 203 includes one or more CPUs. The one or more CPUs are examples of one or more processors. Each of the processor and the CPU is an example of a computer.
[0048] The processing device 203 reads the program PG2 from the storage device 202. The processing device 203 executes the program PG2 to function as an image acquisition unit 211, an object detection unit 212, a reception unit 213, a first generation unit 214, a second generation unit 215, a model data transmission unit 216, a movement information acquisition unit 217, and a change unit 218. At least some of the functions of the image acquisition unit 211, the object detection unit 212, the reception unit 213, the first generation unit 214, the second generation unit 215, the model data transmission unit 216, the movement information acquisition unit 217, and the change unit 218 may be configured by a circuit such as a DSP, an ASIC, a PLD, or an FPGA.
[0049] The image acquisition unit 211 acquires a captured image PC obtained by capturing the real space SR. The image acquisition unit 211 acquires the captured image PC by acquiring imaging data from the imaging terminal 10. As described above, in this embodiment, the virtual space model MV is generated using photogrammetry. Therefore, the image acquisition unit 211 acquires a plurality (a large number) of imaging data obtained by capturing the real space SR to be modeled.
[0050] The object detection unit 212 detects a plurality of objects OR included in the captured image PC. The object detection unit 212 also identifies the type of each of the plurality of objects OR. The object detection unit 212 detects the type of the object OR included in the captured image PC by object recognition using, for example, AI (Artificial Intelligence).
[0051] FIG. 5 is a schematic diagram showing an example of a captured image PC. The real space SR shown in the captured image PC is a part of a room. The captured image PC shows a first wall OR1 with a curtain OR5 (OR5-1, OR5-2) hanging thereon, a second wall OR2 with a picture OR6 hanging thereon, a ceiling OR3, a floor OR4, the curtain OR5, the picture OR6, a lighting fixture OR7, a television OR8, a television stand OR9, and a remote control OR10. That is, the real space SR shown in FIG. 5 is an indoor space surrounded by walls. The first wall OR1, the second wall OR2, the ceiling OR3, and the floor OR4 are examples of walls.
[0052] The object detection unit 212 extracts feature points from the captured image PC, for example, and detects the objects OR included in the captured image PC. This identifies the position of each object OR. The object detection unit 212 also estimates the type of each object OR based on the shape, color, size, etc. of each object OR.
[0053] Note that the detection of an object OR included in the captured image PC may be performed by a person such as a photographer. In this case, the photographer or the like designates an area in the captured image PC that is occupied by one object OR, and repeats the task of designating the type of the one object OR.
[0054] The reception unit 213 receives a setting of whether each of the multiple objects OR detected by the object detection unit 212 can be moved. The reception unit 213 transmits, for example, a captured image PC with a line added to the outer edge of the object OR to the imaging terminal 10. The image capturer displays the captured image PC on the imaging terminal 10 and sets whether each object OR can be moved. Specifically, for example, when a specific object OR on the captured image PC is tapped, the object OR is selected, and a button for selecting "movable" and a button for selecting "immovable" are displayed. When the image capturer taps the button for selecting "movable", the object OR is set as movable. When the image capturer taps the button for selecting "immovable", the object OR is set as immovable. The method of setting whether or not the object OR can be moved is not limited to this, and for example, only movable (or immovable) objects OR may be selected.
[0055] Taking the objects OR shown in Fig. 5 as an example, for example, the first wall OR1, the second wall OR2, the ceiling OR3, and the floor OR4, which are the outer edges of the indoor space, are designated as immovable objects OR. On the other hand, the curtains OR5, the paintings OR6, the lighting fixtures OR7, the television OR8, the television stand OR9, and the remote control OR10 are designated as movable objects OR. At least one of the curtains OR5, the paintings OR6, the lighting fixtures OR7, the television OR8, the television stand OR9, and the remote control OR10 may be designated as an immovable object OR.
[0056] Note that the server 20 may automatically set whether each object OR can be moved based on the type of the object OR, instead of having the reception unit 213 receive the setting as to whether each object OR can be moved. Furthermore, the reception unit 213 may receive the setting as to whether each object model OV can be moved after the object model OV corresponding to each object OR is generated.
[0057] The first generating unit 214 generates a plurality of object models OV corresponding to the plurality of objects OR, respectively. In other words, the first generating unit 214 generates an independent object model OV for each of the plurality of objects OR included in the captured image PC.
[0058] For example, the first generating unit 214 first generates an object model OV of the walls (first wall OR1, second wall OR2, ceiling OR3, and floor OR4) that define the outer edge of the indoor space. When the real space SR is outdoors, the outer edge of the real space SR is the ground, the sky, and the object OR at the backmost position in the captured image PC. Alternatively, the first generating unit 214 may first generate an object model OV of an object OR that is set as immovable among the multiple objects OR.
[0059] Here, in the captured image PC shown in FIG. 5, there may be a portion where the outer edge of the wall is not captured. For example, a portion of the boundary between the first wall OR1 and the floor OR4 is hidden by a curtain OR5, a portion of the boundary between the second wall OR2 and the floor OR4 is hidden by a TV stand OR9, and a portion of the boundary between the second wall OR2 and the ceiling OR3 is hidden by a lighting fixture OR7. The first generating unit 214 estimates the position of the portion of the boundary hidden by these objects OR based on the other portion that is not hidden by the object OR. That is, the first generating unit 214 generates an object model OV of the wall by estimating the position of the portion of the end of the wall forming the indoor space that is not captured in the captured image PC based on the position of the portion of the end that is captured in the captured image PC.
[0060] Fig. 6 is a schematic diagram showing an estimation result of the boundary between the wall surfaces. Fig. 7 is a schematic diagram showing a virtual space model MV1 of an indoor space. The line segment shown by a dotted line in Fig. 6 is the position of the boundary between the wall surfaces estimated based on other parts. The first generating unit 214 generates a first wall model OV1, a second wall model OV2, a ceiling model OV3, and a floor model OV4 based on the position of the boundary estimated as in Fig. 6, and generates a virtual space model MV1 (see Fig. 7) showing the indoor space surrounded by the walls.
[0061] An indoor space is generally rectangular, with walls that meet at right angles. By determining the positions of the walls in the virtual space model MV1 based on this, the distortion of the captured image PC can be corrected. Therefore, by first generating the object models OV of the walls that define the indoor space, the positions of the object models OV other than the walls can also be calculated more accurately.
[0062] Next, the first generating unit 214 generates object models OV of other objects OR present in the indoor space. The object models OV of the other objects OR are a curtain model OV5 corresponding to the curtain OR5, a painting model OV6 corresponding to the painting OR6, a lighting fixture model OV7 corresponding to the lighting fixture OR7, a television model OV8 corresponding to the television OR8, a television stand model OV9 corresponding to the television stand OR9, and a remote control model OV10 corresponding to the remote control OR10.
[0063] Here, it is expected that in many cases, images capturing all of the objects OR from all around will not be obtained. For example, the captured image PC shown in FIG. 5 does not capture the surface (rear surface) of the TV stand OR9 facing the second wall OR2. In this case, the first generation unit 214 refers to the standard shape data DS of the TV stand OR9 and generates a TV stand model OV9 including the rear surface of the TV stand OR9 by interpolating the shape of the part not captured in the captured image PC. That is, the first generation unit 214 generates multiple object models OV by complementing the shapes of the parts of the multiple objects OR that are not captured in the captured image PC based on standard shapes determined for each type.
[0064] Also, for example, the inside of the drawer and hinged door of the TV stand OR9 may be complemented based on the standard shape data DS. In this case, when the user opens (moves) the drawer and hinged door, a storage space appears, making it possible to store other object models OV (for example, remote control model OV10, etc.).
[0065] The second generating unit 215 arranges a plurality of object models OV based on the captured image PC to generate a virtual space model MV. The second generating unit 215 arranges the object models OV of the objects OR generated by the first generating unit 214 in the virtual space model MV1 of the indoor space. The position at which each object model OV is arranged is made the same as the position of the object OR in the indoor space specified using the captured image PC.
[0066] Fig. 8 is a schematic diagram showing a virtual space model MV2 to which an object OR has been added. An object model OV corresponding to the object OR is placed at the same position as the object OR appearing in the captured image PC shown in Fig. 5. The virtual space model MV generated by the second generating unit 215 is stored in the storage device 202 as model data DM.
[0067] When a transmission request for the model data DM is received from the access terminal 30 (model data acquisition unit 311), the model data transmission unit 216 transmits the model data DM to the access terminal 30. Based on identification information of the model data DM (virtual space model MV) included in the transmission request, the model data transmission unit 216 selects the model data DM corresponding to the identification information from the multiple model data DM stored in the storage device 202 and transmits the selected model data DM to the access terminal 30.
[0068] The motion information acquisition unit 217 acquires motion information indicating the motion of a user viewing the virtual space model MV. The user viewing the virtual space model MV is a user using the access terminal 30. The motion information acquisition unit 217 acquires motion information detected by the motion detection unit 313 of the access terminal 30. As described above, the motion of the user is, for example, a motion that moves an object model OV constituting the virtual space model MV, and the motion information includes, for example, the touch position of the user's finger on the touch panel, the duration of touch, and the direction of movement.
[0069] The change unit 218 changes the position of at least one object model OV in the virtual space model MV based on the movement information. The change unit 218 specifies the object model OV touched by the user based on, for example, the touch position of the user's finger on the touch panel. When the touch duration is equal to or longer than a predetermined time, the change unit 218 determines that the user has selected the object model OV and changes the display mode of the object model OV. When the user moves his / her finger in this state, the change unit 218 moves the object model OV in the same direction as the movement direction of the user's finger on the virtual space model MV. When the position of the object model OV is changed by the change unit 218, information specifying the position of the object model OV is updated in the model data DM. When the model data DM is updated, the model data transmission unit 216 transmits the updated model data DM to the access terminal 30. The display control unit 312 of the access terminal 30 updates the display of the display device 301 based on the updated model data DM. This enables the user to move the object model OV in the virtual space model MV.
[0070] The change unit 218 changes the position of the object model OV corresponding to the object OR set to be movable by the reception unit 213 based on the movement information. In other words, the change unit 218 does not change the position of the object model OV of the object OR set to be immovable by the reception unit 213. Typical examples of the object OR set to be immovable are walls, ceilings, and floors. In addition, typical examples of the object OR set to be immovable include a bookshelf attached to a wall, a kitchen sink, and a handrail on a staircase, which are considered to be part of a building. By making the object OR that is generally immovable (not moved) in the real space SR immovable in the virtual space model MV, the user can be given a feeling closer to the real space SR and the user can be prevented from making an erroneous operation.
[0071] 9 is a schematic diagram showing the virtual space model MV3 to which the painting model OV6 has been moved. For example, when the user moves the painting model OV6 to another location, the second wall model OV2 is exposed at the position where the painting model OV6 was. Since the first generating unit 214 also generates the part of the second wall model OV2 that overlaps with the painting model OV6 as shown in FIG. 7, even if the painting model OV6 is moved, the virtual space model MV3 is displayed without any sense of incongruity. Also, for example, even if the remote control model OV10 placed on the top surface of the TV stand model OV9 is moved, the TV stand model OV9 is generated all around, so that the display is also without any sense of incongruity (display is the same as the other parts of the top surface).
[0072] FIG. 10 is a schematic diagram showing a virtual space model MV4 according to a comparative example. When only the surface shape of the real space SR is three-dimensionally modeled as in the prior art, the overlapping parts of the objects OR are not modeled. Therefore, when the painting model OV6 is moved to another location, data is lost, resulting in an unnatural display like the area T. Therefore, the object model OV in the virtual space model MV4 cannot be moved substantially, and the user can only simply view the virtual space model MV4.
[0073] According to this embodiment, the user can move the object model OV in the virtual space model MV, for example, rearrange the interior space, or move the object OR by holding it in his / her hand. Therefore, the user can experience an experience close to the real space SR in the virtual space model MV.
[0074] A-5. Operation of virtual space display system 1 Fig. 11 and Fig. 12 are flow diagrams showing the operation of the virtual space display system 1. Fig. 11 shows the operation of the imaging terminal 10 and the server 20 in the generation process of the virtual space model MV. Fig. 12 shows the operation of the access terminal 30 and the server 20 in the display process of the virtual space model MV.
[0075] First, a description will be given of Fig. 11. An image capturing person captures an image of a real space SR to be recorded as a virtual space using the imaging terminal 10 (step S100). The processing device 106 of the imaging terminal 10 functions as an imaging data transmission unit 111 and transmits imaging data indicating a captured image PC to the server 20 (step S102).
[0076] The processing device 203 of the server 20 functions as the image acquisition unit 211 and acquires imaging data (step S104). The processing device 203 functions as the object detection unit 212 and detects a plurality of objects OR included in the captured image PC (step S105). The processing device 203 functions as the first generation unit 214 and generates an object model OV of a wall surface forming an indoor space (represented as "wall surface model" in the figure) (step S106). The processing device 106 also functions as the first generation unit 214 and generates an object model OV of an object OR other than the wall surface (represented as "non-wall surface model" in the figure) (step S108).
[0077] The processing device 203 functions as the second generating unit 215, and places the object models OV other than the wall surfaces generated in step S108 in the indoor space formed by the object models OV of the wall surfaces (step S110). The processing device 106 stores the model data DM in the storage device 105 (step S112), and ends the process.
[0078] Next, Fig. 12 will be described. In the access terminal 30, the user starts a virtual space access application (indicated as "access app" in the figure) (step S200). The processing device 307 of the access terminal 30 functions as a model data acquisition unit 311, and transmits a transmission request for the model data DM to the server 20 (step S202). The transmission request includes identification information for identifying the model data DM.
[0079] The processing device 203 of the server 20 functions as the model data transmitting unit 216, selects the model data DM based on the identification information included in the transmission request transmitted from the access terminal 30, and transmits the model data DM to the access terminal 30 (step S206).
[0080] When the processing device 307 of the access terminal 30 receives the model data DM, it functions as the display control unit 312 and displays the virtual space model MV on the display device 301 (step S208). The processing device 307 functions as the motion detection unit 313 and detects the user's motion (step S210). If the user's motion is not detected (step S210: NO), the processing device 307 returns the process to step S208 and continues displaying the virtual space model MV. If the user's motion is detected (step S210: YES), the processing device 307 functions as the motion detection unit 313 and transmits motion information to the server 20 (step S212).
[0081] The processing device 203 of the server 20 functions as a movement information acquisition unit 217, and when it acquires the movement information, it functions as a change unit 218 and changes the position of the object model OV in the virtual space model MV based on the user's movement (step S214). The processing device 203 functions as a model data transmission unit 216 and transmits the model data DM in which the position of the object model OV has been changed to the access terminal 30 (step S216). When the processing device 307 of the access terminal 30 receives the model data DM, it functions as a display control unit 312 and updates the display of the virtual space model MV based on the newly received model data DM (step S218). Thereafter, the processing from step S208 onwards is repeated until the user ends the virtual space access application.
[0082] A-6. Summary of embodiments As described above, the server 20 according to the embodiment generates a virtual space model MV that imitates a real space SR. Therefore, for example, by generating a virtual space model MV that imitates a building or the like that is scheduled to be demolished, a user can tour the building or the like in the virtual space even after the building or the like is demolished.
[0083] Furthermore, the server 20 detects a plurality of objects OR included in the captured image PC, and generates a plurality of object models OV corresponding to the plurality of objects OR, respectively. The virtual space model MV is generated by arranging the individually generated object models OV. Therefore, even if one object model OV is moved in the virtual space, the part of the other object models OV that overlaps with the one object model OV is visually recognized by the user, and no discomfort is felt. In other words, according to the virtual space model MV generated by the server 20, it is possible to move the object model OV in the virtual space, and an experience close to the real space SR is possible.
[0084] In addition, the server 20 estimates the positions of the ends of the walls forming the indoor space that are not shown in the captured image PC based on the positions of the ends of the walls that are shown in the captured image PC. Thus, the outer edge of the indoor space is identified by simple processing. In addition, the indoor space is generally a rectangular parallelepiped, and by determining the positions of the walls based on this, the distortion of the captured image PC can be corrected. Thus, the positions of the object models OV other than the walls can also be calculated more accurately.
[0085] Furthermore, the server 20 complements the shapes of the parts of the multiple objects OR that are not shown in the captured image PC based on a standard shape defined for each type of object OR. Therefore, even if an image capturing the entire circumference of the object OR is not available, an object model OV that imitates the entire object OR can be generated.
[0086] In addition, the server 20 acquires motion information indicating the motion of the user viewing the virtual space model MV, and changes the position of the object model OV in the virtual space model MV based on the motion information. Thus, the user can change the position of the object model OV based on his / her own motion, and can experience a virtual space model MV close to the real space SR.
[0087] Furthermore, the server 20 accepts a setting for each object OR as to whether it is movable or not, and changes the position of the object model OV corresponding to the object OR set as movable based on the movement information. Therefore, it is possible to set movable object models OV and immovable object models OV, and it is possible to prevent, for example, the state of the virtual space model MV from being changed unintentionally.
[0088] B: Variation The following are examples of modified aspects of the above-described embodiment. Two or more aspects selected from the following modified aspects may be combined as appropriate within the scope of not being mutually contradictory.
[0089] B1: First modified example In the above-described embodiment, the user's movement is a movement that moves the object model OV that constitutes the virtual space model MV. However, the object model OV may be moved by a user's movement that does not intend to move the object model OV. For example, even if the hand of an avatar in the virtual space model MV accidentally hits the object model OV, the position of the object model OV may be shifted.
[0090] Furthermore, the server 20 may receive not only an instruction to change the position of the object model OV, but also an instruction to delete the object model OV from the virtual space model MV.
[0091] C:Other (1) In the above-described embodiments, ROM and RAM are given as examples of storage devices 105, 202, and 306, but storage devices 105, 202, and 306 may also be a flexible disk, a magneto-optical disk (e.g., a compact disk, a digital versatile disk, a Blu-ray (registered trademark) disc), a smart card, a flash memory device (e.g., a card, a stick, a key drive), a CD-ROM (Compact Disc-ROM), a register, a removable disk, a hard disk, a floppy (registered trademark) disk, a magnetic strip, a database, a server, or any other suitable storage medium.
[0092] (2) In the above-described embodiments, the described information, signals, etc. may be represented using any of a variety of different technologies. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.
[0093] (3) In the above-described embodiment, the input / output information, etc. may be stored in a specific location (e.g., memory) or may be managed using a management table. The input / output information, etc. may be overwritten, updated, or added. The output information, etc. may be deleted. The input information, etc. may be transmitted to another device.
[0094] (4) In the above-described embodiments, the determination may be made based on a value represented by 1 bit (0 or 1), a Boolean (true or false) value, or a numerical comparison (e.g., comparison with a predetermined value).
[0095] (5) The order of the process steps, sequences, flow charts, etc. illustrated in the above-described embodiments may be changed without causing any inconsistency. For example, the methods described in this disclosure present elements of various steps using an example order, and are not limited to the particular order presented.
[0096] (6) Each function illustrated in Figures 2 to 4 is realized by any combination of at least one of hardware and software. Furthermore, the method of realizing each functional block is not particularly limited. That is, each functional block may be realized by using one device that is physically or logically combined, or may be realized by using two or more devices that are physically or logically separated and directly or indirectly connected (for example, by using wires, wirelessly, etc.). The functional block may be realized by combining the one device or the multiple devices with software.
[0097] (7) The programs exemplified in the above-described embodiments should be broadly interpreted to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, execution threads, procedures, functions, etc., regardless of whether they are called software, firmware, middleware, microcode, hardware description language, or by other names.
[0098] Additionally, software, instructions, information, etc. may be transmitted or received over a transmission medium. For example, if the software is transmitted from a website, server, or other remote source using wired technologies (such as coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL)), and / or wireless technologies (such as infrared, microwave, etc.), then these wired and / or wireless technologies are included within the definition of transmission media.
[0099] (8) In each of the above embodiments, the terms “system” and “network” are used interchangeably.
[0100] (9) The information, parameters, etc. described in this disclosure may be expressed using absolute values, relative values from a predetermined value, or corresponding other information.
[0101] (10) In the above-described embodiments, the portable device may be a mobile station (MS). A mobile station may be referred to by those skilled in the art as a subscriber station, a mobile unit, a subscriber unit, a wireless unit, a remote unit, a mobile device, a wireless device, a wireless communication device, a remote device, a mobile subscriber station, an access terminal, a mobile terminal, a wireless terminal, a remote terminal, a handset, a user agent, a mobile client, a client, or some other suitable term. In this disclosure, the terms "mobile station", "user terminal", "user equipment (UE)", "terminal", etc. may be used interchangeably.
[0102] (11) In the above-mentioned embodiments, the terms "connected" and "coupled" or any variation thereof refer to any direct or indirect connection or coupling between two or more elements, and may include the presence of one or more intermediate elements between two elements that are "connected" or "coupled" to each other. The coupling or connection between elements may be physical, logical, or a combination thereof. For example, "connected" may be read as "accessed." As used in this disclosure, two elements may be considered to be "connected" or "coupled" to each other using at least one of one or more wires, cables, and printed electrical connections, as well as electromagnetic energy having wavelengths in the radio frequency range, microwave range, and light (both visible and invisible) range, as some non-limiting and non-exhaustive examples.
[0103] (12) In the above embodiments, the phrase "based on" does not mean "based only on," unless otherwise specified. In other words, the phrase "based on" means both "based only on" and "based at least on."
[0104] (13) As used in this disclosure, the terms "determining" and "determining" may encompass a wide variety of actions. "Determining" and "determining" may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, search, inquiry (e.g., searching in a table, database, or other data structure), ascertaining, and the like. Also, "determining" and "determining" may include receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, accessing (e.g., accessing data in a memory), and the like. In addition, "judgment" and "decision" can include considering resolving, selecting, choosing, establishing, comparing, etc., to be a "judgment" or "decision." In other words, "judgment" and "decision" can include considering some action to be a "judgment" or "decision." In addition, "judgment (decision)" can be interpreted as "assuming," "expecting," "considering," etc.
[0105] (14) In the above embodiments, when "include," "including," and variations thereof are used, these terms are intended to be inclusive, similar to the term "comprising." Furthermore, the term "or" used in this disclosure is not intended to be an exclusive or.
[0106] (15) In this disclosure, where articles have been added by translation, such as a, an, and the in English, the disclosure may include that the nouns following these articles are in the plural.
[0107] (16) In this disclosure, the term "A and B are different" may mean "A and B are different from each other." The term may also mean "A and B are each different from C." Terms such as "separate" and "combined" may also be interpreted in the same way as "different."
[0108] (17) Each aspect / embodiment described in this disclosure may be used alone, in combination, or switched according to execution. In addition, notification of specific information (e.g., notification that "X is the case") is not limited to being done explicitly, but may be done implicitly (e.g., not notifying the specific information). [Explanation of symbols]
[0109] 1...virtual space display system, 10...imaging terminal, 20...server, 30...access terminal, 111...imaging data transmission unit, 211...image acquisition unit, 212...object detection unit, 213...reception unit, 214...first generation unit, 215...second generation unit, 216...model data transmission unit, 217...movement information acquisition unit, 218...change unit, 311...model data acquisition unit, 312...display control unit, 313...motion detection unit, DM...model data, DS...standard shape data, DV...virtual space data, MV (MV1 to MV4)...virtual space model, N...network, OR...object, OV...object model, PC...captured image, SR...real space.
Claims
1. A virtual space generating device that generates a virtual space model that imitates a real space, an image acquisition unit that acquires a captured image of the real space; A detection unit that detects a plurality of objects included in the captured image; a first generation unit that generates a plurality of three-dimensional models corresponding to the plurality of objects, respectively; a second generation unit that arranges the plurality of three-dimensional models based on the captured image and generates the virtual space model; A virtual space generating device comprising:
2. the real space is an indoor space surrounded by walls, The first generation unit generates a three-dimensional model of the wall surface by estimating positions of portions of the ends of the wall surfaces that form the indoor space that are not captured in the captured image based on positions of the portions of the ends that are captured in the captured image. The virtual space generating device according to claim 1.
3. The detection unit identifies a type of each of the plurality of objects; the first generation unit generates the plurality of three-dimensional models by complementing shapes of portions of the plurality of objects that are not captured in the captured image based on a standard shape defined for each type of object. The virtual space generating device according to claim 1.
4. a motion information acquisition unit that acquires motion information indicating a motion of a user viewing the virtual space model; and a change unit that changes a position of at least one three-dimensional model in the virtual space model based on the movement information. The virtual space generating device according to claim 1.
5. a receiving unit configured to receive a setting as to whether or not each of the plurality of objects detected by the detection unit can be moved, the change unit changes a position of a three-dimensional model corresponding to the object set to be movable by the reception unit based on the movement information.
5. The virtual space generating device according to claim 4.
Citation Information
Patent Citations
Information processing device, information processing method, and program
JP7218979B1
