Generation apparatus, generation method, and generation program

Abstract

To simply and properly generate a three-dimensional model that reflects a state of a room, from image data.SOLUTION: A generation apparatus includes: a first generation unit which generates a first three-dimensional model and a floor plan of a part of a room, from image data obtained by imaging the part of the room; a retrieval unit which calculates a similarity between a partial floor plan and an entire floor plan of the room stored in a floor plan information storage unit, to retrieve an entire floor plan having a similarity which is higher than a predetermined threshold; and a second generation unit which uses the first three-dimensional model for a region corresponding to at least a part of the partial floor plan in the entire floor plan, and uses a second three-dimensional model generated based on the entire floor plan for regions other than the above region, to generate an entire three-dimensional model for the imaged room.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present disclosure relates to a generation device, a generation method, and a generation program. [Background technology]

[0002] Common methods for generating a 3D model of the interior state of a room are to use a 3D scanner or photogrammetry.

[0003] For example, Patent Document 1 listed below discloses a floor plan creation system that can accurately read various types of floor plans without error and unify the displays when automatically creating floor plans. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2022-57270 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the technology described in Patent Document 1 requires measurements of the entire room and taking multiple photos to generate a three-dimensional model. There is also technology that considers a room as a simplified rectangular space and estimates its height and width, but this requires complex processing.

[0006] In view of the above problems, the present disclosure aims to provide a generation device, a generation method, and a generation program that can easily and appropriately generate a three-dimensional model from image data. [Means for solving the problem]

[0007] In order to solve the above-mentioned problems and achieve the object, the generation device of the present disclosure includes a first generation unit that generates a first three-dimensional model and a floor plan of the imaged portion of the room from image data of a portion of a room; a search unit that calculates a similarity between an entire floor plan of the room stored in a floor plan information storage unit and the partial floor plan and searches for the entire floor plan whose similarity is higher than a predetermined threshold; and a second generation unit that generates a three-dimensional model of the entire imaged room by using the first three-dimensional model in an area of ​​the entire floor plan that corresponds to at least a portion of the partial floor plan, and by using a second three-dimensional model generated based on the entire floor plan in an area other than the first area.

[0008] In order to solve the above-mentioned problems and achieve the objective, the generation method of the present disclosure includes the steps of generating a first three-dimensional model and a floor plan of a portion of the room from image data in which a portion of a room is captured, calculating a similarity between an entire floor plan of the room stored in a floor plan information storage unit and the partial floor plan, and searching for an entire floor plan of the room whose similarity is higher than a predetermined threshold, and generating a three-dimensional model of the entire captured room by using the first three-dimensional model in an area of ​​the entire floor plan that corresponds to at least a portion of the partial floor plan, and using a second three-dimensional model generated based on the entire floor plan in an area other than the area.

[0009] In order to solve the above-mentioned problems and achieve the object, the generation program of the present disclosure causes a computer to execute the steps of generating a first three-dimensional model and a floor plan of a portion of the room from image data of a portion of the room, and calculating a similarity between the entire floor plan of the room stored in a floor plan information storage unit and the partial floor plan, and searching for an entire floor plan of the room whose similarity is higher than a predetermined threshold, and generates a three-dimensional model of the entire imaged room using the first three-dimensional model in an area of ​​the entire floor plan that corresponds to at least a portion of the partial floor plan, and using a second three-dimensional model generated based on the entire floor plan in an area other than the area. Effect of the Invention

[0010] According to the present disclosure, it is possible to provide a generation device, a generation method, and a generation program that can easily and appropriately generate a three-dimensional model from image data. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram for explaining an overview of a generating device according to this embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of the configuration of a generation system according to the present embodiment. [Diagram 3] FIG. 3 is a diagram illustrating an example of the configuration of a generating device according to the present embodiment. [Figure 4] FIG. 4 is a diagram showing an example of information stored in the image data storage unit of the generating device according to this embodiment. [Diagram 5] FIG. 5 is a diagram showing an example of information stored in the floor plan information storage unit of the generating device according to the present embodiment. [Figure 6] FIG. 6 is a diagram illustrating an example of information stored in the facility information storage unit of the generating device according to the present embodiment. [Figure 7] FIG. 7 is a diagram showing an example of image data that is the basis for generating a 3D model by the generating device according to this embodiment. [Figure 8] FIG. 8 is a diagram showing an example of a partial floor plan of a room generated by the generation device according to the present embodiment. [Figure 9] FIG. 9 is a diagram showing an example of an overall floor plan of a room generated by the generation device according to the present embodiment. [Figure 10] FIG. 10 is a diagram showing an example of a facility model generated by the generating device according to the present embodiment. [Figure 11] FIG. 11 is a flowchart showing the flow of the generation method according to this embodiment. [Figure 12] FIG. 12 is a diagram illustrating a configuration example of a server device according to another example of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Note that the present disclosure is not limited to the embodiments described below.

[0013] (Overview of the generator) First, an overview of a generating device 100 according to this embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram for explaining an overview of a generating device according to this embodiment. As shown in Fig. 1, the generating device 100 according to this embodiment is an information processing device in which, for example, a person P1 captures an image of a part of a room, and generates a three-dimensional model of the entire room based on the captured image data.

[0014] For example, the generating device 100 reproduces the walls W1, W2, W3, ceiling C1, and floor L1 of the room shown in FIG. 1 using a three-dimensional model, and then places three-dimensional models of windows WDL, WDR, air conditioner A1, and other elements arranged in the room within the three-dimensional model to generate a three-dimensional model that reproduces the space of the room depicted in the image data.

[0015] In this way, if a three-dimensional model of a room space can be generated, it is possible to achieve an effect that, in a conversation service using avatars in a virtual space using VR (Virtual Reality) technology, a friend's avatar can be invited to the room space in the generated three-dimensional model and a virtual conversation can be held in the room. However, the use of the generated three-dimensional model may be arbitrary.

[0016] (Generation system configuration) Next, a configuration example of the generation system according to this embodiment will be described with reference to Fig. 2. Fig. 2 is a diagram showing a configuration example of the generation system according to this embodiment. As shown in Fig. 2, the generation system 1 according to this embodiment includes a generation device 100, a server device 200, and a network N. Note that, as shown in Fig. 2, the generation system 1 may include a plurality of generation devices 100. The configuration will be briefly described below.

[0017] The generating device 100 is an information processing device having an imaging function. The generating device 100 may be, for example, an information processing device such as a camera, a smartphone, a tablet terminal, a wearable terminal, a mobile phone, or a PDA (Personal Digital Assistant). In the example shown in FIG. 1, the generating device 100 is a camera. Furthermore, as shown in FIG. 2, the generating system 1 may include a plurality of generating devices 100.

[0018] The server device 200 is an information processing device that executes various information processes. For example, the server device 200 executes a process of generating a three-dimensional model using image data captured by the generating device 100 and transmitted from the generating device 100 via a network N. The server device 200 may be realized by an information processing device such as a PC (Personal Computer), a WS (Work Station), or a computer having a server function.

[0019] The network N connects the generating device 100 and the server device 200 in a wired or wireless manner so that they can communicate with each other. If the network N is wired, it may be realized by ETHERNET (registered trademark) defined in IEEE802.3, or a control signal line of USB (Universal Serial Bus) and a video signal line of SDI (Serial Digital Interface). If the network N is wireless, it may be realized by Bluetooth (registered trademark) or a wireless LAN (Local Area Network) defined in IEEE802.11.

[0020] In this manner, the generation system 1 functions as a single system by multiple devices transmitting and receiving information to and from each other.

[0021] (Configuration of the generating device) Next, a configuration example of the generating device 100 according to this embodiment will be described with reference to Fig. 3. Fig. 3 is a diagram showing a configuration example of the generating device according to this embodiment. As shown in Fig. 3, the generating device 100 according to this embodiment includes a communication unit 110, a storage unit 120, a control unit 130, an input unit 140, an imaging unit 150, a sound output unit 160, and a display unit 170. Below, these configurations will be described in order.

[0022] The communication unit 110 is a communication module that allows the generating device 100 to communicate with an external device such as the server device 200. The communication unit 110 is responsible for transmitting and receiving data to and from the external device via wireless communication or wired communication. In the case of wireless communication, the communication unit 110 may include a communication antenna, an RF (Radio Frequency) circuit, and other circuits for communication processing. In the case of wired communication, the communication unit 110 may include, for example, a LAN (Local Area Network) terminal, a transmission circuit, and other circuits for communication processing.

[0023] The storage unit 120 is a storage device that stores various types of information. The storage unit 120 includes a main storage device and an auxiliary storage device. The main storage device may be realized by a semiconductor memory element such as a random access memory (RAM), a read only memory (ROM), or a flash memory. The auxiliary storage device may be realized by a hard disk, a solid state drive (SSD), an optical disk, or the like.

[0024] 3, the storage unit 120 includes an image data storage unit 121, a floor plan information storage unit 122, and an equipment information storage unit 123. The information stored in these components will be described below in order.

[0025] The image data storage unit 121 stores information related to image data. Here, an example of information stored in the image data storage unit 121 will be described with reference to Fig. 4. Fig. 4 is a diagram showing an example of information stored in the image data storage unit of the generating device according to this embodiment.

[0026] As shown in FIG. 4, the image data storage unit 121 stores information relating to the items "image data ID" and "image data".

[0027] The "image data ID" is an identifier for identifying image data, and is represented by a character string, a number, etc. The "image data" is image data identified by the "image data ID", and may be data in a file format such as JPEG (Joint Photographic Experts Group) or TIFF (Tag Image File Format).

[0028] That is, FIG. 4 shows an example in which image data "IMGDT#1" identified by an image data ID "IMGID#1" is stored.

[0029] It should be noted that the information stored in the image data storage unit 121 is not limited to information relating to the items "image data ID" and "image data", and any other information relating to image data may be stored.

[0030] The floor plan information storage unit 122 stores floor plan information indicating information related to the floor plan. The floor plan information stored in the floor plan information storage unit 122 includes image data of the entire floor plan of the room. In this embodiment, the floor plan information storage unit 122 stores floor plan information for at least the room other than the room imaged by the imaging unit 150. The floor plan information may be acquired by any method, but in this embodiment, it may be acquired from an external server other than the server device 200. That is, for example, the floor plan information storage unit 122 stores floor plan information acquired from an external server. The floor plan information storage unit 122 may store floor plan information acquired from one server, or may store floor plan information acquired from multiple servers. Here, an example of information stored in the floor plan information storage unit 122 will be described with reference to FIG. 5. FIG. 5 is a diagram showing an example of information stored in the floor plan information storage unit of the generating device according to this embodiment.

[0031] As shown in FIG. 5, the floor plan information storage unit 122 stores, as floor plan information, information relating to the following items: “Floor plan ID,” “Floor plan data,” “Total room area,” “Wall length,” “Position and size of pillars,” “Position of doors and windows,” “Door rotation direction,” and “Room type.”

[0032] The "floor plan ID" is an identifier for identifying the floor plan, and is represented by a character string or a number. The "floor plan data" is image data of the floor plan identified by the "floor plan ID". The "total room area" is the total area of ​​the room shown on the floor plan identified by the "floor plan ID", and is, for example, m 2 "Wall length" is the length of the wall of the room shown on the floor plan identified by the "Floor plan ID" and is expressed in units such as m (meters). "Column position and size" is information indicating the position and size of the column in the room shown on the floor plan identified by the "Floor plan ID" and is expressed, for example, by the position coordinates of the column on the floor plan and the cross-sectional area of ​​the column.

[0033] "Door and window position" is information indicating the positions of doors and windows in a room shown on a floor plan identified by a "Floor plan ID", and is represented, for example, by the position coordinates of the door and the position coordinates of the window on the floor plan. "Door rotation direction" is information indicating the rotation direction of a door in a room shown on a floor plan identified by a "Floor plan ID", and is represented, for example, by clockwise or counterclockwise when viewing the room layout from above. "Room type" is information indicating the type of room shown on a floor plan identified by a "Floor plan ID", and is, for example, a living room, kitchen, bathroom, etc.

[0034] That is, Figure 5 shows an example in which the floor plan data "FPDT#1" for a room shown on a floor plan identified by a floor plan ID "FPID#1" stores the total area of ​​the room "RAR#1", the length of the walls of the room "WLG#1", the position and size of the pillars of the room "CLP#1", the position of the door and window of the room "DRP#1", the door rotation direction "DRRD#1", and the type of room "RTYP#1".

[0035] The information stored in the floor plan information storage unit 122 is not limited to information related to the items "floor plan ID", "floor plan data", "total area of ​​the room", "wall length", "position and size of pillars", "position of doors and windows", "rotation direction of doors", and "type of room", and may store information related to any other floor plan. Moreover, the floor plan information storage unit 122 may store at least "floor plan data", which is image data of the entire floor plan of the room, as floor plan information, and may not store other data such as "total area of ​​the room".

[0036] The facility information storage unit 123 stores information related to a three-dimensional model of a facility arranged in a room. The facility here refers to a facility installed in a room, and may be, for example, a door, a window, a toilet, etc. The information related to the three-dimensional model of the facility stored in the facility information storage unit 123 includes three-dimensional model data of the facility. The information related to the three-dimensional model of the facility may be acquired by any method, but in this embodiment, it may be acquired from an external server other than the server device 200. That is, for example, the facility information storage unit 123 stores information related to the three-dimensional model of the facility acquired from an external server. Here, an example of information stored in the facility information storage unit 123 will be described with reference to FIG. 6. FIG. 6 is a diagram showing an example of information stored in the facility information storage unit of the generating device according to this embodiment.

[0037] As shown in FIG. 6, the facility information storage unit 123 stores information relating to the three-dimensional model of the facility, including information on the following items: “floor plan ID”, “facility ID”, “facility category”, “overall facility size”, “three-dimensional model data”, “movable structure”, and “facility location”.

[0038] The "floor plan ID" is an identifier for identifying the floor plan, and is represented by a character string or a number. The "equipment ID" is an identifier for identifying equipment arranged in a room of the floor plan identified by the "floor plan ID", and is represented by a character string or a number. The "equipment category" is information indicating the type of equipment identified by the "equipment ID", and is, for example, furniture, electrical appliances, storage items, and the like. The "size of the entire equipment" is information indicating the size of the equipment, and may be represented by, for example, the width, depth, and height of the entire equipment. The "three-dimensional model data" is three-dimensional model data of the equipment identified by the "equipment ID", and may store data drawn by three-dimensional CG (Computer Graphics) software or three-dimensional CAD (Computer Aided Design). The "movable structure" is information regarding the movable structure of the equipment, and may be represented by, for example, the movable direction and movable range of the movable part of the equipment. "Equipment location" is information indicating the location of equipment arranged in a room shown on a floor plan identified by a "floor plan ID", and is represented, for example, by the location coordinates of the equipment on the floor plan.

[0039] That is, Figure 6 shows an example in which equipment identified by equipment ID "FCID#1-1" is placed on a floor plan identified by floor plan ID "FPID#1", the equipment in question is classified into equipment category "CTG#1", the overall size of the equipment is "EQSZ#1-1", the three-dimensional model data of the equipment in question is stored as "MDLDT#1-1", the movable structure of the equipment in question is stored as "STR#1-1", and the position of the equipment in question is stored as "EQL#1-1".

[0040] The information stored in the facility information storage unit 123 is not limited to information related to the items "floor plan ID", "facility ID", "facility category", "total facility size", "three-dimensional model data", "movable structure", and "facility location", and may store information related to any other facility information. Furthermore, the facility information storage unit 123 may store at least "three-dimensional model data", which is three-dimensional model data of the facility, as information related to the three-dimensional model of the facility, and may not store other data such as "movable structure".

[0041] Next, returning to Fig. 3, the control unit 130 will be described. The control unit 130 is a controller that manages and controls the generating device 100. The control unit 130 is realized by a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or the like, executing various programs stored in the storage unit 120 using a RAM as a working area. The control unit 130 may also be realized by an integrated circuit, such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0042] As shown in FIG. 3, the control unit 130 includes an acquisition unit 131, a first generation unit 132, a search unit 133, a second generation unit 134, and a determination unit 136.

[0043] The control unit 130 realizes these functions by reading and executing a program (software) from the storage unit 120. Note that these functions of the control unit 130 may be realized by electronic circuits. The control unit 130 may execute these processes by one CPU, or may be provided with multiple CPUs and execute these processes in parallel by the multiple CPUs. The details of these processes will be described later.

[0044] Various types of operation information are input from the user to the input unit 140. For example, the input unit 140 may accept various operations from the user via a display surface (e.g., the display unit 170) using a touch panel. The input unit 140 may also accept various operations from the user using various buttons, a keyboard, or a mouse.

[0045] The imaging unit 150 captures various images. The imaging unit 150 includes an optical element and an imaging element. The optical element is an element that constitutes an optical system, such as a lens, a mirror, a prism, and a filter. The imaging element is an element that converts light incident through the optical element into an image signal, which is an electrical signal. The imaging element is, for example, a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor.

[0046] The sound output unit 160 outputs various sounds. For example, when the processing in the generating device 100 is completed, the sound output unit 160 may output a sound notifying the completion of the processing. The sound output unit 160 may be a speaker, and the speaker converts an electric signal into sound using a diaphragm. That is, the speaker vibrates the diaphragm with a predetermined amplitude and frequency based on a control command given by the electric signal, thereby vibrating the air in contact with the diaphragm to output sound.

[0047] The display unit 170 displays various types of information. For example, the display unit 170 may display a GUI (Graphical User Interface) for executing the processing of the generating device 100, a three-dimensional model generated by the generating processing, and the like. The display unit 170 may be realized by a liquid crystal display, an organic EL (Electro Luminescence) display, a micro LED (Light Emitting Diode) display, or the like. The display unit 170 may also be a touch panel that displays various types of information by various methods such as a capacitive method, and receives operation information from a user.

[0048] (Processing of generating device) Next, the process performed by generation device 100 will be described.

[0049] (Image data acquisition) The acquisition unit 131 of the generating device 100 acquires various image data from inside or outside the generating device 100. For example, the acquisition unit 131 acquires image data of a part of a room captured by the imaging unit 150. After acquiring the image data captured by the imaging unit 150, the acquisition unit 131 stores the acquired image data in the image data storage unit 121. Note that when the acquisition unit 131 receives designation of image data and an instruction to generate a three-dimensional model from a user via the input unit 140, it outputs the designated image data to the first generating unit 132.

[0050] (Generation of first 3D model and partial floor plan) The first generating unit 132 generates a first three-dimensional model from image data in which a part of a room is captured. The first three-dimensional model is a three-dimensional model that represents a part of the captured room captured in the image data as a three-dimensional stereoscopic image. Specifically, the first generating unit 132 generates the first three-dimensional model by three-dimensional modeling of the part of the room captured in the image data, which is two-dimensional image data. The first generating unit 132 may generate the first three-dimensional model by any method based on the image data. For example, the first generating unit 132 may calculate the depth using a model that learns the relationship between the pixels of the image data and the depth using a deep neural network, detect the boundaries of the walls and the floor using an object detection technology described later, and generate a three-dimensional model of the part of the room based on the calculated depth. For the part of the room captured in the image data, the three-dimensional model generated here is used as it is as the three-dimensional model of the final output, so a method that can obtain detailed information may be adopted. FIG. 7 shows an example of image data in which a part of a room is captured and input to the first generating unit 132. 7 is a diagram showing an example of image data that is the basis for generating a 3D model by the generating device according to the present disclosure. As shown in FIG. 7, the image data input to the first generating unit 132 may be image data in which a part of a room is captured, and does not need to be image data in which the entire room is captured.

[0051] Further, the first generating unit 132 generates a partial floor plan, which is a floor plan of an area in which a part of the room is shown, based on image data in which a part of the room is captured. The floor plan here is a two-dimensional image showing the layout of the room, and is a two-dimensional image of the layout of the room viewed from above. The first generating unit 132 may generate the partial floor plan by any method based on the image data. For example, the first generating unit 132 may use an object detection technique described later to detect walls and pillars shown in the image data, and generate the partial floor plan by drawing the outline of the floor plan based on the respective position coordinates and positional relationship. Furthermore, if structures such as windows, verandas, and doors are shown in the image data in which a part of the room is captured, the first generating unit 132 detects these by object detection and reflects them in the floor plan based on the respective position coordinates and positional relationship. If the type of room, such as a toilet, a washstand, or a kitchen, can be identified by object detection, the information on the room may also be reflected in the partial floor plan.

[0052] FIG. 8 shows an example of a partial floor plan generated when the image data shown in FIG. 7 is input. FIG. 8 is a diagram showing an example of a partial floor plan of a room generated by a generating device according to this embodiment. As shown in FIG. 8, the first generating unit 132 generates a partial floor plan (a floor plan of the area of ​​the room captured in the image data) based on image data showing a part of the room. FIG. 8 shows that a floor plan of a part of a room has been generated, in which an air conditioner is placed in one corner of the room and there is a balcony behind the window.

[0053] In the above description, the first generating unit 132 generates the first three-dimensional model and the partial floor plan from one image (image data acquired in one image capture). However, the present invention is not limited to this, and the first generating unit 132 may generate the first three-dimensional model and the partial floor plan from multiple images (image data acquired in multiple images capture) in which a portion of a room is captured. However, even in this case, it is preferable that the entire room is not captured even when multiple images are combined.

[0054] (Classification) The first generating unit 132 may classify the class of the room (imaged room) shown in the image data based on image data showing a part of the room. The class here refers to the class of the imaged room when various rooms are classified into a plurality of classes based on a predetermined criterion. The type of class and the method of classifying the classes may be arbitrary. For example, the type of class may be the layout type of the room (one room, one living room, one dining room, etc.). In this case, for example, the first generating unit 132 may estimate the layout type of the room from the image data showing a part of the room, and set the estimated layout type as the class of the room. The method of estimating the layout type from the image data may be arbitrary, but for example, the layout type may be estimated by inputting the currently acquired image data into a machine learning model that has machine-learned the correspondence between the image data of a part of the room and the layout type.

[0055] Also, for example, the first generating unit 132 may detect an object from image data showing a part of a room, and classify the object into a class based on the feature amount of the object. Here, the object refers to an object shown in the image data, such as a desk, a chair, a wall, a pillar, a window, or a door. The detection of an object from the image data may involve, for example, detecting an object captured in the image data, and executing a process of classifying the detected object. For example, this may be realized by semantic segmentation, which is a method of classifying pixels of the image data according to which object class they belong to, using a convolutional neural network (CNN) or the like.

[0056] Further, for example, the first generating unit 132 may classify the image data based on the imaging conditions when the image data showing a part of the room was captured. For example, the first generating unit 132 may classify the image data based on the position information indicating the location where the image was captured. In this case, for example, the image data may be classified by the district where the room exists. The first generating unit 132 may identify the district where the image data was captured from the position information, and set the district as the class of the room. At this time, the first generating unit 132 may use the position information stored as metadata of the image. Further, for example, the first generating unit 132 may classify the image data based on the imaging direction of the image data (the direction in which the camera is pointed) and the object shown in the image. In this case, for example, the image data may be classified by the type of object and the direction in which the object is located (for example, a window and the direction of the window). The first generating unit 132 may identify the type of object and the direction in which the object is located (for example, a window facing south) from the position information, and set the identification result as the class of the room. The imaging conditions may be acquired together with the image data.

[0057] Also for example, the class may be specified by the user via the input unit 140 .

[0058] (Notification of reacquisition of image data) The first generating unit 132 may determine whether reacquisition of image data is necessary based on objects appearing in image data showing a part of the room. For example, when the number of predetermined objects appearing in the image data is equal to or less than a predetermined number, the first generating unit 132 may determine that reacquisition of image data is necessary and output a notification prompting reacquisition of the image data. The type of the predetermined object and the predetermined number here may be arbitrary, but for example, the first generating unit 132 may output a notification prompting reacquisition of the image data when the number of pillars appearing in the image data is two or less. The method of outputting the notification prompting reacquisition of the image data may be arbitrary, and the notification may be displayed on the display unit 170, or the notification may be output by sound from the sound output unit 160.

[0059] After outputting a notification prompting reacquisition of the image data, if the image data is reacquired by a user operation, the first generation unit 132 performs the above-described processing using the reacquired image data to generate a first three-dimensional model, a partial floor plan, class classification, etc. If the number of predetermined objects shown in the image data exceeds a predetermined number, the first generation unit 132 determines that reacquisition of the image data is unnecessary, and performs the processing described below.

[0060] (Search for full floor plan) The search unit 133 acquires an overall floor plan based on a floor plan of the entire room similar to the partial floor plan generated by the first generation unit 132. That is, the overall floor plan is a floor plan showing the entire room similar to the partial floor plan. Specifically, the search unit 133 calculates the similarity between the overall floor plan of the room stored in the floor plan information storage unit 122 and the partial floor plan, and searches for an overall floor plan of the room whose similarity is higher than a predetermined threshold. That is, the search unit 133 searches for an overall floor plan of the room stored in the floor plan information storage unit 122 whose similarity with the partial floor plan is higher than a threshold, and acquires it as the overall floor plan.

[0061] Specifically, the search unit 133 calculates the similarity based on the generated partial floor plan by comparing it with the floor plan stored in the floor plan information storage unit 122. The calculation of the similarity may be performed by any method, but in this embodiment, the search unit 133 calculates the similarity by comparing the partial floor plan with the floor plan stored in the floor plan information storage unit 122 and calculating the difference between them. For example, if there is a part in the entire floor plan of the room that matches the partial floor plan, the similarity is set to 1. Here, the match indicates that the absolute value of the difference in pixel values ​​between a part of the image of the entire floor plan of the room and the image of the partial floor plan is 0. The search unit 133 aligns the position of the image of the partial floor plan with the image of the entire floor plan of the room. In order to align the position, the search unit 133 may perform processing such as moving the image in the width direction, moving in the depth direction, enlarging, reducing, and rotating. The search unit 133 calculates the absolute value of the difference for each position adjustment, and the absolute value of the difference at the position where the value is minimum is set as the absolute value of the difference of the partial floor plan image. The search unit 133 calculates the absolute value of the difference (ratio of difference) for the product of the number of pixels and the number of bits of the partial floor plan image, and subtracts the value from 1 to calculate the similarity. Furthermore, when the search unit 133 adjusts the positions of the image of the entire floor plan of the room and the image of the partial floor plan, the search unit 133 may store an area of ​​the image of the entire floor plan of the room that corresponds to the image of the partial floor plan as the alignment information in the storage unit 120. The search unit 133 may include in the alignment information what kind of processing was performed when the positions of the images were adjusted, such as the position where the absolute value of the difference is minimum. Furthermore, when the entire floor plan of the room has a part that partially matches the partial floor plan, the search unit 133 may calculate the ratio of partial match as the similarity. After the similarity is calculated, the search unit 133 searches for an entire floor plan of the room whose similarity is higher than a predetermined threshold value. The predetermined threshold may be set arbitrarily, for example, in the case of the above-mentioned method of calculating the similarity, it may be set to 0.8. When a plurality of overall floor plans of a room having a similarity higher than the predetermined threshold are searched for, the search unit 133 may select an overall floor plan from among the plurality of floor plans by an arbitrary method.For example, the search unit 133 may determine the one with the highest similarity as the overall floor plan, may randomly select an overall floor plan, or may accept a selection of the overall floor plan of the room from the user via the input unit 140 and determine the overall floor plan based on that.

[0062] The search unit 133 may also search the floor plan information storage unit 122 for an entire floor plan of a room having a part that matches the partial floor plan generated by the first generation unit 132. In the example of searching by calculating the similarity described above, it may search for a floor plan with a similarity of 1. If the search unit 133 finds only one entire floor plan of a room having a matching part, it determines the entire floor plan of the room as the entire floor plan of the room. If the search unit 133 finds multiple entire floor plans of a room having a matching part, it may select the entire floor plan by any method, as in the case where multiple floor plans with a similarity higher than a predetermined threshold are found. If the search unit 133 does not find an entire floor plan of a room having a matching part, it may perform a re-search by relaxing only some of the conditions such as the length of the walls, or may determine a simple floor plan consisting of only walls and an entrance as the entire floor plan of the room. In this way, the search unit 133 determines an entire floor plan of the room including parts of the room that are not shown in the image data.

[0063] In addition, when the image data is classified into classes as described above, the search unit 133 may search for the overall floor plan based on the class as well. Specifically, when searching for an overall floor plan of a room whose similarity to a partial floor plan is higher than a predetermined threshold, the search unit 133 may calculate a high similarity of the overall floor plan of the room having the same class as the class assigned to the image data. In this case, a class is also assigned to the floor plans stored in the floor plan information storage unit 122. A class may be assigned in advance to these floor plans, for example, as in "room type" in FIG. 5, or the search unit 133 may set a class based on the floor plan. For example, the search unit 133 selects, from among the floor plans stored in the floor plan information storage unit 122, a floor plan that is assigned a class that matches the class of the image data and has a similarity higher than a threshold as the overall floor plan. That is, in this case, it can be said that the candidate floor plans are sorted by class, and it is not necessary to calculate a similarity for floor plans of different classes. However, without being limited thereto, the search unit 133 may add a predetermined value to the similarity calculated by the above-mentioned method for a floor plan of a room to which a class matching the class of the image data is assigned. This allows floor plans of the same class to be preferentially selected as the overall floor plan.

[0064] Fig. 9 shows an example of an overall floor plan determined by the search unit 133 searching the floor plan information storage unit 122 for an overall floor plan of a room when the partial floor plan shown in Fig. 8 is input. Fig. 9 is a diagram showing an example of an overall floor plan of a room generated by the generating device according to this embodiment. As shown in Fig. 9, the search unit 133 determines the overall floor plan based on the partial floor plan.

[0065] (Generation of a full 3D model) The second generating unit 134 generates an overall three-dimensional model showing an overall model of the room based on the overall floor plan of the room and the first three-dimensional model (a three-dimensional model of a part of the room). The overall three-dimensional model is a three-dimensional model that represents the entire captured room in a three-dimensional stereoscopic image. Here, an area of ​​the overall floor plan that corresponds to at least a part of the partial floor plan is defined as a first area AR1, and an area of ​​the overall floor plan other than the first area AR1 is defined as a second area AR2. The first area AR1 can be said to be an area that includes at least a part of an area of ​​the overall floor plan that is determined to partially match the partial floor plan, and can also be said to be a part that corresponds to the range of the image data for which the similarity is calculated. For example, in the example of FIG. 9, an area of ​​the overall floor plan where an air conditioner, a balcony, and a window are arranged (an area similar to the partial floor plan of FIG. 8) is defined as the first area AR1. The second generation unit 134 determines an area of ​​the overall floor plan that corresponds to the partial floor plan based on the alignment information stored in the storage unit 120 by the search unit 133. In this case, the second generation unit 134 generates a three-dimensional model of the second area AR2 based on the overall floor plan to set it as a second three-dimensional model. Then, the second generation unit 134 uses a first three-dimensional model generated based on image data showing a part of the room in the first area AR1, and uses a second three-dimensional model generated based on the overall floor plan in the second area AR2 to generate an overall three-dimensional model. In other words, the overall three-dimensional model has the same three-dimensional image data as the first three-dimensional model in the space on the first area AR1, and has the same three-dimensional image data as the second three-dimensional model in the space on the second area AR2.

[0066] The second generating unit 134 may generate the overall three-dimensional model by any method using the first three-dimensional model and the second three-dimensional model. For example, the second generating unit 134 may generate a three-dimensional model of the entire area of ​​the overall floor plan (the entire area including the first area AR1 and the second area AR2) as the second three-dimensional model based on the overall floor plan. Then, the second generating unit 134 may generate the overall three-dimensional model by replacing the three-dimensional image data of the first area AR1 of the second three-dimensional model with the three-dimensional image data of the first three-dimensional model.

[0067] In this case, for example, if the width of a part of the room is W, the depth is D, and the height is H, and the overall width of the room is W' and the depth is D', the second generation unit 134 generates a second three-dimensional model showing the overall model of the room with height H, width W', and depth D'. In generating the second three-dimensional model, the overall width W' and depth D' of the room may be determined based on the information of the overall floor plan of the room determined by the search unit 133, and the second three-dimensional model may be generated. The second generation unit 134 may estimate the width W, depth D, and height H from the captured image by a well-known image recognition method. The second generation unit 134 may set the height H of the second three-dimensional model to be the same as the height H of the first three-dimensional model.

[0068] Then, the second generating unit 134 replaces the portion of the first region AR1 (the portion corresponding to the range of the image data for which the similarity was calculated) in the second three-dimensional model with the first three-dimensional model to generate an entire three-dimensional model. In other words, the second generating unit 134 replaces the portion of the generated second three-dimensional model corresponding to the first three-dimensional model generated by the first generating unit 132 with the first three-dimensional model. For example, in the second three-dimensional model, it is assumed that the range of the width direction coordinate of the first region AR1 is 0 to 10 and the range of the depth direction coordinate is 0 to 20. In this case, the second generating unit 134 replaces the pixel values ​​of the second three-dimensional model corresponding to the range of the width direction coordinate of the second model being 0 to 10 and the range of the depth direction coordinate being 0 to 20 with the pixel values ​​of the first three-dimensional model.

[0069] In the above example, the second generation unit 134 replaces a part of the second three-dimensional model of the entire area of ​​the entire floor plan with the first three-dimensional model to generate the entire three-dimensional model, but is not limited thereto. For example, the second generation unit 134 may generate the entire three-dimensional model by complementing the first three-dimensional model of the first area AR1 with the second three-dimensional model of the second area AR2. In this case, for example, the second generation unit 134 generates a three-dimensional model in the second area AR2 of the entire area of ​​the entire floor plan as the second three-dimensional model. Then, the second generation unit 134 assigns the same three-dimensional image data as the first three-dimensional model to the space on the first area AR1, and assigns the same three-dimensional image data as the second three-dimensional model to the space on the second area AR2 to generate the entire three-dimensional model.

[0070] (First area adjustment) In the above description, the second generating unit 134 applies the first three-dimensional model to the entire area that partially matches the partial floor plan, with the entire area being the first area AR1. However, the present invention is not limited to this. The second generating unit 134 may set the first area AR1 (the area to which the first three-dimensional model is applied) to a narrower area than the area that partially matches the partial floor plan, or may adjust the size of the first area AR1. When adjusting the size of the first area AR1 (the area to which the first three-dimensional model is applied), the second generating unit 134 may adjust the size of the first area AR1 based on image data that shows a part of the room. Specifically, the second generating unit 134 may adjust the size of the first area AR1 based on an object that appears in the image data. For example, when a predetermined object is shown in the image data, the second generating unit 134 may set the first area AR1 to a narrower area than the area that partially matches the partial floor plan.

[0071] (Adjusting the object's appearance) The second generating unit 134 may adjust the aspect of the object in the entire three-dimensional model based on the first three-dimensional model and the second three-dimensional model. The aspect of the object here refers to the display aspect of the object, and may refer to, for example, the color or texture of the object. In this case, the determining unit 136 determines whether or not the same corresponding object exists in the first three-dimensional model and the second three-dimensional model. Specifically, the determining unit 136 collates the type of object in the first three-dimensional model with the type of object in the second three-dimensional model to determine whether or not the same type of object exists.

[0072] When the determination unit 136 determines that the same object exists, the second generation unit 134 sets the display mode of the object in the portion of the entire three-dimensional model corresponding to the second three-dimensional model to the same display mode of the corresponding object in the first three-dimensional model. That is, when an object such as a wall, ceiling, or floor exists in the first three-dimensional model and the same object exists in the second three-dimensional model, the second generation unit 134 inherits the surface color and surface shape of the wall, ceiling, or floor object generated by the first generation unit 132 to the wall, ceiling, or floor object in the portion of the entire three-dimensional model corresponding to the second three-dimensional model. The display mode can be set by adjusting the pixel value of the position corresponding to the object.

[0073] (Equipment model placement) The second generating unit 134 may include a three-dimensional model of the room's equipment in the overall three-dimensional model. Specifically, for home equipment such as a door or a toilet that is not shown in the image data but is shown in the overall floor plan of the room, the second generating unit 134 refers to the equipment information storage unit 123, and uses an appropriate one of the three-dimensional models stored therein to add it to the overall three-dimensional model. The second generating unit 134 adopts equipment that is adopted in a floor plan that is highly similar to the currently used floor plan. Note that general-purpose equipment may be stored in the equipment information storage unit 123 in case no inapplicable equipment is found, and the second generating unit 134 may adopt the stored general-purpose equipment. FIG. 10 is a diagram showing an example of an equipment model generated by the generating device according to the present embodiment. The equipment information storage unit 123 stores a three-dimensional model of a toilet as shown in FIG. 10. The second generating unit 134 reads out a three-dimensional model of the equipment shown in the overall floor plan of the room from the equipment information storage unit 123, and places it in a corresponding position in the overall three-dimensional model of the room.

[0074] (Output of the entire 3D model) In the above manner, the second generating unit 134 generates an overall three-dimensional model reflecting image data captured in a portion of the region (first region AR1). The second generating unit 134 outputs the generated overall three-dimensional model. Specifically, the second generating unit 134 causes the display unit 170 to display the overall three-dimensional model. The second generating unit 134 also transmits the overall three-dimensional model to the server device 200 via the communication unit 110. For example, the server device 200 may generate a virtual space based on the overall three-dimensional model and transmit it to the generating device 100 of a user who enters the virtual space as an avatar.

[0075] (Processing flow) Next, the process flow for generating the entire 3D model described above will be described with reference to Fig. 11. Fig. 11 is a flowchart showing the flow of the generation method according to this embodiment. The generation method according to this embodiment will be described along the flow shown in Fig. 11.

[0076] First, the acquisition unit 131 acquires image data showing a part of the room (step S101). Next, the first generation unit 132 generates a first three-dimensional model and a partial floor plan from the image data showing a part of the room (step S102). Next, the search unit 133 calculates the similarity between the entire floor plan of the room stored in the floor plan information storage unit 122 and the partial floor plan (step S103). Next, the search unit 133 searches the floor plan information storage unit 122 for an entire floor plan of the room having a similarity higher than a predetermined threshold (step S104). Next, the second generation unit 134 generates an entire three-dimensional model showing an entire model of the room based on the searched entire floor plan and the first three-dimensional model (step S105).

[0077] According to this configuration, a 3D model of the entire room can be generated from image data of a portion of the room. Therefore, it is possible to generate a 3D model of the entire room that reflects the state of the portion of the room that has been imaged, without covering the entire room with multiple image data. Therefore, according to this embodiment, a 3D model that reflects the state of the room that has been imaged can be easily and appropriately generated from image data.

[0078] (Other examples) FIG. 12 is a diagram showing a configuration example of a server device according to the present disclosure. In the above embodiment, the generating device 100 having the imaging unit 150 generates a three-dimensional model of a room, but the entity that generates the three-dimensional model of the room is not limited to the generating device having the imaging unit 150, and the server device 200 may generate the three-dimensional model of the room. That is, in an example in which the server device 200 generates a three-dimensional model of the room, the server device 200 functions as a generating device. In this case, as shown in FIG. 12, the server device 200 includes a communication unit 210, a storage unit 220, a control unit 230, and a display unit 240. Note that the communication unit 210, the storage unit 220, the control unit 230, and the display unit 240 of the server device 200 are the same as the communication unit 110, the storage unit 120, the control unit 130, and the display unit 170 of the generating device 100, respectively, and therefore will not be described. That is, for example, the acquisition unit 231, the first generation unit 232, the search unit 233, the second generation unit 234, and the determination unit 236 of the control unit 230 in this example (FIG. 12) execute the same processes as the acquisition unit 131, the first generation unit 132, the search unit 133, the second generation unit 134, and the determination unit 136 in the above-mentioned embodiment (FIG. 3), respectively. Also, the image data storage unit 221, the floor plan information storage unit 222, and the facility information storage unit 223 of the storage unit 220 in this example (FIG. 12) store the same information as the image data storage unit 121, the floor plan information storage unit 122, and the facility information storage unit 123 in the above-mentioned embodiment (FIG. 3), respectively.

[0079] That is, the server device 200 in this example acquires image data of a portion of a room from the generating device 100, and generates a three-dimensional model of the entire room by performing processing similar to that of the generating device 100 described in the above embodiment.

[0080] As a result, even if the generating device 100 does not have a high-performance arithmetic processing device, by installing a high-performance arithmetic processing device in the server device 200, it is possible to execute the generation process of the 3D model at high speed. Also, by aggregating image data transmitted from a plurality of generating devices 100, it is possible to generate 3D models of entire rooms of various shapes, and store the data of the 3D models in the server device 200.

[0081] (Composition and Effects) The generating device 100 according to the present disclosure includes a first generating unit 132 that generates a first three-dimensional model and a floor plan of the imaged portion of the room from image data of a portion of the room, a search unit 133 that calculates a similarity between the entire floor plan of the room stored in the floor plan information storage unit 122 and the partial floor plan, and searches for an entire floor plan having a similarity higher than a predetermined threshold, and a second generating unit 134 that generates a three-dimensional model of the entire imaged room by using the first three-dimensional model in an area of ​​the entire floor plan that corresponds to at least a portion of the partial floor plan, and by using a second three-dimensional model generated based on the entire floor plan in an area other than the area.

[0082] According to this configuration, a 3D model of the entire room can be generated from image data of only a portion of the room, and therefore a 3D model that reflects the state of the captured room can be easily and appropriately generated from the image data.

[0083] The generating device 100 according to the present disclosure further includes a determination unit 136 that determines whether or not a corresponding identical object exists in the first three-dimensional model and the second three-dimensional model, and when the determination unit 136 determines that the same object exists, the second generating unit 134 sets the appearance of the object in the second three-dimensional model to be the same as the appearance of the corresponding object in the first three-dimensional model.

[0084] According to this configuration, an object of the second 3D model existing in a part of the room not captured in the captured image data can inherit the color and texture of the same object of the first 3D model. Therefore, a 3D model of the entire room close to the real room can be generated. Therefore, a 3D model reflecting the state of the captured room can be easily and appropriately generated from the image data.

[0085] The first generation unit 132 of the generation device 100 according to the present disclosure classifies the class of the captured room based on the image data, and the search unit 133, when searching for an overall floor plan from the floor plan information storage unit 122, calculates a high similarity between the overall floor plan classified into the same class as the class of the captured room.

[0086] According to this configuration, the similarity is calculated based on the objects captured in the image data, so that the similarity can be calculated accurately. Therefore, the floor plan of the entire room can be appropriately selected based on the image data of a part of the room. Therefore, a three-dimensional model that reflects the state of the captured room can be easily and appropriately generated from the image data.

[0087] The generation method according to the present disclosure includes the steps of generating a first three-dimensional model and a floor plan of the portion of the room from image data of an image of a portion of the room, calculating a similarity between the entire floor plan of the room stored in the floor plan information storage unit 122 and the partial floor plan, and searching for an entire floor plan of the room whose similarity is higher than a predetermined threshold value, and generating a three-dimensional model of the entire imaged room by using the first three-dimensional model in an area of ​​the entire floor plan that corresponds to at least a portion of the partial floor plan, and using a second three-dimensional model generated based on the entire floor plan in an area other than the area.

[0088] According to this configuration, a three-dimensional model that reflects the captured state of the room can be easily and appropriately generated from the image data.

[0089] The generation program of the present disclosure causes a computer to execute the steps of generating a first three-dimensional model and a floor plan of the portion of the room from image data of an image of a portion of the room, and calculating a similarity between the entire floor plan of the room stored in the floor plan information storage unit 122 and the partial floor plan, and searching for an entire floor plan of the room whose similarity is higher than a predetermined threshold, and generates a three-dimensional model of the entire imaged room using the first three-dimensional model in an area of ​​the entire floor plan that corresponds to at least a portion of the partial floor plan, and using a second three-dimensional model generated based on the entire floor plan in an area other than the area.

[0090] According to this configuration, a three-dimensional model that reflects the captured state of the room can be easily and appropriately generated from the image data.

[0091] Although the embodiment of the present disclosure has been described above, the embodiment is not limited by the contents of this embodiment. In addition, the above-mentioned components include those that a person skilled in the art can easily imagine, those that are substantially the same, and those that are within the so-called equivalent range. Furthermore, the above-mentioned components can be appropriately combined. Furthermore, various omissions, substitutions, or modifications of the components can be made without departing from the gist of the above-mentioned embodiment. [Explanation of symbols]

[0092] 1. Generator System 100 generator 110 Communications Department 120 Storage section 121 Image data storage unit 122 Floor plan information storage unit 123 Equipment information storage section 130 Control section 131 Acquisition Department 132 First generation part 133 Search Department 134 Second generation part 136 Judgment section 140 Input section 150 Imaging unit 160 Sound output section 170 Display section 200 Server device AR1 First area AR2 second area N Network

Claims

1. a first generating unit that generates a first three-dimensional model and a floor plan of the captured part of the room from image data of the captured part of the room; a search unit that calculates a similarity between an entire floor plan of a room stored in a floor plan information storage unit and the partial floor plan, and searches for the entire floor plan having a similarity higher than a predetermined threshold; and a second generation unit that generates a three-dimensional model of the entire captured room by using the first three-dimensional model in an area of ​​the entire floor plan that corresponds to at least a part of the part of the floor plan, and using a second three-dimensional model generated based on the entire floor plan in an area other than the first three-dimensional model. generator.

2. a determination unit that determines whether or not a corresponding object exists between the first three-dimensional model and the second three-dimensional model; when the determination unit determines that the same object exists, the second generation unit sets a state of the object in the second three-dimensional model to be the same as a state of the corresponding object in the first three-dimensional model. The generating device of claim 1 .

3. The first generation unit classifies a class of the captured room based on the image data, the search unit, when searching for the entire floor plan from the floor plan information storage unit, calculates a high similarity between the entire floor plan classified into the same class as the class of the imaged room, A generating device according to claim 1 or 2.

4. generating a first three-dimensional model and a floor plan of the part of the room from image data of the part of the room; calculating a similarity between an entire floor plan of the room stored in a floor plan information storage unit and the partial floor plan, and searching for an entire floor plan of the room having a similarity higher than a predetermined threshold; generating a three-dimensional model of the entire captured room by using the first three-dimensional model in an area of ​​the entire floor plan that corresponds to at least a part of the partial floor plan, and using a second three-dimensional model generated based on the entire floor plan in an area other than the first three-dimensional model; Generation method.

5. generating a first three-dimensional model and a floor plan of the part of the room from image data of the part of the room; calculating a similarity between an entire floor plan of the room stored in a floor plan information storage unit and the partial floor plan, and searching for an entire floor plan of the room having a similarity higher than a predetermined threshold; Run the following on your computer: generating a three-dimensional model of the entire captured room by using the first three-dimensional model in an area of ​​the entire floor plan that corresponds to at least a part of the partial floor plan, and using a second three-dimensional model generated based on the entire floor plan in an area other than the first three-dimensional model; Generator.

Images