Generation apparatus, generation method, and program

Abstract

To provide a generation apparatus, a generation method, and a program capable of generating a model in a natural form.SOLUTION: The generation apparatus according to the present disclosure is provided with: an object detection unit for detecting objects in an image and their positions based on a predetermined image; a layout estimation unit for estimating a layout of a boundary part including at least one of a wall, a floor, and a ceiling that serves as a boundary of a room space based on the predetermined image; and a model interpolation unit for interpolating and generating a missing part of the boundary portion of the layout of the room estimated by the layout estimation unit. The model interpolation unit generates a wall at a position separated by a predetermined distance from the object detected by the object detection unit.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present disclosure relates to a generating device, a generating method, and a program. [Background technology]

[0002] In recent years, there has been active research into techniques for generating 3D models of the interior of a room or other space using AI (Artificial Intelligence) technology from photographic images of the interior of the room. There are a variety of techniques available, including those that use a single ordinary camera photograph to generate a 3D model of the interior of the room within the area captured, those that use multiple camera photographs, and those that generate a 3D model of the entire interior of a room from a single 360-degree panoramic photograph.

[0003] For example, Patent Document 1 listed below discloses an information processing device in which a property information server acquires a panoramic image captured in all directions by an omnidirectional camera installed inside an indoor space, and the property information server extracts, from the acquired panoramic image, boundary lines between the ceiling, floor, and walls that constitute the indoor space, extracts ceiling images, floor images, and wall images from the panoramic image based on the extracted boundary lines, corrects distortions in the extracted ceiling images, etc., and combines the corrected ceiling images, etc. to generate an indoor space image that represents a three-dimensional model equivalent to the indoor space. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 6577557 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the information processing device described in the above Patent Document 1 generates a three-dimensional model of an indoor space using a panoramic image composed of multiple images, and does not generate a three-dimensional model of a room space using a single camera photograph. Furthermore, at present, there is no device that generates a three-dimensional model of a room space that does not appear in a camera photograph in a natural way.

[0006] In view of the above-mentioned problems, the present disclosure aims to provide a generation device, a generation method, and a program capable of generating a model in a natural form. [Means for solving the problem]

[0007] In order to solve the above-mentioned problems and achieve the objective, the generating device of the present disclosure includes an object detection unit that detects objects contained in an image and their positions based on a specified image, a layout estimation unit that estimates a layout of a boundary including at least one of a wall, a floor, and a ceiling that form boundaries of a room space based on the specified image, and a model interpolation unit that generates a layout of the room estimated by the layout estimation unit by interpolating missing parts of the boundary, and the model interpolation unit generates a wall at a position a specified distance away from an object detected by the object detection unit.

[0008] In order to solve the above-mentioned problems and achieve the objective, the generation method of the present disclosure includes the steps of detecting objects and their positions contained in a specified image based on the image, estimating a layout of a boundary including at least one of a wall, a floor, and a ceiling that form the boundary of the room space based on the specified image, and generating a layout of the room estimated in the layout estimation step by interpolating missing portions of the boundary, wherein in the interpolation generation step, a wall is generated at a position a specified distance away from the object detected in the detection step.

[0009] In order to solve the above-mentioned problems and achieve the objective, the program of the present disclosure causes a computer to execute the steps of detecting objects and their positions contained in an image based on a specified image; estimating a layout of a boundary including at least one of a wall, a floor, and a ceiling that form the boundary of a room space based on the specified image; generating a layout of the room estimated in the layout estimation step by interpolating missing parts of the boundary; and generating a wall at a position a specified distance away from the object detected in the detection step in the interpolation generation step. Effect of the Invention

[0010] According to the present disclosure, it is possible to provide a generation device, a generation method, and a program capable of generating a model in a natural form. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram illustrating an overview of a generating device according to the present disclosure. [Diagram 2] FIG. 2 is a diagram illustrating an example configuration of a generation system according to the present disclosure. [Diagram 3] FIG. 3 is a diagram illustrating an example of the configuration of a generating device according to the present disclosure. [Figure 4] FIG. 4 is a diagram illustrating an example of the configuration of a control unit of a generating device according to the present disclosure. [Diagram 5] FIG. 5 is a diagram illustrating the processing of the model interpolation unit of the generating device according to the present disclosure. [Figure 6] FIG. 6 is a diagram illustrating the processing of the door arrangement unit and the window arrangement unit of the generating device according to the present disclosure. [Figure 7] FIG. 7 is a flowchart showing the flow of the generating method according to the present disclosure. [Figure 8] FIG. 8 is a diagram illustrating a configuration example of a server device according to the present disclosure. [Figure 9] FIG. 9 is a diagram illustrating a configuration example of a control unit of a server device according to the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Note that the present disclosure is not limited to the embodiments described below.

[0013] (Overview of the generator) First, an overview of a generating device 100 according to the present disclosure will be described with reference to Fig. 1. Fig. 1 is a diagram for explaining an overview of a generating device according to the present disclosure. As shown in Fig. 1, the generating device 100 according to the present disclosure is an information processing device in which, for example, a person P1 captures an image of a space in a room, and generates a three-dimensional model of the space in the room based on the captured image.

[0014] For example, the generating device 100 reproduces the walls W1, W2, W3, ceiling C1, and floor L1 of the room shown in FIG. 1 using a three-dimensional model, and then places three-dimensional models of windows WDL, WDR, a desk T1, a chair C1, and other elements arranged in the room within the three-dimensional model to generate a three-dimensional model that reproduces the space of the room depicted in the image.

[0015] In this way, if a three-dimensional model of a room space can be generated, it is possible to confirm in advance how the impression of the room will change when the layout of the furniture, etc. arranged in the room is changed. In addition, it is possible to consider the procedure for rearranging the furniture arranged in the room in advance, and to use this to consider efficient work procedures.

[0016] (Generation system configuration) Next, a configuration example of a generating system according to the present disclosure will be described with reference to Fig. 2. Fig. 2 is a diagram showing a configuration example of a generating system according to the present disclosure. As shown in Fig. 2, a generating system 1 according to the present disclosure includes a generating device 100, a server device 200, and a network N. Note that, as shown in Fig. 2, the generating system 1 may include a plurality of generating devices 100. The configuration will be briefly described below.

[0017] The generating device 100 is an information processing device having an imaging function. The generating device 100 may be, for example, an information processing device such as a camera, a smartphone, a tablet terminal, a wearable terminal, a mobile phone, or a PDA (Personal Digital Assistant). In the example shown in FIG. 1, the generating device 100 is a camera.

[0018] The server device 200 is an information processing device that executes various information processes. For example, the server device 200 executes a process of generating a three-dimensional model using an image captured by the generating device 100 and transmitted from the generating device 100 via a network N. The server device 200 may be realized by an information processing device such as a PC (Personal Computer), a WS (Work Station), or a computer having a server function.

[0019] The network N connects the generating device 100 and the server device 200 in a wired or wireless manner so that they can communicate with each other. If the network N is wired, it may be realized by ETHERNET (registered trademark) defined in IEEE802.3, or a control signal line of USB (Universal Serial Bus) and a video signal line of SDI (Serial Digital Interface). If the network N is wireless, it may be realized by Bluetooth (registered trademark) or a wireless LAN (Local Area Network) defined in IEEE802.11.

[0020] In this manner, the generation system 1 functions as a single system by multiple devices transmitting and receiving information to and from each other.

[0021] (Configuration of the generating device) Next, a configuration example of the generating device 100 according to the present disclosure will be described with reference to FIG. 3. FIG. 3 is a diagram showing a configuration example of the generating device according to the present disclosure. As shown in FIG. 3, the generating device 100 according to the present disclosure includes a communication unit 110, a storage unit 120, a control unit 130, an imaging unit 140, a sound output unit 150, and a display unit 160. Below, these configurations will be described in order. Note that in FIG. 3, the configuration of the control unit 130 is omitted. The configuration of the control unit 130 will be described later with reference to FIG. 4.

[0022] The communication unit 110 is a communication module that allows the generating device 100 to communicate with an external device such as the server device 200. The communication unit 110 is responsible for transmitting and receiving data to and from the external device via wireless communication or wired communication. In the case of wireless communication, the communication unit 110 may include a communication antenna, an RF (Radio Frequency) circuit, and other circuits for communication processing. In the case of wired communication, the communication unit 110 may include, for example, a LAN (Local Area Network) terminal, a transmission circuit, and other circuits for communication processing.

[0023] The storage unit 120 is a storage device that stores various types of information. The storage unit 120 includes a main storage device and an auxiliary storage device. The main storage device may be realized by a semiconductor memory element such as a random access memory (RAM), a read only memory (ROM), or a flash memory. The auxiliary storage device may be realized by a hard disk, a solid state drive (SSD), an optical disk, or the like.

[0024] 3, the storage unit 120 includes an image data storage unit 121 and a CAD data storage unit 122. The information stored in these components will be described below in order.

[0025] The image data storage unit 121 stores image data captured by the imaging unit 140. For example, the image data storage unit 121 may store information such as an identification number for identifying image data and the date and time when the image data was captured, in association with the image data.

[0026] The CAD data storage unit 122 stores three-dimensional models of furniture and the like. Specifically, the CAD data storage unit 122 stores three-dimensional models drawn by three-dimensional CAD (Computer Aided Design) that uses CG (Computer Graphics) to assist in the design of three-dimensional structures. The three-dimensional models may be stored in a state of being classified into classes indicating types, etc. The CAD data storage unit 122 may also store ShapeNet data that includes various three-dimensional models.

[0027] The control unit 130 is a controller that manages and controls the generating device 100. The control unit 130 is realized by a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or the like executing various programs stored in the storage unit 120 using a RAM as a working area. The control unit 130 may also be realized by an integrated circuit, such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0028] Here, a detailed configuration of the control unit 130 of the generating device 100 according to the present disclosure will be described with reference to Fig. 4. Fig. 4 is a diagram showing an example of the configuration of the control unit of the generating device according to the present disclosure. As shown in Fig. 4, the control unit 130 includes an input image acquisition unit 131, an object detection unit 132, a taste estimation unit 133, an object model selection unit 134, a layout estimation unit 135, a boundary generation unit 136, an optimization unit 137, a model interpolation unit 138, a taste determination unit 139, a CAD model selection unit 1391, a door arrangement unit 1392, and a window arrangement unit 1393.

[0029] The control unit 130 realizes these functions and executes these processes by reading and executing a program (software) from the storage unit 120. Note that these functions of the control unit 130 may be realized by electronic circuits. The control unit 130 may execute these processes by one CPU, or may be provided with multiple CPUs and execute these processes in parallel by the multiple CPUs. Below, these configurations will be explained in order.

[0030] The input image acquisition unit 131 acquires image data of a predetermined image in which a room is shown. The input image acquisition unit 131 acquires an RGB image, which is data of a photograph of the interior of a room, as the predetermined image data in which a room is shown. Specifically, the input image acquisition unit 131 acquires data of an RGB image captured by the imaging unit 140. After acquiring data of the RGB image captured by the imaging unit 140, the input image acquisition unit 131 stores the acquired image data in the image data storage unit 121. Note that the RGB image is data including information on the luminance of each pixel and the RGB colors, and may be data to which a gradation indicating the number of colors in which the color changes intermittently is assigned to each pixel. Note that, as shown in FIG. 4, the input image acquisition unit 131 also outputs the RGB image to the object detection unit 132 and the layout estimation unit 135.

[0031] The object detection unit 132 detects an object included in a predetermined image (i.e., an RGB image acquired by the input image acquisition unit 131) showing a room and its position (the position of the object in the coordinate system of the image) based on the image. Here, the object refers to an object shown in the image, for example, a wall, a door, a window, furniture, etc. The object detection unit 132 detects information on the type of object and bounding box information of the object indicating the position of the object from the RGB image acquired by the input image acquisition unit 131. For example, the object detection unit 132 executes a process of detecting an object captured in the RGB image and classifying the detected object. The object detection unit 132 may be realized by using, for example, semantic segmentation, which is a method of classifying pixels of an RGB image according to which class the pixels belong to.

[0032] The object detection unit 132 may be realized using a method such as Faster R-CNN (Region-based Convolutional Neural Network), which performs two-stage processing of classifying the detected object after distinguishing the contents reflected in the region into an object and a background, thereby achieving both high-speed processing and improved accuracy. Faster R-CNN uses a CNN structure called RPN (Region Proposal Network) for the first-stage processing, so that the processing can be accelerated. As shown in FIG. 4, the object detection unit 132 outputs the object detection result to the taste estimation unit 133, the object model selection unit 134, the door arrangement unit 1392, and the window arrangement unit 1393. The object detection unit 132 may output the object detection result to the taste estimation unit 133, the object model selection unit 134, the door arrangement unit 1392, and the window arrangement unit 1393 via the configuration shown in FIG. 4, rather than directly.

[0033] The taste estimation unit 133 estimates taste information for each detected object. The taste information here refers to information that represents the basic tone and style of furniture, such as Scandinavian style, Baroque style, Rococo style, Japanese style, and Asian style. The estimation of taste information for each object may be realized by having a model such as a CNN (Convolutional Neural Network) or FCN (Fully Convolutional Network) learn the above-mentioned individual taste features using learning data in which taste information is labeled for each image. As shown in FIG. 4, the taste estimation unit 133 outputs the taste estimation result of the object to the object model selection unit 134 and the taste determination unit 139.

[0034] The object model selection unit 134 selects a matching CAD model for each detected object. Specifically, for each detected object, the object model selection unit 134 selects a CAD model with the closest shape from the CAD model database stored in the CAD data storage unit 122 corresponding to the taste estimated by the object taste estimation unit 133 and the type of object detected by the object detection unit 132, and performs rough position estimation including the pose of the object and size estimation. The object model selection unit 134 may be realized by a CNN that has learned the relationship between the characteristics of the object and the CAD model. As shown in FIG. 4, the object model selection unit 134 outputs the CAD model of the selected object and information on its rough position and size to the boundary generation unit 136.

[0035] The layout estimation unit 135 estimates the layout of the boundary including at least one of the walls, floor, and ceiling that are the boundaries of the space of the room, based on a predetermined image (i.e., an RGB image acquired by the input image acquisition unit 131). Specifically, the layout estimation unit 135 uses FCN or CNN to label the walls, floor, and ceiling of the predetermined image on a pixel-by-pixel basis, and estimates the layout of the boundary by fitting these constituent surfaces of the room into a rectangular box that matches in size and shape. As shown in FIG. 4, the layout estimation unit 135 outputs three-dimensional shape information of the room to the boundary generation unit 136.

[0036] The boundary generation unit 136 generates a three-dimensional model of the boundary of the room. For example, the boundary generation unit 136 generates an initial three-dimensional model in which the walls, floor, and ceiling are configured with planes based on the layout of the boundary estimated by the layout estimation unit 135. The boundary generation unit 136 extracts the representative color and representative texture of each surface in the input image and applies the same color and texture to generate the color and texture of the walls, floor, and ceiling. The boundary generation unit 136 also places the CAD model in the three-dimensional model based on the CAD model, rough position, and size information selected by the object model selection unit 134. As shown in FIG. 4, the boundary generation unit 136 outputs the rough three-dimensional model to the optimization unit 137.

[0037] The optimization unit 137 optimizes the placement and size of the CAD model. For example, the optimization unit 137 optimizes the placement and size of objects in the image of the three-dimensional model so that the difference between the image rendered from the three-dimensional model generated by the boundary generation unit 136 from the camera position where the input RGB image acquired by the input image acquisition unit 131 was taken and the input RGB image is minimized, and outputs the three-dimensional model after the optimization is completed. As shown in FIG. 4, the optimization unit 137 outputs the three-dimensional model after the optimization is completed to the model interpolation unit 138.

[0038] The model interpolation unit 138 generates a model by interpolating missing parts of boundaries in the room layout estimated by the layout estimation unit 135. Specifically, the model interpolation unit 138 generates the walls, floor, and ceiling in the ranges that are not fully generated in the 3D model after the optimization output from the object optimization unit 137 and are not shown in the input RGB image by interpolating them.

[0039] For example, the model interpolation unit 138 generates a wall at a position a predetermined distance away from the foremost object among the objects detected by the object detection unit 132. The foremost object here may refer to the object that is located closest to the position where the RGB image was captured in the coordinate system of the RGB image. Also, the position a predetermined distance away from the foremost object may refer to a position a predetermined distance away from the foremost object further toward the near side (the side toward the position where the RGB image was captured) from the foremost object.

[0040] The model interpolation unit 138 sets the predetermined distance based on the object included in the image. For example, the model interpolation unit 138 may set the above-mentioned predetermined distance to the width of a person×4 as a default. In addition, if the object closest to the user is a low table and there is no chair or sofa around the low table, the model interpolation unit 138 may determine that there is a high possibility that an object of a chair or sofa exists in the foreground, and may set the total value of the gap distance between the low table and the virtual chair + the depth of the virtual chair + (width of a person×4) as the predetermined distance. In addition, if a bed is shown in the RGB image and three walls are shown, the model interpolation unit 138 may set the predetermined distance so that the space is 8 tatami mats (the number of tatami mats may be set arbitrarily) based on the dimensions of the three-dimensional model.

[0041] The model interpolation unit 138 may set the predetermined distance based on the object included in the image as follows. For example, the predetermined distance may be determined based on the type of object (chair, desk, bed). If the object at the forefront is a part of the furniture in the set and the remaining furniture in the set is not included, the remaining furniture object may be interpolated further forward, and the predetermined distance may be "the depth of the interpolated furniture object + the distance that a person can pass through with ease". The distance that a person can pass through with ease may be, for example, the length of a person's width x 4. The size of the room may be specified from the type of object, and the predetermined distance may be determined from the specified size of the room. That is, for example, if a bed is included as an object, the size of a general bedroom may be specified as the size of the room. The predetermined distance may be determined from the distance between objects (the distance between furniture).

[0042] Here, the processing of the model interpolation unit 138 will be specifically described with reference to FIG. 5. FIG. 5 is a diagram for explaining the processing of the model interpolation unit of the generating device according to the present disclosure. For example, it is assumed that a three-dimensional model arranged as shown in (a) on the left side of FIG. 5 has been generated by the processing up to the optimization unit 137. In this case, the model interpolation unit 138 generates an interpolated three-dimensional model as shown in (b) on the right side of FIG. 5. That is, the model interpolation unit 138 generates a left interpolation wall LW1, a rear interpolation wall BW1, and a right interpolation wall RW1 by interpolation. Note that the model interpolation unit 138 generates the rear interpolation wall BW1 at a position a predetermined distance L away from the foremost object OB1. That is, the lengths of the left interpolation wall LW1 and the right interpolation wall RW1 are also determined by the predetermined distance L.

[0043] As shown in FIG. 4, the model interpolation unit 138 outputs the three-dimensional model after the interpolation to a door arrangement unit 1392.

[0044] The taste determination unit 139 determines the taste of the entire room. Specifically, the taste determination unit 139 obtains the composition ratio of the tastes of the objects in the entire room based on the taste information estimated for each object by the taste estimation unit 133, and determines the taste with the highest ratio as the taste of the entire room. For example, if a desk, a chair, and a window are arranged in a room, and the taste information of the desk and the chair is Baroque style and the taste information of the window is Asian style, the taste of the entire room is determined as Baroque style. As shown in FIG. 4, the taste determination unit 139 outputs the determined taste information of the room to the CAD model selection unit 1391.

[0045] The CAD model selection unit 1391 selects the CAD models of the door and the window. The CAD model selection unit 1391 selects the CAD models of the door and the window one by one from the CAD data storage unit 122 in which the CAD models of the door and the window corresponding to the taste information of the whole room determined by the taste determination unit 139 are stored. The CAD models of the door and the window may be determined randomly one by one, or the CAD models of the door and the window may be selected based on the information of the CAD models already selected as the objects in the room. The CAD model selection unit 1391 is not limited to selecting one by one, and any number of CAD models may be selected from the top of the priority as candidates. As shown in FIG. 4, the CAD model selection unit 1391 outputs the CAD models of the selected door and the window to the door arrangement unit 1392. The CAD model selection unit 1391 outputs the CAD model of the selected window to the window arrangement unit 1393 via the door arrangement unit 1392. Alternatively, the CAD model selection unit 1391 may directly output the CAD model of the selected window to the window arrangement unit 1393.

[0046] The door placement unit 1392 determines whether or not there is a door at the boundary based on a predetermined image, and if it is determined that there is no door, adds and places a door on the wall generated by interpolation. First, the door placement unit 1392 determines whether or not a door object exists in the input RGB image based on the result of object detection in the object detection unit 20. Next, if a door object exists in the input RGB image, the door placement unit 1392 determines whether or not a window object exists in the input image. Next, the door placement unit 1392 determines whether or not there is a space in which the CAD model of the door selected by the CAD model selection unit 1391 can be placed in the portion of the interpolated space 3D model of the interpolated space that is not captured and is generated by the model interpolation unit 138. If there is no space in which the CAD model of the door can be placed, the door placement unit 1392 ends the process. If there is a space in which the CAD model of the door can be placed, the door placement unit 1392 places the CAD model of the door in the portion of the interpolation wall where the space in which the CAD model of the door can be placed is available. The door arrangement unit 1392 outputs to the window arrangement unit 1393 a post-interpolation entire space 3D model in which the CAD models of the doors are arranged on the interpolation walls.

[0047] The window arrangement unit 1393 determines whether or not there is a window in the boundary based on a predetermined image, and if it is determined that there is no window, it adds and arranges a window to a wall generated by interpolation in which a door is not arranged and a window can be arranged. First, the window arrangement unit 1393 determines whether or not a window object exists in the input image (input RGB image). If a window object exists in the input image (input RGB image), the window arrangement unit 1394 ends the processing. If a window object does not exist in the input image (input RGB image), the window arrangement unit 1394 determines whether or not there is a space in the interpolated wall where the CAD model of the window selected by the CAD model selection unit 1391 can be arranged from the entire 3D model of the interpolated space in which the CAD model of the door is arranged on the interpolated wall. If there is no space in which the CAD model of the window can be arranged, the window arrangement unit 1394 ends the processing. If there is a space in which the CAD model of the window can be arranged, the CAD model of the window is arranged in the part of the interpolated wall where the space in which the CAD model of the window can be arranged.

[0048] Here, the processing of the door arrangement unit 1392 and the window arrangement unit 1393 will be specifically described with reference to FIG. 6. FIG. 6 is a diagram for explaining the processing of the door arrangement unit and the window arrangement unit of the generating device according to the present disclosure. For example, it is assumed that a three-dimensional model as shown in (c) on the left side of FIG. 6 is generated by the processing up to the model interpolation unit 138. In this case, the door arrangement unit 1392, after determining that there is no door object in the input image, checks whether a door model can be arranged in the interpolation walls, starting from the left interpolation wall LW1, and determines that it can be arranged on the rear interpolation wall BW1, and arranges the door model DR1 on the rear interpolation wall BW1 as shown in (b) in the center of FIG. 6. In addition, the window arrangement unit 1393, after determining that there is no window object in the input image, checks whether a window model can be arranged in the interpolation walls, starting from the walls other than the rear interpolation wall BW1 on which the door model is arranged, and determines that it can be arranged on the left interpolation wall LW1, and arranges the window model WD1 on the left interpolation wall LW1 as shown in (c) on the right side of FIG. 6.

[0049] The imaging unit 140 captures an image. The imaging unit 140 includes an optical element and an imaging element. The optical element is an element that constitutes an optical system, such as a lens, a mirror, a prism, and a filter. The imaging element is an element that converts light incident through the optical element into an image signal, which is an electrical signal. The imaging element is, for example, a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor.

[0050] The sound output unit 150 outputs various sounds. For example, when the processing in the generating device 100 is completed, the sound output unit 150 may output a sound notifying the completion of the processing. The sound output unit 150 may be a speaker, and the speaker converts an electric signal into sound using a diaphragm. That is, the speaker vibrates the diaphragm with a predetermined amplitude and frequency based on a control command given by the electric signal, thereby vibrating the air in contact with the diaphragm to output sound.

[0051] The display unit 160 displays various types of information. For example, the display unit 160 may display a GUI (Graphical User Interface) for executing the processing of the generating device 100, a three-dimensional model generated by the generating processing, etc. The display unit 160 may be realized by a liquid crystal display, an organic EL (Electro Luminescence) display, a micro LED (Light Emitting Diode) display, etc.

[0052] (About the generation method and program) Next, a generation method according to the present disclosure will be described with reference to Fig. 7. Fig. 7 is a flowchart showing the flow of the generation method according to the present disclosure. The generation method according to the present disclosure will be described step by step along the flow shown in Fig. 7.

[0053] First, the generating device 100 acquires RGB image data as input (step S101). Next, the generating device 100 detects objects present in the input image (detects types and bounding boxes) (step S102). Next, the generating device 100 estimates the taste of each object (step S103). Next, the generating device 100 selects the most similar CAD model from a database that matches the taste of each object, and performs rough placement and size estimation (step S104). Next, the generating device 100 estimates the three-dimensional layout of the room (step S105). Next, the generating device 100 generates a rough three-dimensional spatial model (step S106). Next, the generating device 100 optimizes the placement and size of the objects (step S107). Next, the generating device 100 generates an interpolated three-dimensional model of the walls, floor, and ceiling of the area not captured (step S108). Next, the generating device 100 determines the taste of the entire room (step S109). Next, the generating device 100 selects CAD models of doors and windows (step S110). Next, the generating device 100 determines whether or not a door is present in the input image (step S111).

[0054] In step S111, if a door is present in the input image (step S111: Yes), the generating device 100 proceeds to step S114. That is, the generating device 100 determines whether or not a window is present in the input image (step S114). In step S114, if a window is present in the input image, the generating device 100 ends the process.

[0055] In step S111, if a door does not exist in the input image (step S111: No), the generating device 100 judges whether or not a space in which a door can be arranged exists in the interpolated wall portion (step S112). In step S112, if a space in which a door can be arranged exists in the interpolated wall portion (step S112: Yes), the generating device 100 arranges a CAD model of a door in the interpolated wall portion (step S113). Next, the generating device 100 judges whether or not a window exists in the input image (step S114). In step S114, if a window does not exist in the input image (step S114: No), the generating device 100 judges whether or not a space in which a door can be arranged exists in the interpolated wall portion (step S115). In step S115, if a space in which a window can be arranged exists in the interpolated wall portion (step S115: Yes), the generating device 100 arranges a CAD model of a window in the interpolated wall portion (step S116). Then, the generating device 100 ends the process.

[0056] In step S112, if there is no space in the interpolated wall portion in which a door can be placed (step S112: No), the generating device 100 ends the process. Also, in step S115, if there is no space in the interpolated wall portion in which a window can be placed (step S115: No), the generating device 100 ends the process.

[0057] According to this, a three-dimensional model that reproduces a room can be generated based on an image of the room captured by a camera, the boundaries of the room that are not captured in the image can be generated by interpolation, and windows and doors can be placed on the generated boundaries. Therefore, a generation method and program that can generate a model with a natural shape can be provided.

[0058] (Server device configuration) In the above description, the generating device 100 including the imaging unit 140 executes an image processing method for interpolating a wall model from image data showing a room. However, an image processing device not including an imaging unit may acquire image data captured by the imaging unit and execute this image processing method. For example, a server device 200 may be used as an image processing device not including an imaging unit. Next, the configuration of the server device 200 according to the present disclosure will be described with reference to FIG. 8. FIG. 8 is a diagram showing an example of the configuration of the server device according to the present disclosure. As shown in FIG. 8, the server device 200 according to the present disclosure includes a communication unit 210, a storage unit 220, a control unit 230, a sound output unit 240, and a display unit 250. In addition, the communication unit 210, memory unit 220, control unit 230, sound output unit 240, and display unit 250 of the server device 200 according to the present disclosure are the same as the communication unit 110, memory unit 120, control unit 130, sound output unit 150, and display unit 160 of the generating device 100, respectively.

[0059] For example, the configuration of the control unit 230 of the server device 200 will be described with reference to Fig. 9. Fig. 9 is a diagram showing a configuration example of the control unit of the server device according to the present disclosure. As shown in Fig. 9, the control unit 230 of the server device 200 includes an input image acquisition unit 231, an object detection unit 232, a taste estimation unit 233, an object model selection unit 234, a layout estimation unit 235, a boundary generation unit 236, an optimization unit 237, a model interpolation unit 238, a taste determination unit 239, a CAD model selection unit 2391, a door arrangement unit 2392, and a window arrangement unit 2393, which are the same as the configuration of the control unit 130 of the generation device 100.

[0060] This allows the generation device 100 to execute processing at high speed in the server device 200 without having to incorporate a high-performance arithmetic processing device.

[0061] (Composition and Effects) The generating device 100 according to the present disclosure includes an object detection unit 132 that detects objects and their positions contained in an image based on a specified image, a layout estimation unit 135 that estimates a layout of boundaries including at least one of the walls, floor, and ceiling that form boundaries of the room space based on the specified image, and a model interpolation unit 138 that generates a layout of the room estimated by the layout estimation unit 135 by interpolating missing parts of the boundaries, and the model interpolation unit 138 generates a wall at a position a specified distance away from the object detected by the object detection unit 132.

[0062] According to this configuration, a three-dimensional model that reproduces a room can be generated based on an image of the room captured by a camera, and the boundaries of the room that are not captured in the image can be interpolated to generate the model. Therefore, it is possible to provide a generation device 100 that can generate a model with a natural shape.

[0063] The generating device 100 according to the present disclosure further includes a door placement unit 1392 that determines the presence or absence of a door at a boundary portion based on a specified image, and if it is determined that there is no door, places the door on the wall generated by interpolation.

[0064] According to this configuration, a three-dimensional model that reproduces a room can be generated based on an image of the room captured by a camera, the boundaries of the room that are not captured in the image can be generated by interpolation, and a door can be placed at the generated boundaries. Therefore, it is possible to provide a generation device 100 that can generate a model with a natural shape.

[0065] The generating device 100 according to the present disclosure further includes a window placement unit 1393 that determines the presence or absence of a window in a boundary portion based on a predetermined image, and if it is determined that there is no window, places a window in a wall generated by interpolation that does not have a door and in which a window can be placed.

[0066] According to this configuration, a three-dimensional model that reproduces a room can be generated based on an image of the room captured by a camera, the boundaries of the room that are not captured in the image can be generated by interpolation, and windows can be placed on the generated boundaries. Therefore, it is possible to provide a generation device 100 that can generate a model with a natural shape.

[0067] The generation method of the present disclosure includes the steps of detecting objects and their positions contained in a specified image based on the image, estimating a layout of a boundary including at least one of a wall, a floor, and a ceiling that form the boundaries of the room space based on the specified image, and generating the layout of the room estimated in the layout estimation step by interpolating missing portions of the boundary, wherein in the interpolation generation step, a wall is generated at a position a specified distance away from the object detected in the detection step.

[0068] According to this configuration, a three-dimensional model that reproduces a room can be generated based on an image of the room captured by a camera, and the boundaries of the room that are not captured in the image can be interpolated to generate the model. Therefore, a generation method that can generate a model with a natural shape can be provided.

[0069] The program of the present disclosure causes a computer to execute the steps of detecting objects and their positions contained in a specified image based on the specified image; estimating a layout of a boundary including at least one of the walls, floor, and ceiling that form the boundaries of the room space based on the specified image; generating by interpolating missing parts of the boundary of the room layout estimated in the layout estimation step; and generating a wall at a position a specified distance away from the object detected in the detection step in the interpolation generation step.

[0070] According to this configuration, a three-dimensional model that reproduces a room can be generated based on an image of the room captured by a camera, and the boundaries of the room that are not captured in the image can be interpolated to generate the model. Therefore, it is possible to provide a program that can generate a model with a natural shape.

[0071] Although the embodiment of the present disclosure has been described above, the embodiment is not limited by the contents of this embodiment. In addition, the above-mentioned components include those that a person skilled in the art can easily imagine, those that are substantially the same, and those that are within the so-called equivalent range. Furthermore, the above-mentioned components can be appropriately combined. Furthermore, various omissions, substitutions, or modifications of the components can be made without departing from the gist of the above-mentioned embodiment. [Explanation of symbols]

[0072] 100 generator 110 Communications Department 120 Storage section 121 Image data storage unit 122 CAD data storage section 130 Control section 131 Input image acquisition unit 132 Object Detection Unit 133 Taste Estimation Department 134 Object Model Selection Section 135 Layout Estimation Department 136 Boundary generation part 137 Optimization Department 138 Model Interpolation Unit 139 Taste Determination Department 1391 CAD Model Selection Section 1392 Door arrangement section 1393 Window arrangement section 140 Imaging unit 150 Sound output section 160 Display section 200 Server device 210 Communications Department 220 Storage section 221 Image data storage unit 222 CAD data storage unit 230 Control Unit 231 Input image acquisition unit 232 Object Detection Unit 233 Taste Estimation Department 234 Object Model Selection Section 235 Layout Estimation Department 236 Boundary generator 237 Optimization Department 238 Model Interpolation Unit 239 Taste Decision Department 2391 CAD Model Selection Section 2392 Door arrangement section 2393 Window arrangement section 240 Sound output section 250 Display section N Network

Claims

1. an object detection unit that detects an object included in an image and its position based on a predetermined image; a layout estimation unit that estimates a layout of a boundary portion including at least one of a wall, a floor, and a ceiling that are boundaries of a space of a room based on the predetermined image; a model interpolation unit that generates a model by interpolating a missing part of the boundary of the room layout estimated by the layout estimation unit, the model interpolation unit generates a wall at a position a predetermined distance away from the object detected by the object detection unit; generator.

2. and a door placement unit that determines the presence or absence of a door at the boundary portion based on the predetermined image, and places a door with respect to the wall generated by interpolation when it is determined that there is no door. The generating device of claim 1 .

3. a window arrangement unit that determines the presence or absence of a window in the boundary portion based on the predetermined image, and when it is determined that there is no window, arranges a window in a wall in which no door is arranged and in which a window can be arranged, among the walls generated by interpolation, The generating device according to claim 2 .

4. Detecting objects contained in a given image and their positions based on the image; A step of estimating a layout of a boundary portion including at least one of a wall, a floor, and a ceiling that are boundaries of a space of a room based on the predetermined image; generating a layout of the room estimated in the step of estimating the layout by interpolating a missing part of the boundary, In the step of generating by interpolation, a wall is generated at a position a predetermined distance away from the object detected in the step of detecting. Generation method.

5. Detecting objects contained in a given image and their positions based on the image; A step of estimating a layout of a boundary portion including at least one of a wall, a floor, and a ceiling that are boundaries of a space of a room based on the predetermined image; generating a layout of the room estimated in the step of estimating the layout by interpolating a missing part of the boundary; In the step of generating by interpolation, a wall is generated at a position a predetermined distance away from the object detected in the step of detecting; A program that causes a computer to execute the following.

Images