Image processing method, program, and image processing system

Abstract

To provide an image processing method, a program, and an image processing system for automatically disposing a virtual object in an appropriate position in a space inside a structural object.SOLUTION: An image processing method to be executed by an image processing system includes: a structure estimation step of estimating spatial structure from a background image (e.g., omnidirectional image) that shows the space inside a structural object (e.g., room) in all directions; a region estimation step of estimating a region in the space in which a virtual object (e.g., furniture 3D model) can be disposed, based on the estimated structure; and an image processing step of synthesizing the virtual object in the estimated region on the background image.SELECTED DRAWING: Figure 16

Description

Technical Field

[0001] The present disclosure relates to an image processing method, a program, and an image processing system.

Background Art

[0002] There is known a system that distributes image data captured using a photographing device capable of omnidirectional photographing and enables the situation of a remote site to be viewed at another site. Since a panoramic image obtained by photographing a predetermined site omnidirectionally allows a viewer to view in an arbitrary direction, it can convey immersive information. Such a system is used, for example, in fields such as online interior viewings of properties in the real estate industry.

[0003] Such a system is used, for example, in fields such as online interior viewings of properties in the real estate industry. Also, there is a service called "home staging" in which furniture and accessories are placed in a property to create a spatial effect and give viewers an attractive image of a dwelling to smoothly promote transactions. In this service, in order to reduce costs such as expenses or time costs, or the risk of damage to the property, instead of placing actual furniture in the property, a service that synthesizes three-dimensional model (CG) furniture with the image of the property that has been photographed is already known (for example, Patent Documents 1 to 3).

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the conventional method, when synthesizing an image of a virtual object such as furniture with a photographed image, there are cases where the virtual object is placed in an unnatural position for a viewer viewing the image, and there is room for improvement from the perspective of the accuracy of automatic placement of the virtual object.

Means for Solving the Problems

[0005] To solve the above-mentioned problems, the invention according to claim 1 is an image processing method performed by an image processing system, comprising: a structure estimation step of estimating the structure and size of a room from a 360-degree spherical image showing the room of a property in all directions; a detection step of detecting structural objects of the room that are captured in the 360-degree spherical image and further detecting the type of structural objects of the room; and a position estimation step of estimating the position of the detected structural objects of the room in the room, and further, the structure estimation step applies the estimated structure and size of the room and the type of structural objects of the room to the 360-degree spherical image in all directions The image processing method comprises: a determination step of estimating the use of a room indicated by a position and determining furniture to be composited into the 360-degree image based on the estimated use; a region estimation step of estimating a region in the room where the determined furniture can be placed, based on the estimated structure and size of the room, the estimated positions of structural objects in the room, and determined furniture placement rules for the room structure and detected types of structural objects in the room; and an image processing step of compositing a 3D model of the determined furniture into the estimated region of the 360-degree image.

Effects of the Invention

[0006] According to the present invention, it is possible to automatically position virtual objects in appropriate locations within the internal space of a structure. [Brief explanation of the drawing]

[0007] [Figure 1] This figure shows an example of the overall configuration of an image display system. [Figure 2] This figure shows an example of a 360-degree image before virtual objects are placed. [Figure 3] This figure shows an example of a processed image with virtual objects placed within it. [Figure 4] (A) is a hemispherical image (front) taken with the imaging device, (B) is a hemispherical image (back) taken with the imaging device, and (C) is an image represented using equirectangular projection. [Figure 5] (A) A conceptual diagram showing the state of covering the sphere with an equirectangular projection image, and (B) A diagram showing a full-sphere image. [Figure 6] This diagram shows the positions of a virtual camera and a predetermined region when a 360-degree spherical image is treated as a three-dimensional sphere. [Figure 7] This figure shows the relationship between information from a predetermined region and an image of a predetermined region T. [Figure 8] This figure shows an example of the conditions during imaging using the imaging device. [Figure 9] This is a diagram illustrating an example of a 360-degree image. [Figure 10] This figure illustrates an example of a planar image converted from a 360-degree image. [Figure 11] This figure shows an example of the hardware configuration of an image processing device, an image distribution device, and a display device. [Figure 12] This figure shows an example of the functional configuration of an image display system. [Figure 13] This figure shows an example of the functional configuration of an image display system. [Figure 14] This is a schematic diagram showing an example of an image data management table. [Figure 15] This is a conceptual diagram showing an example of a condition information management table. [Figure 16] It is a flowchart showing an example of processing in an image processing apparatus. [Figure 17] It is a flowchart showing an example of structure estimation processing. [Figure 18] It is a diagram for explaining an example of a structure estimation result for a captured image. [Figure 19] It is a diagram showing an example of the shape of a spatial structure estimated by structure estimation processing. [Figure 20] It is a diagram for explaining an example of a method for calculating the size of a space in structure estimation processing. [Figure 21] It is a diagram showing an example of an image when the placement of a virtual object fails. [Figure 22] It is a diagram for explaining an example of a subject detection result for a captured image. [Figure 23] (A)(B) It is a diagram for explaining an example of a process of projecting a subject onto an estimated spatial structure. [Figure 24] It is a flowchart showing an example of the layout processing of a virtual object. [Figure 25] (A)(B) It is a diagram for explaining an example of a layout algorithm for a 3D model of furniture. [Figure 26] (A)(B)(C) It is a diagram for explaining an example of a layout algorithm for a 3D model of furniture. [Figure 27] It is a diagram showing an example of a 3D model of furniture. [Figure 28] It is a diagram showing an example of a layout result of a 3D model of furniture. [Figure 29] It is a diagram showing an example of a layout result of a 3D model of furniture on a 3D space model. [Figure 30] It is a diagram showing an example of a processed image using a shadow catcher. [Figure 31] (A)(B) It is a diagram showing an example of image processing using Image Based lighting. [Figure 32] It is a diagram showing an example of a spatial estimation result and a subject detection result in a processed image in which a virtual object is synthesized. [Figure 33] This figure shows an example where the virtual object is too close to the imaging device. [Figure 34] (A)(B) This figure shows an example of a region where the placement of virtual objects is prohibited. [Figure 35] (A) is a diagram showing an example of a photographed image in which a support member is visible, and (B) is a diagram showing an example of an image in which a virtual object is placed on top of the visible support member. [Figure 36] This is a diagram illustrating an example of the size of a support member. [Figure 37] This is a sequence diagram showing an example of image display processing. [Figure 38] (A) is an example of a screen showing a captured image displayed on the display device, and (B) is an example of a screen showing a processed image displayed on the display device. [Figure 39] This is a conceptual diagram showing an example of an additional information management table. [Figure 40] This is a diagram illustrating an example of where additional information can be placed. [Figure 41] (A)(B) These are examples of processed images with additional information superimposed. [Figure 42] This figure shows another example of a processed image displayed on a display device. [Figure 43] This figure shows another example of a processed image displayed on a display device. [Modes for carrying out the invention]

[0008] The embodiments for carrying out the invention will be described below with reference to the drawings. In the description of the drawings, the same elements will be denoted by the same reference numeral, and redundant explanations will be omitted.

[0009] ●Embodiment● ● Overview of the image display system First, an overview of the configuration of the image display system according to the embodiment will be described using Figure 1. Figure 1 is a diagram showing an example of the overall configuration of the image display system. The image display system 1 shown in Figure 1 is a system that allows viewers to view real estate properties online by displaying images of the interior space of structures such as real estate properties on a display device 90.

[0010] As shown in Figure 1, the image display system 1 includes an image processing device 10, an image distribution device 30, a shooting device 70, a communication terminal 80, and a display device 90. The image processing device 10, image distribution device 30, shooting device 70, communication terminal 80, and display device 90 that constitute the image display system 1 can communicate via a communication network 5. The communication network 5 is constructed using the Internet, a mobile communication network, a LAN (Local Area Network), etc. In addition to wired communication, the communication network 5 may also include wireless communication networks such as 3G (3rd Generation), 4G (4th Generation), 5G (5th Generation), Wi-Fi (Wireless Fidelity) (registered trademark), WiMAX (Worldwide Interoperability for Microwave Access), or LTE (Long Term Evolution).

[0011] The image processing device 10 is a server computer that performs image processing on captured images of the interior space of a structure such as a real estate property. The image processing device 10 synthesizes virtual objects onto the captured images based on, for example, captured image data transmitted from the camera 70, usage information indicating the use of the space captured by the camera 70, and furniture information transmitted from the communication terminal 80. Here, the furniture information includes data showing a 3D model of the furniture, furniture setting data indicating rules regarding the arrangement of the furniture, etc. The 3D model of the furniture is an example of a virtual object, and the furniture information is an example of object information. In addition to 3D models of home appliances, the virtual objects may also be 3D models of, for example, electrical products, decorative items, paintings, lighting, fixtures, or furniture.

[0012] The image distribution device 30 is a server computer that distributes processed image data processed by the image processing device 10.

[0013] Here, the image processing device 10 and the image distribution device 30 are referred to as the image processing system 3. The image processing system 3 may, for example, be a computer that integrates all or part of the functions of the image processing device 10 and the image distribution device 30. Furthermore, the image processing device 10 and the image distribution device 30 may each be configured to distribute their functions across multiple computers. Additionally, while the image processing device 10 and the image distribution device 30 are described as server computers residing in a cloud environment, they may also be servers residing in an on-premises environment.

[0014] The imaging device 70 is a special digital camera (360° spherical imaging device) capable of capturing a 360° spherical image by photographing the interior space of a structure such as a real estate property in all directions. The imaging device 70 is used, for example, by a real estate agent who manages or sells real estate properties. The imaging device 70 may also be a wide-angle camera or stereo camera capable of acquiring wide-angle images with a field of view greater than a predetermined value. A wide-angle image is generally an image taken using a wide-angle lens, an image taken with a lens that can capture a wider area than what the human eye can perceive. In other words, the imaging device 70 is an imaging means capable of acquiring images (360° spherical images, wide-angle images) taken using a lens with a focal length shorter than a predetermined value. A wide-angle image generally means an image taken with a lens with a focal length of 35mm or less in 35mm film equivalent. Furthermore, the images obtained by the imaging device 70 may be videos, still images, or both. The images may also include sound.

[0015] The communication terminal 80 is a computer such as a notebook PC that provides the image processing device 10 with information about virtual objects to be placed in the space captured in the image. The communication terminal 80 is used, for example, by a furniture manufacturer that manufactures or sells furniture to be placed.

[0016] The display device 90 is a computer such as a smartphone used by the viewer of the image. The display device 90 displays the image distributed from the image distribution device 30. Note that the display device 90 is not limited to a smartphone, but may be a PC, tablet device, wearable device, HMD (head-mounted display), PJ (projector), or IWB (Interactive White Board: an electronic whiteboard with interactive blackboard functionality).

[0017] Here, using Figures 2 and 3, we will explain the images displayed on the display device 90 in the image display system 1. Figure 2 is a diagram showing an example of a 360-degree spherical image before virtual objects are placed. The image shown in Figure 2 is a 360-degree spherical image of a room in a real estate property, which is an example of the interior space of a structure, captured by the shooting device 70. 360-degree spherical images are suitable for viewing real estate properties because they can capture the interior of a room in all directions. While there are various forms of 360-degree spherical images, they are often generated using the equirectangular projection method, which will be described later. The advantages of images generated using this equirectangular projection method are that the outline of the image is rectangular, making image data storage efficient and easy, and that there is little distortion near the equator, resulting in a relatively natural appearance because vertical lines are not distorted.

[0018] Figure 3 shows an example of a processed image with virtual objects placed. The image in Figure 3 shows the room shown in Figure 2 with furniture placed in it. In Figure 3, the 360-degree spherical image shown in Figure 2 is used as the background image, and a 3D model of furniture, which is an example of a virtual object, is superimposed on this background image. The image processing device 10 places the 3D model of furniture in a natural state based on the structure of the room, such as the floor, walls, and ceiling, which is captured by the shooting device 70. As shown in Figure 3, the room has a desk and a bed placed along the walls, and the passages used on a daily basis are not blocked by furniture.

[0019] Traditionally, in order to place 3D furniture models onto a 360-degree image of a real estate property (a room), the furniture needed to be placed in a natural position relative to the camera's shooting position. This required manual adjustments by the user to align the placement and orientation. While methods exist for automatically placing furniture models, they still require inputting a floor plan or user input to understand the room's structure, leaving room for improvement in terms of reducing effort and increasing the accuracy of automatic placement of virtual objects.

[0020] Therefore, the image processing system 3 uses a 360-degree spherical image of the room's interior to detect the room's structure and objects installed in the room, and estimates the area where virtual objects can be placed. The image processing system 3 then places the virtual objects in the estimated area where they can be placed and generates a processed image, as shown in Figure 3, which is a composite of the placed virtual objects and the 360-degree spherical image. As a result, the image processing system 3 can naturally arrange furniture based on the general state of the room estimated from the 360-degree spherical image.

[0021] Here, a room, which is a real estate property, is an example of an interior space within a structure. A structure is, for example, a building such as a house, office, or shop. The 360-degree image is an image captured by the imaging device 70, and is an example of a background image showing the interior space of a structure in all directions.

[0022] ○Method for generating a 360-degree image○ Here, we will explain the method for generating a 360-degree image using Figures 4 to 10. First, we will explain the general process from the image captured by the imaging device 70 to the generation of a 360-degree image using Figures 4 and 5. Figure 4(A) shows the hemispherical image (front side) captured by the imaging device, Figure 4(B) shows the hemispherical image (back side) captured by the imaging device, and Figure 4(C) shows the image represented by equirectangular projection (hereinafter referred to as "equistratic projection image"). Figure 5(A) is a conceptual diagram showing the state in which the sphere is covered by the equirectangular projection image, and Figure 5(B) shows the 360-degree image.

[0023] The imaging device 70 has image sensors on both the front and rear sides. These image sensors are used in conjunction with optical components such as lenses that can capture hemispherical images (angle of view of 180° or more). The imaging device 70 can obtain two hemispherical images by capturing subjects around the user with each of the two image sensors.

[0024] As shown in Figures 4(A) and 4(B), the images obtained by the image sensor of the imaging device 70 are curved hemispherical images (front and rear). The imaging device 70 then combines the hemispherical image (front) and the hemispherical image (rear) that has been inverted 180 degrees to create an equirectangular projection image EC as shown in Figure 4(C).

[0025] The imaging device 70 then uses OpenGL ES (Open Graphics Library for Embedded Systems) to overlay the equirectangular projection image EC so that it covers the sphere, as shown in Figure 5(A), and creates a full-sphere image (full-sphere panoramic image) CE as shown in Figure 5(B). In this way, the full-sphere image CE is represented as an image where the equirectangular projection image EC is facing the center of the sphere. OpenGL ES is a graphics library used to visualize 2D (2-Dimensional) and 3D (3-Dimensional) data. The full-sphere image CE may be a still image or a video. Furthermore, the conversion method is not limited to OpenGL ES; any method capable of converting from a hemispherical image to an equirectangular projection method is acceptable, for example, it may be a CPU calculation or an OpenCL calculation.

[0026] As described above, the 360-degree spherical image CE is an image pasted to cover a sphere, which can cause discomfort to the human eye. Therefore, the imaging device 70 can display a predetermined area T of the 360-degree spherical image CE (hereinafter referred to as the "predetermined area image") as a flat image with less curvature, thereby providing a display that does not cause discomfort to the human eye. This will be explained with reference to Figures 6 and 7.

[0027] Figure 6 shows the positions of a virtual camera and a predetermined region when the 360-degree image is treated as a three-dimensional solid sphere. The virtual camera IC corresponds to the viewpoint of the user viewing the 360-degree image CE, which is displayed as a three-dimensional solid sphere. Figure 6 represents the 360-degree image CE as a three-dimensional solid sphere CS. If the generated 360-degree image CE is a solid sphere CS, then, as shown in Figure 6, the virtual camera IC is located inside the 360-degree image CE. The predetermined region T in the 360-degree image CE is the shooting area of ​​the virtual camera IC and is identified by predetermined region information indicating the shooting direction and field of view of the virtual camera IC in the three-dimensional virtual space containing the 360-degree image CE. Furthermore, zooming in on the predetermined region T can also be represented by moving the virtual camera IC closer to or further away from the 360-degree image CE. The predetermined region image Q is an image of the predetermined region T in the 360-degree image CE. Therefore, the predetermined region T can be identified by the field of view α and the distance f from the virtual camera IC to the 360-degree image CE.

[0028] The predetermined region image Q is then displayed on a predetermined display as an image of the shooting area of ​​the virtual camera IC. The following explanation will use the shooting direction (ea, aa) and field of view (α) of the virtual camera IC. Note that the predetermined region T may be represented not by the field of view α and distance f, but by the imaging area (X, Y, Z) of the virtual camera IC which is the predetermined region T.

[0029] Next, we will explain the relationship between the predetermined region information and the image of the predetermined region T using Figure 7. Figure 7 is a diagram showing the relationship between the predetermined region information and the image of the predetermined region T. As shown in Figure 7, "ea" is the elevation angle, "aa" is the azimuth angle, and "α" is the field of view (Angle). That is, the orientation of the virtual camera IC is changed so that the point of fixation of the virtual camera IC, indicated by the shooting direction (ea,aa), becomes the center point CP(x,y) of the predetermined region T, which is the shooting area of ​​the virtual camera IC. As shown in Figure 7, the center point CP(x,y) when the diagonal field of view of the predetermined region T, represented by the field of view α of the virtual camera IC, is α becomes the parameter ((x,y)) of the predetermined region information. The predetermined region image Q is the image of the predetermined region T in the 360-degree spherical image CE. f is the distance from the virtual camera IC to the center point CP(x,y). L is the distance between any vertex of the predetermined region T and the center point CP(x,y) (2L is the diagonal). In Figure 7, the trigonometric functions generally hold true, as shown in (Equation 1) below.

[0030]

number

[0031] Next, Figure 8 will be used to explain the state of the imaging device 70 during imaging. Figure 8 is a diagram showing an example of the state of imaging by the imaging device. In order to capture an entire room of a real estate property, it is preferable to install the imaging device 70 at a height close to the height of human eyes. For this reason, as shown in Figure 8, the imaging device 70 is generally fixed with a support member 7 such as a monopod or tripod for imaging. As described above, the imaging device 70 is a 360-degree imaging device capable of acquiring light rays from all directions in the entire surrounding area, and it can also be said that it acquires an image (360-degree image CE) on a unit sphere around the imaging device 70. When the imaging direction is determined, the coordinates of the 360-degree image are determined. For example, in Figure 8, point A is at a distance of (d, -h) from the center point C of the imaging device 70, and if the angle made between line segment AC and the horizontal direction is θ, then the angle θ can be expressed by the following (Equation 2).

[0032]

number

[0033] Then, assuming that point A is at a depression angle θ, the distance d between point A and point B can be expressed by the following equation (Equation 3), using the installation height h of the imaging device 70.

[0034]

number

[0035] Here, we will briefly explain the process of converting positional information on a spherical image to coordinates on a planar image converted from a spherical image. Figure 9 is a diagram illustrating an example of a spherical image. Figure 9(A) shows the hemispherical image shown in Figure 4(A) with lines connecting points where the angles of incidence in the horizontal and vertical directions relative to the optical axis are equal. Hereafter, the angle of incidence in the horizontal direction relative to the optical axis will be referred to as "θ", and the angle of incidence in the vertical direction relative to the optical axis will be referred to as "φ".

[0036] Figure 10(A) illustrates an example of an image processed using equirectangular projection. Specifically, the image shown in Figure 9 is matched using a pre-generated LUT (Look Up Table), processed using equirectangular projection, and the processed images shown in Figures 9(A) and (B) are combined to generate the planar image shown in Figure 10(A), which corresponds to the 360-degree spherical image, by the imaging device 70. The equirectangular projection image EC shown in Figure 4(C) is an example of the planar image shown in Figure 10(A).

[0037] As shown in Figure 10(A), in an image processed using equirectangular projection, latitude (θ) and longitude (φ) are orthogonal. In the example shown in Figure 10(A), the center of the image is (0,0), and the latitude direction is represented as -90 to +90, and the longitude direction as -180 to +180, allowing any position in the spherical image to be indicated. For example, the coordinates of the top left of the image are (-180,-90). The coordinates of the spherical image may be expressed in a 360-degree format as shown in Figure 10(A), or in radians or in pixels like a real image. Alternatively, the coordinates of the spherical image may be converted to two-dimensional coordinates (x,y) as shown in Figure 10(B).

[0038] Furthermore, the image compositing process shown in Figure 10(A) or (B) is not limited to simply arranging the hemispherical images shown in Figures 9(A) and (B) in sequence. For example, if the horizontal center of the full-sphere image is not θ=180°, in the compositing process, the imaging device 70 first preprocesses the hemispherical image shown in Figure 4(C) and places it at the center of the full-sphere image. Next, the imaging device 70 divides the image to be generated into left and right portions, with the preprocessed hemispherical image shown in Figure 4(B) being sized to fit into the left and right portions, and then composites the hemispherical images to generate the equirectangular projection image EC shown in Figure 4(C).

[0039] Furthermore, in the planar image shown in Figure 10(A), the points corresponding to the poles (PL1 or PL2) of the hemispherical images (global images) shown in Figures 9(A) and (B) are line segments CT1 or CT2. This is because, as shown in Figures 5(A) and 5(B), the global image (for example, global image CE) is created by using OpenGL ES to superimpose the planar image (equirectangular projection image EC) shown in Figure 10(A) onto a sphere.

[0040] ● Hardware configuration Next, the hardware configuration of each device constituting the image display system according to the embodiment will be described using Figure 11. Note that the hardware configuration shown in Figure 11 may have components added or removed as needed.

[0041] ○Hardware configuration of the image processing device○ First, the hardware configuration of the image processing device 10 will be explained using Figure 11. Figure 11 is a diagram showing an example of the hardware configuration of the image processing device. Each hardware component of the image processing device 10 is indicated by a code in the 100s. The image processing device 10 is built by a computer and, as shown in Figure 11, includes a CPU (Central Processing Unit) 101, ROM (Read Only Memory) 102, RAM (Random Access Memory) 103, HD (Hard Disk) 104, HDD (Hard Disk Drive) controller 105, display 106, external device connection I / F (Interface) 108, network I / F 109, bus line 110, keyboard 111, pointing device 112, DVD-RW (Digital Versatile Disk Rewritable) drive 114, and media I / F 116.

[0042] Of these, the CPU 101 controls the operation of the entire image processing device 10. The ROM 102 stores programs used to drive the CPU 101, such as the IPL (Initial Program Loader). The RAM 103 is used as the work area for the CPU 101. The HD 104 stores various data such as programs. The HDD controller 105 controls the reading or writing of various data to the HD 104 according to the control of the CPU 101. The display 106 displays various information such as cursors, menus, windows, characters, or images. The display 106 may be a touch panel display equipped with input means. The external device connection I / F 108 is an interface for connecting various external devices. In this case, external devices include, for example, a USB memory or a printer. The network I / F 109 is an interface for data communication using the communication network 5. The bus line 110 is an address bus or data bus, etc., for electrically connecting each component such as the CPU 101 shown in Figure 11.

[0043] Furthermore, the keyboard 111 is a type of input means equipped with multiple keys for inputting characters, numbers, various instructions, etc. The pointing device 112 is a type of input means for selecting or executing various instructions, selecting a processing target, or moving a cursor, etc. Note that the input means may not be limited to the keyboard 111 and the pointing device 112, but may also be a touch panel or an audio input device, etc. The DVD-RW drive 114 controls the reading or writing of various data to the DVD-RW 113, which is an example of a removable recording medium. Note that the removable recording medium is not limited to DVD-RW, but may also be DVD-R or Blu-ray® Disc, etc. The media I / F 116 controls the reading or writing (storage) of data to the recording medium 115, such as flash memory.

[0044] ○Hardware configuration of the image distribution device○ Figure 11 shows an example of the hardware configuration of an image distribution device. Each hardware component of the image distribution device 30 is indicated by a 300-series code in parentheses. The image distribution device 30 is built using a computer and has a configuration similar to that of the image processing device 10, as shown in Figure 11; therefore, a description of each hardware component is omitted.

[0045] ○Hardware configuration of the display device○ Figure 11 shows an example of the hardware configuration of the display device. Each hardware component of the display device 90 is indicated by a number in the 900s in parentheses. The display device 90 is built by a computer and has a configuration similar to that of the image processing device 10, as shown in Figure 11, so a description of each hardware component is omitted.

[0046] Furthermore, each of the above programs may be distributed as an installable or executable file recorded on a computer-readable recording medium. Examples of recording media include CD-R (Compact Disc Recordable), DVD (Digital Versatile Disk), Blu-ray Disc, SD card, USB memory, etc. The recording media can also be provided domestically or internationally as a program product. For example, the image processing system 3 realizes the image processing method according to the present invention by executing the program according to the present invention.

[0047] ●Functional Configuration Next, the functional configuration of the image display system according to the embodiment will be described using Figures 12 to 15. Figures 12 and 13 are diagrams showing an example of the functional configuration of the image display system. Figures 12 and 13 show the devices or terminals shown in Figure 1 that are related to the processing or operation described later.

[0048] ○Functional Configuration of Image Processing Device○ First, the functional configuration of the image processing device 10 will be explained using Figure 12. The image processing device 10 includes a transmitting / receiving unit 11, a receiving unit 12, a judgment unit 13, a structure estimation unit 14, a detection unit 15, a position estimation unit 16, a region estimation unit 17, a determination unit 18, a placement unit 19, an image processing unit 20, an input unit 21, and a storage / reading unit 29. Each of these units is a function or means realized by any of the components shown in Figure 11 operating according to instructions from the CPU 101 that follow the image processing device program deployed from HD 104 onto RAM 103. The image processing device 10 also has a storage unit 1000 constructed from ROM 102, RAM 103, and HD 104 shown in Figure 11.

[0049] The transmitting / receiving unit 11 is mainly implemented by the processing of the CPU 101 with respect to the network I / F 109, and transmits and receives various data or information with other devices or terminals via the communication network 5.

[0050] The reception unit 12 is mainly implemented by the CPU 101 processing the keyboard 111 or pointing device 112, and accepts various selections or inputs from the user. The decision unit 13 is implemented by the CPU 101 and makes various decisions.

[0051] The structure estimation unit 14 is implemented by the CPU 101 and estimates the structure of the space based on a background image that shows the internal space of the structure in all directions.

[0052] The detection unit 15 is implemented by the processing of the CPU 101 and detects the subject shown in the background image.

[0053] The position estimation unit 16 is implemented by the CPU 101 and estimates the spatial position of the subject detected by the detection unit 15.

[0054] The region estimation unit 17 is implemented by the processing of the CPU 101 and estimates the region in space where virtual objects can be placed based on the structure of the space estimated by the structure estimation unit 14.

[0055] The determination unit 18 is implemented by the CPU 101 and determines the virtual objects to be placed in the space based on the use of the space shown in the background image.

[0056] The placement unit 19 is implemented by the processing of the CPU 101 and places virtual objects in the region estimated by the region estimation unit 17. For example, the placement unit 19 lays out the virtual objects determined by the determination unit 18 in the placeable region estimated by the region estimation unit 17.

[0057] The image processing unit 20 is implemented by the CPU 101 and composites virtual objects onto the background image in the region estimated by the region estimation unit 17. The image processing unit 20 then performs rendering processing on the placed virtual objects, for example, based on the layout result of the virtual objects by the placement unit 19.

[0058] The input unit 21 is mainly implemented by the processing of the CPU 101 with respect to the external device connection I / F 108, and accepts various data or information input from external devices.

[0059] The memory / read unit 29 is mainly implemented by the CPU 101 and stores various data (or information) in the memory unit 1000 and reads various data (or information) from the memory unit 1000.

[0060] ○Image data management table figure 14 This is a conceptual diagram showing an example of an image data management table. The storage unit 1000 contains, 14 An image data management DB1001 is constructed, which consists of the image data management tables shown. These image data management tables manage the image ID that identifies the image data, the condition ID that identifies the selection criteria for the virtual object, the captured image data, and the processed image data in association with each other.

[0061] ○Condition Information Management Table figure 15This is a conceptual diagram showing an example of a condition information management table. The condition information management table manages condition information that indicates the placement conditions of virtual objects. The storage unit 1000 contains, 15 A condition information management DB1002 is constructed, which is composed of the condition information management tables shown. This condition information management table manages information such as a condition ID that identifies the selection criteria for a virtual object, the use and size of the room, and the style and set of furniture, which are examples of virtual objects to be selected.

[0062] ○Functional Configuration of Image Distribution Device○ Next, the functional configuration of the image distribution device 30 will be described using Figure 13. The image distribution device 30 includes a transmitting / receiving unit 31, a display control unit 32, a determination unit 33, a coordinate detection unit 34, a calculation unit 35, an image processing unit 36, and a storage / reading unit 39. Each of these units is a function or means realized by any of the components shown in Figure 11 operating according to instructions from the CPU 301 that follow the image distribution device program deployed from the HD 304 onto the RAM 303. The image distribution device 30 also has a storage unit 3000 constructed from the ROM 302, RAM 303, and HD 304 shown in Figure 11.

[0063] The transmitting / receiving unit 31 is mainly implemented by the processing of the network interface 309 by the CPU 301, and transmits and receives various data or information with other devices or terminals via the communication network 5.

[0064] The display control unit 32 is primarily implemented by the CPU 301 and displays various images or characters on the display device 90. The display control unit 32 displays various screens on the display device 90 by distributing (transmitting) image data to the display device 90, for example, using a web browser or a dedicated application. The various screens displayed on the display device 90 are defined, for example, by HTML (HyperText Markup Language), XHTML (Extensible HyperText Markup Language), CSS (Cascading Style Sheets), or JavaScript (registered trademark). The judgment unit 33 is implemented by the CPU 301 and performs various judgments.

[0065] The coordinate detection unit 34 is implemented by the CPU 101 and detects the coordinate position of a virtual object shown in the processed image generated by the image processing device 10. The calculation unit 35 is implemented by the CPU 301 and calculates the center position of the virtual object for superimposing the additional information described later onto the processed image, based on the coordinate position detected by the coordinate detection unit 34. The image processing unit 36 ​​is implemented by the CPU 301 and performs predetermined image processing on the processed image generated by the image processing device 10.

[0066] The memory / read unit 39 is mainly implemented by the CPU 301 and stores various data (or information) in the memory unit 3000 and reads various data (or information) from the memory unit 3000.

[0067] ○Functional Configuration of Display Device○ Next, the functional configuration of the display device 90 will be described using Figure 13. The display device 90 has a transmitting / receiving unit 91, a receiving unit 92, and a display control unit 93. Each of these units is a function or means realized by any of the components shown in Figure 11 operating according to instructions from the CPU 901 that follow the display device program deployed from the HD 904 onto the RAM 903.

[0068] The transmitting / receiving unit 91 is mainly implemented by the processing of the network interface 909 by the CPU 901, and transmits and receives various data or information with other devices or terminals via the communication network 5.

[0069] The reception unit 92 is mainly implemented by the processing of the keyboard 911 or pointing device 912 by the CPU 901, and accepts various selections or inputs from the user.

[0070] The display control unit 93 is mainly implemented by the processing of the CPU 901 and displays various images or characters on the display 906. The display control unit 93 accesses the image distribution device 30 using, for example, a web browser or a dedicated application, and displays images corresponding to the data distributed from the image distribution device 30 on the display 906, which is an example of a display means.

[0071] ●Processing or operation of the embodiment ○Image Synthesis Processing○ Next, the processing or operation of the image display system according to the embodiment will be described using Figures 16 to 43. First, the image synthesis processing in the image processing device 10 will be described using Figures 16 to 32. In the following description, an example of a room in a real estate property will be shown as an example of the space inside a structure, and an example of furniture placed in the room will be shown as an example of a virtual object. Figure 16 is a flowchart showing an example of processing in the image processing device.

[0072] First, the image processing device 10 receives input of a captured image of a predetermined room, which is an example of the interior space of a structure (step S1). Specifically, the transmitting and receiving unit 11 of the image processing device 10 receives, for example, an image of the interior space of a predetermined structure captured by the imaging device 70 from the imaging device 70 via the communication network 5. The image processing device 10 may be configured to receive input of the captured image to be processed from the imaging device 70 when performing image synthesis processing, or it may be configured to store captured images received from the imaging device 70 in advance in the storage unit 1000 and read the stored captured images when performing image synthesis processing. Alternatively, the image processing device 10 may be configured to receive input of captured images via the input unit 21 by directly connecting to the imaging device 70 via the external device connection I / F 108. Furthermore, since there may be cases where the imaging device 70 does not have a communication function, the image processing device 10 is not limited to directly receiving input of captured images from the imaging device 70, but may also be configured to receive input of captured images via a predetermined communication device owned by the real estate agent.

[0073] Next, the determination unit 13 uses the captured image input in step S1 to determine whether the furniture arrangement in the room shown in the captured image is suitable (step S2). Preferably, the room shown in the captured image is, for example, an empty room that does not already have any furniture in it, or a room that has a predetermined space for arranging furniture. Therefore, if the determination unit 13 determines that the room shown in the captured image is an outdoor space or an external space outside of a structure, or that it is extremely narrow or has objects in it and therefore does not have space to arrange furniture, it determines that the room shown in the captured image is not suitable for furniture arrangement.

[0074] If the determination unit 13 determines that the furniture arrangement is suitable (YES in step S2), it proceeds to step S3. On the other hand, if the determination unit 13 determines that the furniture arrangement is not suitable (NO in step S2), it proceeds to step S9. In step S9, the image processing device 10 outputs an error message indicating that the furniture arrangement is unsuitable without performing image synthesis processing. Specifically, the storage / reading unit 29 of the image processing device 10 associates the error message with the captured image input in step S1 and stores it in the storage unit 1000. This allows viewers viewing the corresponding captured image to understand the error message along with the captured image. The image processing device 10 may also be configured to execute the processing from step S3 onward after outputting the error message. However, in this case, there is a high possibility that there will be no furniture that can be placed in the image synthesis processing in step S7, described later, and the processed image data stored in step S8, described later, will be an image without furniture.

[0075] Next, the structure estimation unit 14 uses the captured image input in step S1 to estimate the structure of the room as seen in the captured image (step S3). One known method for estimating the structure of a room is to detect straight lines of objects in the captured image using image processing, find the vanishing points of the detected lines, and estimate the structure of the room from the boundaries of the floor, walls, or ceiling. When using a 360-degree image, all elements necessary for estimating the structure of a room, such as the ceiling, floor, and walls, are captured. This has the advantage of increasing the accuracy of structure estimation compared to using a normal planar image, where only a part of the room is captured and estimation other than detection of vanishing points is difficult. In addition, methods using machine learning for detecting vanishing points, detecting boundaries between floor and walls or ceiling and walls, or estimating the three-dimensional structure from the detection results are also known. The structure estimation unit 14 may perform structure estimation using any of the known methods.

[0076] ○Structural estimation process Here, an example of the structure estimation process in the image processing device 10 will be explained in detail using Figures 17 to 20. Figure 17 is a flowchart of an example of the structure estimation process.

[0077] First, the structure estimation unit 14 uses the captured image to estimate the vertices of the space captured in the image (step S31). Specifically, the structure estimation unit 14, for example, as described above, detects the lines of the subject captured in the image by image processing of the captured image, and estimates the vanishing points calculated from the detected lines as the vertices of the space.

[0078] Figure 18 is a diagram illustrating an example of the structural estimation result for a captured image. Figure 18 shows an example of a room structure represented by equirectangular projection. As mentioned above, in equirectangular projection, vertical lines are projected as straight lines, and horizontal lines are projected as curves. When these are applied to the structure of a room, many rooms have a shape in which straight lines intersect each other vertically. Therefore, the structural estimation unit 14 can estimate the general structure of a room by using an image represented by equirectangular projection. The structural estimation unit 14 estimates the general structure of a room by detecting the elements, lines, and surfaces that make up the room. The example in Figure 18 is an example of a rectangular room being photographed, and the structural estimation unit 14 estimates that there are four surfaces in the horizontal direction and two surfaces in the vertical direction.

[0079] Next, if the structure estimation unit 14 determines that it is possible to classify the shape of the room from the vertex estimation results in step S31 (YES in step S32), it proceeds to step S33. On the other hand, if the structure estimation unit 14 determines that it is not possible to classify the shape of the room (NO in step S32), it continues the process in step S31. Figure 19 shows an example of the shape of the spatial structure estimated by the structure estimation process. The shapes of real rooms are diverse, and obtaining detailed three-dimensional information requires measurements using laser scanners, total stations, etc., but this is a time-consuming and expensive process. When virtually arranging furniture, detailed reconstruction is not necessary, and a simplified version with narrowed conditions is sufficient. That is, since it is sufficient to know the general structure of the room, the structure estimation unit 14 narrows down the conditions by using, for example, the assumption that the room is composed of straight lines and planes, and that these straight lines basically intersect at 90° (Manhattan World Assumption). Furthermore, in order to restore the structure to a degree that allows for furniture placement, the structural estimation unit 14 classifies it, for example, as a rectangular prism with eight vertices, as shown in Figure 19, or as an L-shaped room with 12 vertices.

[0080] Next, the structure estimation unit 14 estimates the size (scale) of the space captured in the image (step S33). Specifically, the structure estimation unit 14 obtains the coordinates of each vertex of the room on an equirectangular projection using the methods of steps S31 and S32. The structure estimation unit 14 converts the obtained equirectangular projection coordinates into coordinates in three-dimensional space.

[0081] The structural estimation unit 14 detects whether the imaging device 70 is installed vertically or detects the direction of gravitational acceleration and makes corrections. The structural estimation unit 14 can then estimate the structure of the room by assuming that the South Pole on the equirectangular projection (for example, PL1 shown in Figure 9(A)) coincides with the direction of gravitational acceleration and that the Manhattan World hypothesis is followed.

[0082] The Manhattan World Hypothesis posits that most man-made structures are constructed parallel to a Cartesian coordinate system, allowing us to assume that walls, ceilings, etc., are constrained to be parallel to the x, y, and z directions. Following this assumption, as shown in Figure 20, assuming the camera 70 is at a height h from the floor, the distance between point A, where the floor meets the wall, and point B, where the ceiling meets the wall, can be expressed using the installation height h of the camera 70. On the other hand, this method only reveals the general shape of the room, and does not accurately determine its size (scale). In extreme cases, it is not even possible to determine whether the room is a miniature with a height of 20 cm or a typical room with a height of 2 m, so it is necessary to have some understanding of the room's scale in order to place furniture.

[0083] As a method for calculating the scale of the room, the structural estimation unit 14, for example, assumes that the installation height h of the imaging device 70 is known and calculates point A located at the depression angle θ using the above (Equation 3) shown in Figure 8. Alternatively, as a method for measuring the installation height of the imaging device 70 by physical means, the structural estimation unit 14 may measure the distance to the optical center of the imaging device 70 using a method such as laser ranging. Furthermore, as a method for measuring the installation height of the imaging device 70 by image processing, the structural estimation unit 14 may measure the distance to the scale by preparing a scale of known length on the floor and photographing it with the imaging device 70.

[0084] Furthermore, the structural estimation unit 14 may estimate the scale of a room assuming the room height is known. In Japan, the Building Standards Act stipulates that the ceiling height must be 210 cm or more, but the ceiling height of a typical apartment is 240 cm to 250 cm. In the United States, it is about 8 feet (243 cm), which is close to the case in Japan. Although there is variation in room height, if it is within ±10 cm, the accuracy of the scale will match to 5% or less, and it will function as a rough scale. In addition, as a method of measuring the distance to an object using stereo vision, the structural estimation unit 14 may take advantage of the parallax of the optical centers of the multiple lenses of the imaging device 70 and measure the distance to a predetermined object using the common part between the lenses. Furthermore, as a method of estimating the scale from IMU (Inertial Measurement Unit) data, which is so-called Structure from motion that estimates a three-dimensional structure from multiple images, the structural estimation unit 14 may estimate the scale based on the value of the distance traveled, which can be roughly estimated from the IMU data of the imaging device 70.

[0085] The structural estimation unit 14 may use any of the above methods as a method for calculating the room scale in step S33.

[0086] Then, the structure estimation unit 14 acquires coordinate information for each vertex based on the room structure estimated in steps S31 to S33 (step S34). As a result of the series of processes, the structure estimation unit 14 acquires coordinate information for each vertex of n rooms (n=8 or 12). The structure estimation unit 14 acquires coordinates Cn (Cn=((x0,y0,z0),(x1,y1,z1),…(xn,yn,zn)) expressed in XYZ coordinates as shown in Figure 10(B), for example, with the optical center of the imaging device 70 as the limit. Alternatively, the structure estimation unit 14 may acquire coordinates in polar coordinates as shown in Figure 10(A).

[0087] In this way, the structure estimation unit 14 can estimate the general structure of the room captured in the captured image using the captured image input to the image processing device 10.

[0088] Returning to Figure 16, the detection unit 15 of the image processing device 10 detects subjects present in the room as captured in the image input in step 1 (step S4). However, the image processing device 10 may not be able to properly arrange furniture if it only knows the structure of the room. Figure 21 shows an example of an image when the placement of virtual objects fails. As shown in Figure 21, furniture may be placed in positions that are not suitable for actual furniture placement, such as a bed being placed in the hallway of the room. Therefore, in order to realize a natural furniture layout, the image processing device 10 uses the detection unit 15 to detect subjects as captured in the image and estimates natural places where furniture can be placed.

[0089] Here, the objects detected by the detection unit 15 are structural objects of the room that appear in the captured image, such as objects installed in the room, that is, objects that are pre-installed in the room and relate to the room's layout. Examples of objects detected by the detection unit 15 include doors, windows, frames, sliding doors, light switches, closets, cupboards, kitchens, corridors, air conditioners, electrical outlets, lighting outlets, fireplaces, ladders, stairs, or fire alarms.

[0090] Since the development of machine learning, many object detection algorithms have been developed as methods for detecting objects in images. A typical method is to represent the detected object with a rectangle (bounding box). In addition, by using a method called semantic segmentation, which represents objects as regions, objects can be detected with higher accuracy. The detection unit 15 may use any of the known methods for detecting objects in step S4. The detection unit 15 also detects the type of object in the image using a known method. The type of object in the image is information used to identify what the object in the image is (for example, whether it is a door or a window). Figure 22 is a diagram illustrating an example of the object detection result for a captured image. Figure 22 shows the results of the detection unit 15 detecting a kitchen, air conditioner, window, door, and passageway from the objects in the captured image.

[0091] In this way, the image processing device 10 can estimate the state of the room shown in the captured image by using the input captured image to estimate the structure of the room and detect the subject in the captured image. Furthermore, by performing subject detection based on the room structure estimated in step S3, the image processing device 10 can estimate the locations in the room structure where the subject may be installed, thereby improving the efficiency of processing. Note that the image processing device 10 may perform steps S3 and S4 in parallel, or the order of steps S3 and S4 may be reversed.

[0092] Next, the position estimation unit 16 of the image processing device 10 estimates the position of the object detected in step S4 within the room (step S5). The result of object detection is represented as a rectangle when a bounding box is used, or as a filled pixel in the corresponding area when semantic segmentation is used. These are representations on the unit sphere of the imaging device 70 as shown in Figure 23(A), and can also be represented on an equirectangular projection as shown in Figure 22. The position estimation unit 16 projects the result of object detection on the unit sphere as shown in Figure 23(A) onto the three-dimensionally reconstructed shape of the room. For example, among the detected objects, the position estimation unit 16 projects objects that are normally located on walls, such as doors, windows, and corridors as shown in Figure 23(B), onto the structure of the room estimated by the structure estimation unit 14.

[0093] In this case, the position estimation unit 16 projects a virtual object corresponding to the type of subject detected by the detection unit 15. The position estimation unit 16 can, for example, place a virtual object that acts as a light source at the detected window position, and then, by synthesizing the image of the placed virtual object with the image processing unit 20 (described later), it can more naturally represent ambient light entering the room. In this way, the position estimation unit 16 ultimately estimates and assigns the position of the subject in relation to the structure of the room.

[0094] Furthermore, the estimated position of the object in relation to the room structure by the position estimation unit 16 does not necessarily have to be accurate. In the case of object detection on an equirectangular projection, there will be a discrepancy with the actual position of the object, but since the object detection result is detected as larger than the object itself, there is a margin for estimating the area in which furniture can be placed, as described later, and this does not pose a major problem in terms of layout.

[0095] Next, the image processing device 10 performs furniture layout processing (step S6). When a human actually arranges furniture, the layout is based on the structure of the room and the position of the objects within that structure. There are general rules based on conventions and so on when humans arrange furniture, methods are known to either follow the rules of human-based layout or optimize the layout using machine learning based on a large number of past layout results. The image processing device 10 arranges the furniture based on simple rules for the room structure estimated by the structure estimation unit 14 and the objects detected by the detection unit 15.

[0096] ○ Layout processing Here, an example of layout processing in the image processing device 10 will be explained in detail using Figures 24 to 29. Figure 24 is a flowchart showing an example of layout processing of a virtual object. Figure 24 shows the process of determining the furniture to be placed according to the purpose of the room and automatically placing the furniture in order in the available areas.

[0097] First, the determination unit 18 of the image processing device 10 determines the furniture to be placed (step S61). The determination unit 18 determines the furniture to be placed based, for example, on the use and size of the room. Specifically, the determination unit 18 determines the furniture to be placed based on the condition information stored in the condition information management DB 1002 and the use information indicating the use of the room. The use information is information specified by the real estate agent or the like who photographed the target room. The image processing device 10 receives the use information transmitted from an external device, such as the shooting device 70, via the transmission / reception unit 11. The use information may be input to the image processing device 10 together with the captured image input in step S1, or it may be information directly specified to the image processing device 10.

[0098] Here, the usage information includes, for example, information on the room's purpose and size. The room's purpose is the intended use of the room, such as a living room, bedroom, or children's room. Generally, it is difficult to determine the room's purpose from the condition of the room itself, so it is preferable to configure the system to allow users, such as real estate agents who photograph the room, to select it according to their intentions. The room's purpose may be automatically estimated by the structure estimation unit 14 according to the structure of the room and the subject matter captured in the image, for example, a large room with a kitchen may be estimated as a living room, and a room with few windows may be estimated as a bedroom.

[0099] Furniture layouts are diverse, and the types of furniture used vary depending on individual preferences and cultural backgrounds. Furthermore, home staging aims to make a room look beautiful, requiring an aesthetic perspective, and thus, a wide variety of furniture arrangement patterns exist. The type of furniture chosen is also determined by various factors such as the room's intended use, size, style, season, and color coordination.

[0100] Therefore, the decision unit 18 searches the condition information management DB 1002 (see Figure 15) using the usage information as a search key, for example, to retrieve condition information associated with the same usage and size as the usage information. Then, based on the furniture style or furniture set indicated in the retrieved condition information, the decision unit 18 selects the furniture to be placed from among the furniture stored in the storage unit 1000 or from the furniture information transmitted from the communication terminal 80.

[0101] In the example shown in Figure 15, the condition information defines different furniture sets for each room's purpose and size. For example, living rooms and bedrooms are each classified into three stages (L, M, S) according to their size. For instance, a furniture set for a large room includes a dining table and a large sofa, while a furniture set for a small room includes a single sofa and a table. Alternatively, the condition information can define furniture styles instead of furniture sets. In this case, the determination unit 18 selects a furniture set from the furniture stored in the memory unit 1000 or transmitted from the communication terminal 80 that corresponds to the defined furniture style. Furniture styles include, for example, natural, pop, modern, Japanese, Scandinavian, or Asian. In addition to furniture sets or furniture styles, the condition information may also include information such as color coordination or season.

[0102] Next, the image processing device 10 acquires furniture information, which is information about the furniture to be placed as determined in step S61 (step S62). The furniture information includes data showing the 3D model of the furniture and furniture setting data showing the rules for furniture placement. Specifically, the storage / reading unit 29 of the image processing device 10 acquires the furniture information of the determined furniture by reading the furniture information stored in the storage unit 1000. The furniture information is transmitted from a communication terminal 80 owned by a furniture manufacturer, etc., to the image processing device 10 and stored in the storage unit 1000 in advance. Alternatively, the transmitting / receiving unit 11 of the image processing device 10 may be configured to acquire the furniture information of the determined furniture by receiving the furniture information transmitted from the communication terminal 80 in response to a request from the image processing device 10 in step S62.

[0103] Next, the area estimation unit 17 estimates the area where furniture can be placed based on the room structure estimated in step S3 and the position of the subject estimated in step S5 (step S63). The area where furniture can be placed will be specifically explained using Figures 25 and 26. Figure 25 is a diagram showing the layout algorithm for a 3D model of a rug, which is an example of furniture, as a specific example. Figure 25(A) shows the position of the shooting device 70 and the room structure estimated by the structure estimation unit 14, and Figure 25(B) shows the state in which the rug is placed in the center of the room, which is the area where furniture can be placed estimated by the area estimation unit 17. Rugs and carpets are placed on the floor, so if the structure of the room is known, they are furniture that can be placed regardless of the position of the subject, such as doors and windows.

[0104] Figure 26 shows a layout algorithm for a 3D model of a bed, representing a more complex case where the surrounding environment needs to be considered. Figure 26(A) shows the room structure estimated by the position and structure estimation unit 14 of the imaging device 70, Figure 26(B) shows the placeable area estimated by the area estimation unit 17, and Figure 26(C) shows the state with the bed placed in the placeable area. It also shows the state with a rug placed in the center of the room.

[0105] The area estimation unit 17 estimates the area in which the target furniture can be placed based on the basic rules for furniture placement shown in the furniture setting data acquired in step S62. Rules for bed placement include, for example, placing the bed on the floor (not in mid-air), placing it along a wall, and not placing it in front of a passageway or door (it may be placed in front of a window). Based on these rules, the area estimation unit 17 estimates the area in which the bed can be placed, as shown in Figure 26(B), based on the room structure estimated by the structure estimation unit 14 and the position of the subject detected by the detection unit 15.

[0106] Furthermore, the rules regarding bed placement include sub-rules such as placing them randomly, placing beds in the corners of the room, and placing them in the middle of the edges of the room. The area estimation unit 17 determines the position of the bed based on the available placement area and the sub-rules. If the area estimation unit 17 cannot place furniture according to the rules shown in the furniture setting data, it cancels the placement of that furniture and proceeds to place other furniture.

[0107] Next, the placement unit 19 of the image processing device 10 determines the placement of the 3D model of the furniture based on the furniture information acquired in step S63 (step S64). 3D models of furniture can be in various file formats such as 3ds.max, .blend, .stl, or .fbx, and any format may be used. However, since the installation direction and center position are not usually defined for 3D models of furniture, it is necessary to set initial installation direction and center point rules and edit the 3D model according to the set rules, or to prepare separate data for conversion. The rules for furniture installation direction and center point may be included in the furniture setting data, or they may be set separately as a database when selecting the 3D model.

[0108] Figure 27 shows an example of a 3D model of furniture. In the 3D model of the table shown in Figure 27, the front is defined as the coordinates (0,-1,0) in virtual space, and the direction a person faces is aligned with the front. Also, in the 3D model of the table shown in Figure 27, the center of the surface facing the floor is defined as the center of the furniture. The center of the furniture is determined based on the surface of the furniture that touches the ground. For example, the center of a light fixture hanging from the ceiling would be the point where it touches the ceiling.

[0109] Figure 28 shows an example of the layout result of a 3D furniture model. The placement unit 19 calculates the coordinates and orientation of the furniture placement position based on the placement area estimated in step S62 and the placement rules shown in the furniture setting data. This allows the placement unit 19 to determine the placement of the furniture determined in step S61. Here, the layout information indicating the furniture placement determined by the placement unit 19 includes information on the type of furniture, as well as the orientation, position, and size of the furniture.

[0110] Then, the placement unit 19 terminates the process if the placement of all the furniture determined in step S71 is complete (YES in step S75). On the other hand, if the placement of all the furniture determined in step S71 is not complete (NO in step S75), the placement unit 19 repeats the process in step S74 until the placement of all the furniture is complete. Note that by placing the furniture in order from largest to smallest, more furniture can be placed.

[0111] In this way, the image processing device 10 can automatically place 3D models of furniture suitable for the purpose of the room captured in the image. Furthermore, by placing the determined 3D models of furniture in the placeable area estimated based on the estimated room structure and the position of the detected subjects, the image processing device 10 can achieve a more natural furniture arrangement for the viewer.

[0112] Returning to Figure 16, the image processing unit 20 of the image processing device 10 performs image synthesis processing between the captured image input in step S1 and the 3D model of the furniture placed in step S6 (step S7). Specifically, the image processing unit 20 performs rendering using the captured image input in step S1, the room structure estimated in step S3, and the furniture layout information in step S6. For rendering, a CG tool such as 3dsMax, Blender, Maya, various CAD tools, Unity, or a web browser is used, preferably a tool equipped with a function that allows the operation of the CG tool to be controlled by scripting. Furthermore, rendering is preferably done in equirectangular projection format, but rendering may also be done using a projection method that converts part of it using perspective projection or other projection methods. In addition, rendering may be done using either rasterization or ray tracing, but ray tracing is preferred from the viewpoint of improving quality.

[0113] First, the placement unit 19 places the 3D furniture models on the CG tool based on the layout result in step S6. Figure 29 shows an example of the layout result of the 3D furniture models on the 3D space model. The furniture layout information includes, as described above, the type of furniture, as well as the orientation, position, and size of the furniture. The image processing unit 20 places the 3D furniture models, which are decoded and specified by the script on the CG tool, into the 3D space. Note that the furniture layout information may also include correction information for the 3D furniture models, such as the color or texture of the furniture. The example in Figure 29 shows a bed, rug, desk, and potted plants placed in the 3D space.

[0114] Furthermore, the placement unit 19 may represent all or part of the room structure estimated in step S3 on the CG tool. The room structure, i.e., the floor, ceiling, and walls, may be represented using textures or transparently. The example in Figure 29 shows all but the front wall and ceiling. If transparent representation is used, the image processing unit 20 can set the transparent surface to function as a shadow catcher and render shadows to enhance the texture of the CG. The image processing unit 20 then performs a process to composite the 3D model placed by the placement unit 19 with the captured image input in step S1.

[0115] Figure 30 shows an example of a processed image using the shadow catcher. As shown in Figure 30, the image processing unit 20 can use the shadow catcher function to cast shadows on empty areas or on subjects in the captured image. In this way, the image processing unit 20 can maintain consistent shadow quality as a single rendered image by performing rendering processing.

[0116] Figure 31 shows an example of image processing using image-based lighting. As shown in Figure 31(B), the image processing unit 20 can, for example, set the captured image as the data background for image-based lighting, thereby expressing more natural light rays and improving texture compared to the normal lighting process shown in Figure 30(A).

[0117] Then, the storage / reading unit 29 stores the processed image data synthesized in step S7 in the image data management DB 1001 (see Figure 14) (step S8). In this case, the storage / reading unit 29 stores the processed image data synthesized in step S7 in the image data management DB 1001 in association with the captured image data before the synthesis process and the condition ID that identifies the furniture selection conditions.

[0118] In this way, the image processing device 10 can estimate the structure of the room and the position of the subject using the input captured image, and by placing 3D models of furniture in the placement area according to the estimation result, it can achieve a more natural placement of virtual objects.

[0119] Figure 32 shows an example of spatial estimation results and subject detection results in a processed image into which virtual objects have been synthesized. In the example in Figure 32, the dotted lines represent the spatial estimation results, which represent the structure of the room, and the thick borders represent the subject detection results. As shown in Figure 32, the image processing device 10 can place furniture in the configurable area based on the spatial estimation results and subject detection results.

[0120] Examples of image synthesis applications Next, using Figures 33 to 36, we will explain examples of the application of the image synthesis process in the image processing device 10 described above. First, using Figures 33 and 34, we will explain the process of estimating areas where the placement of virtual objects such as furniture is prohibited.

[0121] Figure 33 shows an example where the virtual object is too close to the imaging device. As shown in Figure 33, when the virtual object is placed in the placement area based on the room structure and the subject detection results as described above, the virtual object may be too close to the imaging device 70, resulting in an unsightly appearance. Therefore, as shown in Figures 34(A) and (B), the image processing device 10 can improve the appearance of the processed image after image synthesis by estimating the area around the imaging device 70 as a placement prohibited area where the virtual object cannot be placed.

[0122] In this case, in step S4 described above, the detection unit 15 detects the position of the imaging device 70. In step S63, the area estimation unit 17 estimates the area around the imaging device 70 detected by the detection unit 15 as a no-placement area. Then, the area estimation unit 17 estimates the area where the virtual object can be placed, taking into account the estimated no-placement area along with the structure of the room estimated by the structure estimation unit 14 and the position of the subject estimated by the position estimation unit 16. The no-placement area may be defined in two dimensions or in three dimensions.

[0123] Next, Figures 35 and 36 will be used to explain the process for concealing subjects in captured images. Figures 35 and 36 show examples where the shooting device 70 or support member 7 are visible in the captured image. As shown in Figure 35(A), the shooting device 70 captures the entire surrounding area, so the image captured by the shooting device 70 will include the photographer's hand or a support member 7 such as a tripod or monopod, which is undesirable from the standpoint of making the room look beautiful. Therefore, the image processing device 10 detects the support member 7 using, for example, the detection unit 15, and places a virtual object of an arbitrary size that can conceal the detected support member 7. Then, the image processing unit 20 of the image processing device 10 uses the captured image as the background and combines the image of the placed virtual object to eliminate the visibility of the support member 7, as shown in Figure 35(B).

[0124] Figure 36 is a diagram illustrating an example of the size of the support member. The detection unit 15 detects the support member 7 in step S4. The detection of the support member 7 may be performed as is in the equirectangular projection, or it may be performed after transforming the vertical downward direction with perspective projection. As a result, the detection unit 15 obtains the field of view angle φ of the support member 7. As shown in Figure 36, if the installation height of the imaging device 70 from the floor is h and the width of the support member 7 is w, then the width w of the support member 7 can be expressed as w = 2tan(φ / 2). The placement unit 19 places any virtual object with a size greater than or equal to the width w on the floor, and the image processing unit 20 can prevent the support member 7 from being reflected by combining the image of the placed virtual object with the captured image. Note that the object detected by the detection unit 15 is not limited to the support member 7; the imaging device 70 or the photographer taking pictures with the imaging device 70 may be detected, and a virtual object of some kind may be placed to hide the detected object.

[0125] ○Image display processing○ Next, the image display process in the image processing system 3 will be explained using Figures 37 to 43. Figure 37 is a sequence diagram showing an example of the image display process. Figure 37 shows the process of distributing the processed image data stored in the image processing device 10 by the above-described process to the viewer using the image distribution device 30.

[0126] First, the transmitting / receiving unit 91 of the display device 90 transmits an image display request to the image distribution device 30, indicating that it requests the display of an image, based on the viewer's input operation to the input device, etc. (step S51). This image display request includes an image ID that identifies the image of the structure to be displayed. As a result, the transmitting / receiving unit 31 of the image distribution device 30 receives the image display request transmitted from the display device 90.

[0127] Next, the transmitting / receiving unit 31 of the image distribution device 30 transmits an image acquisition request to the image processing device 10 indicating that it is requesting the acquisition of image data to be distributed to the display device 90 (step S52). This image acquisition request includes the image ID received in step S51. As a result, the transmitting / receiving unit 11 of the image processing device 10 receives the image acquisition request transmitted from the image distribution device 30.

[0128] Next, the storage / reading unit 29 of the image processing device 10 searches the image data management DB 1001 (see Figure 14) using the image ID received in step S52 as a search key, and reads the captured image data and processed image data associated with the same image ID as the received image ID (step S54). The transmitting / receiving unit 11 transmits the captured image data and processed image data read in step S54 to the image distribution device 30. As a result, the transmitting / receiving unit 31 of the image distribution device 30 receives the captured image data and processed image data transmitted from the image processing device 10.

[0129] The display control unit 32 then transmits (distributes) the received captured image data or processed image data to the display device 90 via the transmitting / receiving unit 31, causing the display device 90 to display the captured image or processed image (step S55). The display control unit 93 of the display device 90 then displays the captured image or processed image corresponding to the data transmitted (distributed) from the image distribution device 30 on the display 906 (step S56).

[0130] Figure 38(A) shows an example of a captured image displayed on a display device, and Figure 38(B) shows an example of a processed image displayed on the display device. The captured image 400 shown in Figure 38(A) is an image showing the state of the room before furniture was placed. On the other hand, the processed image 600 shown in Figure 38(B) is an image of the captured image 400 shown in Figure 38(A) after furniture has been placed.

[0131] Furthermore, the reception unit 92 of the display device 90 accepts the selection of whether or not furniture is placed in response to a predetermined input operation using the input means of the display device 90 (step S57). This allows the viewer to select whether or not furniture is placed in the image displayed on the display device 90, and to view the appearance of the room before and after the furniture is placed by switching the image.

[0132] In this way, the image processing system 3 can convey a more concrete image of the room to the viewer by displaying the processed image, which is a composite of 3D models of furniture, on the display device 90.

[0133] ○ Display of additional information Next, using Figures 39 to 41, we will explain the process of overlaying additional information corresponding to the placed furniture onto the processed image 600 shown in Figure 38(B). When the image distribution device 30 displays the processed image 600 shown on the display device 90, it can display additional information such as a warning icon, a link to a website selling the furniture, or a description of the furniture on the rendered image with the furniture model placed on it.

[0134] Figure 39 is a conceptual diagram showing an example of an additional information management table. As shown in Figure 13, the storage unit 3000 has an additional information management DB 3001 constructed, which is composed of the additional information management table shown in Figure 39. This additional information management table manages the association of an additional ID that identifies the additional information, the type of furniture, coordinate information indicating the placement location of the additional information, and a link to a website for each image ID that identifies the image data. Of these, the coordinate location is calculated by the calculation unit 35 based on the coordinate location of the 3D model of the furniture corresponding to the additional information. The link to the website is included, for example, in the furniture information transmitted from the communication terminal 80 mentioned above.

[0135] Figure 40 is a diagram illustrating an example of the placement of additional information. When additional information such as explanatory text, icons, or links is superimposed on a captured image, it is necessary to accurately superimpose the additional information on the position of the furniture. Therefore, as shown in Figure 40, the coordinate detection unit 34 detects the coordinates of the furniture shown in the processed image. The calculation unit 35 calculates the coordinates of the center position D of the furniture and calculates the direction pointing to the center position D calculated from the shooting device 70. Then, the image processing unit 36 ​​superimposes the additional information corresponding to the furniture onto the coordinate position on the processed image indicating the direction calculated by the calculation unit 35.

[0136] The additional information to be superimposed is, for example, an icon to draw the viewer's attention, a description of furniture, or an image to accept access to a website link. The image distribution device 30, for example, when it receives processed image data from the transmitting / receiving unit 31 in step S54 shown in Figure 36, performs the superimposition process of the above-mentioned additional information, and in step S55, displays the processed image with the superimposed additional information on the display device 90.

[0137] Figure 41 shows an example of a processed image screen with superimposed additional information. The processed image 600a shown in Figure 41(A) displays an image 710 as additional information, which allows access to a website corresponding to the furniture shown in the processed image 600a. Image 710 includes, for example, a link to a website such as an e-commerce (EC) site where the furniture shown in the processed image 600a can be purchased. A viewer viewing the processed image 600a displayed on the display device 90 can access the corresponding EC site page by, for example, pressing image 710.

[0138] The processed image 600a shown in Figure 41(B) displays an icon 730 as additional information, indicating that the furniture shown in processed image 600a is a composite image. When images of virtual objects are composited using ray tracing or image-based lighting technology, it can be difficult for viewers to determine which parts of the processed image are computer graphics. Therefore, processed image 600b displays a warning icon 730 on top of the composite image of furniture, allowing viewers to clearly distinguish between objects that are actually in the room and the composited objects.

[0139] Furthermore, the icon 730 does not have to be displayed at all times; it may be hidden after a certain period of time, or its display may be toggled on and off by the viewer's input. In addition, the icon 730 may have effects such as flashing to draw the viewer's attention. Moreover, the processed image 600b may be configured to display a description of the furniture when the viewer selects the icon 730.

[0140] ○ Examples of image display applications Next, using Figures 42 and 43, we will explain an example of the application of the processed image displayed on the display device 90. The processed image 600c shown in Figure 42 shows an image of placed furniture with a border. The image processing unit 36 ​​of the image distribution device 30, for example, when it receives processed image data from the transmitting / receiving unit 31 in step S54 shown in Figure 36, generates a composite image that borders the image of the furniture, and displays the generated processed image 600c on the display device 90. As a result, viewers of the processed image 600c can clearly identify the location of the composited virtual object (furniture).

[0141] Furthermore, the processed image 600d shown in Figure 43 shows the state after the color tones of the placed furniture have been changed. The image processing unit 36 ​​of the image distribution device 30, for example, when it receives processed image data from the transmitting / receiving unit 31 in step S54 shown in Figure 36, performs a process to change the color tones of the furniture image and displays the processed image 600d on the display device 90. As a result, viewers of the processed image 600c can clearly identify the locations of the synthesized virtual objects (furniture) by looking at an image that has been deliberately altered in color to make it appear unnatural.

[0142] ● Effects of the embodiment As described above, the image display system 1 estimates the structure of the room and the position of the subject using the captured image taken by the shooting device 70, and by placing 3D models of furniture in the placement area according to the estimation result, it can achieve a more natural placement of virtual objects.

[0143] Furthermore, the image display system 1 displays the processed image, in which the 3D model of the furniture has been synthesized by the image processing system 3, on the display device 90. This allows viewers to see not only the empty room but also the room with furniture in place, thus conveying a more concrete image of the room to the viewer.

[0144] ●Summary● As described above, the image processing method according to one embodiment of the present invention is an image processing method executed by an image processing system 3, which includes: a structure estimation step of estimating the structure of the space from a background image (e.g., a 360-degree image) showing the space inside a structure (e.g., a room) in all directions; a region estimation step of estimating a region in the space where a virtual object (e.g., a 3D model of furniture) can be placed based on the estimated structure; and an image processing step of compositing the virtual object onto the estimated region in the background image. As a result, the image processing method can automatically place the virtual object in an appropriate position in the space inside the structure.

[0145] Furthermore, an image processing method according to one embodiment of the present invention further includes a detection step of detecting an object shown in a background image (e.g., a 360-degree image), a position estimation step of estimating the spatial position of the detected object, and a region estimation step of estimating a region based on the estimated spatial structure and the estimated position of the object. As a result, the image processing method can estimate the state of space by estimating the spatial structure shown in the background image and detecting an object. In addition, by performing object detection based on the estimated spatial structure, the image processing method can estimate locations on the spatial structure where an object may be installed, thereby improving the efficiency of processing.

[0146] Furthermore, in an image processing method according to one embodiment of the present invention, the image processing system 3 includes a condition information management DB 1002 (an example of a storage means) that stores condition information indicating the placement conditions of virtual objects (e.g., 3D models of furniture). The image processing method executed by the image processing system 3 then performs a decision step of selecting virtual objects according to the purpose of the space inside the structure (e.g., a room) from the stored condition information. As a result, the image processing method can automatically place virtual objects suitable for the purpose according to the purpose of the space shown in the background image.

[0147] Furthermore, an image processing system according to one embodiment of the present invention includes a structure estimation unit 14 (an example of structure estimation means) that estimates the structure of a space from a background image (e.g., a 360-degree image) showing the interior space of a structure (e.g., a room) in all directions; a region estimation unit 17 (an example of region estimation means) that estimates a region in the space where a virtual object (e.g., a 3D model of furniture) can be placed based on the estimated structure; and an image processing unit 20 (an example of image processing means) that composites the virtual object into the estimated region on the background image. As a result, the image processing system 3 can automatically place the virtual object in an appropriate position in the interior space of the structure.

[0148] Furthermore, an image processing system according to one embodiment of the present invention includes a display control unit 32 (an example of display control means) that displays the processed image synthesized by the image processing unit 20 (an example of image processing means) on the display device 90. As a result, the image processing system 3 can allow the viewer to view the appearance of the space before and after the virtual object is placed by switching the images.

[0149] ●Additional Information● Each function of the embodiment described above can be realized by one or more processing circuits. Here, "processing circuit" in this embodiment includes processors programmed to execute each function by software, such as processors implemented by electronic circuits, as well as devices such as ASICs (Application Specific Integrated Circuits), DSPs (digital signal processors), FPGAs (field programmable gate arrays), SOCs (System on a chip), GPUs (Graphics Processing Units), and conventional circuit modules designed to execute each of the functions described above.

[0150] Furthermore, the various tables in the embodiments described above may be generated by the learning effect of machine learning, and tables may not be used if the data of each related item is classified by machine learning. Here, machine learning is a technique for enabling computers to acquire human-like learning abilities, and it refers to a technique in which a computer autonomously generates algorithms necessary for judgments such as data identification from pre-incorporated training data, and applies these to new data to make predictions. The learning method for machine learning may be supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, or deep learning, or a combination of these learning methods may be used, and the learning method for machine learning is not limited.

[0151] Although an image processing method, program, and image processing system according to one embodiment of the present invention have been described so far, the present invention is not limited to the embodiments described above, and can be modified to the extent that a person skilled in the art can imagine, such as by adding, changing, or deleting other embodiments, and any embodiment that achieves the function and effect of the present invention is included within the scope of the present invention. [Explanation of Symbols]

[0152] 1. Image display system 3 Image Processing System 5. Communication Network 7. Support Member 10 Image Processing Device 11 Transmitter / Receiver 14. Structural estimation unit (an example of structural estimation means) 15 Detection unit 16 Position estimation part 17. Region Estimation Unit (An Example of Region Estimation Means) 18. Decision Section 19 Placement section 20 Image Processing Unit (An example of image processing means) 30 Image distribution device 31 Transmitter / Receiver 32 Display Control Unit (Example of Display Control Means) 35 Calculation Unit (Example of Calculation Means) 70 Imaging device 80 Communication terminals 90 Display device 1002 Condition Information Management Database (Example of a Storage Method) [Prior art documents] [Patent Documents]

[0153] [Patent Document 1] Patent No. 6570161 [Patent Document 2] Patent No. 6116746 [Patent Document 3] Patent No. 3720587

Claims

1. An image processing method performed by an image processing system, A structural estimation step in which the structure and size of a room are estimated from a 360-degree spherical image showing the room in all directions of the property, A detection step that detects structural objects in the room that appear in the aforementioned 360-degree image, and further detects the type of structural object in the room, A position estimation step for estimating the position of a structural object in the room that has been detected in the room, Execute, Furthermore, The structural estimation step estimates the use of the room as shown in all directions on the 360-degree image, based on the estimated structure and size of the room and the types of structural objects detected in the room. A decision step to determine the furniture to be composited into the 360-degree image based on the estimated use, A region estimation step for estimating the area in the room where the determined furniture can be placed, based on the estimated structure and size of the room, the estimated position of structural objects in the room, and the determined furniture placement rules for the structure of the room and the detected types of structural objects in the room; Image processing step of compositing the determined 3D model of the furniture onto the estimated region of the panoramic image, An image processing method that performs this task.

2. The image processing system includes a storage means for storing conditional information relating the use and size of a room to furniture, The image processing method according to claim 1, wherein the decision step involves selecting furniture according to the estimated use of the room from the stored condition information.

3. The detection step involves detecting a support member that supports the imaging device for photographing the room, The image processing method according to claim 1 or 2, wherein the image processing step involves compositing a predetermined image onto the 360-degree image so as to conceal the detected support member.

4. On the computer, A structural estimation step in which the structure and size of a room are estimated from a 360-degree spherical image showing the room in all directions of the property, A detection step that detects structural objects in the room that appear in the aforementioned 360-degree image, and further detects the type of structural object in the room, A position estimation step for estimating the position of a structural object in the room that has been detected in the room, Execute, Furthermore, The structural estimation step estimates the use of the room as shown in all directions on the 360-degree image, based on the estimated structure and size of the room and the types of structural objects detected in the room. A decision step to determine the furniture to be composited into the 360-degree image based on the estimated use, A region estimation step for estimating the area in the room where the determined furniture can be placed, based on the estimated structure and size of the room, the estimated position of structural objects in the room, and the determined furniture placement rules for the structure of the room and the detected types of structural objects in the room; Image processing step of compositing the determined 3D model of the furniture onto the estimated region of the panoramic image, A program that executes the command.

5. A structural estimation means for estimating the structure and size of a room from a 360-degree image showing the room in all directions, A detection means for detecting structural objects in the room that appear in the aforementioned 360-degree image, and further for detecting the type of structural object in the room, A position estimation means for estimating the position of a structural object in the room that has been detected, Equipped with, The structure estimation means estimates the use of the room shown in all directions on the 360-degree image based on the estimated structure and size of the room and the types of structural objects detected in the room. Furthermore, A determination means for determining the furniture to be composited onto the 360-degree image based on the estimated use, An area estimation means for estimating an area in a room where the determined furniture can be placed, based on the estimated structure and size of the room, the estimated position of structural objects in the room, and the determined furniture placement rules for the structure of the room and the detected types of structural objects in the room; Image processing means for compositing the determined 3D model of the furniture onto the estimated region of the aforementioned 3D spherical image, An image processing system equipped with the following features.

6. Furthermore, The image processing system according to claim 5, further comprising a display control means for displaying the processed image synthesized by the image processing means on a display device.

7. The image processing system according to claim 6, wherein the display control means switches the display of the 360-degree image and the processed image.

8. The system includes a calculation means for calculating the center position of the 3D model of the furniture on the displayed processed image, and a direction indicating the calculated center position. The display control means overlays additional information onto the calculated center position of the coordinate position of the processed image indicating the calculated direction and displays it. The image processing system according to claim 6 or 7.

9. The image processing system according to claim 8, wherein the additional information is an icon corresponding to the furniture or a link to a website.

10. The image processing system according to any one of claims 6 to 9, wherein the display control means causes the color of the 3D model of the furniture shown in the processed image to be changed or an image in which the 3D model of the furniture is outlined.

Images