Image processing method, program, and image processing system

Abstract

To provide an image processing method, a program, and an image processing system that makes a virtual object automatic placed in an inner space of a structure.SOLUTION: An image processing method executed by an image processing system executes a structure estimation step of estimating a structure of a space from a background image (e.g. an entire celestial sphere image) where an inner space of s structure (e.g. a room) is omnidirectional shown, an area estimation step of estimating an area at which a virtual object (e.g. a 3D model of a furniture) in the space can be placed, and an image processing step of synthesizing the virtual object into the estimated area for the background image.SELECTED DRAWING: Figure 16

Description

Technical Field

[0001] The present disclosure relates to an image processing method, a program, and an image processing system.

Background Art

[0002] There is known a system that distributes image data captured using a photographing device capable of photographing in all directions, and allows the situation of a remote site to be viewed at another site. Since a panoramic image obtained by photographing a predetermined site in all directions can allow a viewer to view in an arbitrary direction, it is possible to convey information with a sense of presence. Such a system is used, for example, in the field of online interior view of properties in the real estate industry.

[0003] Such a system is used, for example, in the field of online interior view of properties in the real estate industry. In addition, there is a service called "home staging" in which the space is decorated by arranging furniture and accessories in the property to give the viewer an attractive image of living and to smoothly promote transactions. In this service, in order to reduce costs such as fees or time, or the risk of damage to the property, instead of arranging actual furniture in the property, a service that synthesizes three-dimensional model (CG) furniture with the image of the property that has been photographed is already known (for example, Patent Documents 1 to 3).

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the conventional method, when synthesizing an image of a virtual object such as furniture with a photographed image, the virtual object may be arranged at an unnatural position for a viewer who views the image, and there is room for improvement from the viewpoint of the accuracy of automatic arrangement of the virtual object.

Means for Solving the Problems

[0005] In order to solve the above-described problems, the invention according to claim 1 is an image processing method executed by an image processing system, including a structure estimation step of estimating the structure of the space from a background image in which the space inside the structure is shown in all directions, and a subject shown in the background image and the type of the subject a detection step of detecting, a position estimation step of estimating the position of the detected subject in the space, and the estimated structure 、 the estimated position of the subject in the space , and the rule of arrangement for each type of virtual object in the space with respect to the structure and the type of the subject based on, a region estimation step of estimating a region where a virtual object can be placed, and an image processing step of synthesizing the virtual object into the estimated region with respect to the background image. Temporary note It is an image processing method that executes

Effect of the Invention

[0006] According to the present invention, there is an effect that a virtual object can be automatically placed at an appropriate position in the space inside the structure.

Brief Description of the Drawings

[0007]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Figure 29

Figure 30

Figure 31

Figure 32

Figure 33

Figure 34

Figure 35

Figure 36

Figure 37

Figure 38

Figure 39

Figure 40

Figure 41

Figure 42

Figure 43

Embodiments for Carrying Out the Invention

[0008] Hereinafter, embodiments for carrying out the invention will be described with reference to the drawings. In the description of the drawings, the same reference numerals are given to the same elements, and redundant descriptions are omitted.

[0009] ●Embodiment● ●Outline of the Image Display System First, with reference to FIG. 1, an outline of the configuration of the image display system according to the embodiment will be described. FIG. 1 is a diagram showing an example of the overall configuration of the image display system. The image display system 1 shown in FIG. 1 is a system that allows a viewer to view a real estate property online by displaying an image of the interior space of a structure such as a real estate property on a display device 90.

[0010] As shown in FIG. 1, the image display system 1 includes an image processing device 10, an image distribution device 30, a photographing device 70, a communication terminal 80, and a display device 90. The image processing device 10, the image distribution device 30, the photographing device 70, the communication terminal 80, and the display device 90 that constitute the image display system 1 can communicate via a communication network 5. The communication network 5 is constructed by the Internet, a mobile communication network, a LAN (Local Area Network), etc. Note that the communication network 5 may include not only wired communication but also a network by wireless communication such as 3G (3rd Generation), 4G (4th Generation), 5G (5th Generation), Wi-Fi (Wireless Fidelity) (registered trademark), WiMAX (Worldwide Interoperability for Microwave Access), or LTE (Long Term Evolution).

[0011] The image processing device 10 is a server computer that performs image processing on a captured image of the interior space of a structure such as a real estate property. The image processing device 10 synthesizes a virtual object with the captured image based on, for example, the captured image data transmitted from the imaging device 70, the usage information indicating the usage of the space captured by the imaging device 70, and the furniture information transmitted from the communication terminal 80. Here, the furniture information includes data indicating a 3D model of the furniture, furniture setting data indicating rules regarding the arrangement of the furniture, and the like. The 3D model of the furniture is an example of a virtual object, and the furniture information is an example of the object information. The virtual object may be, in addition to the 3D model of the home appliance, for example, a 3D model of an electrical product, a decoration, a painting, lighting, equipment, or a fixture.

[0012] The image distribution device 30 is a server computer that distributes the processed image data processed by the image processing device 10.

[0013] Here, the image processing device 10 and the image distribution device 30 are referred to as the image processing system 3. Note that the image processing system 3 may be a computer that aggregates all or part of the functions of the image processing device 10 and the image distribution device 30, for example. Also, each of the image processing device 10 and the image distribution device 30 may be configured such that each function is realized by being distributed among a plurality of computers. Furthermore, although the image processing device 10 and the image distribution device 30 are described as server computers existing in a cloud environment, they may be servers existing in an on-premises environment.

[0014] The imaging device 70 is a special digital camera (omnidirectional imaging device) that can capture the interior space of a structure such as a real estate property in all directions to obtain a full-sphere (360°) image. The imaging device 70 is used, for example, by a real estate agent who manages or sells real estate properties. Note that the imaging device 70 may be a wide-angle camera or a stereo camera that can obtain a wide-angle image having an angle of view equal to or greater than a predetermined value. A wide-angle image is generally an image captured using a wide-angle lens, and is an image captured using a lens that can capture a range wider than that perceived by the human eye. That is, the imaging device 70 is an imaging means that can obtain an image (omnidirectional image, wide-angle image) captured using a lens having a focal length shorter than a predetermined value. A wide-angle image generally means an image captured using a lens having a focal length of 35 mm or less in terms of 35 mm film. Furthermore, the captured image obtained by the imaging device 70 may be a moving image, a still image, or both a moving image and a still image. The captured image may also include sound together with the image.

[0015] The communication terminal 80 is a computer such as a notebook PC that provides information on virtual objects to be arranged in the space shown in the captured image to the image processing device 10. The communication terminal 80 is used, for example, by a furniture manufacturer or the like that manufactures or sells furniture to be arranged.

[0016] The display device 90 is a computer such as a smartphone used by a viewer of the image. The display device 90 displays the image distributed from the image distribution device 30. Note that the display device 90 is not limited to a smartphone, and may be, for example, a PC, a tablet terminal, a wearable terminal, an HMD (head mount display), a PJ (Projector), or an IWB (Interactive White Board: a whiteboard having an electronic blackboard function capable of mutual communication).

[0017] Here, with reference to FIGS. 2 and 3, the image displayed on the display device 90 in the image display system 1 will be described. FIG. 2 is a diagram showing an example of a full-sphere image before arranging virtual objects. The image shown in FIG. 2 is a full-sphere image of a room of a real estate property, which is an example of the interior space of a structure, taken by the imaging device 70. Since the full-sphere image can capture the interior of the room in all directions, it is suitable for viewing real estate properties. The form of the full-sphere image varies, but it is often generated by the Equirectangular projection method described later. The image generated by this Equirectangular projection method has the advantages that the outer shape of the image is rectangular, making it efficient and easy to store image data, and there is less distortion near the equator and vertical lines are not distorted, so it looks relatively natural.

[0018] FIG. 3 is a diagram showing an example of a processed image with virtual objects arranged. The image shown in FIG. 3 shows a state where furniture is arranged in the room shown in the image of FIG. 2. The image of FIG. 3 has the full-sphere image shown in FIG. 2 as the background image, and a 3D model of furniture, which is an example of a virtual object, is synthesized on this background image. The image processing device 10 arranges the 3D model of the furniture in a natural state based on the structure of the floor, walls, ceiling, etc. of the room photographed by the imaging device 70. As shown in FIG. 3, a desk, a bed, etc. are arranged along the walls in the room, and the passage for daily use is not blocked by furniture.

[0019] Conventionally, in order to arrange a 3D model of furniture on a full-sphere image of a room of a real estate property, since the furniture needs to be arranged in a natural position as seen from the imaging position of the imaging device, manual operations by the user were required to adjust the arrangement position and orientation. Also, although there are methods to automatically arrange the furniture model, in order to grasp the structure of the room where the furniture is to be arranged, it is necessary to input a floor plan of the room or perform input operations by the user, and there was room for improvement from the perspective of improving the accuracy of automatic arrangement of virtual objects without much effort.

[0020] Therefore, the image processing system 3 detects the structure of the room and the objects provided in the room using the omnidirectional image of the interior of the room, and estimates the area where the virtual object can be placed. Then, the image processing system 3 places the virtual object in the estimated placement area, and generates a processed image as shown in FIG. 3 in which the placed virtual object and the omnidirectional image are combined. Thereby, the image processing system 3 can naturally arrange the furniture based on the rough state of the room estimated based on the omnidirectional image.

[0021] Here, the room, which is a real estate property, is an example of the space inside a structure. The structure is, for example, a building such as a house, an office, or a store. The omnidirectional image is a captured image captured by the imaging device 70, and is an example of a background image showing the space inside the structure in all directions.

[0022] ○ Method for generating omnidirectional image ○ Here, with reference to FIGS. 4 to 10, the method for generating the omnidirectional image will be described. First, with reference to FIGS. 4 and 5, an overview of the process from the image captured by the imaging device 70 to the generation of the omnidirectional image will be described. FIG. 4(A) shows a hemispherical image (front side) captured by the imaging device, FIG. 4(B) shows a hemispherical image (rear side) captured by the imaging device, and FIG. 4(C) is a diagram showing an image represented by the orthographic cylindrical projection method (hereinafter referred to as an "orthographic cylindrical projection image"). FIG. 5(A) is a conceptual diagram showing a state in which a sphere covers the orthographic cylindrical projection image, and FIG. 5(B) is a diagram showing the omnidirectional image.

[0023] The imaging device 70 is provided with image sensors on the front side (front) and the back side (rear), respectively. These image sensors (image sensors) are used in combination with optical members such as lenses capable of capturing hemispherical images (angle of view of 180° or more). The imaging device 70 can obtain two hemispherical images by imaging the objects around the user with the two image sensors, respectively.

[0024] As shown in FIGS. 4(A) and 4(B), the images obtained by the imaging device 70 of the imaging element are curved hemispherical images (front side and rear side). Then, the imaging device 70 synthesizes the hemispherical image (front side) and the hemispherical image (rear side) inverted by 180 degrees to create an orthographic cylindrical projection image EC as shown in FIG. 4(C).

[0025] Then, the imaging device 70 uses OpenGL ES (Open Graphics Library for Embedded Systems) to paste the orthographic cylindrical projection image EC so as to cover the spherical surface as shown in FIG. 5(A), and creates an omnidirectional image (omnidirectional panorama image) CE as shown in FIG. 5(B). In this way, the omnidirectional image CE is represented as an image in which the orthographic cylindrical projection image EC faces the center of the sphere. Note that OpenGL ES is a graphics library used to visualize 2D (2-Dimensions) and 3D (3-Dimensions) data. Also, the omnidirectional image CE may be a still image or a moving image. Furthermore, the conversion method is not limited to OpenGL ES, and any method that can convert from a hemispherical image to an orthographic cylindrical projection method may be used. For example, it may be an operation by a CPU or an operation by OpenCL.

[0026] As described above, since the omnidirectional image CE is an image pasted so as to cover the spherical surface, it will feel uncomfortable to humans. Therefore, the imaging device 70 can display a part of a predetermined region T (hereinafter referred to as a "predetermined region image") of the omnidirectional image CE as a planar image with less curvature, so as not to give an uncomfortable feeling to humans. This will be described with reference to FIGS. 6 and 7.

[0027] FIG. 6 is a diagram showing the position of a virtual camera and a predetermined region when the entire sky image is a three-dimensional spherical object. The virtual camera IC corresponds to the position of the viewpoint of the user who views the image with respect to the entire sky image CE displayed as a three-dimensional spherical object. FIG. 6 represents the entire sky image CE as a three-dimensional spherical object CS. When the entire sky image CE generated in this way is the spherical object CS, as shown in FIG. 6, the virtual camera IC is located inside the entire sky image CE. The predetermined region T in the entire sky image CE is the shooting region of the virtual camera IC, and is specified by predetermined region information indicating the shooting direction and the angle of view of the virtual camera IC in the three-dimensional virtual space including the entire sky image CE. Further, the zoom of the predetermined region T can also be expressed by moving the virtual camera IC closer to or farther from the entire sky image CE. The predetermined region image Q is an image of the predetermined region T in the entire sky image CE. Therefore, the predetermined region T can be specified by the angle of view α and the distance f from the virtual camera IC to the entire sky image CE.

[0028] Then, the predetermined region image Q is displayed on a predetermined display as an image of the shooting region of the virtual camera IC. Hereinafter, the description will be made using the shooting direction (ea, aa) and the angle of view (α) of the virtual camera IC. Note that the predetermined region T may be indicated by the imaging region (X, Y, Z) of the virtual camera IC which is the predetermined region T instead of the angle of view α and the distance f.

[0029] Next, with reference to FIG. 7, the relationship between the predetermined region information and the image of the predetermined region T will be described. FIG. 7 is a diagram showing the relationship between the predetermined region information and the image of the predetermined region T. As shown in FIG. 7, "ea" represents the elevation angle, "aa" represents the azimuth angle, and "α" represents the angle of view (Angle). That is, the posture of the virtual camera IC is changed so that the point of gaze of the virtual camera IC indicated by the shooting direction (ea, aa) becomes the center point CP(x, y) of the predetermined region T which is the shooting region of the virtual camera IC. As shown in FIG. 7, when the diagonal angle of view of the predetermined region T represented by the angle of view α of the virtual camera IC is α, the center point CP(x, y) becomes the parameter ((x, y)) of the predetermined region information. The predetermined region image Q is an image of the predetermined region T in the omnidirectional image CE. f is the distance from the virtual camera IC to the center point CP(x, y). L is the distance between an arbitrary vertex of the predetermined region T and the center point CP(x, y) (2L is the diagonal). And in FIG. 7, generally, the trigonometric function represented by the following (Equation 1) holds.

[0030]

Equation

[0031] Next, with reference to FIG. 8, the state during shooting by the shooting device 70 will be described. FIG. 8 is a diagram showing an example of the state during shooting by the shooting device. In order to shoot so as to overlook the entire room of a real estate property or the like, it is preferable to install the shooting device 70 at a position close to the height of a human eye. Therefore, as shown in FIG. 8, it is common for the shooting device 70 to perform shooting by fixing the shooting device 70 with a support member 7 such as a monopod or a tripod. As described above, the shooting device 70 is an omnidirectional shooting device capable of acquiring light rays in all directions of the entire circumference, and it can also be said that an image (omnidirectional image CE) on the unit sphere around the shooting device 70 is acquired. When the shooting direction of the shooting device 70 is determined, the coordinates of the omnidirectional image are determined. For example, in FIG. 8, point A is at a distance of (d, -h) from the center point C of the shooting device 70. At this time, if the angle formed by the line segment AC and the horizontal direction is θ, the angle θ can be expressed by the following (Equation 2).

[0032]

Number

[0033] And, assuming that point A is at a depression angle θ, the distance d between point A and point B can be expressed by the following (Equation 3) using the installation height h of the imaging device 70.

[0034]

Number

[0035] Here, the process of converting the position information on the omnidirectional image into coordinates on the planar image converted from the omnidirectional image will be briefly described. FIG. 9 is a diagram for explaining an example of an omnidirectional image. Note that FIG. 9(A) is a diagram showing the hemispherical image shown in FIG. 4(A) by connecting with lines the portions where the incident angles in the horizontal and vertical directions with respect to the optical axis are equal. Hereinafter, the incident angle in the horizontal direction with respect to the optical axis is referred to as “θ”, and the incident angle in the vertical direction with respect to the optical axis is referred to as “φ”.

[0036] Further, FIG. 10(A) is a diagram for explaining an example of an image processed by the orthographic cylindrical projection method. Specifically, the image shown in FIG. 9 is associated with a pre-generated LUT (Look Up Table) or the like, processed by the orthographic cylindrical projection method, and the respective images shown in the processed FIG. 9(A) and (B) are synthesized, then the planar image shown in FIG. 10(A) corresponding to the omnidirectional image is generated by the imaging device 70. The orthographic cylindrical projection image EC shown in FIG. 4(C) is an example of the planar image shown in FIG. 10(A).

[0037] As shown in Fig. 10(A), in the image processed by the orthographic cylindrical projection method, the latitude (θ) and longitude (φ) are orthogonal. In the example shown in Fig. 10(A), with the center of the image being (0, 0), and expressing the latitude direction as -90 to +90 and the longitude direction as -180 to +180, any position in the full - sphere image can be indicated. For example, the coordinates of the upper - left corner of the image are (-180, -90). Note that the coordinates of the full - sphere image may be represented in a format using 360 - degree numbers as shown in Fig. 10(A), or may be represented in radian notation or in terms of the number of pixels like a real - world image. Also, the coordinates of the full - sphere image may be converted and represented as two - dimensional coordinates (x, y) as shown in Fig. 10(B).

[0038] Note that the compositing process for the planar image shown in Fig. 10(A) or (B) is not limited to simply continuously arranging the hemisphere images shown in Fig. 9(A) and (B). For example, when the horizontal center of the full - sphere image is not θ = 180°, in the compositing process, the imaging device 70 first pre - processes the hemisphere image shown in Fig. 4(C) and arranges it at the center of the full - sphere image. Next, the imaging device 70 divides the pre - processed image of the hemisphere image shown in Fig. 4(B) into a size that can be arranged in the left - and - right parts of the generated image, and may generate the orthographic cylindrical projection image EC shown in Fig. 4(C) by compositing the hemisphere images.

[0039] Also, in the planar image shown in Fig. 10(A), the locations corresponding to the poles (PL1 or PL2) of the hemisphere images (full - sphere images) shown in Fig. 9(A) and (B) are line segments CT1 or CT2. This is because, as shown in Fig. 5(A) and Fig. 5(B), the full - sphere image (for example, the full - sphere image CE) is created by pasting the planar image (orthographic cylindrical projection image EC) shown in Fig. 10(A) onto a spherical surface by using OpenGL ES.

[0040] ●Hardware Configuration Next, the hardware configuration of each device constituting the image display system according to the embodiment will be described with reference to FIG. 11. Note that the hardware configuration shown in FIG. 11 may have components added or deleted as necessary.

[0041] ○Hardware Configuration of Image Processing Device○ First, the hardware configuration of the image processing device 10 will be described with reference to FIG. 11. FIG. 11 is a diagram showing an example of the hardware configuration of the image processing device. Each hardware configuration of the image processing device 10 is indicated by a reference numeral in the 100s. The image processing device 10 is constructed by a computer and includes, as shown in FIG. 11, a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, a RAM (Random Access Memory) 103, an HD (Hard Disk) 104, an HDD (Hard Disk Drive) controller 105, a display 106, an external device connection I / F (Interface) 108, a network I / F 109, a bus line 110, a keyboard 111, a pointing device 112, a DVD-RW (Digital Versatile Disk Rewritable) drive 114, and a media I / F 116.

[0042] Among these, the CPU 101 controls the operations of the entire image processing apparatus 10. The ROM 102 stores programs used for driving the CPU 101 such as the IPL (Initial Program Loader). The RAM 103 is used as a work area for the CPU 101. The HD 104 stores various data such as programs. The HDD controller 105 controls the reading or writing of various data to and from the HD 104 according to the control of the CPU 101. The display 106 displays various information such as a cursor, menu, window, characters, or images. Note that the display 106 may be a touch panel display having an input means. The external device connection I / F 108 is an interface for connecting various external devices. The external devices in this case are, for example, a USB memory or a printer. The network I / F 109 is an interface for performing data communication using the communication network 5. The bus line 110 is an address bus or a data bus for electrically connecting each component such as the CPU 101 shown in FIG. 11.

[0043] Also, the keyboard 111 is a kind of input means having a plurality of keys for inputting characters, numerical values, various instructions, etc. The pointing device 112 is a kind of input means for selecting or executing various instructions, selecting a processing target, or moving a cursor. Note that the input means may be not only the keyboard 111 and the pointing device 112 but also a touch panel or a voice input device. The DVD-RW drive 114 controls the reading or writing of various data to and from the DVD-RW 113 as an example of a removable recording medium. Note that the removable recording medium is not limited to the DVD-RW and may be a DVD-R or a Blu-ray (registered trademark) Disc (Blu-ray disk). The media I / F 116 controls the reading or writing (storage) of data to and from the recording medium 115 such as a flash memory.

[0044] ○Hardware Configuration of Image Distribution Device○ FIG. 11 is a diagram showing an example of the hardware configuration of the image distribution device. Each hardware configuration of the image distribution device 30 is indicated by a reference numeral in the 300s in parentheses. The image distribution device 30 is constructed by a computer and has the same configuration as the image processing device 10 as shown in FIG. 11, so the description of each hardware configuration is omitted.

[0045] ○Hardware Configuration of the Display Device○ FIG. 11 is a diagram showing an example of the hardware configuration of the display device. Each hardware configuration of the display device 90 is indicated by a reference numeral in the 900s in parentheses. The display device 90 is constructed by a computer and has the same configuration as the image processing device 10 as shown in FIG. 11, so the description of each hardware configuration is omitted.

[0046] Note that each of the above programs may be in an installable format or an executable format file and may be recorded on a computer-readable recording medium and distributed. Examples of the recording medium include CD-R (Compact Disc Recordable), DVD (Digital Versatile Disk), Blu-ray Disc, SD card, USB memory, etc. Also, the recording medium can be provided as a program product, either domestically or abroad. For example, the image processing system 3 realizes the image processing method according to the present invention when the program according to the present invention is executed.

[0047] ●Functional Configuration Subsequently, with reference to FIGS. 12 to 15, the functional configuration of the image display system according to the embodiment will be described. FIGS. 12 and 13 are diagrams showing an example of the functional configuration of the image display system. Note that FIGS. 12 and 13 show those of the devices or terminals shown in FIG. 1 that are related to the processes or operations described later.

[0048] ○Functional Configuration of the Image Processing Device○ First, the functional configuration of the image processing apparatus 10 will be described with reference to FIG. 12. The image processing apparatus 10 includes a transmission / reception unit 11, a reception unit 12, a determination unit 13, a structure estimation unit 14, a detection unit 15, a position estimation unit 16, a region estimation unit 17, a decision unit 18, an arrangement unit 19, an image processing unit 20, an input unit 21, and a storage / reading unit 29. Each of these units is a function or means realized by any of the components shown in FIG. 11 operating according to instructions from the CPU 101 in accordance with the image processing apparatus program expanded onto the RAM 103 from the HD 104. Further, the image processing apparatus 10 has a storage unit 1000 constructed by the ROM 102, the RAM 103, and the HD 104 shown in FIG. 11.

[0049] The transmission / reception unit 11 is mainly realized by the processing of the CPU 101 for the network I / F 109, and performs transmission and reception of various data or information with other devices or terminals via the communication network 5.

[0050] The reception unit 12 is mainly realized by the processing of the CPU 101 for the keyboard 111 or the pointing device 112, and receives various selections or inputs from the user. The determination unit 13 is realized by the processing of the CPU 101 and performs various determinations.

[0051] The structure estimation unit 14 is realized by the processing of the CPU 101, and estimates the structure of the space based on a background image in which the internal space of the structure is shown in all directions.

[0052] The detection unit 15 is realized by the processing of the CPU 101, and detects a subject shown in the background image.

[0053] The position estimation unit 16 is realized by the processing of the CPU 101, and estimates the spatial position of the subject detected by the detection unit 15.

[0054] The region estimation unit 17 is realized by the processing of the CPU 101, and estimates a region in the space where a virtual object can be arranged based on the structure of the space estimated by the structure estimation unit 14.

[0055] The determination unit 18 is realized by the processing of the CPU 101, and determines virtual objects to be arranged in the space based on the use of the space shown in the background image.

[0056] The arrangement unit 19 is realized by the processing of the CPU 101, and arranges virtual objects in the area estimated by the area estimation unit 17. For example, the arrangement unit 19 performs layout of the virtual objects determined by the determination unit 18 on the arrangeable area estimated by the area estimation unit 17.

[0057] The image processing unit 20 is realized by the processing of the CPU 101, and synthesizes virtual objects into the area estimated by the area estimation unit 17 for the background image. For example, the image processing unit 20 performs rendering processing on the arranged virtual objects based on the layout result of the virtual objects by the arrangement unit 19.

[0058] The input unit 21 is mainly realized by the processing of the CPU 101 for the external device connection I / F 108, and receives input of various data or information from external devices.

[0059] The storage / reading unit 29 is mainly realized by the processing of the CPU 101, and stores various data (or information) in the storage unit 1000 and reads out various data (or information) from the storage unit 1000.

[0060] ○ Image Data Management Table FIG. 13 is a conceptual diagram showing an example of the image data management table. In the storage unit 1000, an image data management DB 1001 configured by the image data management table shown in FIG. 13 is constructed. This image data management table manages by associating an image ID for identifying image data, a condition ID for identifying selection conditions of virtual objects, photographed image data, and processed image data.

[0061] ○ Condition Information Management Table FIG. 14 is a conceptual diagram showing an example of a condition information management table. The condition information management table manages condition information indicating the arrangement conditions of virtual objects. In the storage unit 1000, a condition information management DB 1002 configured by the condition information management table shown in FIG. 14 is constructed. This condition information management table manages by associating a condition ID for identifying the selection conditions of virtual objects, the use and size of the room, and information on the style and set of furniture, which is an example of the virtual object to be selected.

[0062] ○Functional Configuration of Image Distribution Device○ Next, the functional configuration of the image distribution device 30 will be described with reference to FIG. 13. The image distribution device 30 includes a transmission / reception unit 31, a display control unit 32, a determination unit 33, a coordinate detection unit 34, a calculation unit 35, an image processing unit 36, and a storage / reading unit 39. Each of these units is a function or means realized by the operation of any of the components shown in FIG. 11 according to instructions from the CPU 301 in accordance with the image distribution device program developed on the RAM 303 from the HD 304. Also, the image distribution device 30 has a storage unit 3000 constructed by the ROM 302, the RAM 303, and the HD 304 shown in FIG. 11.

[0063] The transmission / reception unit 31 is mainly realized by the processing of the CPU 301 for the network I / F 309, and transmits and receives various data or information to and from other devices or terminals via the communication network 5.

[0064] The display control unit 32 is mainly realized by the processing of the CPU 301, and causes the display device 90 to display various images, characters, etc. The display control unit 32 causes the display device 90 to display various screens by, for example, distributing (transmitting) image data to the display device 90 using a web browser or a dedicated application. The various screens displayed on the display device 90 are defined by, for example, HTML (HyperText Markup Language), XHTML (Extensible HyperText Markup Language), CSS (Cascading Style Sheets), or JavaScript (registered trademark), etc. The determination unit 33 is realized by the processing of the CPU 301 and makes various determinations.

[0065] The coordinate detection unit 34 is realized by the processing of the CPU 101 and detects the coordinate position of the virtual object shown in the processed image generated by the image processing apparatus 10. The calculation unit 35 is realized by the processing of the CPU 301 and calculates the center position of the virtual object for superimposing the additional information described later on the processed image based on the coordinate position detected by the coordinate detection unit 34. The image processing unit 36 is realized by the processing of the CPU 301 and performs predetermined image processing on the processed image generated by the image processing apparatus 10.

[0066] The storage / reading unit 39 is mainly realized by the processing of the CPU 301 and stores various data (or information) in the storage unit 3000 and reads out various data (or information) from the storage unit 3000.

[0067] ○Functional Configuration of Display Device○ Next, with reference to FIG. 13, the functional configuration of the display device 90 will be described. The display device 90 includes a transmission / reception unit 91, a reception unit 92, and a display control unit 93. Each of these units is a function or means realized by any of the components shown in FIG. 11 operating in accordance with an instruction from the CPU 901 according to the display device program expanded from the HD 904 onto the RAM 903.

[0068] The transmission / reception unit 91 is mainly realized by the processing of the CPU 901 with respect to the network I / F 909, and transmits and receives various data or information to and from other devices or terminals via the communication network 5.

[0069] The reception unit 92 is mainly realized by the processing of the CPU 901 with respect to the keyboard 911 or the pointing device 912, and receives various selections or inputs from the user.

[0070] The display control unit 93 is mainly realized by the processing of the CPU 901, and causes the display 906 to display various images, characters, etc. The display control unit 93 accesses the image distribution device 30, for example, using a web browser or a dedicated application, and causes the display 906, which is an example of a display means, to display an image corresponding to the data distributed from the image distribution device 30.

[0071] ●Processing or operations of the embodiment ○Image composition processing○ Subsequently, with reference to FIGS. 16 to 43, the processing or operations of the image display system according to the embodiment will be described. First, with reference to FIGS. 16 to 32, the image composition processing in the image processing apparatus 10 will be described. In the following description, as an example of the internal space of a structure, an example of a room in a real estate property is shown, and as an example of a virtual object, an example of furniture arranged in the room is shown. FIG. 16 is a flowchart showing an example of the processing in the image processing apparatus.

[0072] First, the image processing apparatus 10 receives an input of a captured image of a predetermined room which is an example of the internal space of a structure (step S1). Specifically, the transmission / reception unit 11 of the image processing apparatus 10 receives, for example, a captured image of the internal space of a predetermined structure captured by the imaging device 70 from the imaging device 70 via the communication network 5. Note that the image processing apparatus 10 may be configured to receive an input of a captured image to be processed from the imaging device 70 when performing the image synthesis process, or may be configured to store the captured image received in advance from the imaging device 70 in the storage unit 1000 and read out the stored captured image when performing the image synthesis process. Further, the image processing apparatus 10 may be configured to receive an input of a captured image by directly connecting to the imaging device 70 via the external device connection I / F 108. Furthermore, since there may be a case where the imaging device 70 does not have a communication function, for example, the image processing apparatus 10 is not limited to directly receiving an input of a captured image from the imaging device 70, and may be configured to receive an input of a captured image via a predetermined communication device owned by a real estate agent.

[0073] Next, the determination unit 13 determines the suitability of the furniture arrangement for the room shown in the captured image using the captured image input in step S1 (step S2). The room shown in the captured image is preferably, for example, an empty room where furniture has not been placed yet, and a room having a predetermined space for arranging furniture. Therefore, when the determination unit 13 determines that the room shown in the captured image is an external space of a structure such as outdoors, or a space where it is impossible to secure a furniture arrangement space due to being extremely narrow or having objects placed thereon, the determination unit 13 determines that the room shown in the captured image is not suitable for furniture arrangement.

[0074] When the determination unit 13 determines that it is suitable for furniture arrangement (YES in step S2), it causes the process to proceed to step S3. On the other hand, when the determination unit 13 determines that it is not suitable for furniture arrangement (NO in step S2), it causes the process to proceed to step S9. In step S9, the image processing apparatus 10 outputs an error message indicating that it is not suitable for furniture arrangement without performing image synthesis processing. Specifically, the storage / reading unit 29 of the image processing apparatus 10 stores the error message in association with the captured image input in step S1 in the storage unit 1000. Thereby, a viewer who views the corresponding captured image can grasp the error message together with the captured image. Note that the image processing apparatus 10 may be configured to execute the processing after step S3 after outputting the error message. However, in this case, there is no furniture that can be arranged in the image synthesis processing in step S7 described later, and the processed image data stored in step S8 described later is likely to be an image in which no furniture is arranged.

[0075] Next, the structure estimation unit 14 estimates the structure of the room shown in the captured image using the captured image input in step S1 (step S3). As a method for estimating the structure of the room, for example, a method is known in which a straight line of a subject shown in the captured image is detected by image processing, a vanishing point of the detected straight line is obtained, and the structure of the room is estimated from the boundary such as the floor, wall, or ceiling. When using an omnidirectional image, since all the ceiling, floor, and walls, which are elements necessary for estimating the structure of the room, are captured, compared with the case of using a normal planar image in which only a part of the room is shown and it is difficult to perform estimation other than the detection of the vanishing point, there is an advantage that the accuracy of structure estimation is increased. Also, a method using machine learning is known for detecting the vanishing point, detecting the boundary between the floor and the wall or between the ceiling and the wall, or estimating the three-dimensional structure from the detection results. The structure estimation unit 14 may perform structure estimation using any known method.

[0076] ○ Structure estimation process Here, an example of the structure estimation process in the image processing apparatus 10 will be described in detail with reference to FIGS. 17 to 20. FIG. 17 is a flowchart showing an example of the structure estimation process.

[0077] First, the structure estimation unit 14 estimates the vertices of the space shown in the captured image using the captured image (step S31). Specifically, for example, as described above, the structure estimation unit 14 detects the lines of the subject shown in the captured image by performing image processing on the captured image, and estimates the vanishing point calculated from the detected lines as the vertex of the space.

[0078] FIG. 18 is a diagram for explaining an example of the structure estimation result for the captured image. FIG. 18 shows an example of a room structure represented by the orthographic cylindrical projection method. As described above, in the orthographic cylindrical projection method, vertical lines are projected as straight lines and horizontal lines are projected as curves. When these are applied to the structure of the room, many rooms have a shape that intersects vertically with each other based on straight lines. Therefore, the structure estimation unit 14 can estimate the approximate structure of the room by using the image represented by the orthographic cylindrical projection method. The structure estimation unit 14 estimates the approximate structure of the room by detecting the elements, lines, and surfaces composed of them that make up the room. The example of FIG. 18 is an example of capturing a rectangular parallelepiped room, and the structure estimation unit 14 estimates that there are four surfaces in the horizontal direction and two surfaces in the vertical and horizontal directions.

[0079] Next, if the structure estimation unit 14 can classify the shape of the room from the vertex estimation result in step S31 (YES in step S32), the process proceeds to step S33. On the other hand, if the structure estimation unit 14 cannot classify the shape of the room (NO in step S32), the process in step S31 is continued. FIG. 19 is a diagram showing an example of the shape of the space structure estimated by the structure estimation process. The shapes of actual rooms are diverse, and in order to obtain detailed three-dimensional information, measurement with a laser scanner, total station, etc. is necessary, but this is a laborious and expensive process. When virtually arranging furniture, detailed restoration is not required, and it is sufficient to narrow down the conditions and simplify the process. That is, since it is only necessary to know the approximate structure of the room, the structure estimation unit 14 uses, for example, the assumption that the room is composed of straight lines and planes, and the straight lines basically intersect at 90° (Manhattan World Assumption) to narrow down the conditions. Furthermore, in order to aim for restoration to the extent that furniture can be arranged, the structure estimation unit 14 classifies, for example, as a rectangular parallelepiped composed of eight vertices or an L-shaped room composed of 12 vertices as shown in FIG. 19.

[0080] Next, the structure estimation unit 14 estimates the size (scale) of the space shown in the captured image (step S33). Specifically, the structure estimation unit 14 obtains the coordinates of each vertex of the room on the orthographic cylindrical projection method by the methods of steps S31 and S32. The structure estimation unit 14 converts the obtained coordinates on the orthographic cylindrical projection method into the coordinates of the three-dimensional space.

[0081] The structure estimation unit 14 checks whether the imaging device 70 is installed vertically or detects the direction of the gravitational acceleration and performs correction. Then, the structure estimation unit 14 can estimate the structure of the room by assuming that the south pole on the orthographic cylindrical projection method (for example, PL1 shown in FIG. 9(A)) coincides with the direction of gravitational acceleration and follows the Manhattan World Assumption.

[0082] The Manhattan World hypothesis is the hypothesis that many man-made artifacts are created parallel to a Cartesian coordinate system, and based on this, it can be assumed that there are constraints parallel to the x, y, and z directions for things such as walls or ceilings. According to such an assumption, as shown in FIG. 20, assuming that the height of the imaging device 70 is at a position h from the floor, the distance between point A at the boundary of the floor and the wall and point B at the boundary of the ceiling and the wall can be expressed using the installation height h of the imaging device 70. On the other hand, by this method, only the approximate shape of the room can be known, and its size (scale) cannot be accurately known. In an extreme example, it is not known whether the room is a 20 cm high miniature or the size of a general 2 m room, so it is necessary to grasp the scale of the room to some extent for arranging furniture.

[0083] As a method for calculating the scale of the room, for example, the structure estimation unit 14 calculates point A located at the depression angle θ using the above (Equation 3) represented in FIG. 8, assuming that the installation height h of the imaging device 70 is known. In addition, as a method for measuring the installation height of the imaging device 70 by physical means, the distance to the optical center of the imaging device 70 may be measured by a technique such as laser ranging. Further, as a method for measuring the installation height of the imaging device 70 by image processing, the distance to the scale may be measured by preparing a scale of a known length on the floor and imaging it with the imaging device 70.

[0084] In addition, the structure estimation unit 14 may estimate the scale of the room with the height of the room being known. In Japan, the Building Standards Act stipulates that the ceiling height should be 210 cm or more, but the ceiling height of a general condominium is 240 cm to 250 cm. In the United States, it is about 8 feet (243 cm), which is similar to the case in Japan. Although there is variation in the height of the room, if it is about ±10 cm, the scale accuracy will match within 5%, and it will function as a rough scale. Furthermore, as a method for measuring the distance to an object by stereo vision, the structure estimation unit 14 may utilize the presence of the parallax of the optical centers of a plurality of lenses included in the imaging device 70 and measure the distance to a predetermined object using the common portion between the lenses. Additionally, as a method for estimating the scale from so-called Structure from motion for estimating the three-dimensional structure from a plurality of images and IMU (Inertial Measurement Unit) data, since the moving distance can be roughly estimated from the IMU data of the imaging device 70, the structure estimation unit 14 may estimate the scale based on that value.

[0085] As a method for calculating the scale of the room in step S33, the structure estimation unit 14 may use any of the above methods.

[0086] Then, based on the structure of the room estimated in steps S31 to S33, the structure estimation unit 14 acquires the coordinate information of each vertex (step S34). As a result of a series of processes, the structure estimation unit 14 acquires the coordinate information of each vertex of n rooms (n = 8 or 12). For example, with the optical center of the imaging device 70 as a reference, the structure estimation unit 14 acquires coordinates Cn (Cn = ((x0, y0, z0), (x1, y1, z1), …(xn, yn, zn))) represented in XYZ coordinates as shown in FIG. 10(B). Note that the structure estimation unit 14 may acquire coordinates in polar coordinate representation as shown in FIG. 10(A).

[0087] In this way, the structure estimation unit 14 can estimate the approximate structure of the room shown in the captured image using the captured image input to the image processing device 10.

[0088] Returning to FIG. 16, the detection unit 15 of the image processing apparatus 10 detects a subject existing in the room shown in the captured image input in step 1 (step S4). Here, if the image processing apparatus 10 only knows the structure of the room, there may be cases where furniture cannot be appropriately arranged. FIG. 21 is a diagram showing an example of an image when the arrangement of virtual objects fails. As shown in FIG. 21, it may happen that furniture is arranged at a position not suitable for the actual arrangement of furniture such as a bed being placed in the passage of the room. Therefore, in order to realize a natural layout of furniture, the image processing apparatus 10 detects the subject shown in the captured image by the detection unit 15 and estimates locations where natural furniture arrangement is possible.

[0089] Here, the subject detected by the detection unit 15 is an object on the structure of the room shown in the captured image, such as an object installed in the room, that is, among the objects pre-installed in the room, those related to the layout of the room. The subjects detected by the detection unit 15 are, for example, doors, windows, frames, fusumas, electrical switches, closets, cupboards, kitchens (kitchenettes), passages, air conditioners, power outlets, lighting outlets, heaters, ladders, stairs, or fire alarms, etc.

[0090] As a method for detecting a subject shown in an image, many object detection algorithms have been known since the development of machine learning. As a typical method, there is one that represents the detection result of a subject with a rectangle (bounding box). Also, by using a method called semantic segmentation that indicates a subject with a region, the subject can be detected with higher accuracy. The detection method of the subject in step S4 by the detection unit 15 may use any known method. Also, the detection unit 15 detects the type of the subject shown in the image by a known method. The type of the subject shown in the image is information for specifying, for example, what the subject shown in the image is (e.g., whether it is a door or a window, etc.). FIG. 22 is a diagram for explaining an example of the subject detection result for a captured image. FIG. 22 shows the result of the kitchen, air conditioner, window, door, and passage being detected by the detection unit 15 from the subjects shown in the captured image.

[0091] In this way, the image processing apparatus 10 can estimate the state of the room shown in the captured image by estimating the structure of the room shown in the captured image and detecting the subject using the input captured image. Further, the image processing apparatus 10 can estimate the locations where the subject may be installed on the structure of the room by performing subject detection based on the structure of the room estimated in step S3, so that the processing efficiency can be improved. Note that the image processing apparatus 10 may perform the processes of step S3 and step S4 in parallel, or may change the order of step S3 and step S4.

[0092] Next, the position estimation unit 16 of the image processing apparatus 10 estimates the position of the subject detected in step S4 inside the room (step S5). The result of the subject detection is represented in the form of a rectangle when using a bounding box, or the filled pixels in the corresponding range when using semantic segmentation. These are representations on the unit sphere of the imaging device 70 as shown in Fig. 23(A), and can also be represented on the orthographic cylindrical projection as shown in Fig. 22. The position estimation unit 16 projects the result of the subject detection on the unit sphere as shown in Fig. 23(A) onto the shape of the room reconstructed in three dimensions. The position estimation unit 16 projects, for example, the subject existing on a normal wall such as a door, window, and passage as shown in Fig. 23(B) among the detected subjects, onto the structure of the room estimated by the structure estimation unit 14.

[0093] In this case, the position estimation unit 16 projects a virtual object according to the type of the subject detected by the detection unit 15. The position estimation unit 16, for example, arranges a virtual object serving as a light source at the detected window position, and synthesizes the image of the virtual object arranged by the image processing unit 20 described later, so that the external light incident on the room can be expressed more naturally. In this way, as a result, the position estimation unit 16 estimates and assigns the position of the subject on the structure of the room.

[0094] Note that the estimated position of the object in the room structure in the position estimation unit 16 does not necessarily have to be accurate. In the case of subject detection using the orthographic cylindrical projection method, there is a deviation from the actual position of the subject. However, since the result of subject detection is detected to be larger than the subject itself, there is a margin for estimating the furniture placement available area described later, and it does not pose a major problem in terms of layout.

[0095] Next, the image processing apparatus 10 performs furniture layout processing (step S6). When a human actually performs furniture layout, the layout is performed based on the room structure and the position of the object in the room structure. There are rough rules for furniture layout performed by humans based on conventions and the like. When automatically performing furniture layout, there are known methods of performing layout according to the rules of layout performed by humans or methods of optimizing layout by machine learning from a large number of past layout results. The image processing apparatus 10 performs furniture layout based on a simple rule for the room structure estimated by the structure estimation unit 14 and the object detected by the detection unit 15.

[0096] ○ Layout processing Here, an example of the layout processing in the image processing apparatus 10 will be described in detail with reference to FIGS. 24 to 29. FIG. 24 is a flowchart showing an example of the layout processing of virtual objects. FIG. 24 shows a process of determining the furniture to be arranged according to the use of the room and automatically arranging the furniture in order in the available area.

[0097] First, the determination unit 18 of the image processing apparatus 10 determines the furniture to be arranged (step S61). The determination unit 18 determines the furniture to be arranged, for example, according to the use and size of the room. Specifically, the determination unit 18 determines the furniture to be arranged based on the condition information stored in the condition information management DB 1002 and the use information indicating the use of the room. The use information is information specified by a real estate agent or the like who has photographed the target room. The image processing apparatus 10 receives, for example, the use information transmitted from an external device such as the photographing apparatus 70 by the transmission / reception unit 11. Note that the use information may be input to the image processing apparatus 10 together with the photographed image input in step S1, or may be information directly specified for the image processing apparatus 10.

[0098] Here, the use information includes, for example, information on the use of the room and the size of the room. The use of the room is the purpose of using the room, and is, for example, a classification such as a living room, a bedroom, or a children's room. Since it is generally difficult to determine the use of the room from the state of the room itself, it is preferably configured to be selected according to the intention of a user such as a real estate agent who has photographed the room. Note that the use of the room may be, for example, a configuration in which a large room with a kitchen is presumed to be a living room, a room with few windows is presumed to be a bedroom, and is automatically estimated by the structure estimation unit 14 according to the structure and subject of the room shown in the photographed image.

[0099] The layout of furniture is diverse, and the types of furniture to be arranged also vary depending on personal preferences and cultural circles. Also, it is necessary to make the room look beautiful for the purpose of home staging, and an aesthetic perspective is also required. There are various arrangement patterns for the layout of furniture. Also, the types of furniture are determined by various factors such as the use of the room, the size of the room, the style of the furniture, the season, or color coordination.

[0100] Therefore, the determination unit 18 reads out the condition information associated with the same use and size as the use information by, for example, searching the condition information management DB 1002 (see FIG. 15) using the use information as a search key. Then, the determination unit 18 selects the furniture to be arranged from among the furniture indicated in the furniture information stored in the storage unit 1000 or transmitted from the communication terminal 80 based on the style of the furniture or the furniture set indicated in the read-out condition information.

[0101] In the example shown in FIG. 15, the condition information defines different furniture sets for each use of the room and the size of the room. For example, for each of the living room and the bedroom, they are classified into three levels (L, M, S) according to the size of the room. For example, for the furniture set for a large room, a dining table and a large sofa, etc. are defined, and for the furniture set for a small room, a single sofa and a table are defined. Also, instead of the furniture set, the condition information defines the style of the furniture. In this case, the determination unit 18 selects a furniture set corresponding to the defined furniture style from among the furniture indicated in the furniture information stored in the storage unit 1000 or transmitted from the communication terminal 80. The style of the furniture is, for example, natural, pop, modern, Japanese, Nordic style or Asian, etc. Note that the condition information may include information such as color coordination or season in addition to the furniture set or the style of the furniture.

[0102] Next, the image processing apparatus 10 acquires furniture information, which is the information of the furniture to be arranged determined in step S61 (step S62). The furniture information includes data indicating a 3D model of the furniture and furniture setting data indicating rules regarding the arrangement of the furniture. Specifically, the storage / reading unit 29 of the image processing apparatus 10 reads out the furniture information stored in the storage unit 1000 to acquire the furniture information of the determined furniture. The furniture information is transmitted from a communication terminal 80 possessed by a furniture manufacturer or the like to the image processing apparatus 10 and is stored in the storage unit 1000 in advance. Note that the transmission / reception unit 11 of the image processing apparatus 10 may be configured to acquire the furniture information of the determined furniture by receiving the furniture information transmitted from the communication terminal 80 in response to a request from the image processing apparatus 10 in step S62.

[0103] Next, based on the structure of the room estimated in step S3 and the position of the subject estimated in step S5, the area estimation unit 17 estimates an area where furniture can be arranged (step S63). The arrangeable area will be specifically described with reference to FIGS. 25 and 26. FIG. 25 is a diagram showing a layout algorithm of a 3D model of a rug, which is an example of furniture, as a specific example. FIG. 25(A) shows the position of the imaging device 70 and the room structure estimated by the structure estimation unit 14, and FIG. 25(B) shows a state where the rug is placed at the center of the room, which is the arrangeable area estimated by the area estimation unit 17. Since rugs and carpets are laid on the floor, they are furniture that can be arranged regardless of the position of subjects such as doors and windows as long as the structure of the room is known.

[0104] FIG. 26 is a diagram showing a layout algorithm of a 3D model of a bed as furniture for which it is necessary to grasp the surrounding situation as a more complicated case. FIG. 26(A) shows the position of the imaging device 70 and the room structure estimated by the structure estimation unit 14, FIG. 26(B) shows the arrangeable area estimated by the area estimation unit 17, and FIG. 26(C) shows a state where the bed is placed in the arrangeable area. It shows a state where a rug is placed at the center of the room.

[0105] The area estimation unit 17 estimates the deployable area of the target furniture based on the basic rules regarding the installation of the furniture indicated in the furniture setting data acquired in step S62. The rules regarding the installation of a bed are, for example, placing the bed on the floor (not hanging in the air), placing it along the wall, not placing it in front of a passage or a door (it may be placed in front of a window), etc. The area estimation unit 17 estimates the deployable area of the bed as shown in Fig. 26(B) based on such rules, the structure of the room estimated by the structure estimation unit 14, and the position of the subject detected by the detection unit 15.

[0106] Also, the rules regarding the installation of a bed include sub-rules such as placing it randomly, placing the bed in a corner of the room, and placing it in the middle of the side of the room. The area estimation unit 17 determines the position to place the bed based on the deployable area and the sub-rules. Note that when the furniture cannot be placed according to the rules indicated in the furniture setting data, the area estimation unit 17 aborts the placement of that furniture and proceeds with the placement of other furniture.

[0107] Next, the placement unit 19 of the image processing apparatus 10 determines the placement of the 3D model of the furniture based on the furniture information acquired in step S63 (step S64). There are various file format formats for the 3D model of furniture, such as 3ds.max,.blend,.stl, or.fbx, and any format may be used. On the other hand, since the installation direction and the center position of a normal furniture 3D model are not defined, it is necessary to set the rules for the initial installation direction and the center point, and edit the 3D model according to the set rules, or prepare data for separate conversion. The rules for the installation direction and the center point of the furniture may be included in the furniture setting data, or may be separately set as a database when selecting the 3D model.

[0108] FIG. 27 is a diagram showing an example of a 3D model of furniture. The 3D model of the table shown in FIG. 27 has the coordinates (0, -1, 0) in the virtual space as the front, and the direction facing the person is set as the front. Also, the 3D model of the table shown in FIG. 27 defines the center of the surface facing the floor as the center of the furniture. The center of the furniture is determined based on the grounding surface of the furniture. Also, for example, the center of the light descending from the ceiling is the point in contact with the ceiling.

[0109] FIG. 28 is a diagram showing an example of the layout result of the 3D model of furniture. The arrangement unit 19 calculates the coordinates and orientation as the arrangement position of the furniture based on the available arrangement area estimated in step S62 and the arrangement rules shown in the furniture setting data. Thereby, the arrangement unit 19 can determine the arrangement of the furniture determined in step S61. Here, the layout information indicating the arrangement of the furniture determined by the arrangement unit 19 includes the type of furniture, and information on the orientation, position, and size of the furniture.

[0110] Then, when the arrangement of all the furniture determined in step S71 is completed (YES in step S75), the arrangement unit 19 ends the process. On the other hand, when the arrangement of all the furniture determined in step S71 is not completed (NO in step S75), the arrangement unit 19 repeats the process of step S74 until the arrangement of all the furniture is completed. Note that by arranging the larger furniture first, more furniture can be arranged.

[0111] In this way, the image processing apparatus 10 can automatically arrange the 3D model of the furniture suitable for the use according to the use of the room shown in the photographed image. Also, the image processing apparatus 10 can realize a more natural arrangement of furniture for the viewer by arranging the determined 3D model of the furniture in the available arrangement area estimated based on the estimated room structure and the detected position of the subject.

[0112] Returning to FIG. 16, the image processing unit 20 of the image processing apparatus 10 performs image synthesis processing on the captured image input in step S1 and the 3D model of the furniture arranged in step S6 (step S7). Specifically, the image processing unit 20 performs rendering using the captured image input in step S1, the structure of the room estimated in step S3, and the layout information of the furniture in step S6. For rendering, for example, CG tools such as 3dsMax, Blender, Maya, various CAD tools, Unity, and web browsers are used. Preferably, a tool equipped with a function that can operate the operation of the CG tool with a script is used. Also, the rendering is preferably performed in a format on an orthographic cylindrical projection method, but a part may be rendered in a projection method converted by a perspective projection or other projection methods. Further, the rendering may be either a rasterization method or a ray tracing method, but the ray tracing method is preferred from the viewpoint of improving the quality.

[0113] First, the placement unit 19 places the 3D model of the furniture on the CG tool based on the layout result in step S6. FIG. 29 is a diagram showing an example of the layout result of the 3D model of the furniture on the 3D space model. The layout information of the furniture includes, as described above, the type of furniture, and the orientation, position, and size of the furniture. The image processing unit 20 places the 3D model of the furniture decoded and specified by the script on the CG tool in the 3D space. Note that the layout information of the furniture may include correction information for the 3D model of the furniture such as the color or texture of the furniture. The example of FIG. 29 is an example in which a bed, a rug, a desk, and a potted plant are arranged in the 3D space.

[0114] Further, the placement unit 19 may represent all or part of the room structure estimated in step S3 on the CG tool. The room structure, namely the floor, ceiling, and walls, may be represented using textures or may be represented transparently. The example in FIG. 29 shows the representation of areas other than the front wall and ceiling. In the case of a transparent representation, the image processing unit 20 is set to function the transparent surface as a shadow catcher, and shadows are rendered to enhance the quality of the CG. Then, the image processing unit 20 performs a process of compositing the 3D model arranged by the placement unit 19 into the captured image input in step S1.

[0115] FIG. 30 is a diagram showing an example of a processed image using a shadow catcher. As shown in FIG. 30, by using the shadow catcher function, the image processing unit 20 can cast a shadow on an empty area or on a subject captured in the captured image. In this way, the image processing unit 20 can maintain consistent shadow quality as a single rendered image by performing the rendering process.

[0116] FIG. 31 is a diagram showing an example of image processing using Image Based lighting. As shown in FIG. 31(B), the image processing unit 20 can set, for example, the captured image as the data background of Image Based lighting, and can represent more natural light rays and enhance the texture compared to the normal lighting process shown in FIG. 30(A).

[0117] Then, the storage / reading unit 29 stores the processed image data synthesized in step S7 in the image data management DB 1001 (see FIG. 14) (step S8). In this case, the storage / reading unit 29 stores the processed image data synthesized in step S7 in the image data management DB 1001 in association with the captured image data before the compositing process and the condition ID for identifying the selection conditions of the furniture.

[0118] In this way, the image processing apparatus 10 can estimate the structure of the room and the position of the subject using the input captured image, and by arranging the 3D model of the furniture in the available placement area according to the estimation result, a more natural placement of the virtual object can be realized.

[0119] FIG. 32 is a diagram showing an example of the spatial estimation result and the subject detection result in the processed image in which the virtual object is synthesized. In the example of FIG. 32, the dotted line indicates the spatial estimation result of the room structure, and the thick frame indicates the subject detection result. As shown in FIG. 32, the image processing apparatus 10 can arrange the furniture in the available placement area based on the spatial estimation result and the subject detection result.

[0120] 〇Application examples of image synthesis processing Subsequently, with reference to FIGS. 33 to 36, application examples of the image synthesis processing in the above-described image processing apparatus 10 will be described. First, with reference to FIGS. 33 and 34, the process of estimating the prohibited placement area of the virtual object such as furniture will be described.

[0121] FIG. 33 is a diagram showing an example when the virtual object is too close to the imaging device. As shown in FIG. 33, when the virtual object is arranged in the available placement area based on the room structure and the subject detection result as described above, the virtual object may be too close to the imaging device 70 and may look bad. Therefore, as shown in FIGS. 34(A) and (B), the image processing apparatus 10 can estimate the area around the imaging device 70 as a prohibited placement area for the virtual object, thereby improving the appearance of the processed image after image synthesis.

[0122] In this case, in the above-described step S4, the detection unit 15 detects the position of the imaging device 70. In step S63, the region estimation unit 17 estimates the periphery of the imaging device 70 detected by the detection unit 15 as a placement prohibited region. Then, the region estimation unit 17 estimates the placement possible region of the virtual object in consideration of the estimated placement prohibited region together with the room structure estimated by the structure estimation unit 14 and the subject position estimated by the position estimation unit 16. Note that the placement prohibited region may be defined two-dimensionally or three-dimensionally.

[0123] Next, with reference to FIGS. 35 and 36, the processing when it is desired to hide a subject appearing in a captured image will be described. FIGS. 35 and 36 show examples in which the imaging device 70 and the support member 7 are reflected in the captured image. As shown in FIG. 35(A), since the entire surrounding area is captured by the imaging device 70, the captured image captured by the imaging device 70 includes the hand of the photographer supporting the imaging device 70 or a support member 7 such as a tripod or a monopod, which is not preferable from the viewpoint of making the room look beautiful. Therefore, the image processing apparatus 10, for example, detects the support member 7 by the detection unit 15 and arranges an arbitrary virtual object having a size capable of hiding the detected support member 7. Then, the image processing unit 20 of the image processing apparatus 10 synthesizes the image of the arranged virtual object with the captured image as the background, thereby eliminating the reflection of the support member 7 as shown in FIG. 35(B).

[0124] FIG. 36 is a diagram for explaining an example of the size of the support member. The detection unit 15 detects the support member 7 in step S4. The detection of the support member 7 may be performed while remaining in the orthographic cylindrical projection method, or may be performed by converting the vertically downward direction by perspective projection. As a result, the detection unit 15 acquires the viewing angle φ of the support member 7. As shown in FIG. 36, when the installation height of the imaging device 70 from the floor is h and the width of the support member 7 is w, the width w of the support member 7 can be expressed as w = 2tan(φ / 2). The placement unit 19 places an arbitrary virtual object having a size equal to or larger than the width w on the floor, and the image processing unit 20 can prevent the support member 7 from being reflected by synthesizing the image of the placed virtual object and the captured image. Note that the object detected by the detection unit 15 is not limited to the support member 7, and the imaging device 70 or the photographer who performs imaging with the imaging device 70 may be detected, and an arbitrary virtual object that hides the detected object may be placed.

[0125] ○Image display processing○ Next, with reference to FIGS. 37 to 43, the image display processing in the image processing system 3 will be described. FIG. 37 is a sequence diagram showing an example of the image display processing. FIG. 37 shows the processing when the processed image data stored in the image processing device 10 by the above-described processing is distributed to the viewer using the image distribution device 30.

[0126] First, the transmission / reception unit 91 of the display device 90 transmits an image display request indicating a request for image display to the image distribution device 30 based on an input operation on the input device or the like of the viewer (step S51). This image display request includes an image ID for identifying an image in which the structure to be requested is photographed. Thereby, the transmission / reception unit 31 of the image distribution device 30 receives the image display request transmitted from the display device 90.

[0127] Next, the transmission / reception unit 31 of the image distribution device 30 transmits an image acquisition request indicating a request to acquire image data to be distributed to the display device 90 to the image processing device 10 (step S52). This image acquisition request includes the image ID received in step S51. Accordingly, the transmission / reception unit 11 of the image processing device 10 receives the image acquisition request transmitted from the image distribution device 30.

[0128] Next, the storage / reading unit 29 of the image processing device 10 searches the image data management DB1001 (see FIG. 14) using the image ID received in step S52 as a search key, and reads out the captured image data and the processed image data associated with the same image ID as the received image ID (step S54). The transmission / reception unit 11 transmits the captured image data and the processed image data read out in step S54 to the image distribution device 30. Accordingly, the transmission / reception unit 31 of the image distribution device 30 receives the captured image data and the processed image data transmitted from the image processing device 10.

[0129] Then, the display control unit 32 transmits (distributes) the received captured image data or processed image data to the display device 90 via the transmission / reception unit 31, thereby causing the display device 90 to display the captured image or the processed image (step S55). Then, the display control unit 93 of the display device 90 causes the display 906 to display the captured image or the processed image corresponding to the data transmitted (distributed) from the image distribution device 30 (step S56).

[0130] FIG. 38(A) is an example of a screen of a captured image displayed on the display device, and FIG. 38(B) is an example of a screen of a processed image displayed on the display device. The captured image 400 shown in FIG. 38(A) is an image showing the state of the room before the furniture is arranged. On the other hand, the processed image 600 shown in FIG. 38(B) is an image after the furniture is arranged with respect to the captured image 400 shown in FIG. 38(A).

[0131] In addition, the reception unit 92 of the display device 90 receives a selection of whether furniture is arranged or not in response to a predetermined input operation using the input means of the display device 90 (step S57). Thereby, the viewer can select whether furniture is arranged or not for the image displayed on the display device 90, and can view the state of the room before and after the furniture is arranged by switching the images.

[0132] In this way, the image processing system 3 can convey a specific image of a room to the viewer by causing the display device 90 to display a processed image in which a 3D model of furniture is synthesized.

[0133] ○ Display of additional information Next, with reference to FIGS. 39 to 41, a process of superimposing and displaying additional information corresponding to the arranged furniture on the processed image 600 as shown in FIG. 38(B) will be described. When the image distribution device 30 causes the processed image 600 displayed on the display device 90 to be displayed, it can display additional information such as an icon prompting attention, a link to a website selling the furniture, or an explanation of the furniture for the image rendered by arranging the furniture model.

[0134] FIG. 39 is a conceptual diagram showing an example of an additional information management table. As shown in FIG. 13, in the storage unit 3000, an additional information management DB 3001 configured by the additional information management table shown in FIG. 39 is constructed. This additional information management table manages by associating an additional ID for identifying additional information, the type of furniture, coordinate information indicating the arrangement position of the additional information, and a link to a website for each image ID for identifying image data. Among these, the coordinate position is calculated by the calculation unit 35 based on the coordinate position of the 3D model of the furniture corresponding to the additional information. Also, the link to the website is included, for example, in the furniture information transmitted from the communication terminal 80 described above.

[0135] FIG. 40 is a diagram for explaining an example of the arrangement position of additional information. When superimposing additional information such as explanatory text, an icon, or a link on a photographed image, it is necessary to accurately superimpose the additional information on top of the position of the furniture. Therefore, as shown in FIG. 40, the coordinate detection unit 34 detects the coordinates on the processed image of the furniture shown in the processed image. The calculation unit 35 calculates the coordinates of the center position D of the furniture and calculates the direction indicating the center position D calculated from the imaging device 70. Then, the image processing unit 36 superimposes the additional information corresponding to the furniture on the coordinate position on the processed image indicating the direction calculated by the calculation unit 35.

[0136] The additional information to be superimposed is, for example, an icon that prompts the viewer's attention, explanatory text of the furniture, or an image for receiving access to a link to a website. When the image distribution device 30 receives the processed image data by the transmission / reception unit 31 in step S54 shown in FIG. 36, for example, the image distribution device 30 executes the above-described superimposing process of the additional information and causes the display device 90 to display the processed image on which the additional information has been superimposed in step S55.

[0137] FIG. 41 is an example screen of a processed image on which additional information is superimposed. In the processed image 600a shown in FIG. 41(A), an image 710 for receiving access to a website corresponding to the furniture shown in the processed image 600a is displayed as additional information. The image 710 includes, for example, a link to a website such as an EC (Electronic Commerce) site where the furniture shown in the processed image 600a can be purchased. A viewer who views the processed image 600a displayed on the display device 90 can access the page of the corresponding EC site by pressing the image 710, for example.

[0138] In the processed image 600a shown in FIG. 41(B), an icon 730 indicating that the furniture shown in the processed image 600a is a synthesized image is displayed as additional information. When synthesizing an image of a virtual object using ray tracing or Image Based lighting technology, it may be difficult for viewers to understand which part of the processed image is CG. Therefore, the processed image 600b can clearly distinguish for the viewer between the objects combined with the objects installed in the room by displaying the icon 730 prompting attention above the synthesized furniture image.

[0139] Note that the icon 730 may not be always displayed, and may be made non-displayed after a certain period of time has elapsed, or may be switched between display and non-display by an input operation by the viewer. Also, the icon 730 may be given effects such as blinking display so as to further prompt the viewer's attention. Furthermore, the processed image 600b may be configured to display an explanation of the furniture when the viewer selects the icon 730.

[0140] ○ Application Example of Image Display Next, with reference to FIGS. 42 and 43, an application example of the processed image displayed on the display device 90 will be described. The processed image 600c shown in FIG. 42 shows a state where the image of the arranged furniture is outlined. When the image processing unit 36 of the image distribution device 30 receives the processed image data by the transmission / reception unit 31 in step S54 shown in FIG. 36, for example, it generates a composite image that outlines the furniture image, and causes the generated processed image 600c to be displayed on the display device 90. Thereby, the viewer of the processed image 600c can clearly grasp the location of the synthesized virtual object (furniture).

[0141] In addition, the processed image 600d shown in FIG. 43 shows a state in which the color tone of the image of the arranged furniture is changed. When the image processing unit 36 of the image distribution apparatus 30 receives the processed image data by the transmission / reception unit 31 in step S54 shown in FIG. 36, for example, it performs a process of changing the color tone of the image of the furniture, and causes the display device 90 to display the processed processed image 600d. As a result, the viewer of the processed image 600c can clearly grasp the location of the synthesized virtual object (furniture) by looking at the image whose color tone has been changed and deliberately made unnatural.

[0142] ● Effects of the Embodiment As described above, the image display system 1 estimates the structure of the room and the position of the subject using the captured image captured by the imaging device 70, and arranges the 3D model of the furniture in the arrangeable area according to the estimation result, thereby realizing a more natural arrangement of the virtual object.

[0143] In addition, the image display system 1 causes the display device 90 to display the processed image in which the 3D model of the furniture is synthesized by the image processing system 3, so that the viewer can view not only the state of the empty room but also the state in which the furniture is arranged, and thus can convey a more specific image of the room to the viewer.

[0144] ● Summary ● As described above, the image processing method according to an embodiment of the present invention is an image processing method executed by the image processing system 3, and includes a structure estimation step of estimating the structure of a space (for example, a room) inside a structure from a background image (for example, an omnidirectional image) in which the space is shown in all directions, a region estimation step of estimating a region in which a virtual object (for example, a 3D model of furniture) can be arranged in the space based on the estimated structure, and an image processing step of synthesizing the virtual object in the estimated region with the background image. Thereby, the image processing method can automatically arrange the virtual object at an appropriate position in the space inside the structure.

[0145] Furthermore, the image processing method according to an embodiment of the present invention further includes a detection step of detecting a subject shown in a background image (e.g., a panoramic image), a position estimation step of estimating the spatial position of the detected subject, and an execution. The region estimation step estimates a region based on the estimated spatial structure and the position of the estimated subject. Thereby, the image processing method can estimate the state of the space by estimating the structure of the space shown in the background image and detecting the subject. In addition, the image processing method can estimate the location where the subject may be provided on the structure of the space by performing subject detection based on the estimated spatial structure, so that the processing efficiency can be improved.

[0146] Furthermore, in the image processing method according to an embodiment of the present invention, the image processing system 3 includes a condition information management DB 1002 (an example of a storage means) that stores condition information indicating the arrangement conditions of virtual objects (e.g., 3D models of furniture). Then, the image processing method executed by the image processing system 3 executes a determination step of selecting a virtual object according to the use of the internal space (e.g., a room) of the structure from the stored condition information. Thereby, the image processing method can automatically arrange a virtual object suitable for the use according to the use of the space shown in the background image.

[0147] In addition, the image processing system according to an embodiment of the present invention includes a structure estimation unit 14 (an example of a structure estimation means) that estimates the structure of a space from a background image (e.g., a panoramic image) in which the internal space (e.g., a room) of a structure is shown in all directions, a region estimation unit 17 (an example of a region estimation means) that estimates a region where a virtual object (e.g., a 3D model of furniture) can be arranged in the space based on the estimated structure, and an image processing unit 20 (an example of an image processing means) that synthesizes the virtual object into the estimated region with respect to the background image. Thereby, the image processing system 3 can automatically arrange the virtual object at an appropriate position in the internal space of the structure.

[0148] Furthermore, an image processing system according to an embodiment of the present invention includes a display control unit 32 (an example of display control means) that causes a display device 90 to display a processed image synthesized by an image processing unit 20 (an example of image processing means). Thereby, the image processing system 3 can switch images and allow a viewer to view the state of the space before and after the virtual object is placed.

[0149] ●Supplementary● Each function of the embodiment described above can be realized by one or more processing circuits. Here, the "processing circuit" in the present embodiment refers to a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, and an ASIC (Application Specific Integrated Circuit), DSP (digital signal processor), FPGA (field programmable gate array), SOC (System on a chip), GPU (Graphics Processing Unit), and a device such as a conventional circuit module designed to execute each function described above.

[0150] Also, the various tables of the embodiment described above may be generated by the learning effect of machine learning, and by classifying the data of the associated items by machine learning, it may not be necessary to use the tables. Here, machine learning is a technology for enabling a computer to acquire learning ability like a human. The computer autonomously generates an algorithm necessary for judgments such as data identification from the pre-loaded learning data and applies this to new data to make predictions. The learning method for machine learning may be any of supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, and deep learning, and furthermore, a learning method combining these learning methods may be used. The learning method for machine learning is not limited.

[0151] Although the image processing method, program, and image processing system according to an embodiment of the present invention have been described so far, the present invention is not limited to the above-described embodiment, and can be changed within the scope that those skilled in the art can conceive, such as addition, modification, or deletion of other embodiments. As long as the effects of the present invention are achieved in any aspect, it is included in the scope of the present invention.

Explanation of Signs

[0152] 1 Image display system 3 Image processing system 5 Communication network 7 Support member 10 Image processing apparatus 11 Transmission / reception unit 14 Structure estimation unit (an example of structure estimation means) 15 Detection unit 16 Position estimation unit 17 Region estimation unit (an example of region estimation means) 18 Decision unit 19 Arrangement unit 20 Image processing unit (an example of image processing means) 30 Image distribution apparatus 31 Transmission / reception unit 32 Display control unit (an example of display control means) 35 Calculation unit (an example of calculation means) 70 Imaging device 80 Communication terminal 90 Display device 1002 Condition information management DB (an example of storage means)

Prior Art Documents

Patent Documents

[0153]

Patent Document 1

Patent Document 2

Patent Document 3

Claims

1. An image processing method executed by an image processing system, comprising: a structure estimation step of estimating the structure of the space from a background image in which the space inside the structure is shown in all directions; a detection step of detecting a subject shown in the background image and the type of the subject; a position estimation step of estimating the position of the detected subject in the space; a region estimation step of estimating a region where the virtual object can be arranged based on the estimated structure, the estimated position of the subject in the space, and the rule of arrangement for each type of virtual object in the space with respect to the structure and the type of the subject; an image processing step of synthesizing the virtual object into the estimated region with respect to the background image; An image processing method for executing the above steps.

2. The detection step detects a support member that supports an imaging device for imaging the space, The image processing step synthesizes a predetermined image with respect to the background image so as to hide the detected support member. The image processing method according to claim 1.

3. The detection step detects a specific subject that serves as a light source among the subjects shown in the background image, The position estimation step estimates the position of the detected specific subject in the space, The image processing step synthesizes an image showing a virtual object serving as a light source at the position of the specific subject in the space estimated by the position estimation step. The image processing method according to claim 1.

4. The structure estimation step estimates the size of the space. The image processing method according to any one of claims 1 to 3.

5. An image processing method according to any one of claims 1 to 4, wherein the structure estimation step estimates the use of the space, and further executes a determination step of determining the virtual object based on the estimated use, The image processing step synthesizes the determined virtual object with respect to the background image.

6. An image processing method according to any one of claims 1 to 4, further comprising: a reception step of receiving use information indicating the use of the space from an external device; a determination step of determining the virtual object based on the received use information; and The image processing step synthesizes the determined virtual object with respect to the background image.

7. An image processing method according to claim 5 or 6, wherein The image processing system includes a storage means for storing condition information indicating the arrangement conditions of virtual objects. The determination step is an image processing method for selecting the virtual object corresponding to the use of the space from the stored condition information.

8. An image processing method according to any one of Claims 1 to 7, further comprising: Performing an arrangement step of arranging the virtual object with respect to the area estimated by the area estimation step; The image processing step is an image processing method for synthesizing the arranged virtual object into the estimated area with respect to the background image.

9. A program for causing a computer to execute the image processing method according to any one of Claims 1 to 8.

10. A structure estimation means for estimating the structure of the space from a background image showing the space inside the structure in all directions; A detection means for detecting the subject shown in the background image and the type of the subject; A position estimation means for estimating the position of the detected subject in the space; An area estimation means for estimating an area where the virtual object can be arranged based on the estimated structure, the estimated position of the subject in the space, and the rule of arrangement for each type of virtual object in the space with respect to the structure and the type of the subject; An image processing means for synthesizing the virtual object into the estimated area with respect to the background image; An image processing system comprising:

11. The image processing system according to Claim 10, further comprising: A display control means for causing a display device to display the processed image synthesized by the image processing means.

12. The display control means is the image processing system according to Claim 11, which switches the display of the background image and the processed image.

13. The image processing system according to Claim 11 or 12, Comprising a calculation means for calculating the center position of the virtual object on the displayed processed image; The display control means is an image processing system for superimposing and displaying additional information at the calculated center position with respect to the processed image.

14. The additional information is an icon corresponding to the virtual object or a link to a website, according to the image processing system of Claim 13.

15. The image processing system according to any one of claims 11 to 14, wherein the display control means displays a change in the color of the virtual object shown in the processed image or an image in which the virtual object is outlined.

Images