Image combining system, image combining method, and image combining program

Abstract

To provide a technique for easily creating a composite image of a background and a commodity.SOLUTION: The image synthesizing system 100 includes an inputting unit for receiving a first image 130 obtained by photographing, an image synthesizing unit for generating a synthesized image 150 from the first image 130 and a 3D (three dimensional) model, and an outputting unit for outputting the synthesized image 150. The image composition part detects an eye level from the first image 130, creates a second image 140 of the 3D model by adjusting the visual point of the imaginary camera in the 3D space to the eye level, and generates a composite image 150 by overlapping the first image 130 and the second image 140.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an image synthesis system, an image synthesis method, and an image synthesis program. [Background technology]

[0002] In business such as proposing renovations for buildings, there is a task of generating an image in which an image of the building is combined with an image of the equipment (hereinafter referred to as a composite image). Such composite images are sometimes created by processing photographs of the building taken on-site. By viewing the composite image, customers can get an idea of ​​what the new equipment will look like when installed in the building. However, creating composite images requires specialized software, and operating this software can be difficult. Therefore, there is a need for technology that makes it easier to create composite images.

[0003] Regarding the creation of composite images, for example, Japanese Patent Application Laid-Open No. 2012-208756 (Patent Document 1) discloses an image composition device that "determines the positional relationship between a marker for determining the product's position and the product in the image composition device. A marker for product placement is placed on a background CG scene created based on a rough sketch of the background, and the background CG is translated so that the center of the marker is the three-dimensional coordinate center of the background CG scene. The product and marker are photographed with a camera, the captured image is input into the image composition device, the position of the camera is calculated from the shape of the marker in the captured image, and a camera view from the same position as the camera is applied to the background CG to create a background image for composition. The area around the product in the photographed image of the product, which does not include the marker, is trimmed as a trimming area, and the resulting composite image is composed with the background image" (see [Abstract]). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2012-208756 Summary of the Invention [Problem to be solved by the invention]

[0005] According to the technology disclosed in Patent Document 1, a product image is composited into a background CG (Computer Graphics) scene. Therefore, the composite image created using the technology disclosed in Patent Document 1 may look significantly different from a real building. Therefore, it is desirable that the background material of the composite image be a real background photograph rather than CG. Furthermore, in order to use the technology disclosed in Patent Document 1, the person creating the composite image must prepare a background CG scene. Therefore, there is a need for a technology that allows for easy creation of composite images of backgrounds and products (equipment, etc.).

[0006] The present disclosure has been made in view of the above-described background, and an object of one aspect is to provide a technique for easily creating a composite image of a background and a product (such as equipment). [Means for solving the problem]

[0007] According to one embodiment, there is provided an image synthesis system. The image synthesis system includes an input unit for receiving an input of a first image obtained by photographing, an image synthesis unit for generating a synthetic image from the first image and a 3D (three-dimensional) model, and an output unit for outputting the synthetic image. The image synthesis unit detects an eye level from the first image, creates a second image of the 3D model by aligning the viewpoint of a virtual camera in 3D space with the eye level, and generates the synthetic image by overlaying the first image and the second image.

[0008] In one aspect, detecting the eye level from the first image includes extracting a plurality of straight lines from the first image, detecting the positions of two vanishing points that exist on the extensions of the plurality of straight lines, and determining the straight line that passes through the two vanishing points as the eye level.

[0009] In one aspect, the input unit receives an input of the length of a line segment on the first image, and the image synthesis unit adjusts the size of the second image based on the length of the line segment.

[0010] In one aspect, adjusting the size of the second image includes adjusting the size of the 3D model based on a ratio between the length of one side of the 3D model and the length of the line segment.

[0011] In one aspect, generating the composite image includes placing the second image on the first image such that one side is positioned on the specified line.

[0012] In one aspect, the output unit further outputs a UI (User Interface) for editing the composite image, and the UI moves the second image on the composite image based on the received command to move the second image.

[0013] According to another embodiment, there is provided an image synthesis method including: receiving an input of a first image obtained by photographing, detecting an eye level from the first image, creating a second image of a 3D model by aligning a viewpoint of a virtual camera in 3D space with the eye level, overlaying the first image and the second image to generate a synthetic image, and outputting the synthetic image.

[0014] According to another embodiment, there is provided an image synthesis program that causes a computer to receive an input of a first image obtained by photographing, detect an eye level from the first image, create a second image of a 3D model by aligning the viewpoint of a virtual camera in 3D space with the eye level, overlay the first image and the second image to generate a synthesized image, and output the synthesized image. [Effects of the Invention]

[0015] According to one embodiment, a composite image of a background and a product can be easily created.

[0016] The above and other objects, features, aspects and advantages of the present disclosure will become apparent from the following detailed description of the disclosure taken in conjunction with the accompanying drawings. [Brief explanation of the drawings]

[0017] [Figure 1] FIG. 1 is a diagram showing an example of an outline of the operation of image synthesis system 100 according to the present embodiment. [Figure 2] FIG. 2 is a diagram illustrating an example of functional blocks of the image synthesis system 100. [Figure 3] 1 is a diagram illustrating an example of a hardware configuration of an image synthesis system 100. FIG. [Figure 4] FIG. 10 is a diagram showing an example of data used to create a composite image 150. [Figure 5] 10 is a diagram showing an example of a first process of a composite image 150. FIG. [Figure 6] 10 is a diagram showing an example of a second process of the composite image 150. FIG. [Figure 7] 10 is a diagram illustrating an example of a third process of the composite image 150. FIG. [Figure 8] 10 is a diagram showing an example of a fourth process of the composite image 150. FIG. [Figure 9] FIG. 10 is a diagram illustrating an example of a fifth process of the composite image 150. [Figure 10] FIG. 10 is a diagram illustrating an example of a sixth process of the composite image 150. [Figure 11] FIG. 10 is a diagram showing an example of an editing function for a composite image 150. [Figure 12] 1 is a diagram showing an example of a series of procedures from creation to output of a composite image 150 in the image synthesis system 100. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0018] Hereinafter, embodiments of the technical concept of the present disclosure will be described with reference to the drawings. In the following description, identical components are assigned the same reference numerals. Their names and functions are also the same. Therefore, detailed descriptions thereof will not be repeated. Furthermore, each embodiment, each modification, each software configuration, each hardware configuration, each function, each process, etc. may be selectively combined as appropriate.

[0019] <A.システム> FIG. 1 is a diagram illustrating an example of an outline of the operation of an image synthesis system 100 according to the present embodiment. The image synthesis system 100 creates a synthesized image 150 by overlaying a second image 140, which is an image of a 3D model, on a first image 130, which is a background. As an example, the image synthesis system 100 can be used to create an image of a building after renovation or an image of a building after new equipment has been installed. In this case, an operator such as a renovation contractor can use the image synthesis system 100 to easily present an image of the building after renovation to a client who is the building owner. The image synthesis system 100 can also be used to create an image of a product (equipment) installed in any location, such as a vehicle (car, train, airplane, etc.) or outdoors.

[0020] In one aspect, the image synthesis system 100 is configured by a single device or a combination of multiple devices. The device may include a personal computer, a workstation, a server device, a tablet, or a smartphone. The device may also include a system-on-a-chip (SoC) or a system-on-module (SoM). The device may also include any peripheral devices such as a switch, a router, a display, a keyboard, and a mouse. Furthermore, the image synthesis system 100 or a device constituting the image synthesis system 100 may include a virtual machine or instance built in a cloud environment. In one aspect, the image synthesis system 100 may be connected to input / output devices such as a display and a keyboard and used as a standalone device. In another aspect, the image synthesis system 100 may provide various functions as a service or web application via a network. In this case, a user may use the functions of the image synthesis system 100 via a browser or client software installed on their terminal. Furthermore, the image synthesis system 100 may function as a single device as a whole and may be referred to as an image synthesis device.

[0021] Next, an overview of the operation of the image synthesis system 100 will be described. The image synthesis system 100 is configured to allow a first image 130 of a location where a product (such as equipment) is installed to be input from a terminal 110. In one aspect, the image synthesis system 100 may receive the first image 130 from the terminal 110 via a network. In another aspect, the image synthesis system 100 may be connected to the terminal 110 via an arbitrary input interface and configured to allow the first image 130 to be input. In yet another aspect, the image synthesis system 100 is configured to allow the first image 130 to be copied from an arbitrary storage medium. In this case, the image synthesis system 100 includes an interface for the storage medium. In the example of FIG. 1, the first image 130 is a photograph of a building entrance, but this is merely an example. The first image 130 may be an image of the location where the product (such as equipment) is installed, such as the interior of a building, the interior of a vehicle, outdoors such as a park, or any other location.

[0022] For example, the terminal 110 may be a terminal of a renovation contractor, a terminal of the building owner, or a terminal of a contractor commissioned to photograph the building. The terminal 110 also includes any terminal with a camera, such as a smartphone, tablet, or wearable computer. The terminal 110 may also be a terminal without a camera, such as a personal computer. In this case, a photographer photographs the building with a regular camera or the like, captures a first image 130 from the camera into the terminal 110, and transmits the first image 130 from the terminal 110 to the image synthesis system 100.

[0023] The image synthesis system 100 may be configured to accept a 3D (three-dimensional) model of a product separately from the first image 130. The image synthesis system 100 may also be configured to access a database 202 (see FIG. 2 ) that stores the 3D (three-dimensional) model of the product. The database 202 may be located within the image synthesis system 100 or may be located outside the image synthesis system 100.

[0024] The image synthesis system 100 reads from the database 202 a 3D model to be superimposed on the first image 130 and adjusts the position of the virtual camera (user's viewpoint) in the 3D space in which the 3D model is placed. More specifically, the image synthesis system 100 analyzes the first image 130 to obtain a vanishing point and an eye level. Then, the image synthesis system 100 adjusts the position (height) of the virtual camera (user's viewpoint) in the 3D space to the eye level. This makes the 3D model of the product appear to have been photographed from the virtual camera in the virtual space. Hereinafter, the 3D model viewed from the virtual camera (user's viewpoint) fixed at eye level will be referred to as the "second image 140." Since the viewpoint from which the 3D model is observed is adjusted, the second image 140 may be interpreted as a 3D model. In some aspects, the image synthesis system 100 may obtain a screenshot (image) of the 3D model. In this case, the "second image 140" refers to an image of the 3D model photographed from the virtual camera (user's viewpoint).

[0025] The image synthesis system 100 also has a function for adjusting the size of the second image 140. More specifically, the image synthesis system 100 is configured to be able to acquire length information of a portion of the first image 130. The image synthesis system 100 may adjust the size of the 3D model or the second image based on the length information of the portion. If the second image 140 is a screenshot, the image synthesis system 100 may adjust the size of the 3D model before photographing the 3D model and acquiring the second image 140. Alternatively, the image synthesis system 100 may directly adjust the size of the second image 140. In some aspects, the image synthesis system 100 may adjust the size of the second image 140 based on an operation by an operator.

[0026] Furthermore, the image synthesis system 100 creates the composite image 150 by overlaying the second image 140 on the first image 130. The image synthesis system 100 may receive input from a user specifying the location of the second image 140. As an example, the user may specify the location of the second image 140 on the first image 130 using a line, a point, an area, or the like. When the location is specified, the image synthesis system 100 overlays the second image 140 at the specified location on the first image 130. In one aspect, the image synthesis system 100 may create the composite image 150 by overlaying a 3D model on the first image 130. In another aspect, the image synthesis system 100 may create the composite image 150 by overlaying a screenshot of the 3D model on the first image 130.

[0027] Image synthesis system 100 is configured to be able to output the created composite image 150. In one aspect, image synthesis system 100 may be configured to be able to transmit composite image 150 to terminal 120 via a network. Terminal 120 includes any information processing terminal such as a smartphone. Terminal 110 may also function as terminal 120. In another aspect, image synthesis system 100 may be configured to be able to display composite image 150 on a display. Furthermore, in another aspect, image synthesis system 100 may have both of these functions.

[0028] As described with reference to FIG. 1 , the image synthesis system 100 creates a composite image 150 by overlaying a second image 140 (product) on a first image 130 (background). The eye levels of the first image 130 and the second image 140 are equal. Therefore, in the composite image 150, the product appears to be positioned in the background at a natural angle. Furthermore, the image synthesis system 100 can adjust the size of the second image 140 (product) to match the scale of the background and the product. In certain aspects, the image synthesis system 100 may be configured to synthesize multiple types of second images 140 (products) on the first image 130 (background). As an example, the image synthesis system 100 may synthesize an image of a mailbox (product), an image of an auto-lock door panel (product), and an image of a surveillance camera (product) on an image of an apartment building entrance (background). Furthermore, two or more of each product may be overlaid on the background. As an example, the image synthesis system 100 may synthesize images of multiple mailboxes (products) onto an image of an entrance to an apartment building (background).

[0029] FIG. 2 is a diagram illustrating an example of functional blocks of image synthesis system 100. In one aspect, each functional block illustrated in FIG. 2 may be realized as a program. In this case, each functional block may be realized by executing a program on hardware illustrated in FIG. 3. In another aspect, some of the functional blocks may be realized as hardware. In this case, image synthesis system 100 further includes, in addition to the configuration of FIG. 3, a hardware configuration for realizing the functional blocks realized as hardware.

[0030] The image synthesis system 100 includes an input unit 201, a database 202, an image synthesis unit 203, and an output unit 204. The image synthesis unit 203 includes a line detection unit 211, a vanishing point detection unit 212, an eye level detection unit 213, a length acquisition unit 214, and a synthesis unit 215.

[0031] The input unit 201 acquires the first image 130 from the terminal 110 and stores the first image 130 in the database 202. The input unit 201 may also directly output the first image 130 to the image synthesis unit 203. The input unit 201 may also directly output the first image 130 to the image synthesis unit 203 and store the first image 130 in the database 202.

[0032] Furthermore, the input unit 201 may accept input of a 3D model of a product and store the 3D model in the database 202. In another aspect, the input unit 201 may appropriately acquire a 3D model stored in the database 202 outside the image synthesis system 100.

[0033] The database 202 is configured to be able to store first images 130, which are photographs of product placement locations, and 3D models of the products. In one aspect, the database may be represented as a relational database table or in any other data format, such as JSON (JavaScript (registered trademark) Object Notation). The image synthesis system 100 may reuse the first images 130 stored in the database 202 any number of times. In another aspect, the image synthesis system 100 may be provided with a separate database of 3D models. In yet another aspect, the image synthesis system 100 may acquire 3D data by appropriately accessing an external database of 3D models.

[0034] The image synthesis unit 203 creates a synthesized image. In this case, the image synthesis unit 203 may read out the first image 130 and the 3D model from the database 202. Alternatively, the image synthesis unit 203 may acquire the first image 130 and the 3D model from the input unit 201. Alternatively, the image synthesis unit 203 may acquire the first image 130 from the input unit 201 and read out the 3D model from the database 202. The image synthesis unit 203 may read out the 3D model from an external database.

[0035] The line detection unit 211 analyzes the first image 130 and detects a plurality of lines from the first image 130. The line detection unit 211 outputs information about the detected plurality of lines to the vanishing point detection unit 212. The vanishing point detection unit 212 detects the intersection of the acquired plurality of lines as a vanishing point. The vanishing point detection unit 212 outputs information about the detected vanishing point to the eye level detection unit 213.

[0036] The eye level detection unit 213 detects or determines the eye level based on the acquired vanishing point information. More specifically, many photographs (images) are two-point perspective drawings using perspective projection or perspective. Two-point perspective drawings have two vanishing points. The straight line passing through these two vanishing points becomes the eye level. In a certain situation, if only one vanishing point is detected in the first image 130 (if the first image 130 is a one-point perspective drawing), the eye level detection unit 213 may detect or determine the horizon line passing through the one vanishing point as the eye level. The eye level detection unit 213 outputs the eye level information to the synthesis unit 215.

[0037] The length acquisition unit 214 acquires, via the input unit 201, a designation of a portion within the first image 130 and length information of the portion. As an example, the portion within the first image 130 may be a portion of a wall, a floor, or the like. The user can designate the portion by inputting a line segment or two points constituting a line segment on the first image 130 via the UI on the terminal 110. The length acquisition unit 214 outputs the information of the designated portion and length information of the portion to the synthesis unit 215. Furthermore, the length acquisition unit 214 stores the information of the designated portion and the length information of the designated portion in the database 202 in association with the first image 130.

[0038] The composition unit 215 creates the composite image 150. The creation of the composite image 150 includes processes (1) to (3). Process (1) is a process of aligning the virtual camera (user's viewpoint) in the 3D space with the eye level of the first image 130. The composition unit 215 sets the position of the virtual camera to the eye level detected by the eye level detection unit 213. As a result, the second image 140 (3D model) appears on the display as an image taken from the same eye level as the first image 130.

[0039] Process (2) is adjusting the size of the 3D model. The synthesis unit 215 adjusts the size of the 3D model placed in the 3D space based on the length information of the specified part. As an example, suppose the length of the specified part (e.g., part of the floor) in the first image 130 is "1000 mm." Also, suppose the length of a certain side of the 3D model (e.g., one side of a mailbox) is "500 mm." In this case, the synthesis unit 215 adjusts the size of the 3D model so that when the first image 130 and the second image 140 are superimposed, the ratio between the specified part and the certain side of the 3D model becomes "1000:500."

[0040] Process (3) is adjusting the placement of the second image 140 (3D model). The composition unit 215 may receive, via the input unit 201, an input specifying the placement location of the second image 140 (3D model). The placement location may be specified using a point, a line, an area, or the like. The user may specify the placement location (e.g., a line) on the first image 130 via the UI on the terminal 110, and may also specify a portion of the 3D model (e.g., one side of a mailbox) to be superimposed on the placement location (e.g., the line). As an example, the composition unit 215 may adjust the position of the 3D model so that a certain side of the 3D model (e.g., a side of a mailbox) is placed at the specified placement location (e.g., a specified line) on the first image 130. In some aspects, the portion used to adjust the size of the 3D model may be used as the placement location.

[0041] Assume that the second image 140 is a screenshot of a 3D model. Also, assume that the placement location is specified by a line. In this case, the composition unit 215 adjusts the angle of the 3D model so that a certain side of the 3D model is parallel to the specified line (specified placement location) on the first image 130. Then, the composition unit 215 takes a screenshot (second image 140) of the 3D model after the angle adjustment. The composition unit 215 overlays the screenshot (second image 140) on the first image 130 so that the screenshot (second image 140) appears to be placed in the specified placement location.

[0042] The compositing unit 215 outputs the created composite image 150 to the output unit 204. In one aspect, the composite image 150 may include layer information corresponding to each of the first image 130 and the second image 140. In this case, the composite image 150 is configured so that the position of the second image 140 is adjustable. In another aspect, the composite image 150 may be created as a single image. In this case, the data size of the composite image 150 is reduced by the amount that the layer information is not included. In still another aspect, the image compositing system 100 may accept input of a creation setting (presence or absence of layers) for the composite image 150 via the input unit 201. The image compositing system 100 may select the format of the composite image 150 (presence or absence of layer information) based on the creation setting.

[0043] The output unit 204 presents the composite image 150 to the user. In one aspect, the output unit 204 may output the composite image 150 to the terminal 120, a display, or both. The output unit 204 may also output a UI for editing the composite image 150. The user may move the second image 140 on the composite image 150 via the UI. In one aspect, the UI may provide a function for editing the composite image 150 using JavaScript or the like. In this case, the terminal 120 may edit the composite image 150 using only a browser function without communicating with the image synthesis system 100. In another aspect, the UI may request the image synthesis system 100 to perform editing processing on the composite image 150 based on receiving an editing operation on the composite image 150. In this case, the image synthesis unit 203 executes editing processing on the composite image 150 and returns the edited composite image 150 to the terminal 120. At least one of the input unit 201 and the output unit 204 may mediate communication between the image synthesis unit 203 and the UI.

[0044] Fig. 3 is a diagram showing an example of the hardware configuration of the image synthesis system 100. The image synthesis system 100 may not include some of the components shown in Fig. 3. The image synthesis system 100 may also include components not shown in Fig. 3. Furthermore, the image synthesis system 100 may include two or more of the same components. The various functions of the image synthesis system 100 described in this specification can be realized by executing a program on the hardware shown in Fig. 3.

[0045] The image synthesis system 100 includes a processor 301 , a memory 302 , a storage 303 , an external device IF 304 , an input IF 305 , an output IF 306 , and a communication IF 307 .

[0046] The processor 301 can execute programs for implementing various functions of the image synthesis system 100. The processor 301 is configured, for example, with at least one integrated circuit. According to an embodiment, the integrated circuit may include at least one central processing unit (CPU), at least one graphics processing unit (GPU), at least one field programmable gate array (FPGA), at least one application specific integrated circuit (ASIC), at least one artificial intelligence (AI) chip, or a combination thereof.

[0047] The memory 302 functions as a workspace for the processor 301. The memory 302 stores programs executed by the processor 301 and data referenced by the processor 301. In one aspect, the memory 302 can be realized by a dynamic random access memory (DRAM), a static random access memory (SRAM), or the like.

[0048] Storage 303 is a non-volatile memory that stores programs executed by processor 301 and data referenced by processor 301. Processor 301 executes programs read from storage 303 to memory 302 and references data read from storage 303 to memory 302. In one aspect, storage 303 may be realized by a hard disk drive (HDD), a solid state drive (SSD), an erasable programmable read only memory (EPROM), an electrically erasable programmable read only memory (EEPROM), a flash memory, or the like. Database 202 may be stored in storage 303.

[0049] The external device IF 304 can be connected to any external device such as a printer, a scanner, an external HDD, etc. In one aspect, the external device IF 304 can be realized by a USB (Universal Serial Bus) terminal or the like.

[0050] The input IF 305 can be connected to any input device such as a keyboard, a mouse, a touchpad, a gamepad, etc. In one aspect, the input IF 305 can be realized by a USB terminal, a PS / 2 terminal, a Bluetooth (registered trademark) module, etc.

[0051] Output IF 306 can be connected to any output device such as a cathode ray tube display, a liquid crystal display, an organic electroluminescence (EL) display, etc. In one aspect, output IF 306 can be realized by a USB terminal, a D-sub terminal, a DVI (Digital Visual Interface) terminal, an HDMI (High-Definition Multimedia Interface) terminal, a DisplayPort terminal, etc.

[0052] Communication IF 307 is connected to other devices via a wired network or a wireless network. In one aspect, communication IF 307 may be implemented by a wired local area network (LAN) port, a Wi-Fi (registered trademark) (Wireless Fidelity) module, or the like. In another aspect, communication IF 307 may transmit and receive data using a communication protocol such as TCP / IP (Transmission Control Protocol / Internet Protocol) or UDP (User Datagram Protocol).

[0053] As described with reference to FIGS. 1 to 3, the image synthesis system 100 includes an input unit 201 for receiving an input of a first image 130 obtained by photographing, an image synthesis unit 203 for generating a synthesized image 150 from the first image 130 and a 3D model, and an output unit 204 for outputting the synthesized image 150. The image synthesis unit 203 detects the eye level from the first image 130, obtains or creates a second image 140 of the 3D model by adjusting the viewpoint of the virtual camera in the 3D space to the eye level, and overlays the first image 130 and the second image 140 to generate the synthesized image 150. In the synthesized image 150, the product appears to be placed at the background location at a natural angle.

[0054] <B. Procedure for Creating Synthesized Image> FIG. 4 is a diagram showing an example of data used for creating a synthesized image. The image synthesis system 100 generates a synthesized image 150 from a 3D model 400 of a product and a first image 130 which is a photograph of the installation location of the product. At this time, the image synthesis system 100 executes a synthesis process 420 using the 3D model 400 and the first image 130. The synthesis process 420 includes each process shown in FIGS. 5 to 10. The image synthesis system 100 can obtain the second image 140 by adjusting the eye level, size and orientation of the 3D model 400.

[0055] FIG. 5 is a diagram showing an example of a first process of the synthesized image 150. The first process is a process of receiving data necessary for creating the synthesized image 150. The first process is executed by the input unit 201. The image synthesis system 100 receives an input of at least the first image 130 which is a photograph of the installation location of the product via a data upload screen 500.

[0056] In one aspect, the upload screen 500 may be provided as a screen of a web application. In this case, the image compositing system 100 transmits the upload screen 500 to the terminal 110 based on the terminal 110 accessing a predetermined uniform resource locator (URL). In another aspect, the upload screen 500 may be a screen of an application installed on the user's terminal 110, 120. The image compositing system 100 may further receive input of the 3D model 400 via the upload screen 500. Alternatively, the image compositing system 100 may separately provide an upload screen for the 3D model 400.

[0057] FIG. 6 is a diagram illustrating an example of the second processing of the composite image 150. The second processing is processing for detecting straight lines in the first image 130. The second processing is executed by the line detection unit 211. For example, the image synthesis system 100 may detect straight lines or line segments in the first image 130 using a Hough transform or a line segment detection algorithm. In the example of FIG. 6, the image synthesis system 100 detects straight lines 600A, 600B, 600C, and 600D. In practice, the image synthesis system 100 may detect more straight lines, such as from a lattice pattern on a wall.

[0058] FIG. 7 is a diagram showing an example of the third processing of the composite image 150. The third processing is processing for detecting a vanishing point in the first image 130. The third processing is executed by the vanishing point detection unit 212. The image synthesis system 100 detects or determines the intersection of the multiple lines detected in the second processing as a vanishing point. In the example of FIG. 7, the image synthesis system 100 detects vanishing point 700, which is the point where lines 600A and 600B intersect at right angles. Similarly, the image synthesis system 100 detects vanishing point 710, which is the point where lines 600C and 600D intersect at right angles. In other words, it can be said that the image synthesis system 100 detects two vanishing points in two-point perspective projection from the first image.

[0059] In one aspect, the image synthesis system 100 may calculate the coordinates of the vanishing points 700 and 710. The image synthesis system 100 may set an arbitrary location, such as the center or an edge of the first image 130, as the origin of the coordinates and calculate the coordinates of each vanishing point based on the distance from the origin to each vanishing point. In the example of Figure 7, the image synthesis system 100 may set the lower left edge of the first image 130 as the origin 720 and calculate the coordinates of the vanishing points 700 and 710 based on the horizontal and vertical directions (also referred to as X and Y directions) from the origin 720 to each vanishing point.

[0060] 8 is a diagram showing an example of the fourth processing of the composite image 150. The fourth processing is processing for detecting the eye level in the first image 130. The fourth processing is executed by the eye level detection unit 213. The image synthesis system 100 sets a straight line 800 passing through the vanishing points 700 and 710 detected in the third processing as the eye level. In one aspect, the image synthesis system 100 can calculate the distance from an origin (such as the origin 720) on a coordinate system set at an arbitrary position on the first image 130 to the straight line 800 as the eye level.

[0061] As explained with reference to Figures 6 to 8, detecting the eye level from the first image 130 includes extracting a plurality of straight lines from the first image 130, detecting the positions of two vanishing points 700, 710 that exist on the extensions of the plurality of straight lines, and determining the straight line 800 that passes through the two vanishing points 700, 710 as the eye level.

[0062] 9 is a diagram showing an example of the fifth process of the composite image 150. The fifth process is a process of acquiring the length of a portion of the first image 130. The fifth process is executed by the terminal 110 and the length acquisition unit 214. The user can draw a line segment 910 at a desired position on the first image 130 on the terminal 110 and set length information 920 (length information of the portion of the first image 130) for the line segment 910 (the portion of the first image 130).

[0063] The user may transmit the line segment 910 and the length information 920 together with the first image 130 from the terminal 110 to the image synthesis system 100. The length acquisition unit 214 acquires the line segment 910 and the length information 920 via the input unit 201. The length acquisition unit 214 associates the line segment 910 and the length information 920 with the first image 130 and stores them in the database 202. The line segment 910 and the length information 920 are used to adjust the size of the 3D model 400.

[0064] The length information 920 is used as a basis for adjusting the size of the 3D model 400. Therefore, the length information 920 may alternatively be referred to as scale information or dimension information. In one aspect, the functions of the terminal 110 described with reference to FIG. 9 may be implemented by an application installed on the terminal 110. In another aspect, the functions of the terminal 110 described with reference to FIG. 9 may be implemented by a screen of a web application provided by the image synthesis system 100.

[0065] 9, the input unit 201 receives an input of the length of a line segment on the first image 130. The image synthesis unit 203 adjusts the size of the second image 140 based on the length of the line segment.

[0066] 10 is a diagram showing an example of the sixth process of the composite image 150. The sixth process is a process of adjusting the position of the virtual camera and the size and position of the 3D model. The sixth process is executed by the composition unit 215.

[0067] First, the image synthesis system 100 sets the position of the virtual camera (user's viewpoint) at eye level (straight line 800) in 3D space. Next, the image synthesis system 100 enlarges or reduces the 3D model based on the specified length (length information 920). That is, the image synthesis system 100 adjusts the size of the 3D model. As an example, assume that the length of side 1050 is "400 mm." In this case, the image synthesis system 100 adjusts the size of the 3D model 400 so that the ratio of the lengths of line segment 910 and side 1050 becomes "2250:400" when the first image 130 and the second image 140 are superimposed. The 3D model 1000 is the 3D model after the size adjustment.

[0068] Next, the image synthesis system 100 adjusts the position of the 3D model 1000. As an example, the image synthesis system 100 may align the position of any edge of the 3D model with the line segment 910. In the example of FIG. 9 , the image synthesis system 100 aligns the edge 1050 with the line segment 910 or an extension thereof. In one aspect, the image synthesis system 100 may provide a UI for the sixth process. In this case, the image synthesis system 100 may accept an input for selecting an edge of the 3D model via the UI. The image synthesis system 100 may also accept an input for manually adjusting the size of the 3D model 400 via the UI. In another aspect, the image synthesis system 100 may automatically select one of the edges of the 3D model and align the orientation of the selected edge with the line segment 910.

[0069] As another example, the image synthesis system 100 may be configured to accept input of an arrangement location separately from the line segment 910 via a UI for the sixth process. The arrangement location may be defined as a straight line or a line segment. In this case, the image synthesis system 100 places the edge 1050 of the 3D model 1000 along the specified line (arrangement location). Furthermore, in a certain aspect, the image synthesis system 100 may be configured to accept input of an arrangement location consisting of two lines (e.g., two line segments 910 and 1040, which may be straight lines) and two edges of the 3D model 1000 (e.g., edges 1050 and 1060) via the UI. In this case, the image synthesis system 100 places each of the edges 1050 and 1060 along each of the line segments 910 and 1040.

[0070] 10, the image synthesis system 100 can place the second image 140 (3D model of the product) at a natural angle and size at a desired position on the first image 130 (background). In one aspect, the image synthesis system 100 can place a screenshot of the 3D model (second image 140) adjusted to a natural angle and size at a desired position on the first image 130 (background).

[0071] 10 , adjusting the size of the second image 140 includes adjusting the size of the 3D model based on the ratio between the length of one side 1050 of the 3D model and the length of the line segment 910. Furthermore, generating the composite image 150 includes placing the second image 140 on the first image 130 so that the one side 1050 of the 3D model is positioned on a specified line. The specified line may be the line segment 910 used to adjust the size of the 3D model 400, or may be a line separately specified by the user.

[0072] FIG. 11 is a diagram illustrating an example of an editing function for the composite image 150. The image synthesis system 100 provides a user with the editing function for the composite image 150. A screen 1100 is an editing screen for the composite image 150. The image synthesis system 100 may transmit the screen 1100 to the terminal 120 as a screen of a web application. Alternatively, the image synthesis system 100 may display the screen 1100 on the display of the image synthesis system 100. Alternatively, the screen 1100 may be a screen of an application installed on the terminal 120. A user may adjust the position, size, etc. of the second image 140 by operating the screen 1100 displayed on the display of the terminal 120. As an example, a user may grab and move the second image 140 or the coordinate icon 1110 with a mouse or the like. In some aspects, the editing function for the composite image 150 may be implemented using JavaScript or the like. In this case, the terminal 120 may update the position, size, or both of the second image 140 without communicating with the image synthesis system 100 each time an operation is received on the screen 1100. In another aspect, the terminal 120 may request the image synthesis system 100 to adjust the position, size, or both of the second image 140. In this case, the terminal 120 sends a request to adjust the second image 140 to the image synthesis system 100 each time an operation is received on the screen 1100. The image synthesis system 100 adjusts the second image 140 and returns the adjusted second image 140 to the terminal 120.

[0073] 11 , the output unit 204 further outputs a UI for editing the composite image 150. The UI moves the second image 140 on the composite image 150 based on the receipt of the command to move the second image 140.

[0074] <C.シーケンス> Fig. 12 is a diagram showing an example of a series of procedures from creation to output of a composite image 150 in the image synthesis system 100. In the example of Fig. 12, the terminal processing before step S1225 will be described as being executed by the terminal 110. Furthermore, the terminal processing after step S1245 will be described as being executed by the terminal 120. In reality, the terminals 110 and 120 may be the same terminal.

[0075] In step S1205, the terminal 110 uploads the first image 130, which is a photograph of the product placement location, to the image synthesis system 100. In step S1210, the image synthesis system 100 analyzes the first image 130 and detects straight lines, vanishing points, and eye levels within the first image 130. In step S1215, the image synthesis system 100 transmits eye level information to the terminal 110.

[0076] In step S1220, terminal 110 receives input of information on the target product, the line segment, and the length of the line segment from the user. The target product here corresponds to the product shown in second image 140. The line segment and the information on the length of the line segment here correspond to line segment 910 and length information 920. Terminal 110 also displays the eye level on the display, superimposed on first image 130.

[0077] In step S1225, the terminal 110 uploads information about the target product, the line segment, and the length of the line segment to the image synthesis system 100. In some aspects, the terminal 110 may simultaneously execute the processes of steps S1205 and S1225. That is, the terminal 110 may simultaneously upload information about the first image 130, the target product, the line segment, and the length of the line segment to the image synthesis system 100. In this case, the image synthesis system 100 does not need to execute the process of step S1215.

[0078] In step S1230, the image synthesis system 100 requests a 3D model of the target product from the database 202. In one aspect, the image synthesis system 100 may reference the database 202 within the image synthesis system 100. In another aspect, the image synthesis system 100 may send the request to a database 202 external to the image synthesis system 100. In step S1235, the database 202 outputs or transmits a list of URLs of the 3D models of the requested target products to the image synthesis system 100. If one target product is requested, the database 202 may output one URL to the image synthesis system 100. Alternatively, if multiple target products are requested, the database 202 may output a URL list including multiple URLs to the image synthesis system 100. Alternatively, the database 202 may output or transmit a 3D model of each of the one or more requested target products to the image synthesis system 100.

[0079] In step S1240, the image synthesis system 100 synthesizes the first image 130 and the second image 140. More specifically, the image synthesis system 100 adjusts the virtual camera (user's viewpoint) in the 3D space to eye level. The image synthesis system 100 also adjusts the size, orientation, and position of the 3D model based on the line segments and their lengths acquired in step S1225. As a result, the image synthesis system 100 acquires the adjusted 3D model (second image 140). The image synthesis system 100 creates a synthesized image 150 by overlaying the second image 140 on the first image 130. In some aspects, the image synthesis system 100 may place the second image 140 at a specified position on the first image 130. Alternatively, the image synthesis system 100 may automatically select an arbitrary position on the first image 130 and place the second image 140 at the selected position.

[0080] In step S1245, the image synthesis system 100 transmits the composite image 150 to the terminal 120. The image synthesis system 100 also transmits a UI for editing the composite image 150 to the terminal 120. The terminal 120 displays the received composite image 150 and UI on a display. In one aspect, the image synthesis system 100 may transmit the composite image 150 itself to the terminal 120. In another aspect, the image synthesis system 100 may transmit the second image 140 and its coordinate information to the terminal 120. In this case, the terminal 120 creates the composite image 150 by superimposing the second image 140 on the first image 130 in the terminal 120 based on the received coordinate information.

[0081] In step S1250, the terminal 120 updates the position of the second image 140 within the composite image 150 based on receiving input to change the position of the second image 140 (3D model) via the UI for editing the composite image 150.

[0082] In one aspect, the above functions may be realized by having a computer constituting image synthesis system 100 execute an image synthesis program. That is, the image synthesis program may cause the computer to receive input of first image 130 obtained by shooting, detect the eye level from first image 130, acquire or create second image 140 of the 3D model by aligning the viewpoint of a virtual camera in 3D space with the eye level, overlay first image 130 and second image 140 to generate composite image 150, and output composite image 150.

[0083] <D.まとめ> The image synthesis system 100 according to this embodiment is configured to be able to create a composite image 150 by overlaying a second image 140 (product) on a first image 130 (background). When creating the composite image 150, the image synthesis system 100 adjusts the eye level of the second image 140 to that of the first image 130. This makes the product appear to be placed in the background at a natural angle in the composite image 150. The image synthesis system 100 can also adjust the size of the second image 140 (product) based on the line segment whose length is specified in the first image 130. This allows the image synthesis system 100 to appropriately adjust the size ratio between the first image 130 (background) and the second image 140 (product). A user can easily create the composite image 150 by using the image synthesis system 100. As a result, the user can easily confirm an image (composite image 150) in which a desired product is placed in a desired location (background).

[0084] The embodiments disclosed herein should be considered to be illustrative in all respects and not restrictive. The scope of the present disclosure is defined by the claims, not by the above description, and is intended to include all modifications within the meaning and scope equivalent to the claims. Furthermore, the disclosures described in the embodiments and each modification are intended to be implemented, as far as possible, either alone or in combination. [Explanation of symbols]

[0085] 100 Image synthesis system, 110, 120 Terminal, 130 First image, 140 Second image, 150 Synthesized image, 201 Input unit, 202 Database, 203 Image synthesis unit, 204 Output unit, 211 Line detection unit, 212 Vanishing point detection unit, 213 Eye level detection unit, 214 Length acquisition unit, 215 Synthesis unit, 301 Processor, 302 Memory, 303 Storage, 304 External device IF, 305 Input IF, 306 Output IF, 307 Communication IF, 400, 1000 3D model, 420 Synthesis processing, 500 Upload screen, 600A, 600B, 600C, 600D, 800 Straight line, 700, 710 Vanishing point, 720 Origin, 910, 1040 Line segment, 920 Length information, 1050,1060 sides, 1100 screen, 1110 coordinate icon.

Claims

1. an input unit for receiving an input of a first image obtained by photographing; an image synthesis unit for generating a synthetic image from the first image and a 3D (Three-Dimensional) model; an output unit for outputting the composite image, The image synthesis unit Detecting an eye level from the first image; creating a second image of the 3D model by aligning a viewpoint of a virtual camera in 3D space with the eye level; an image synthesis system that overlays the first image and the second image to generate the composite image;

2. Detecting the eye level from the first image includes: Extracting a plurality of straight lines from the first image; Detecting the positions of two vanishing points that exist on the extensions of the plurality of straight lines; and determining a straight line passing through the two vanishing points as the eye level.

3. the input unit accepts an input of a length of a line segment on the first image; The image synthesis system according to claim 1 , wherein the image synthesis unit adjusts a size of the second image based on the length of the line segment.

4. 4. The image synthesis system of claim 3, wherein adjusting the size of the second image includes adjusting the size of the 3D model based on a ratio between a length of one side of the 3D model and a length of the line segment.

5. The image synthesis system of claim 4 , wherein generating the synthetic image includes placing the second image on the first image so that the one side is positioned on a specified line.

6. the output unit further outputs a UI (User Interface) for editing the composite image; The image compositing system according to claim 1 , wherein the UI moves the second image on the composite image based on receipt of a command to move the second image.

7. Accepting an input of a first image obtained by photographing; Detecting an eye level from the first image; creating a second image of the 3D model by aligning a viewpoint of a virtual camera in 3D space with the eye level; superimposing the first image and the second image to generate a composite image; and outputting the composite image.

8. Accepting an input of a first image obtained by photographing; Detecting an eye level from the first image; creating a second image of the 3D model by aligning a viewpoint of a virtual camera in 3D space with the eye level; superimposing the first image and the second image to generate a composite image; and outputting the composite image.

Images