Composite image generating method, server and program

Abstract

To provide a method, a server, and a program for generating a composite image.SOLUTION: A composite image generation method provided herein comprises: identifying a composition target object included in an input image; determining an insertion content associated with the identified composition target object; and synthesizing the insertion content with an area of the composition target object to generate an output image.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a method, a server, and a program for generating a composite image by synthesizing other content with an input image. Specifically, the present invention relates to a method, a server, and a program capable of executing the method, which can provide a user with various personalized custom images from the same input image by identifying one or more objects included in the input image, determining related content, and synthesizing the related content into the object area to generate a composite image. [Background technology]

[0002] Chroma key is a widely known technique for creating a new image by combining two images. The chroma key technique uses the principle that when a subject to be combined is photographed against a single-color board and the background color is removed from the screen, only the subject remains. The single-color board used as the background is called a chroma back. Chroma backs usually use one of the colors RGB (red, green, blue), with blue being the most commonly used. However, it is not necessary to use only certain colors such as blue or green, and any color can be used as a chroma back.

[0003] According to the conventional chromakey technique, there is no relationship between the area that corresponds to the chromakey back and is removed or made transparent from the original image (hereinafter referred to as the "chromakey area") and the insert content that is composited into the chromakey area (hereinafter referred to as the "composite insert content"). Therefore, even if there are multiple chromakey areas in an image, there is a limit to freely composite different related contents into each of the multiple chromakey areas. Summary of the Invention [Problem to be solved by the invention]

[0004] SUMMARY OF THE PRESENT EMBODIMENTS The present invention aims to provide a synthetic image generation method for generating a personalized custom output image from an input image.

[0005] Another object of the present invention is to provide a method for generating a composite image by identifying one or more objects contained in an input image and synthesizing the identified object regions using related content to generate a composite image.

[0006] Another object of the present invention is to provide a server or system as a composite image generating device that performs the composite image generating method according to the present invention.

[0007] Another object of the present invention is to provide a program for executing the synthetic image generating method according to the present invention.

[0008] The technical problems of the present invention are not limited to the above-mentioned technical problems, and other technical problems not described above will be clearly understood by a person having ordinary knowledge in the technical field to which the present disclosure pertains (hereinafter referred to as "ordinary engineer") from the following description. [Means for solving the problem]

[0009] According to one aspect of the present invention, a method for generating a composite image performed by a computer device including at least one processor may include steps of identifying an object to be composited contained in an input image, determining insertion content associated with the identified object, and compositing the insertion content into a region of the object in the input image to generate an output image.

[0010] According to another aspect of the present invention, a server for generating a composite image may include an image receiving unit for acquiring an input image, an object identification unit for identifying an object to be composited that is included in the input image, a content determination unit for determining an insert content related to the identified object, a content synthesis unit for synthesizing the insert content in an area of ​​the object in the input image to generate an output image, and an image transmission unit for transmitting the output image to a user device via a network.

[0011] A program according to another aspect of the present invention can cause a computer device to execute the synthetic image generating method according to the present invention.

[0012] The above briefly summarized features of the present disclosure are merely illustrative aspects of the following detailed description of the present disclosure and are not intended to limit the scope of the present disclosure. Effect of the Invention

[0013] According to the present invention, a personalized and custom output image can be generated from an input image.

[0014] The present invention also provides a method for generating a composite image by identifying one or more objects contained in an input image and compositing identified object regions using related content.

[0015] It is also possible to provide a user device, a server, or a system as a synthetic image generating apparatus that performs the synthetic image generating method according to the present invention.

[0016] In addition, a program for executing the synthetic image generating method according to the present invention can be provided.

[0017] The effects obtained from the present disclosure are not limited to those described above, and other effects not described above will be clearly understood by those skilled in the art from the following description. [Brief description of the drawings]

[0018] [Figure 1] FIG. 1 illustrates a system in which a synthetic image generation method according to an embodiment of the present invention can be used. [Diagram 2] 1 is a block diagram showing an embodiment of a synthetic image generating device for performing a synthetic image generating method according to the present invention; [Diagram 3] FIG. 2 is a diagram showing an example of an input image according to the present invention. [Figure 4] FIG. 4 is a diagram showing objects in an input image identified by an object identification unit. [Diagram 5] FIG. 2 illustrates example candidate content that can be composited into each identified object region in the input image. [Figure 6] 13 is an example of an output image generated by combining the content determined by the content determination unit with each of the identified object regions. [Figure 7] 1 is a diagram for explaining a synthetic image generating method according to the present invention; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] <Summary of the Invention> According to one aspect of the present invention, a method for generating a composite image performed by a computer device including at least one processor may include steps of identifying an object to be composited contained in an input image, determining insertion content associated with the identified object, and compositing the insertion content into a region of the object in the input image to generate an output image.

[0020] In a composite image generating method according to the present invention, the input image may include one or more chromakey areas, and the step of identifying the object to be composited may include the steps of detecting the chromakey areas and identifying an object associated with the detected chromakey areas as the object to be composited.

[0021] In the composite image generating method according to the present invention, the step of identifying the object to be composited may identify the object to be composited based on at least one of a color key, a size, and a shape of the detected chromakey area.

[0022] In the synthetic image generating method according to the present invention, the step of identifying the object to be synthesized can identify the object to be synthesized by applying an object recognition technique to an object included in the input image.

[0023] The synthetic image generating method according to the present invention may further include a step of associating at least one accessible content with a target object, and a step of storing content information including association information with the target object for each of the accessible contents in the computer device.

[0024] In the synthetic image generation method according to the present invention, the step of determining the insertion content may include a step of determining at least one of the accessible contents related to the identified synthetic target object as a candidate content based on the content information, and a step of determining one of the candidate contents as the insertion content based on user profile information.

[0025] In the synthetic image generating method according to the present invention, the user profile information may include at least one of user personal information, preference information, and user history information.

[0026] In the synthetic image generation method according to the present invention, the step of determining the insertion content may include a step of determining at least one of the accessible contents related to the identified synthetic target object as a candidate content based on the content information, a step of displaying the candidate content, a step of receiving a selection for one of the candidate contents from a user of the computer device, and a step of determining the one of the candidate contents as the insertion content based on the received selection.

[0027] In the composite image generation method according to the present invention, the step of generating the output image may include a step of deforming the insert content based on the area of ​​the object to be composited, and a step of composite the deformed insert content into the area of ​​the object to be composited.

[0028] In the synthetic image generating method according to the present invention, the step of transforming the insertion content may transform at least one of the size, inclination, and shape of the insertion content so that the insertion content matches the area of ​​the object to be synthesized.

[0029] According to another aspect of the present invention, a server for generating a composite image may include an image receiving unit for acquiring an input image, an object identification unit for identifying an object to be composited that is included in the input image, a content determination unit for determining an insert content related to the identified object, a content synthesis unit for synthesizing the insert content in an area of ​​the object in the input image to generate an output image, and an image transmission unit for transmitting the output image to a user device via a network.

[0030] In the server according to the present invention, the input image includes one or more chromakey areas, and the object identification unit may perform a step of detecting the chromakey areas and a step of identifying an object associated with the detected chromakey areas as the object to be composited.

[0031] In the server according to the present invention, the object identification unit may identify the composite target object based on at least one of a color key, a size, and a shape of the detected chromakey area.

[0032] In the server according to the present invention, the object identification unit can identify the object to be synthesized by applying an object recognition technique to an object included in the input image.

[0033] The server according to the present invention can associate at least one accessible content with a target object, and store content information including association information with the target object for each of the accessible contents.

[0034] In the server according to the present invention, the content determination unit can determine at least one of the accessible contents related to the identified object to be synthesized as a candidate content based on the content information, and can determine one of the candidate contents as the insertion content based on user profile information of the user device.

[0035] In the server according to the present invention, the content determination unit can determine at least one of the accessible contents related to the identified object to be combined as a candidate content based on the content information, transmit the candidate content to the user device, receive a selection for any one of the candidate contents from a user of the user device, and determine the any one of the candidate contents as the insertion content based on the received selection.

[0036] In the server according to the present invention, the content combining unit can transform the insert content based on the area of ​​the object to be combined, and combine the transformed insert content with the area of ​​the object to be combined.

[0037] In the server according to the present invention, the content synthesis unit may transform at least one of a size, a tilt, and a shape of the insert content so that the insert content matches an area of ​​the object to be synthesized.

[0038] A program according to another aspect of the present invention can cause a computer device to execute the synthetic image generating method according to the present invention.

[0039] The above briefly summarized features of the present disclosure are merely illustrative aspects of the following detailed description of the present disclosure and are not intended to limit the scope of the present disclosure.

[0040] <Details of the Invention> DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present disclosure will be described in detail below with reference to the accompanying drawings, so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be embodied in various different forms and is not limited to the embodiments described herein.

[0041] In describing the embodiments of the present disclosure, if it is determined that a specific description of a known configuration or function may obscure the gist of the present disclosure, the detailed description thereof will be omitted. In addition, in the drawings, parts that are not related to the description of the present disclosure are omitted, and similar parts are denoted by similar reference numerals.

[0042] In the present disclosure, when a certain component is "coupled," "coupled," or "connected" to another component, this includes not only a direct connection relationship, but also an indirect connection relationship in which another component is interposed between them. Furthermore, when a certain component is described as "including" or "having" another component, this does not mean that the other component is excluded, but that the other component can be further included, unless otherwise specified to the contrary.

[0043] In this disclosure, terms such as "first" and "second" are used only for the purpose of distinguishing one component from other components, and do not limit the order or importance of the components unless otherwise specified. Therefore, within the scope of this disclosure, a first component in one embodiment may be referred to as a second component in another embodiment, and similarly, a second component in one embodiment may be referred to as a first component in another embodiment.

[0044] In this disclosure, components that are distinguished from one another are used to clearly describe the characteristics of each component, and do not necessarily mean that the components are separate. In other words, multiple components may be integrated into one hardware or software unit, and one component may be distributed into multiple hardware or software units. Therefore, unless otherwise specified, such integrated or distributed embodiments are also included in the scope of this disclosure.

[0045] In this disclosure, the components described in the various embodiments are not necessarily essential components, and some of them may be optional components. Therefore, an embodiment consisting of a subset of the components described in one embodiment is also included in the scope of this disclosure. In addition, an embodiment including other components in addition to the components described in the various embodiments is also included in the scope of this disclosure.

[0046] In this specification, the term "network" is a concept that includes both wired and wireless networks. In this case, the term "network" may refer to a network in which data is exchanged between a device and a system, and between devices, and is not limited to a specific network.

[0047] In addition, in this specification, the device may include not only mobile devices such as smartphones, tablet PCs, wearable devices, and HMDs (Head Mounted Displays), but also fixed devices such as PCs and home appliances with display functions. For example, the device may be a computing device capable of operating on a server, a vehicle, or an IoT (Internet of Things) device. In other words, in this specification, the device may refer to an apparatus capable of performing the synthetic image generation method according to the present invention, and is not limited to a specific type.

[0048] In addition, in this specification, "image" may include not only still images but also all types of information that a user can visually recognize through a device display, such as moving images and streaming images.

[0049] System and device configuration FIG. 1 is a diagram illustrating a system in which a synthetic image generation method according to an embodiment of the present invention can be used.

[0050] A system according to the present invention can include one or more user devices 101 , 102 , 103 and a server 110 connected via a network 104 .

[0051] Each of the user devices 101, 102, 103, sometimes referred to as a client, can connect to the server 110 via the network 104 to download and output desired images or content.

[0052] The server 110 can store a huge amount of images and content in a storage space within the server 110 or in a separate database. The server 110 can also identify users and accumulate and store various information, such as information about the users and information about the images and content.

[0053] For example, when a user connects to the server 110 by inputting specific connection information (ID and password) via the user device 101, 102, or 103, the server 110 can identify the connected user using the connection information input from the user device 101, 102, or 103.

[0054] The history of an identified user connecting to the server 110 and using a service may be stored in the server 110 as user history information. The user history information may include, for example, a search history, a request history, a playback history, an upload history, etc. A user may connect to the server 110 and input information about his / her gender, date of birth, age, health condition, occupation, address, etc., and this information may be stored in the server 110 as the user's personal information. In addition, a user may directly input his / her hobbies, areas of interest, etc. to the server 110, and this information may be stored in the server 110 as preference information.

[0055] The user history information, personal information and / or preference information may be collectively referred to herein as user profile information. Some or all of the user profile information may be stored on the user device 101, 102, 103 and / or the server 110 and may be used in the method for generating a synthetic image according to the present invention.

[0056] The synthetic image generating method according to the present invention can be performed by various types of devices, for example, all steps may be performed by the server 110 or the user devices 101, 102, 103, or some steps may be performed by the server 110 and some steps may be performed by the user devices 101, 102, 103.

[0057] The synthetic image generation method according to the present invention may be performed on the server 110 .

[0058] Specifically, the server 110 can determine an image to be transmitted to a user. The image to be transmitted to a user can be determined according to a request from the user. Alternatively, the image can be determined according to a request from the server 110 or a service provider. For example, according to a request from a service provider, an image that meets a specific requirement or a specific image can be determined as an image to be transmitted to a user. The server 110 can generate a composite image by performing a composite image generating method according to the present invention using an image to be transmitted to a user as an input image. The server 110 can transmit the generated composite image to the user devices 101, 102, and 103 via the network 104, and the user devices 101, 102, and 103 can output the transmitted composite image.

[0059] If input from a user is required to generate a composite image or if information stored in the user devices 101, 102, and 103 needs to be consulted, the server 110 can obtain the required information by exchanging data with the user devices 101, 102, and 103 via the network 104. For example, if a user selection is required to determine an insert content to be composited from at least one candidate content related to an object to be composited in an image, the server 110 can provide the candidate content to the user devices 101, 102, and 103 and receive the user selection. The server 110 can perform subsequent steps based on the received user selection. Similarly, if user profile information is required to determine an insert content to be composited and the information is stored in the user devices 101, 102, and 103, the server 110 can request and receive the required information from the user devices 101, 102, and 103 to perform subsequent steps.

[0060] The synthetic image generation method according to the present invention can also be performed on a client.

[0061] Specifically, the user devices 101, 102, and 103 may receive an image transmitted from the server 110. As described above, the transmitted image may be determined according to a request from a user or a request from the server 110 or a service provider, as described above. The user devices 101, 102, and 103 may generate a composite image by using the received image as an input image and performing a composite image generation method according to the present invention. The user devices 101, 102, and 103 may display the generated composite image via a display unit, thereby allowing the user to consume the composite image.

[0062] When an image, content, or information stored in the server 110 is required to generate a composite image, the user devices 101, 102, and 103 can obtain the required image, content, or information by exchanging data with the server 110 via the network 104. For example, when content related to an object in an image is stored in the server 110, the user devices 101, 102, and 103 can request and receive content related to the object from the server 110. When a plurality of contents are received, the user devices 101, 102, and 103 can display the plurality of contents as candidate contents on the display unit and determine one of the candidate contents as an insertion content to be composited according to a user selection or based on user history information. When a single content is received, the user devices 101, 102, and 103 can determine the received content as an insertion content to be composited. When the insertion content to be composited is determined, the user devices 101, 102, and 103 can generate a composite image using the content. Similarly, if user profile information is required to determine the insertion content to be synthesized and that information is stored on the server 110, the user devices 101, 102, 103 can perform subsequent steps by requesting and receiving the necessary information from the server 110.

[0063] Some steps of the synthetic image generation method according to the present invention may be performed on the server 110 and the remaining steps may be performed on the user devices 101, 102, 103.

[0064] For example, among the steps of the synthetic image generating method according to the present invention, the object identification step may be performed by the server 110, and the content determination step and the content synthesis step may be performed by the user devices 101, 102, and 103. Alternatively, the object identification step and the content synthesis step may be performed by the server 110, and the content determination step may be performed by the user devices 101, 102, and 103. The steps performed by the server 110 and the user devices 101, 102, and 103, respectively, are not limited to the above examples, and any step constituting the synthetic image generating method according to the present invention may be performed by the server 110 or the user devices 101, 102, and 103. Which steps the server 110 or the user devices 101, 102, and 103 perform may be adaptively determined in consideration of the computing power, data capacity, network environment, etc. of the server 110 or the user devices 101, 102, and 103.

[0065] FIG. 2 is a block diagram showing an embodiment of a synthetic image generating apparatus for performing the synthetic image generating method according to the present invention.

[0066] As described above, the method for generating a synthetic image according to the present invention may be performed solely in a user device or a server, so the synthetic image generation device 200 in Fig. 2 may be provided in the user device or the server. Since some steps of the synthetic image generation method according to the present invention may be performed in a server and the remaining steps in a user device, part of the synthetic image generation device 200 in Fig. 2 may be provided in the server and the remaining steps in the user device.

[0067] As shown in FIG. 2, the synthetic image generating apparatus 200 according to the present invention may include an image receiving unit 210, an object identifying unit 220, a content determining unit 230, and a content synthesizing unit 240. The synthetic image generated by the synthetic image generating apparatus 200 may be provided to a user as an output image via an output image providing unit 250. When the synthetic image is generated in a user device, the output image providing unit 250 may be a display unit 260 that displays the output image. The display unit 260 may be a display screen provided in the user device. When the synthetic image is generated in a server, the output image providing unit 250 may be an image transmitting unit 270 that transmits the output image to the user device. The image transmitting unit 270 may be a communication module provided in the server.

[0068] The image receiving unit 210 can receive an input image to be synthesized. The image receiving unit 210 provided in the user device can receive an image stored in a storage space in the server or in another database via a network as an input image. Alternatively, the user device can receive an image newly acquired by an image acquisition device such as a camera as an input image. Similarly, when provided in the server, the image receiving unit 210 can receive an image stored in a storage space in the server or in another database as an input image.

[0069] FIG. 3 is a diagram showing an example of an input image according to the present invention.

[0070] As shown in FIG. 3, the input image 300 may include various objects such as a display screen 310, a canned drink 320, a car 330, a table 340, and a person 350. The input image 300 may include metadata such as information about the type of image and information about objects in the image. For example, the information about the type of image may be information indicating whether the input image includes an object to be composited (hereinafter, referred to as a "composite object"). For example, the information about whether the input image includes a chromakey region may be information about the type of image. Based on the information about the type of image, it may be determined whether to perform the composite image generation method of the present invention on the input image. Also, for example, the information about the object in the image may include information about the position, type, size, area, etc. of the object included in the input image. As another example, even if the input image does not include information about the chromakey region in the type of image information, the composite image generation method may be performed when the composite image generation device 200 receives a message requesting identification of a composite object to generate a composite image through approval, request, etc. of a user device and / or a server.

[0071] 2 again, the object identification unit 220 can identify objects to be synthesized included in the input image. For example, objects to be synthesized included in the input image can be identified for each input image. As another example, when the input image is an image composed of a plurality of frames (e.g., a video, a time-lapse image, an image including a plurality of images, etc.), the method for identifying objects to be synthesized is performed for each frame, for each predetermined frame group, or at a predetermined time interval.

[0072] In this case, various methods can be applied to identify the objects to be synthesized included in the input images on an input image basis or on a frame basis. For example, as described above, when information on the objects to be synthesized included in the input images is included as metadata, the objects to be synthesized included in the input images can be identified using the metadata.

[0073] As another example, metadata for each frame constituting the input image may include information about a compositing target object. For example, information indicating that the display screen is a compositing target object in the 10th frame may be included as metadata for the 10th frame. As another example, upon receiving a message requesting identification of a compositing target object, the object identification unit 220 may use an object recognition technique to identify an image area of ​​the display screen included in the image of the 10th frame as a compositing target object.

[0074] The object recognition method for identifying a composite object may be a technology for recognizing various objects such as a display screen 310, a canned drink 320, a car 330, a table 340, and a human 350 from an input image 300 and identifying a composite object from among the various objects. Specifically, the object recognition method may include image classification, object localization, object detection, and a determination as to whether a detected object is a composite object. The image classification may predict and generate a class associated with a category list for each object in the input image 300. The object localization may provide a bounding box indicating the position and scale of one instance corresponding to the category list of each object for each object in the input image 300. The object detection may provide a bounding box for all instances corresponding to each category list for all objects in the input image 300 based on the result of the image classification and information from the object localization, and may generate a label including a predicted detailed object type and a predicted probability for each bounding box. The determination as to whether an object is a composite object may determine whether an object of a predicted detailed type is adopted as a composite object according to a preset condition. For example, when detailed types of objects in the input image 300 are detected as a display screen 310, a canned drink 320, a car 330, a table 340, and a human 350, the display screen 310, the canned drink 320, and the car 330 can be determined as objects to be synthesized according to conditions for adopting other objects other than the human 350 and the table 340 as objects to be synthesized. In addition, objects that satisfy a predetermined value or a predetermined range according to the position, size, motion, etc. of the detected object in the input image 300 may be adopted as objects to be synthesized. At least a part of the process of the object recognition technology can be realized by applying a deep learning model. The object recognition technology to which the deep learning model is applied may be an R-CNN (Region-Based Convolutional Neural Network) model group, a YOLO model group, etc.The R-CNN models may be one of R-CNN, Fast R-CNN, and Faster R-CNN. The YOLO models may be one of YOLO, YOLOv2, and YOLOv3. In addition to object detection, object segmentation may be performed to recognize objects into instances that are highlighted and pointed to by specific pixels for each object associated with a bounding box instead of a bounding box.

[0075] As another example, the object identification unit 220 can identify a chromakey region included in an input image to identify a composite target object included in the input image. According to an embodiment of the present invention, each chromakey region is associated with a composite target object, and by identifying the chromakey region, the associated composite target object can be identified.

[0076] The identification of the chromakey area may be performed by various methods. As described above, the chromakey area is an area into which other content is composited, and may be represented in a special form so as to be easily identified or removed. For example, the chromakey area may be represented and identified by a predetermined color key. Typically, the chromakey area is represented in a bluish color, but is not limited thereto, and may be represented in a predetermined color such as a greenish color, a reddish color, etc. If the input image includes multiple chromakey areas, each of the multiple chromakey areas may be represented in a different color from each other.

[0077] For example, among objects to be composited included in the input image 300, the chromakey areas to be the targets of image composition may be three object areas, namely, a display screen 310, a canned drink 320, and a car 330. In this case, the three chromakey areas may all be expressed in the same color series (e.g., blue-based), and the chromakey areas may be identified using the color key. Alternatively, the three chromakey areas may be expressed in two or more different color series (e.g., blue-based and green-based), and the chromakey areas may be identified using the respective color keys. Information regarding the color series in which the chromakey areas are expressed or information regarding the color keys may be predefined in the server and the device, may be transmitted from the server to the device, or may be included as metadata of the input image 300.

[0078] The color key used to identify the chromakey area may not only indicate one color, but may indicate a range of colors that can be expressed in a similar range to the color. For example, if blue is used as the chroma back, the color key may not only indicate the case where (R, G, B) = (0, 0, 255), but may indicate a range of colors such as (R, G, B) = (0 to 10, 0 to 10, 245 to 255). In this way, the chromakey area may be more reliably identified and removed. However, if the range of colors is too wide, there is a risk that a non-chromakey area may be mistakenly identified as a chromakey area, so the range of similar colors may be determined taking this into consideration. After identifying the chromakey areas in the image using the color key, the number or area of ​​pixels in each chromakey area may be compared with a predetermined threshold value. For example, if the area of ​​a chromakey area is smaller than a predetermined threshold value, the area may be determined to be not a chromakey area. That is, in order to more accurately identify the chromakey area, only an area having a size equal to or larger than a predetermined threshold value among a plurality of chromakey areas identified using a color key can be finally identified as a chromakey area. At this time, information regarding the predetermined threshold value may be predefined in the server and the device, may be transmitted from the server to the device, or may be included as metadata of the input image 300.

[0079] When multiple chromakey areas are expressed in different colors, a color key for each chromakey area can be used to identify a composite object related to the corresponding chromakey area. For example, as shown in Table 1, a color key (color) expressing a chromakey area can be associated with an object, and the color key can be used to identify a composite object.

[0080] [Table 1] For example, if a chromakey area expressed in a bluish color is identified in the input image 300, the composite target object corresponding to the chromakey area can be identified as related to the display screen. Also, if a chromakey area whose color key indicates green is identified in the input image 300, the chromakey area can be determined to be related to canned drinks. Similarly, a chromakey area using a red chroma back can be determined to be related to automobiles.

[0081] According to another embodiment of the present invention, the size and shape of the identified chromakey region may be used to identify an object to be composited with the corresponding chromakey region. For example, as shown in Table 2, the shape of the identified chromakey region may be associated with an object, and the object to be composited may be identified using the associated shape.

[0082] [Table 2] For example, if the identified chromakey region has a rectangular shape, the chromakey region can be determined to be an area associated with the display screen, and if the identified chromakey region has a cylindrical shape, the object associated with the chromakey region can be identified as a canned drink.

[0083] In addition, as shown in Table 3, the size of the identified chromakey region is associated with an object, and this can be used to identify the object to be composited.

[0084] [Table 3] For example, a chromakey area with a size of 350*200 pixels in the input image 300 can be determined to be related to the display screen of a large TV. If the chromakey area is identified with a size of 100*60 pixels in the input image 300, the chromakey area can be associated with the display screen of a laptop. If the chromakey area is identified with a size of 50*30 pixels in the input image 300, the object associated with the chromakey area can be determined to be the display screen of a mobile phone.

[0085] Or, for example, a chromakey area having a size of 350*200 pixels or more in the input image 300 can be determined to be the display screen of a large TV. A chromakey area having a size of 50*30 pixels or less can be determined to be the display screen of a mobile phone. A chromakey area having other sizes can be determined to be the display screen of a notebook computer. The size of the chromakey area for each object is not limited to the above examples, and can be set to various sizes or ranges of sizes.

[0086] In an embodiment using Table 3, a determination as to the size of the chromakey region may be made using the actual size of the chromakey region and a predefined threshold, which may be provided as metadata for the image, may be predefined, or may be calculated taking into account the size of a reference object in the image, for example, if the image contains a human, the reference object may be a human.

[0087] The above methods for identifying a compositing target object associated with an identified chromakey region may be combined in two or more ways. For example, as shown in Table 4, a combination of color key, size, and shape may be associated with a compositing target object, and the associated combination may be used to identify the compositing target object.

[0088] [Table 4] That is, if the color key for a chromakey region is blue and has a rectangular shape, it can be determined that the chromakey region is related to one of the display screens of a large TV, a laptop, or a mobile phone based on the size of the chromakey region. If the color key for a chromakey region is blue and has a cylindrical shape, the chromakey region can be identified as a canned drink. If the color key for a chromakey region is green, the object related to the chromakey region can be identified as a car.

[0089] In addition to the above-mentioned methods, various methods for identifying objects in an image can be applied. For example, a method of detecting and classifying objects contained in an image using a deep learning-based artificial neural network such as a Convolutional Neural Network (CNN) may be used.

[0090] The objects to be composited in the input image can be identified by analyzing the images of each frame contained in the input image, and the methods described above for identifying the objects to be composited in the input image can also be used to identify the objects to be composited in the images of each frame.

[0091] FIG. 4 is a diagram showing objects to be synthesized in the input image identified by the object identification unit.

[0092] For example, an input image 400 may include, among a plurality of objects, a display screen 410, a canned drink 420, and a car 430 as objects to be synthesized. Fig. 4 shows a result of identifying objects to be synthesized 410, 420, and 430 among the objects included in the input image 400.

[0093] Referring back to FIG. 2, the content determining unit 230 can determine an inserting content to be composited into the area of ​​the identified composite target object.

[0094] In this case, the insertion content may be any of the contents accessible from the synthetic image generating device 200. The synthetic image generating device 200 according to the present invention may associate the contents accessible from the synthetic image generating device 200 with the target object, and may store content information including association information with the target object for each accessible content. Table 5 shows an example of the stored content information.

[0095] [Table 5] In Table 5, Content ID (Identifier) ​​is an identifier of a piece of content accessible from synthetic image generating device 200, and can be used to identify each piece of accessible content.

[0096] The Content Type may include information about the type of the content. For example, the Content Type may be information indicating whether the content is a video or an image. Alternatively, the Content Type may be expressed as an extension of the content file. For example, the extension of the content file may be stored as the Content Type, such as mp4, avi, png, jpeg, tif, etc. In this case, the Content Type may indicate not only whether the content file is a video or an image, but also the encoding method of the content file.

[0097] The Target Object may refer to a target object related to the content. For example, Content 1 may be a content related to a display screen. Furthermore, the Content Provider may refer to a provider of the content.

[0098] The Content Route may include information regarding the location of the content. For example, in the case of Content 1, a URL (universal resource locator) may be included as the Content Route. Content 1 related to the display screen may be provided by accessing the URL. In this case, the Content Provider may easily update the content provided to the user by changing the content at the URL location, and Content 1 may not be stored in the synthetic image generating device 200. Alternatively, for example, as in the case of Content 2 or Content 3, the content may be stored in a storage device within the synthetic image generating device 200. In this case, the Content Route may refer to a storage path of the content within a storage device.

[0099] The content information may include various information related to the content in addition to the information exemplified in Table 5. For example, in the case of video content, information such as resolution, frame rate, and playback time may be included, and in the case of image content, information related to resolution, etc. may be included.

[0100] The content information may also include content profile information as an item used when determining the insert content in association with the user profile information. For example, information about the user who mainly consumes the content (age, sex, preferences, hobbies, history, etc.), or information about the environment in which each content is mainly consumed (season, weather, time period, region, etc.), may be included in the content information of Table 5 as the content profile information of the content. The content profile information may later be compared with the user profile information, etc., to determine the insert content to be synthesized. For example, in Table 5, if Content 1 is an animation image with a child as the main consumer, the main consumer may be set to "child" and saved as the content profile information of Content 1. Later, if the user who receives the synthesized image is identified as a "child" based on the user profile information, Content 1, whose main consumer is a "child," may be determined as the insert content to be synthesized based on the content profile information. Similarly, if the time period in which Content 2 is mainly consumed is at night, the main consumption time period may be set to "night" and saved as the content profile information of Content 2. Subsequently, if the time period when the composite image is provided is identified as "night", Content 2, the main consumption time period of which is "night", can be determined as the insertion content to be composited based on the content profile information.

[0101] Although Table 5 illustrates one content for each target object, the present invention is not limited thereto, and multiple contents can be provided for each target object. In addition, the information regarding the multiple contents may be the same or may be partially or completely different. One or more pieces of content profile information may be used to determine the insert content to be synthesized, and the content selected based on the content profile information may be provided to the user as candidate content.

[0102] The insertion content to be composited may be determined by selecting one of the candidate contents from one or more candidate contents associated with the identified composite target object. For example, one or more candidate contents associated with the identified composite target object may be displayed to a user. The user may view the displayed candidate contents and select one of the candidate contents. Upon receiving the user's selection, the selected candidate content may be determined as the insertion content to be composited into the region of the identified composite target object.

[0103] The content determination unit 230 included in the user device may receive a plurality of candidate contents related to the identified composite target object from the server and then display them to the user. The content determination unit 230 included in the server may transmit a plurality of candidate contents to the user device and then receive a user selection of the candidate contents.

[0104] The candidate content to be displayed or the inserted content to be combined may be determined based on user profile information. For example, when determining candidate content related to canned beverage 420, the user's age may be taken into consideration. That is, if the user is a minor, the candidate content may be determined to be only content related to non-alcoholic beverages. The inserted content to be combined may be determined similarly. For example, when the candidate content related to canned beverage 420 is two, a canned beer content and a canned cola content, if the user is a minor, the inserted content to be combined may be determined to be a canned cola content. In addition to the user's age, various user profile information regarding the user, such as personal information such as the user's gender and address, preference information such as hobbies and areas of interest, and history information such as search history, request history, and playback history, may be used to determine the candidate content and / or the inserted content to be combined. For example, the candidate content and / or the inserted content to be combined may be determined based on an image that the user has played back in a history. In this case, an image or content related to the image that has the playback history may be used. As a specific example, if the user has played back images of a particular genre the most, the content related to that genre may be determined as the inserted content.

[0105] The displayed candidate content or the inserted content to be synthesized may be determined based on environmental information such as the time, place, season, and weather when the image is provided. For example, if the season is winter, content related to beverages that are statistically primarily consumed in winter may be selected as content related to canned beverage 420. In this case, for each piece of content accessible from synthetic image generating device 200, the attributes of the content may be stored as content information, and the attributes of the content may be used to determine whether the beverage is statistically primarily consumed in winter.

[0106] The displayed candidate content or the synthesized insertion content may also be determined by the selection of a service provider offering the relevant service.

[0107] The candidate content to be displayed or the inserted content to be synthesized may also be determined by a combination of two or more of the above methods.

[0108] FIG. 5 is a diagram illustrating example candidate content that can be composited into each identified object region in the input image.

[0109] 5A shows an example of candidate content that can be synthesized in the object area 410 of the display screen. For example, a sports image 511, a performance image 512, an animation image 513, etc. can be provided as candidate content.

[0110] 5B shows examples of candidate content that can be synthesized into the canned drink object region 420. For example, an image 521 of canned beer, an image 522 of canned cola, an image 523 of canned coffee, etc. can be provided as candidate content.

[0111] 5C shows an example of candidate content that can be synthesized into the automobile object region 430. For example, an image 531 of a blue four-door automobile, an image 532 of a silver two-door automobile, and an image 533 of a red four-door automobile can be provided as candidate content.

[0112] For example, the content determining unit 230 can determine insertion content to be synthesized for each of the object regions from the candidate contents shown in FIG. 5 based on the various methods and criteria described above.

[0113] Referring again to FIG. 2, the content combiner 240 can generate the output image by combining the determined insertion content into each identified object region in the input image 400 .

[0114] FIG. 6 is an example of an output image generated by combining the inserting content determined by the content determining unit 230 with each of the identified object regions.

[0115] The output image 600 of FIG. 6 is an image generated by, for example, selecting a sports image 511 for the object region 410 of the display screen, an image of a canned beer 521 for the canned drink object region 420, and an image of a silver two-door car 532 for the car object region 430 in the example shown in FIG. 5, and synthesizing them in each object region. For example, for the object region 410 of the display screen, the sports image 511 can be determined as the insertion content to be synthesized from among a plurality of candidate contents by using the fact that the user's preference for sports is the highest as a result of referring to the user's preference information. In addition, for the canned drink object region 420, the image of a canned beer 521 can be determined as the insertion content to be synthesized by using the fact that the user is an adult male and enjoys beer as a result of referring to the user's personal information. In addition, for the car object region 430, the content 531 of a blue four-door car, the content 532 of a silver two-door car, the content 533 of a red four-door car, and the like can be provided to the user as candidate contents, and then the content 532 of the silver two-door car can be determined as the insertion content to be synthesized according to the user's selection.

[0116] There are various ways to composite the insert content into the object region. For example, the region of the identified object can be defined based on the contour of the object to be composited, and the insert content can be transformed to match the object region. For example, the size, skew, aspect ratio, shape, etc. of the insert content can be changed so that the composited insert content matches the object region. Once the insert content is transformed to match the object region, the transformed content can be composited in place of the object region.

[0117] In this case, if the input image is an image consisting of multiple frames (e.g., a moving image, a time-lapse image, or an image including multiple other images), the insertion content to be synthesized into the region of the object to be synthesized may be determined for each frame, for each predetermined frame group, or at a predetermined time interval. For example, if the object to be synthesized is a canned drink, the insertion content may be determined to be different for each frame. Alternatively, the insertion content from the first frame to the nth frame (first frame group) may be an image of a can of cola, and the insertion content from after the nth frame to the mth frame (second frame group) may be an image of a can of beer. Alternatively, the insertion content may be determined to be different at intervals of, for example, one second.

[0118] 2 again, as described above, when the output image is synthesized within the user device, the generated image can be consumed by the user by displaying the output image on the display unit 260 of the user device. When the output image is synthesized within the server, the output image can be transmitted to the user device connected to the network via the image transmission unit 270 of the server, thereby allowing the user to consume the image.

[0119] Synthetic image generation method FIG. 7 is a diagram for explaining a synthetic image generating method according to the present invention.

[0120] As described above, the synthetic image generating method according to the present invention may be performed independently in a user device or a server, and therefore the synthetic image generating method of FIG. 7 may be performed independently in a user device or a server. Also, some steps of the synthetic image generating method according to the present invention may be performed in a server, and the remaining steps in a user device. Also, at least one of the steps shown in FIG. 7 may be performed by data exchange between a user device and a server. For example, when a user selection is required, if content or user profile information, etc., is stored in a server or a user device, data exchange between the server and the user device may be performed as described above.

[0121] In step S710, an input image to be synthesized can be received. The user device can perform step S710 by receiving an image stored in a storage space in the server or in another database as an input image via a network, or by acquiring a new image using an image acquisition device such as a camera. The server can perform step S710 by reading an image stored in a storage space in the server or in another database. The input image of the synthetic image generating method according to the present invention is the same as the input image of the synthetic image generating device according to the present invention. Therefore, detailed description of the input image will be omitted below.

[0122] In step S720, a composite target object included in the input image can be identified. Various methods for identifying a composite target object included in the input image have already been described in relation to the object identification unit 220, and therefore a duplicate description will be omitted.

[0123] In step S730, an insert content to be composited into the region of the identified composite target object can be determined. The above-described contents related to the content determination unit 230 can be similarly applied to step S730, and therefore a duplicated description will be omitted.

[0124] For example, if multiple candidate contents are stored in a storage space in a server or a server-side database, and the insertion content to be synthesized is determined by a user's selection from the multiple candidate contents, step S730 may be performed as follows.

[0125] When the synthetic image generating method according to the present invention is performed in a user device, when a synthetic target object is identified in step S720, the user device can transmit information about the identified synthetic target object to a server. The server can identify a plurality of candidate contents based on the information about the identified synthetic target object and provide the candidate contents to the user device. Thereafter, the user device can perform step S730 by selecting one candidate content from the plurality of candidate contents.

[0126] In the case where the synthetic image generating method according to the present invention is performed by a server, when a synthetic target object is identified in step S720, the server may identify a plurality of candidate contents based on information about the identified synthetic target object, and provide the candidate contents to the user device. Thereafter, the server may perform step S730 in a manner of determining an insert content to be synthesized in the identified object region by receiving, from the user device, selection information of a user selecting one candidate content from the plurality of candidate contents.

[0127] Although the above example illustrates a case where one candidate content is selected from multiple candidate contents by user selection, the present invention is not limited to this example. In other words, step S730 may be performed by data exchange between the server and the user device depending on the storage location of various information (user selection information, user profile information, environmental information, information from the service provider, etc.) used to determine the insertion content to be synthesized.

[0128] For example, if multiple candidate contents are stored in a storage space in a server or in a server-side database, and the insertion content to be synthesized is determined based on user profile information, and the user profile information is stored in a user device, step S730 is performed as follows.

[0129] When the synthetic image generating method according to the present invention is performed in a user device, when a synthetic target object is identified in step S720, the user device may transmit information about the identified synthetic target object to a server. The server may identify a plurality of candidate contents based on the information about the identified synthetic target object and provide the candidate contents to the user device. Thereafter, the user device may perform step S730 in a manner of determining an insert content to be synthesized in the identified object region by selecting one candidate content from the plurality of candidate contents based on user profile information.

[0130] When the synthetic image generation method according to the present invention is performed on a server, when an object to be synthesized is identified in step S720, the server can perform step S730 in a manner that determines an insert content to be synthesized in the identified object area by identifying a plurality of candidate contents based on information about the identified object to be synthesized, and then requesting and receiving user profile information from the user device for selecting one candidate content from the plurality of candidate contents.

[0131] In step S740, the output image can be generated by synthesizing the determined insertion content to each identified object region in the input image. Various methods for synthesizing the content have already been described in relation to the content synthesizer 240, and therefore repeated description will be omitted.

[0132] As another example, when the synthetic image generating method according to the present invention is performed by a server, it may further include a step of identifying a user device to which the synthetic image including the inserted content is to be transmitted. The identification of the user device may be performed before step S710. The server may identify the user device to which the synthetic image is to be transmitted based on a user's selection, a default setting, a targeted user based on the type or content of the inserted content, a request from an external system other than the user, or the like. The identification of the user device may also be performed between steps S710 to S740 or after step S740 in a manner substantially similar to that described above.

[0133] According to the present disclosure, various output images 600 can be generated by synthesizing different content for each object to be synthesized from the input image 300. The inserted content to be synthesized can be determined for each user. In other words, instead of providing the same image to all users, a user-customized output image can be generated taking into consideration factors attributable to the user, such as user selection and user profile information, or various other factors. This makes it possible to maximize the impact of the created image on the user or to adjust it to a certain level. For example, by providing a user-customized image, the effect of the image, such as the educational effect or advertising effect, can be maximized.

[0134] Although the exemplary method of the present disclosure is expressed as a series of operations for clarity of explanation, this is not intended to limit the order in which the steps are performed, and each step may be performed simultaneously or in a different order, if necessary. To realize the method according to the present disclosure, the steps illustrated may include other steps, some steps may be omitted but the remaining steps may be included, and some steps may be omitted but additional other steps may be included.

[0135] The various embodiments of the present disclosure are not intended to enumerate all possible combinations, but are intended to describe representative aspects of the present disclosure, and the matters described in the various embodiments may be applied independently or in combination of two or more.

[0136] In addition, the method according to the present invention may be realized in the form of program instructions that can be executed by various computer devices and recorded on a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions recorded on the medium may be those specially designed and constructed for the present invention, or those known and available to those of ordinary skill in the computer software art. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program instructions, such as ROMs, RAMs, and flash memories. Examples of program instructions include not only machine language code produced by a compiler, but also high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

[0137] Additionally, various embodiments of the present disclosure may be realized by hardware, firmware, software, or a combination thereof, etc. In the case of a hardware implementation, the implementation may be realized by one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), general purpose processors, controllers, microcontrollers, microprocessors, etc.

[0138] The scope of the present disclosure includes software or machine-executable instructions (e.g., operating systems, applications, firmware, programs, etc.) that cause operations according to the methods of the various embodiments to be performed on a device or computer device, and non-transitory computer-readable media on which such software or instructions or the like are stored and executable on a device or computer.

[0139] The present invention further has the following object.

[0140] The present invention aims to provide a composite image generation method that generates a composite image by identifying one or more chromakey regions contained in an input image, identifying objects associated with the chromakey regions, and combining the object regions with content associated with the objects.

[0141] The present invention further has the following effects.

[0142] In accordance with the present invention, a composite image can be generated by identifying one or more chromakey regions contained in an input image, identifying objects associated with the chromakey regions, and compositing the object regions with content associated with the objects.

Claims

1. A program for executing a program on a computer device including at least one processor, associating at least one accessible piece of content with a composite target object; storing content information including association information with the composite target object for each of the accessible contents in the computer device; identifying the object to be synthesized contained in an input image; determining inserted content associated with the identified object; compositing the insertion content onto a region of the identified object in the input image to generate an output image; Run the command, The step of determining the insertion content comprises: determining, based on the content information, at least one of the accessible contents related to the identified composite target object as a candidate content; displaying the candidate content; receiving a selection for any one of the candidate contents from a user of the computing device; determining, based on the received selection, any one of the candidate contents as the insertion content.

2. the input image includes one or more chromakey regions; The step of identifying objects to be composited comprises: detecting the chromakey area; and identifying an object associated with the detected chromakey region as the object to be composited.

3. The step of identifying objects to be composited comprises: The program of claim 2 , further comprising: identifying the object to be composited based on at least one of a color key, a size, and a shape of the detected chromakey area.

4. The step of identifying objects to be composited comprises: The program according to claim 1 , further comprising: applying an object recognition technique to an object included in the input image to identify the object to be synthesized.

5. The step of generating an output image comprises: transforming the insertion content based on a region of the object to be composited; The program according to claim 1 , further comprising: a step of compositing the transformed insertion content into a region of the object to be composited.

6. The program according to claim 5 , wherein the step of transforming the insertion content transforms at least one of a size, an inclination, and a shape of the insertion content so that the insertion content matches an area of ​​the object to be composited.

7. A server that performs synthetic image generation, Associating at least one of the accessible contents with a composite target object, and storing content information for each of the accessible contents including association information with the composite target object; an image receiving unit for acquiring an input image; an object identification unit that identifies the object to be synthesized that is included in the input image; a content determiner for determining insertion content associated with the identified object; a content synthesis unit for synthesizing the insertion content in a region of the identified object in the input image to generate an output image; an image transmission unit that transmits the output image to a user device via a network; The content determination unit determining, based on the content information, at least one of the accessible contents related to the identified composite object as a candidate content; transmitting the candidate content to the user device; receiving a selection of any one of the candidate contents from a user of the user device; A server that determines, based on the received selection, any one of the candidate contents as the insertion content.

8. the input image includes one or more chromakey regions; The object identification unit Detecting the chromakey area; The server of claim 7 , further comprising: a processor configured to: identify an object associated with the detected chromakey region as the object to be composited;

9. The object identification unit The server of claim 8 , further comprising: a server configured to identify the object to be composited based on at least one of a color key, a size, and a shape of the detected chromakey area.

10. The object identification unit The server of claim 7 , further comprising: a server for identifying the object to be synthesized by applying an object recognition technique to objects contained in the input image.

11. The content synthesis unit Transforming the insertion content based on a region of the composite target object; The server according to claim 7 , further comprising: a processor for generating a composite image of the transformed insertion content in a region of the composite target object.

12. The content synthesis unit The server according to claim 7 , further comprising: a processor configured to transform at least one of a size, a tilt, and a shape of the insertion content so that the insertion content matches a region of the object to be composited.

Images