Method and device for obtaining images using image generation model

Abstract

According to the present disclosure, a method which includes obtaining concept information, a reference image, additional condition information, and a customized target image for a target image; applying a reference keyword obtained from the reference image and the concept information to a language generation model; applying an image generation prompt obtained from the language generation model, the additional condition information, and the customized target image to an image generation model upon applying the reference keyword and the concept information; performing post-processing on an initial image generated by the image generation model; and obtaining a target image by performing post-processing on the initial image, and a device and a recording medium using which the method is performed are disclosed.

Description

METHOD AND DEVICE FOR OBTAINING IMAGES USING IMAGE GENERATION MODEL

[0001] The technical field of the present disclosure relates to a method of obtaining images using an image generation model and relates to a technical field of a method of generating an image corresponding to a condition using an image generation model upon obtaining a plurality of conditions for obtaining an image.

[0002] Among conventional generative artificial intelligence (AI) technologies, image generation models are mostly used to generate images by inputting text or to change existing images by inputting images and text. However, in order to create designs or new content using an image generation model, the image generation model should allow users to make and modify a new image while accurately reflecting a desired image, and thus the image generation model may be meaningfully used by the users. To this end, conventional image generation models should be well controlled, and the image generation models should be able to be integrated with various other AI technologies. Although OpenAI's DALL-E image generation model was previously released through the ChatGPT service, it does not include a technology related to a learning or editing function to properly control the image generation model, and thus there are many cases where images other than those appropriate for users are generated. Therefore, there is a need for a method and system for learning an image generation model so that a user may obtain a desired image.

[0003] The object of the present disclosure to be solved is to provide a target image by obtaining an image generation model that generates one or more images to which a user preference is applied after a plurality of conditions for generating an image are obtained when the image generation model is used.

[0004] Objects of the present disclosure are not limited to the above-described technical objects and other technical objects may be present.

[0005] As a technical means to achieve the above-mentioned technical objects, a first aspect of the present disclosure provides a method for a device to obtain images using an image generation model, which includes obtaining concept information, a reference image, additional condition information, and a customized target image for a target image; applying a reference keyword obtained from the reference image and the concept information to a language generation model; applying an image generation prompt obtained from the language generation model, the additional condition information, and the customized target image to an image generation model upon applying the reference keyword and the concept information; performing post-processing on an initial image generated by the image generation model; and obtaining a target image by performing post-processing on the initial image.

[0006] Further, the reference keyword may be obtained from a keyword extraction model by applying the reference image to the keyword extraction model.

[0007] Further, the language generation model may generate the image generation prompt using the concept information in a text format and the reference keyword.

[0008] Further, general characteristic information on the target image may be obtained from the reference image, and unique characteristic information on the target image may be obtained from the customized target image.

[0009] Further, the additional condition information may include at least one of a sketch-type image, a skeleton image, and a masking image.

[0010] Further, the language generation model may provide an output in a prompt format on the basis of an input in a text format, and the image generation model may provide an output in an image format on the basis of the input in the text format and an input in an image format.

[0011] Further, the method may further include applying a preference response prompt for reflecting a preference in the target image to an image reinforcement learning model; applying an updated target image, in which the target image is updated by the image reinforcement learning model, to the image generation model; and obtaining a customized target image, in which the preference is reflected, by performing post-processing on the updated target image.

[0012] Further, the obtaining of the customized target image may include obtaining a first preference learning model that learns characteristics of a structure in which the updated target image is changed from the target image; obtaining a second preference learning model that learns characteristics of a style in which the updated target image is changed from the target image; and obtaining the customized target image on the basis of the first preference learning model and the second preference learning model.

[0013] Further, the image generation model may generate a target shape for the target image on the basis of an outline for an image shape corresponding to at least one of the sketch-type image, the skeleton image, and the masking image and / or a numerical value for at least one of a width, a height, and an angle of the image shape, and the target image may include an image of the target shape.

[0014] A second aspect of the present disclosure provides a device for obtaining images using an image generation model, which includes a receiver configured to obtain concept information, a reference image, additional condition information, and a customized target image for a target image; and a processor configured to apply a reference keyword obtained from the reference image and the concept information to a language generation model, apply an image generation prompt obtained from the language generation model, the additional condition information, and the customized target image to an image generation model upon applying the reference keyword and the concept information, perform post-processing on an initial image generated by the image generation model, and obtain a target image by performing post-processing on the initial image.

[0015] Further, the reference keyword may be obtained from a keyword extraction model by applying the reference image to the keyword extraction model.

[0016] Further, the language generation model may generate the image generation prompt using the concept information in a text format and the reference keyword.

[0017] Further, general characteristic information on the target image may be obtained from the reference image, and unique characteristic information on the target image may be obtained from the customized target image.

[0018] Further, the additional condition information may include at least one of a sketch-type image, a skeleton image, and a masking image.

[0019] Further, the processor may allow the language generation model to provide an output in a prompt format on the basis of an input in a text format, and allow the image generation model to provide an output in an image format on the basis of the input in the text format and an input in an image format.

[0020] Further, the processor may apply a preference response prompt for reflecting a preference in the target image to an image reinforcement learning model, apply an updated target image, in which the target image is updated by the image reinforcement learning model, to the image generation model, and obtain the customized target image, in which the preference is reflected, by performing post-processing on the updated target image.

[0021] Further, the processor may obtain a first preference learning model that learns characteristics of a structure in which the updated target image is changed from the target image, obtain a second preference learning model that learns characteristics of a style in which the updated target image is changed from the target image, and obtain the customized target image on the basis of the first preference learning model and the second preference learning model.

[0022] Further, the image generation model may generate a target shape for the target image on the basis of an outline for an image shape corresponding to at least one of the sketch-type image, the skeleton image, and the masking image and / or a numerical value for at least one of a width, a height, and an angle of the image shape, and the target image may include an image of the target shape.

[0023] A third aspect of the present disclosure provides a non-transitory computer-readable recording medium on which a program for implementing the method of the first aspect is recorded.

[0024] According to an embodiment of the present disclosure, since an image generation artificial intelligence (AI) model is controlled to fit the purpose of generation, the efficiency of image generation can be increased, and a user's satisfaction can be improved.

[0025] Further, in companies, image generation models can be used to generate designs or new content that fit their company's characteristics simply by inputting keywords.

[0026] Further, since an image generation model is generated as a personalized model in which each user preference is reflected so that personalized results customized for each user can be generated, the user's satisfaction can be improved.

[0027] Effects of the present disclosure are not limited to the above-described effects, and other effects that are not described may be clearly understood by those skilled in the art from the following descriptions.

[0028] FIG. 1 is a block diagram schematically illustrating a configuration of a device according to an embodiment of the present disclosure.

[0029] FIG. 2 is a flowchart illustrating each operation of a method of obtaining, by a device according to an embodiment of the present disclosure, a target image.

[0030] FIG. 3 is a diagram schematically illustrating an example in which a device according to an embodiment of the present disclosure uses a language generation model and an image generation model.

[0031] FIG. 4 is a diagram schematically illustrating an example in which a device according to an embodiment of the present disclosure uses a keyword extraction model.

[0032] FIG. 5 is a diagram schematically illustrating an example in which a device according to an embodiment of the present disclosure obtains a target image.

[0033] FIG. 6 is a diagram schematically illustrating an example in which a device according to an embodiment of the present disclosure uses an image reinforcement learning model.

[0034] FIG. 7 is a diagram schematically illustrating an example in which an image generation model is trained to reflect a user preference through an image reinforcement learning model according to an embodiment of the present disclosure.

[0035] FIG. 8 is a diagram schematically illustrating an example in which a device according to an embodiment of the present disclosure determines a target shape of a target image.

[0036] FIG. 9 is a diagram schematically illustrating an example in which a device according to an embodiment of the present disclosure obtains a plurality of first target images using additional condition information, concept information, and a reference image.

[0037] FIG. 10 is a diagram schematically illustrating an example in which a device according to an embodiment of the present disclosure applies result values, which are derived by applying a plurality of first target images and a customized target image to an encoder model, to an image generation model.

[0038] FIG. 11 is a diagram schematically illustrating an example in which a device according to an embodiment of the present disclosure obtains a plurality of second target images using a plurality of first target images and a customized target image.

[0039] FIG. 12 is a diagram schematically illustrating functions performed by a device according to an embodiment of the present disclosure.

[0040] Advantages and features of the present disclosure and methods of achieving the same will be clearly understood with reference to the accompanying drawings and embodiments described in detail below. However, the present disclosure is not limited to the embodiments to be disclosed below but may be implemented in various different forms. The embodiments are provided in order to make the present embodiments complete and inform those skilled in the art of the scope of the present disclosure.

[0041] Terms used herein are provided only to describe the embodiments of the present disclosure and not for purposes of limitation. In this specification, the singular forms include the plural forms unless the context clearly indicates otherwise. It will be understood that terms “comprise”and / or “comprising”used herein specify some stated components, but do not preclude the presence or addition of one or more other components. Like reference numerals throughout the specification denote like components, and“and / or”includes each and every combination of one or more of the above-describe components. It should be understood that, although the terms “first,”“second,”etc. may be used herein to describe various components, these components are not limited by these terms. The terms are only used to distinguish one component from another component. Therefore, it should be understood that a first component to be described below may be a second component within the technical scope of the present disclosure.

[0042] Unless otherwise defined, all terms including technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art. Further, it should be further understood that terms, such as those defined in commonly used dictionaries, should not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0043] Spatially relative terms “below,”“beneath,”“lower,”“above,”“upper,”etc., may be used to facilitate the description of a relationship between one component and other components as illustrated in the accompanying drawings. The spatially relative terms should be understood to include different directions of the element during use or operation in addition to the direction illustrated in the accompanying drawings. For example, when a component illustrated in the drawing are flipped, a component described as “below”or “beneath" another component may end up being placed “above”the other component. Therefore, an exemplary term “below” may include both downward and upward directions. Components may be arranged in different directions so that spatially relative terms may be interpreted according to the arrangement.

[0044] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings.

[0045] FIG. 1 is a block diagram schematically illustrating a configuration of a device 100 according to an embodiment of the present disclosure.

[0046] Referring to FIG. 1, the device 100 may include a receiver 110 and a processor 120.

[0047] According to an embodiment, the receiver 110 may obtain concept information, a reference image, additional condition information, and a customized target image for a target image.

[0048] The processor 120 according to the embodiment may apply a reference keyword obtained from the reference image and concept information to a language generation model. Further, the processor 120 may apply an image generation prompt obtained from the language generation model, the additional condition information, and the customized target image to an image generation model upon applying the reference keyword and the concept information. Further, the processor 120 may perform post-processing on an initial image generated by the image generation model. Further, the processor 120 may obtain a target image by performing post-processing on the initial image.

[0049] As illustrated in FIG. 1, the processor 120 may obtain the target image by controlling the language generation model, a keyword extraction model, the image generation model, and an image post-processing model.

[0050] Further, the device 100 may be combined with a combination of various conventional networks, such as the Internet, mobile communication networks, etc., in the process in which the receiver 110 obtains the concept information, the reference image, the additional condition information, and the customized target image, and the processor 120 obtains the image generation prompt obtained by controlling the language generation model, the keyword extraction model, the image generation model, and the image post-processing model, and the target image corresponding to the initial image, and it should be noted that there are no specific limitation on this.

[0051] In addition, it will be understood by those skilled in the art that, in addition to the components illustrated in FIG. 1, other general components may be included in the device 100. For example, the device 100 may further include a memory (not illustrated) for storing the concept information, the reference image, the additional condition information, the customized target image, the initial image, etc., and may further include a display unit (not illustrated) for providing the target image. Alternatively, according to another embodiment, it will be understood by those skilled in the art that some components illustrated in FIG. 1 may be omitted.

[0052] The device 100 according to the embodiment may be used by a user, may be linked with any type of handheld-based wireless communication device equipped with a touch screen panel, such as a mobile phone, a smartphone, a personal digital assistant (PDA), a portable multimedia player (PMP), a tablet personal computer (PC), etc., and may also be included in or linked with a device that has a foundation for installing and executing an application, such as a desktop PC, a tablet PC, a laptop PC, an Internet Protocol television (IPTV) including a set-top box.

[0053] The device 100 may be implemented as a terminal such as a computer that operates through a computer program to realize the functions described in this specification.

[0054] The device 100 according to the embodiment may include a target image generation system (not illustrated) and a related server (not illustrated), but the present disclosure is not limited thereto. The device 100 according to the embodiment and the server may support an application that generates and provides a target image.

[0055] Hereinafter, the description will focus on an embodiment in which the device 100 according to the embodiment independently generates a target image and an embodiment in which the device 100 according to the embodiment obtains a target image generation model, but as described above, the device 100 may be performed in conjunction with a server. That is, it can be seen that the device 100 and server according to the embodiment may be integrally implemented in terms of their functions, the server may be omitted, and the present disclosure is not limited to any one embodiment.

[0056] In an embodiment, the device 100 and the server may be linked, and the function for providing a target image by performing a target image generation process may be performed by the server or by the device 100. For example, the device 100 may operate as a server, and will be collectively referred to as the device 100 in the following descriptions.

[0057] FIG. 2 is a flowchart illustrating each operation of a method of obtaining, by a device 100 according to the embodiment of the present disclosure, a target image.

[0058] In operation S210, the device 100 according to the embodiment may obtain concept information, a reference image, additional condition information, and a customized target image for a target image.

[0059] In an embodiment, the concept information is information in the text format on a concept used for obtaining the target image and may be type-specific text indicating the type of object that is a target of the target image. For example, when the object is a “refrigerator,”type-specific text indicating concept information may be “mini-fridge,”"double-door refrigerator," etc., and when the object is a “car,”the type-specific text may be “sedan,”“sport utility vehicle (SUV),”etc. Further, the concept information may include image-specific text for determining an image of the object that is the target of the target image. For example, an example may be performed in which, when the image-specific text indicating concept information on any object is “cool,” sky blue is determined as the color of the object so that cool concept may be reflected according to a preset algorithm corresponding to the case in which text such as season, weather, etc. is input, and when the image-specific text is “mantis,”green, which is the color of the mantis, is determined as the color of the object, or the shape of the object is determined to correspond to the appearance characteristics of the mantis, according to a preset algorithm corresponding to the case in which text such as animal, insect, etc. is input. In an embodiment, the reference image may be an image that directly indicates the object that is the target of the target image or an image for determining the type of the object. When an example image containing the object is obtained as the reference image, the target image may be obtained as an image containing the same object as the object contained in the reference image. For example, when a reference image containing a “refrigerator”is obtained, the object of the target image may be determined to be a “refrigerator,”and when a reference image containing a “double-door refrigerator”is obtained, the type of the object of the target image may be determined to be a “double-door refrigerator.”That is, general characteristic information on the target image may be obtained from the reference image. The general characteristic information may be information corresponding to the type of object. In an embodiment, the additional condition information may include at least one of a sketch-type image, a skeleton image, and a masking image. For example, the additional condition information may be additional information for determining the schematic form and shape of the object within the target image. The device 100 may obtain a target image including an object having a form and shape corresponding to an outline and skeleton of at least one of a sketch-type image, a skeleton image, and a masking image. In an embodiment, the customized target image may be an image for obtaining a design, logo, or representative identity characteristic corresponding to the brand of the object (product). For example, an object (product) of the same brand as the object (product) included in the customized target image may be determined as the object included in the target image, or a special shape (e.g., a doughnut shape) of the object included in the customized target image may be reflected in the shape of the object included in the target image. That is, unique characteristic information on the target image may be obtained from the customized target image. The unique characteristic information may be information corresponding to the object's brand or unique special shape.

[0060] In operation S220, the device 100 according to the embodiment may apply a reference keyword obtained from the reference image and the concept information to a language generation model. In an embodiment, the device 100 may obtain one or more reference keywords that can indicate the characteristics of the reference image from the reference image. The concept information may be in the text format. The language generation model may obtain an image generation prompt using the concept information in the text format and the reference keywords. The image generation prompt may include text (e.g., a command, a question, etc.) that requests the generation of an image.

[0061] In operation S230, the device 100 according to the embodiment may apply an image generation prompt, additional condition information, and a customized target image that are obtained from the language generation model to an image generation model upon applying the reference keyword and the concept information. That is, when the image generation prompt generated by the language generation model is applied to the image generation model by applying the concept information in the text format and the reference keyword, the image generation model may obtain one or more initial images for obtaining the target image using the concept information, the reference image, the additional condition information, and the customized target image. This will be described with reference to FIGS. 3 and 4.

[0062] FIG. 3 is a diagram schematically illustrating an example in which the device 100 according to the embodiment of the present disclosure uses the language generation model and the image generation model.

[0063] Referring to FIG. 3, the image generation prompt, which is generated by applying the concept information and the reference image to the language generation model, may be applied to the image generation model. In an embodiment, the language generation model may provide an output in the prompt format on the basis of an input in the text format. That is, the language generation model may receive concept information in the text format and reference keywords as inputs and output an image generation prompt in the prompt format, which is generated in response to the concept information and the reference keywords. Further, the customized target image and the additional condition information may be applied together with the image generation prompt to the image generation model. The image generation model may provide an output in the image format on the basis of the input in the text format and an input in the image format. In an embodiment, the customized target image may be in the image format containing unique characteristic information, and the additional condition information may be in the form of both an image and text format. That is, the image generation model may receive the additional condition information which may be in the text format, the image generation prompt, and the customized target image in the image format as inputs and output an initial image in the image format, which is generated in response to the additional condition information, the image generation prompt, and the customized target image.

[0064] Specifically, the device 100 may perform an image generation process in the order of applying the customized target image to the image generation model, applying the concept information and the reference image to the language generation model, and applying the image generation prompt and the additional condition information to the image generation model. In an embodiment, the image generation prompt applied to the image generation model may be in the form of prompt that enables a target image to be obtained that is generated in a direction preferred by a user in response to concept information and reference keywords that are input based on a preset learning algorithm.

[0065] FIG. 4 is a diagram schematically illustrating an example in which the device 100 according to the embodiment of the present disclosure uses a keyword extraction model.

[0066] Referring to FIG. 4, the device 100 according to the embodiment may apply the reference image to the keyword extraction model. Therefore, reference keywords may be obtained from the keyword extraction model by applying the reference image to the keyword extraction model. The reference keywords obtained from the keyword extraction model may include one or more keywords corresponding to an object and / or shape included in the reference image.

[0067] In operation S240, the device 100 according to the embodiment may perform post-processing on the initial image generated by the image generation model.

[0068] In operation S250, the device 100 according to the embodiment may obtain the target image by performing post-processing on the initial image. This will be described with reference to FIG. 5.

[0069] FIG. 5 is a diagram schematically illustrating an example in which the device 100 according to the embodiment of the present disclosure obtains the target image.

[0070] Referring to FIG. 5, the image generation model may apply the image generation prompt, the customized target image, and the additional condition information, which are input by the language generation model, to the image post-processing model. The image post-processing model may process the initial image, which is generated by the image generation prompt, the customized target image, and the additional condition information, according to a learning algorithm to generate a plurality of target images. The plurality of target images may include images of the same type of object but with different shapes, different locations, or different background areas other than the object. That is, a plurality of different target images may be generated according to a preset learning algorithm. In an embodiment, it may include an example in which the image post-processing model is included in the image generation model.

[0071] The device 100 according to the embodiment may apply a preference response prompt for reflecting a preference in the target image to an image reinforcement learning model. This will be described with reference to FIG. 6.

[0072] FIG. 6 is a diagram schematically illustrating an example in which the device 100 according the an embodiment of the present disclosure uses the image reinforcement learning model.

[0073] Referring to FIG. 6, the device 100 according to the embodiment may perform post-processing on the initial image to obtain the target image. The device 100 may store the obtained target image in a generated output image database. The device 100 may apply the stored target image to the image reinforcement learning model, and the image reinforcement learning model may output an updated target image updated after a preference is reflected in the target image. The device 100 may apply the updated target image, in which the target image is updated by the image reinforcement learning model, to the image generation model. This will be described with reference to FIG. 7.

[0074] FIG. 7 is a diagram schematically illustrating an example in which the image generation model is trained to reflect a user preference through the image reinforcement learning model according to an embodiment of the present disclosure.

[0075] Referring to FIG. 7, the device 100 according to the embodiment may receive feedback information for reflecting a preference from a user account for the plurality of target images obtained through operations S210 to S250. For example, the device 100 may provide the plurality of target images obtained based on the concept information, the reference image, the additional condition information, and the customized target image that are obtained from the user account to the user account and obtain feedback information for each of the plurality of target images from the user account. The device 100 may display the plurality of target images and provide characteristic information corresponding to each target image. The user may input feedback information indicating good or bad for the displayed target images and the characteristic information (e.g., color information, size information, etc.) of the target images. Therefore, the device 100 may obtain preference information of the user account on the basis of the feedback information for the plurality of target images received from the user account using the image reinforcement learning model. The image reinforcement learning model may correspond to a first image generation model of FIG. 7 and may be a learning model that is continuously trained so that the preference information of the user account is applied. Specifically, the first image generation model may be a diffusion-based image generative AI model. The device 100 may train the image reinforcement learning model so that the customized target image in which the preference information of the user account is reflected may be obtained by continuously applying the plurality of target images to the image reinforcement learning model (first image generation model) to obtain a plurality of updated target images. Further, the device 100 may obtain an optimized image reinforcement learning model that corresponds to the user account by being reinforced so that so that the user preference may be reflected in detail by continuously and repeatedly performing a process of applying the plurality of updated target images or a plurality of newly obtained target images to the image reinforcement learning model, which is learned and reinforced so that the preference is reflected. Therefore, the image reinforcement learning model in which the user's preferred outline shape, degree of color reflection, degree of characteristics deformation, direction of characteristics reflection, etc., are generated to be customized to each individual may be obtained. The device 100 may obtain a customized target image, in which the preference is reflected, by performing post-processing on the updated target image.

[0076] FIG. 8 is a diagram schematically illustrating an example in which the device 100 according to the embodiment of the present disclosure determines a target shape of the target image.

[0077] Referring to FIG. 8, the image generation model according to the embodiment may obtain the target shape of the target image on the basis of the outline for the image shape corresponding to at least one of the sketch-type image, skeleton image, and masking image that are obtained as the additional condition information, and / or the numerical value for at least one of a width, a height, and an angle of the image shape. Specifically, a reflection image in which a viewpoint / ratio is reflected may be generated by applying the sketch-type image, the skeleton image, and the masking image to a generalized depth control model. An output value output from the depth control model may include a target output value that enables the generation of an image of a new shape and form, and the target output value may be applied to a second image generation model as an input value. The device 100 may apply information on the target shape obtained through the depth control model to the second image generation model (DRAG-human preference diffusion-based image generative AI model) as an input value. In an embodiment, the second image generation model may correspond to an image reinforcement learning model which is reinforced so that the user preference may be reflected in detail, according to the process of FIG. 7. Therefore, the device 100 may obtain a target image having a target shape corresponding to the image obtained with the additional condition information. That is, the target image may include an image of the target shape.

[0078] FIG. 9 is a diagram schematically illustrating an example in which the device 100 according to the embodiment of the present disclosure obtains a plurality of first target images using the additional condition information, the concept information, and the reference image.

[0079] Referring to FIG. 9, the device 100 according to the embodiment may apply the information on the target shape obtained through the depth control model to the second image generation model (DRAG-human preference diffusion-based image generative AI model), and apply the concept information and the reference image that are obtained from the user account to the second image generation model. Therefore, the second image generation model may obtain a plurality of first target images (Image A, Image B, and Image C) by performing post-processing as described in operations S230 and S240. Specifically, the device 100 may obtain a reference image corresponding to an object to be included in the target image as the reference image from the user account and obtain animal-corresponding text including an animal keyword and an adjective-corresponding text including an adjective keyword as the concept information. The device 100 may automatically reinforce a text prompt required for text-to-image model inference on the basis of a design generation prompt auxiliary model included in the image generation model. When the animal-corresponding text is obtained as the concept information, the device 100 may allow the obtained characteristics of the animal to be reflected in the image according to the user preference. For example, when the obtained animal-corresponding text is “mantis,”a direction in which the shape of the mantis is reflected according to the user preference may be determined differently, rather than the shape of the mantis being reflected as is on the object. Specifically, the device 100 may allow the shape of the object to be reflected according to an algorithm for reflecting the shape of an animal for each user, which is determined differently for each user according to the user preference, such as changing the shape of an existing object to a sharp and pointed shape according to the sharp and pointed shape of the mantis. Further, when the adjective-corresponding text is obtained as the concept information, the obtained characteristics of the adjective may be reflected in the image according to the user preference. For example, the obtained adjective-corresponding text may include sharp, high, wide, etc., and the shape indicated by the adjective may be reflected in the shape of the object. As another example, the type of object may be determined through an adjective. For example, when the adjective-corresponding text is “wide”and the object is “bed,”the device 100 may allow the shape of a queen, king, or large king size to be reflected in the shape of the object.

[0080] Therefore, the device 100 may obtain a 1-1 target image (Image A) obtained in response to the reference image and the additional condition information, a 1-2 target image (Image B) obtained in response to the reference image and the animal-corresponding text, and a 1-3 target image (Image C) obtained in response to the reference image and the adjective-corresponding text. In FIG. 9, three first target images are illustrated, but the present disclosure is not limited thereto, and three or more first target images may be obtained.

[0081] FIG. 10 is a diagram schematically illustrating an example in which the device 100 according to the embodiment of the present disclosure applies result values, which are derived by applying the plurality of first target images and the customized target image to an encoder model, to the image generation model.

[0082] Referring to FIG. 10, the device 100 according to the embodiment may apply the plurality of first target images and the customized target image to the encoder model. Specifically, the encoder model may be an image-text embedding encoder model. That is, in a process of deriving semantic relationships between the image and the text, style information, such as color and material relationships, may be obtained between the plurality of first target images and the customized target image through the learned encoder model, and structural information, such as structure and composition relationships, may be obtained. Therefore, the style information and structural information output from the encoder model may be applied to the second image generation model.

[0083] FIG. 11 is a diagram schematically illustrating an example in which the device 100 according to the embodiment of the present disclosure obtains a plurality of second target images using the plurality of first target images and the customized target image.

[0084] Referring to FIG. 11, the device 100 according to the embodiment may provide the style information and structural information applied to the second image generation model to the ControlNet Detph. For example, ControlNet may be an example of an additional adapter neural network that can apply input values, such as outlines, DepthMap, etc., when generating images based on an image generation model.

[0085] The device 100 may obtain a plurality of second target images (Image D, Image E, and Image F) in which the plurality of first target images are updated by the second image generation model.

[0086] In an embodiment, customized target images obtained by performing post-processing on the updated images may include the plurality of first target images and the plurality of second target images. That is, both the second image generation model and the image post-processing model are included in each of the process of obtaining the plurality of first target images and the process of obtaining the plurality of second target images performed after obtaining the plurality of first target images, so that post-processing may be performed on the plurality of first updated target images and the plurality of second updated target images that are updated by the second image generation model generated to reflect the preference of the user account. Therefore, the plurality of first target images and the plurality of second target images may be obtained as customized target images.

[0087] In an embodiment, the device 100 may obtain a first preference learning model that learns the characteristics of a structure in which the updated target image is changed from the target image. Further, the device 100 may obtain a second preference learning model that learns the characteristics of a style in which the updated target image is changed from the target image. The device 100 may obtain customized target images on the basis of the first preference learning model and the second preference learning model. The first preference learning model may be a model trained to reflect the characteristics of the corresponding structure in an image in response to the structural information output from the encoder model. Further, the second preference learning model may be a model trained to reflect the characteristics of the corresponding style in an image in response to the style information output from the encoder model. Therefore, the device 100 may use the first preference learning model and the second preference learning model to obtain a customized target image in which the user preference is reflected. Specifically, the first preference learning model and the second preference learning model may be models to which a personalization fine-tuning technique is applied. That is, according to the personalization fine-tuning technique, it is possible to train to identify individual characteristics (identity) of specific objects that cannot be expressed by a general-purpose generative model. For example, the device 100 may obtain a 2-1 target image (Image D) generated by the second image generation model and the image post-processing model on the basis of the style information and structural information output from the encoder model. The device 100 may obtain a 2-2 target image (Image E) from the second image generation model and the image post-processing model on the basis of preference structure information output by applying the 2-1 target image to the first preference learning model. Further, the device 100 may obtain a 2-3 target image (Image F) from the second image generation model and the image post-processing model on the basis of preference style information output by applying the 2-1 target image to the second preference learning model. Therefore, a plurality of customized target images in which user preferences are reflected may be obtained. In FIG. 11, an example in which the 2-2 target image and the 2-3 target image are obtained using the first preference learning model and the second preference learning model, respectively, is illustrated, but the present disclosure is not limited thereto, and a 2-4 target image, in which both the preference structure information and the preference style information are reflected may be obtained, using both the first preference learning model and the second preference learning model. As another example, a 2-4-1 target image may be obtained by applying images in the order of the first preference learning model and the second preference learning mode, and a 2-4-2 target image may be obtained by applying images in the order of the second preference learning model and the first preference learning model, according to the importance of structure and style.

[0088] FIG. 12 is a diagram schematically illustrating functions performed by the device 100 according to the embodiment of the present disclosure.

[0089] Referring to FIG. 12, the device 100 according to the embodiment may perform an AI-based image generation model fine-tuning function (image Gen_AI model fine-tuning function), an image generation function, an image editing function, a prompt assistance function, and an evaluation function.

[0090] Specifically, as described above, the device 100 may apply a personalization fine-tuning technique to the image generation model and perform the image generation model fine-tuning function (image Gen_AI Model fine-tuning function) that reflects feedback information of a user. Further, the device 100 may perform an image generation function in which text is received as an input through the image generation model to generate an image, an image is received as an input to automatically generate a similar image or generate image content, and outlines, masking areas, etc., which correspond to objects included in the additional condition information, are generated as input values other than the text and the image. Further, the device 100 may perform an image editing function in which partial regeneration or extended regeneration is performed on an image, a background is removed from an image, an object is automatically extracted, an object is separated and extracted based on a prompt, and an image is converted into a high-resolution image. Further, the device 100 may perform a prompt assistance function in which an image generation prompt is automatically reinforced or a prompt for extracting a reference keyword from a reference image is assisted. Further, the device 100 may continuously train and update the image generation model by performing an evaluation function for the AI-based image generation model.

[0091] According to an embodiment, since an image generation AI model can be controlled to fit the purpose of generation, the efficiency of image generation can be increased, and the user's satisfaction can be improved. Further, in companies, an image generation model can be used to generate designs or new content that fit their company's characteristics simply by inputting keywords, and since the image generation model is generated as a personalized model in which each user preference is reflected so that personalized results customized for each user can be generated, the user's satisfaction can be improved.

[0092] Various embodiments of the present disclosure may be implemented as software including one or more instructions stored in a storage medium (e.g., a memory) that can be read by a machine (e.g., a display device or a computer). For example, a processor (e.g., the processor 120) of the machine may call at least one of the stored instructions from the storage medium and execute the instructions. This enables the device to operate to perform at least one function in accordance with the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The storage medium readable by the device may be provided in the form of a non-transitory storage medium. Here, the term “y” means only that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and this term does not distinguish between a case where data is stored semi-permanently and a case where data is stored temporarily in the storage medium.

[0093] According to one embodiment, the method according to various embodiments disclosed in the present disclosure may be included in a computer program product and provided. The computer program product may be traded between a seller and a buyer as a commodity. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., by download or upload) through an application store (e.g., Play Store TM) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as a memory of a manufacturer's server, an application store's server, or an intermediary server.

[0094] While the present disclosure has been described with reference to the accompanying drawings, it is not limited to the disclosed embodiments and drawings, and it will be understood by those skilled in the art that various changes in form and details may be made without departing from the spirit and scope of the present disclosure. Therefore, the disclosed methods should be considered from an exemplary point of view for description rather than a limiting point of view. Even when the embodiments are described and the effects according to the configuration of the present disclosure are not explicitly described, effects that may be predicted by the configuration may also be recognized. The scope of the present disclosure is defined not by the detailed description of the present disclosure but by the appended claims and encompasses all modifications and equivalents that fall within the scope of the appended claims and will be construed as being included in the present disclosure.

Claims

1.A method of obtaining images using an image generation model, the method comprising:obtaining concept information, a reference image, additional condition information, and a customized target image for a target image;applying a reference keyword obtained from the reference image and the concept information to a language generation model;applying an image generation prompt obtained from the language generation model, the additional condition information, and the customized target image to an image generation model upon applying the reference keyword and the concept information;performing post-processing on an initial image generated by the image generation model; andobtaining a target image by performing post-processing on the initial image.2.The method of claim 1, wherein the reference keyword is obtained from a keyword extraction model by applying the reference image to the keyword extraction model.3.The method of claim 1, wherein the language generation model generates the image generation prompt using the concept information in a text format and the reference keyword.4.The method of claim 1, wherein general characteristic information on the target image is obtained from the reference image, andunique characteristic information on the target image is obtained from the customized target image.5.The method of claim 1, wherein the additional condition information includes at least one of a sketch-type image, a skeleton image, and a masking image.6.The method of claim 1, wherein the language generation model provides an output in a prompt format on the basis of an input in a text format, andthe image generation model provides an output in an image format on the basis of the input in the text format and an input in an image format.7.The method of claim 1, further comprising:applying a preference response prompt for reflecting a preference in the target image to an image reinforcement learning model;applying an updated target image, in which the target image is updated by the image reinforcement learning model, to the image generation model; andobtaining a customized target image, in which the preference is reflected, by performing post-processing on the updated target image.8.The method of claim 7, wherein the obtaining of the customized target image comprising:obtaining a first preference learning model that learns characteristics of a structure in which the updated target image is changed from the target image;obtaining a second preference learning model that learns characteristics of a style in which the updated target image is changed from the target image; andobtaining the customized target image on the basis of the first preference learning model and the second preference learning model.9.The method of claim 5, wherein the image generation model generates a target shape for the target image on the basis of an outline for an image shape corresponding to at least one of the sketch-type image, the skeleton image, and the masking image and / or a numerical value for at least one of a width, a height, and an angle of the image shape, andthe target image includes an image of the target shape.10.A device for obtaining images using an image generation model, the device comprising:a receiver configured to obtain concept information, a reference image, additional condition information, and a customized target image for a target image; anda processor configured to apply a reference keyword obtained from the reference image and the concept information to a language generation model,apply an image generation prompt obtained from the language generation model, the additional condition information, and the customized target image to an image generation model upon applying the reference keyword and the concept information,perform post-processing on an initial image generated by the image generation model, andobtain a target image by performing post-processing on the initial image.11.The device of claim 10, wherein the reference keyword is obtained from a keyword extraction model by applying the reference image to the keyword extraction model.12.The device of claim 10, wherein the language generation model generates the image generation prompt using the concept information in a text format and the reference keyword.13.The device of claim 10, wherein general characteristic information on the target image is obtained from the reference image, andunique characteristic information on the target image is obtained from the customized target image.14.The device of claim 10, wherein the additional condition information includes at least one of a sketch-type image, a skeleton image, and a masking image.15.The device of claim 10, wherein the processor is configured to:allow the language generation model to provide an output in a prompt format on the basis of an input in a text format; andallow the image generation model to provide an output in an image format on the basis of the input in the text format and an input in an image format.16.The device of claim 10, wherein the processor is configured to:apply a preference response prompt for reflecting a preference in the target image to an image reinforcement learning model;apply an updated target image, in which the target image is updated by the image reinforcement learning model, to the image generation model; andobtain the customized target image, in which the preference is reflected, by performing post-processing on the updated target image.17.The device of claim 16, wherein the processor is configured to:obtain a first preference learning model that learns characteristics of a structure in which the updated target image is changed from the target image;obtain a second preference learning model that learns characteristics of a style in which the updated target image is changed from the target image; andobtain the customized target image on the basis of the first preference learning model and the second preference learning model.18.The device of claim 14, wherein the image generation model generates a target shape for the target image on the basis of an outline for an image shape corresponding to at least one of the sketch-type image, the skeleton image, and the masking image and / or a numerical value for at least one of a width, a height, and an angle of the image shape, andthe target image includes an image of the target shape.19.A non-transitory computer-readable recording medium on which a program for implementing the method of claims 1 is recorded.

Images