Method and electronic device for providing personalized image

Abstract

Provided is a method for an electronic device to generate a personalized image. The method comprises the steps of: obtaining a prompt for generating an image; extracting a keyword from the prompt; searching for one or more reference images corresponding to the keyword among a plurality of images stored in the electronic device; segmenting an area related to the keyword within each of the one or more reference images; obtaining image feature information on the basis of the area related to the keyword; transmitting the prompt and the image feature information to a server; and receiving, from the server, a personalized image generated by providing the prompt and the image feature information as inputs to a generative model.

Description

Method and electronic device for providing personalized images

[0001] The present disclosure relates to a method, electronic device, server and system for generating a personalized image and providing it to a user.

[0002] Generative AI can refer to technologies that learn the structure and patterns of large-scale data and generate new synthetic data based on input data. In some implementations, generative AI can be used to produce human-level results in a variety of tasks involving text, images, speech, video, music, and more. For example, a generative model can be used to generate new images based on given text, or to generate new images based on given images.

[0003] However, even if an image generation model can generate an image containing content that matches the given input text or image, it may be difficult to obtain results tailored to a specific user.

[0004] According to one aspect of the present disclosure, a method for an electronic device to provide a personalized image may be provided. The method may include obtaining a prompt for image generation. The method may include extracting a keyword from the prompt. The method may include searching for one or more reference images corresponding to the keyword among a plurality of images stored in the electronic device. The method may include segmenting an area associated with the keyword within the reference image. The method may include obtaining image feature information based on the area associated with the keyword within the reference image. The method may include transmitting the prompt and the image feature information to a server. The method may include receiving, from the server, a personalized image generated using a generation model that receives the prompt and the image feature information as inputs.

[0005] According to one aspect of the present disclosure, an electronic device for providing personalized images may be provided. The electronic device may include a communication interface; a memory storing one or more instructions; and one or more processors executing the one or more instructions stored in the memory. The one or more processors may obtain a prompt for image generation by executing the one or more instructions. The one or more processors may extract a keyword from the prompt by executing the one or more instructions. The one or more processors may search for one or more reference images corresponding to the keyword among a plurality of images stored in the electronic device by executing the one or more instructions. The one or more processors may segment an area associated with the keyword within the reference image by executing the one or more instructions. The one or more processors may obtain image feature information based on an area associated with the keyword within the reference image by executing the one or more instructions. The one or more processors may transmit the prompt and the image feature information to a server through the communication interface by executing the one or more instructions. The one or more processors may receive, from the server through the communication interface, a personalized image generated using a generation model that receives the prompt and the image feature information as input.

[0006] According to one aspect of the present disclosure, a method for providing a personalized image by a server may be provided. The method may include obtaining a prompt for image generation. The method may include extracting a keyword from the prompt. The method may include searching for one or more reference images corresponding to the keyword among a plurality of images stored in the electronic device. The method may include segmenting an area associated with the keyword within the reference image. The method may include obtaining image feature information based on the area associated with the keyword within the reference image. The method may include generating the personalized image using a generative model. The generative model may be an artificial intelligence model that receives the prompt and the image feature information as inputs and outputs the personalized image.

[0007] According to one aspect of the present disclosure, a server for generating a personalized image may be provided. The server may include a communication interface; a memory storing one or more instructions; and one or more processors for executing the one or more instructions stored in the memory. The one or more processors may obtain a prompt for image generation by executing the one or more instructions. The one or more processors may extract a keyword from the prompt by executing the one or more instructions. The one or more processors may search for one or more reference images corresponding to the keyword among a plurality of images stored in the electronic device by executing the one or more instructions. The one or more processors may segment an area associated with the keyword within the reference image by executing the one or more instructions. The one or more processors may obtain image feature information based on an area associated with the keyword within the reference image by executing the one or more instructions. The one or more processors may generate a personalized image using a generative model by executing the one or more instructions. The generative model may be an artificial intelligence model that receives the prompt and the image feature information as input and outputs the personalized image.

[0008] According to one aspect of the present disclosure, a computer-readable recording medium having recorded thereon a program for causing an electronic device and / or a server to execute any one of the aforementioned and hereinafter described methods for providing a personalized image may be provided.

[0009] FIG. 1 is a diagram schematically illustrating an electronic device according to one embodiment of the present disclosure providing a personalized image.

[0010] FIG. 2 is a flowchart illustrating an operation of an electronic device generating a personalized image according to one embodiment of the present disclosure.

[0011] FIG. 3 illustrates the operation of an electronic device and a server according to one embodiment of the present disclosure.

[0012] FIG. 4 is a diagram for explaining an operation of an electronic device extracting a keyword according to one embodiment of the present disclosure.

[0013] FIG. 5A is a diagram for explaining an operation of an electronic device searching for an image according to one embodiment of the present disclosure.

[0014] FIG. 5b is a diagram for explaining an operation of an electronic device searching for an image according to one embodiment of the present disclosure.

[0015] FIG. 6 is a diagram for explaining an operation of an electronic device segmenting an image according to one embodiment of the present disclosure.

[0016] FIG. 7 is a diagram illustrating an operation of an electronic device generating a vector representation according to one embodiment of the present disclosure.

[0017] FIG. 8 is a diagram illustrating an operation of a server generating a personalized image according to one embodiment of the present disclosure.

[0018] FIG. 9 is a diagram illustrating an example of an electronic device providing a personalized image according to one embodiment of the present disclosure.

[0019] FIG. 10A is a flowchart illustrating the operation of an electronic device and a server according to one embodiment of the present disclosure.

[0020] FIG. 10b is a flowchart illustrating additional operations of an electronic device and a server according to one embodiment of the present disclosure.

[0021] FIG. 11 is a diagram illustrating an operation of an electronic device according to one embodiment of the present disclosure to provide image feature information to a user.

[0022] FIG. 12 is a diagram for explaining an operation of an electronic device generating or processing a prompt according to one embodiment of the present disclosure.

[0023] FIG. 13 is a diagram illustrating an operation of an electronic device providing a personalized image according to one embodiment of the present disclosure.

[0024] FIG. 14A is a diagram illustrating an operation of an electronic device providing a personalized image according to one embodiment of the present disclosure.

[0025] FIG. 14b is a diagram illustrating an operation of an electronic device providing a personalized image according to an embodiment of the present disclosure.

[0026] FIG. 14c is a diagram illustrating an operation of an electronic device providing a personalized image according to one embodiment of the present disclosure.

[0027] FIG. 14d is a diagram illustrating an operation of an electronic device providing a personalized image according to an embodiment of the present disclosure.

[0028] FIG. 15 is a diagram illustrating an operation of an electronic device providing a personalized image according to one embodiment of the present disclosure.

[0029] FIG. 16 is a diagram for explaining an operation of an electronic device according to one embodiment of the present disclosure to provide information related to a personalized image.

[0030] FIG. 17 is a diagram illustrating an operation of an electronic device providing a personalized image according to one embodiment of the present disclosure.

[0031] FIG. 18a is a block diagram illustrating a configuration of an electronic device according to one embodiment of the present disclosure.

[0032] FIG. 18b is a block diagram illustrating a configuration of an electronic device according to one embodiment of the present disclosure.

[0033] FIG. 19a is a block diagram illustrating a configuration of a server according to one embodiment of the present disclosure.

[0034] FIG. 19b is a block diagram illustrating a configuration of a server according to one embodiment of the present disclosure.

[0035] Hereinafter, terms used in this specification will be briefly described, and the present disclosure will be described in detail. In this disclosure, the expression “at least one of a, b, or c” can refer to “a,” “b,” “c,” “a and b,” “a and c,” “b and c,” “all of a, b, and c,” or variations thereof.

[0036] The terms used in this disclosure are selected from widely used, common terms, taking into account the functions of the disclosure. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, in which case their meanings will be described in detail in the relevant description. Therefore, the terms used in this disclosure should not be defined simply as names, but rather based on the meanings of the terms and the overall content of the disclosure.

[0037] Singular expressions may include plural expressions unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art described herein. Furthermore, terms containing ordinal numbers, such as "first" or "second," used herein may be used to describe various components, but such components should not be limited by such terms. Such terms are used solely to distinguish one component from another.

[0038] When a part of the specification is said to "include" a component, unless otherwise specifically stated, this does not exclude other components but rather implies the inclusion of other components. Furthermore, terms such as "part" and "module" used in the specification refer to a unit that processes at least one function or operation, which may be implemented in hardware, software, or a combination of hardware and software.

[0039] The present disclosure relates to a method for generating a personalized image by reflecting an image owned by a user (e.g., an image stored on the user's device, such as a smartphone) when generating an image according to a user's request using a generative model. In embodiments, a second image generated by reflecting a first image may refer to a second image generated with respect to the first image, taking the first image into consideration, or based on the first image. According to an embodiment of the present disclosure, a synthetic image can be obtained by reflecting an image that was not used during the training process of the generative model during the inference (e.g., image generation) of the generative model.

[0040] In this disclosure, a "personalized image" may refer to synthetic data (e.g., a synthetic image) generated using a generation model in response to a user's image generation request. Furthermore, in this disclosure, a "personalized image" may refer to an image generated by a generation model that reflects an image held by the user. For example, a personalized image may be generated to include the user's personalized characteristics (e.g., a specific object, an image style, etc.). A personalized image may also be referred to as a "custom image."

[0041] In the present disclosure, "generative artificial intelligence" may refer to an artificial intelligence technology capable of generating new text, images, etc. in response to input data (e.g., text, images, etc.). A "generative model" may refer to a neural network model that implements generative AI technology. The generative model can generate new data having similar characteristics to the input data or new data corresponding to the input data by learning the patterns and structures of training data. For example, the generative model may generate an image using an image-to-image method or a text-to-image method.

[0042] In this disclosure, a "reference image" may refer to data referenced during the process of creating a personalized image. The reference image may include elements that may represent the user's characteristics (e.g., the user or people around the user, objects owned by the user, the space around the user, etc.).

[0043] Below, with reference to the attached drawings, embodiments of the present disclosure are described in detail so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In the drawings, irrelevant parts have been omitted for clarity, and similar parts have been designated with similar reference numerals throughout the disclosure.

[0044] The present disclosure will be described below with reference to the attached drawings.

[0045] FIG. 1 is a diagram schematically illustrating an electronic device according to one embodiment of the present disclosure providing a personalized image.

[0046] In one embodiment, the electronic device (2000) and / or the server (3000) may provide a user (100) with an image personalization service that generates an image using a generative model. The image personalization technology of the present disclosure allows the user (100) to generate a new image, which is synthetic data that includes his / her own personalized features, using the image(s) he / she owns.

[0047] For example, when a user (100) inputs the prompt "draw me a puppy" to the electronic device (2000) to request the creation of an image, the electronic device (2000) may search and retrieve, or otherwise acquire, one or more images (120) related to the keyword "puppy" from among a personal image collection (110) stored in the media storage of the electronic device (2000). Here, the personal image collection (110) may include personal images owned by the user that have not been used in the training process of the generative model.

[0048] The electronic device (2000) can process one or more retrieved images (120) to generate data for image personalization. The data for image personalization may refer to embedded data that converts one or more images (120) into vector representation data. The data for image personalization generated from images held by the user is transmitted to the server (3000), and the server (3000) can generate a personalized image (130) by using the data for image personalization received from the electronic device (2000) when performing an inference operation using a generation model.

[0049] The server (3000) can transmit a personalized image (130) to the electronic device (2000), and the electronic device (2000) can provide the personalized image (130) to the user (100). The personalized image (130) can include features of one or more images (120) retrieved from the electronic device (2000). For example, based on a request from the user (100) to “draw a puppy,” a personalized image (130) depicting a puppy (e.g., a puppy raised by the user) included in an image held by the user (100) can be provided to the user.

[0050] In one embodiment, the server (3000) can provide an image personalization service to multiple different users. When the server (3000) provides image personalization to multiple users, the generation model does not need to be additionally trained on each user's individual image collection (110). In other words, the server (3000) does not provide each user with a generation model that has been custom-trained with each user's individualized features. Instead, the server can use a commonly available, deployed generation model to generate a personalized image (130) that reflects features unique to each user's image. To this end, the electronic device (2000) generates data for image personalization and provides it to the server (3000). Here, "the model has been deployed" means that the model has been transferred to an actual operating environment so that it can perform inference operations, rather than an environment in which the model is trained (e.g., pre-trained and / or fine-tuned trained) or tested. In other words, "deployment" refers to the process of making a model ready for service after training, performance verification, and service provision. However, since the service may be provided to individuals and / or a limited group, distribution does not necessarily mean public release.

[0051] In the present disclosure, an image held by a user that is used as reference data for image personalization is referred to as a “reference image,” and data for image personalization generated using the “reference image” is referred to as “image feature information.”

[0052] In one embodiment, the electronic device (2000) may be various types of devices that provide a personalized image (130) to the user (100). For example, the electronic device (2000) may be implemented as various types and forms of electronic devices that include a display. The electronic device (2000) may include, but is not limited to, devices capable of displaying the personalized image (130) through a display, such as a smart TV, a smart phone, a tablet PC, a laptop PC, a glasses-type display, a head-mounted display (HMD), etc. For example, the electronic device (2000) may be implemented as various types and forms of electronic devices that can be connected to a display via wired / wireless. For example, the electronic device (2000) may include, but is not limited to, devices capable of displaying the personalized image (130) through a wired / wireless connection to a display, such as a set-top box, a desktop PC, etc.

[0053] In one embodiment, the server (3000) may be a device that generates a personalized image (130) using a generative model. The server (3000) may be a device capable of processing complex operations and tasks using large amounts of data, such as training, inference, management, and distribution of the generative model. According to one embodiment, the training of the generative model executed on the server (3000) may be performed by another computing device. The server (3000) may receive data for generating a personalized image from an electronic device (2000), which is a client device, and return the personalized image (130) to the electronic device (2000).

[0054] In one embodiment, the electronic device (2000) and server (3000) providing the image personalization service may be referred to as an "image personalization system." Alternatively, for convenience, they may simply be referred to as a "system." This disclosure describes an embodiment in which a personalized image is generated and provided to a user by the image personalization system. However, the operations of this disclosure do not necessarily have to be performed by the image personalization system.

[0055] In one embodiment, the image personalization operations can be performed independently by the electronic device (2000). In this case, a generation model for generating a personalized image (130) can be stored in the electronic device (2000). The electronic device (2000) can perform the operations of the present disclosure using the generation model to generate the personalized image (130). Meanwhile, since the electronic device (2000) has relatively low computing performance compared to the server (3000), the generation model used by the electronic device (2000) can be a relatively lightweight AI model with resource requirements tailored to the computing performance of the electronic device (2000).

[0056] In one embodiment, the image personalization operations may be performed independently by the server (3000). In this case, the server (3000) may receive only an image personalization request from the electronic device (2000), perform the operations of the present disclosure to generate a personalized image (130), and return the generated result to the electronic device (2000).

[0057] In one embodiment, image personalization operations may be performed separately by multiple electronic devices. For example, among the operations of the present disclosure, "Operation A" may be performed by "electronic device A," and "Operation B" may be performed by "electronic device B." Examples of various implementation methods of the present disclosure, such as those described above, are self-evident and therefore, for the sake of brevity, a detailed description thereof will be omitted.

[0058] Exemplary operations in which an electronic device (2000) provides a personalized image (130) to a user (100) will be described in more detail through the drawings and descriptions thereof described below.

[0059] FIG. 2 is a flowchart illustrating an operation of an electronic device generating a personalized image according to one embodiment of the present disclosure.

[0060] Referring to FIG. 2, operations for an electronic device (2000) to generate a personalized image are schematically described, and a detailed description of each operation will be described with reference to the drawings that follow.

[0061] In operation S210, the electronic device (2000) may obtain a prompt for image generation.

[0062] A prompt may be an input that initiates an interaction with a generative model that generates an image. The prompt may be a text input or a speech input containing one or more words and / or one or more sentences.

[0063] In one embodiment, the prompt may include natural language text. The natural language text may include various information that the generative model can utilize to generate an image, such as context, intent, task, and constraints. The electronic device (2000) may process the natural language text using a natural language processing (NLP) model.

[0064] In one embodiment, the prompt may be received in the form of user input. For example, the electronic device (2000) may receive text input from the user. Alternatively, the electronic device (2000) may receive voice input from the user. The electronic device (2000) may convert the user's voice input into text using Automatic Speech Recognition (ASR).

[0065] In one embodiment, the prompt may be generated by the electronic device (2000). For example, the electronic device (2000) may receive an image input from a user. The electronic device (2000) may extract a text description of the image from the image. The electronic device (2000) may generate the prompt using various techniques for extracting text from the image (e.g., image captioning, etc.).

[0066] Meanwhile, prompts can correspond to various expressions representing the same or similar concepts. Prompts can correspond to expressions such as "input," "user input," "input phrase," "user command," "directive," "starting sentence," "task query," "trigger sentence," and "message," but are not limited to the examples described above.

[0067] In operation S220, the electronic device (2000) can extract a keyword from the prompt.

[0068] Keywords may include words, phrases, or clauses that represent the core content of a prompt. Among the extracted keywords, at least one keyword may be a keyword for image personalization. If there is more than one keyword, a keyword for image personalization may be selected from among the keywords. The keyword for image personalization may correspond to a personalized element reflected in the personalized image. For example, if the keyword for image personalization is "A," personalization may be applied to an element corresponding to "A" among the elements in the personalized image. There may be one or more keywords for image personalization. In the present disclosure, the keyword for image personalization may also be referred to as a "personalization keyword." Alternatively, it may be referred to simply as a "keyword" for convenience.

[0069] In one embodiment, the electronic device (2000) can parse the prompt to break down the text into multiple components and extract one or more keywords from among the multiple components.

[0070] In one embodiment, the electronic device (2000) may extract one or more keywords from a prompt using an NLP model. For example, the electronic device (2000) may analyze a sentence using a pre-trained language model, determine the meaning and context of words within the sentence, and extract one or more keywords.

[0071] The electronic device (2000) can determine a personalized keyword from among one or more keywords extracted from the prompt.

[0072] In one embodiment, the electronic device (2000) may determine one or more extracted keywords as personalized keywords. For example, the electronic device (2000) may determine all or some of the extracted keywords as personalized keywords. In this case, the method by which the electronic device (2000) determines the personalized keywords may be, but is not limited to, a rule-based method that predefines personalized keywords (e.g., predefined / pre-registered / recently used / frequently used / personalized keywords determined based on context analysis, etc.).

[0073] In one embodiment, the electronic device (2000) may determine personalized keywords based on user input. For example, the electronic device (2000) may display one or more keywords extracted from a prompt on the screen and receive user input selecting at least one of the one or more keywords displayed on the screen. The electronic device (2000) may determine the keyword corresponding to the user input as a personalized keyword.

[0074] Meanwhile, keywords can correspond to various expressions representing identical or similar concepts. Keywords can correspond to expressions such as "word," "term," "main word," "search term," "core term," "primary term," "related term," and "recommendations," but are not limited to the examples described above.

[0075] In operation S230, the electronic device (2000) may search for one or more reference images corresponding to a keyword among a plurality of images stored in the electronic device (2000). The operation of the electronic device (2000) searching for one or more reference images corresponding to the keyword may include a process in which the electronic device (2000) searches for one or more images corresponding to the keyword and selects a reference image from among the one or more searched images.

[0076] In one embodiment, the electronic device (2000) can search for one or more images based on a keyword in the electronic device (2000). For example, the electronic device (2000) can search for one or more images that match the keyword. Alternatively, the electronic device (2000) can search for one or more images among a plurality of images stored in the electronic device (2000) with reference to the keyword. For example, the electronic device (2000) can search for one or more images that match words related to the keyword (e.g., synonyms, similar words, etc.).

[0077] The electronic device (2000) can search for one or more images corresponding to a keyword among a plurality of images stored in the local media storage of the electronic device (2000). That is, the electronic device (2000) can retrieve one or more images corresponding to the keyword from among the user's personal images stored in the local media storage of the electronic device (2000), rather than searching for an image corresponding to the keyword among images published in the public domain. Meanwhile, the images stored in the local media storage may include images directly taken by the user and / or images acquired by the user from an external source (e.g., images downloaded from images known in the public domain and images received from other users, etc.). In one embodiment, the electronic device (2000) may search for an image matching the keyword and / or an image matching a synonym, similar word, etc. of the keyword among various images published in the public domain through an external database (e.g., the web, etc.).

[0078] In one embodiment, the electronic device (2000) can search for an image corresponding to a keyword using image metadata. Image metadata may include, but is not limited to, the image title, description, shooting location, shooting date, person information, tags, etc. The image metadata may also include information about various objects included in the image. The information about the various objects included in the image metadata may be obtained by the electronic device (2000) classifying the image or detecting objects in the image and performing object recognition. The image metadata may be automatically generated when a photo is taken by the electronic device (2000). Alternatively, the image metadata may be manually input by a user. For example, the electronic device (2000) may provide an interface through which a user can input additional information about an image when or after taking the image. In this case, the interface through which additional information about the image can be input may be provided through various applications (e.g., a camera app, a gallery app, a photo editing app, etc.).

[0079] In one embodiment, the electronic device (2000) can generate image metadata. The electronic device (2000) can utilize various image processing / analysis algorithms to generate the image metadata. For example, the electronic device (2000) can cluster images by subject and tag the subject of the image in the image cluster as metadata. Alternatively, the electronic device (2000) can extract a text description of the image from the image, extract one or more keywords from the text description, and tag the extracted keywords as metadata. Alternatively, the electronic device (2000) can perform image classification and tag the class of the image, which is the classification result, as metadata. However, the method by which the electronic device (2000) generates image metadata is not limited to the examples described above.

[0080] In one embodiment, the electronic device (2000) can select a reference image from among one or more retrieved images.

[0081] A reference image may be an image selected from among images corresponding to a searched keyword, and refers to an image to be used for image personalization. For example, data representing a reference image (e.g., image feature information, described below) is used as input data for a generation model, and accordingly, a personalized image generated by the generation model may include features of the reference image. For example, various features of the reference image, such as background, object, style, subject, and context, may be included in the personalized image. In an embodiment, one or more reference images may be selected.

[0082] Reference images can correspond to various expressions representing the same or similar concepts. Reference images can correspond to expressions such as "image," "photo," "reference photo," "standard image / photo," "benchmark image / photo," etc., but are not limited to the examples described above.

[0083] In one embodiment, the electronic device (2000) may determine a reference image based on predefined or predetermined conditions. For example, the electronic device (2000) may select, from among one or more searched images corresponding to a keyword, images whose similarity to the keyword exceeds a threshold value as reference images. However, the conditions for selecting a reference image are not limited to the examples described above.

[0084] In one embodiment, the electronic device (2000) may determine or select a reference image based on a user input. For example, the electronic device (2000) may display one or more images corresponding to a searched keyword on the screen and receive a user input for selecting at least one of the one or more images displayed on the screen. The electronic device (2000) may determine the image corresponding to the user input as the reference image.

[0085] In one embodiment, once a reference image is determined, a keyword and a reference image corresponding to the keyword may be stored as a text-image pair.

[0086] In operation S240, the electronic device (2000) may segment an area related to a keyword within a reference image. In other words, the electronic device (2000) may segment the reference image into a plurality of areas. The electronic device (2000) may determine a segment corresponding to an area related to the keyword among segments of the reference image based on the keyword. For example, the electronic device (2000) may select a segment matching the keyword among segments of the reference image. The electronic device (2000) may determine a segment related to the keyword among segments of the reference image by referring to the keyword. For example, the electronic device (2000) may select a segment matching a word related to the keyword (e.g., a synonym, an analogue, etc.).

[0087] The keyword-related region may be a region within the image that represents the visual characteristics of the keyword. For example, if the keyword is "object A," the keyword-related region may refer to a region corresponding to "object A" within the image. The electronic device (2000) may perform image segmentation to segment an image into multiple regions based on objects, foregrounds, backgrounds, etc. contained within the image from a reference image. In this case, the keyword-related region may be selected from among the multiple segmented regions.

[0088] In one embodiment, the electronic device (2000) can analyze a reference image to determine an area associated with a keyword. For example, the electronic device (2000) can use semantic segmentation to separate multiple semantic elements within the image. The electronic device (2000) can classify the multiple semantic elements and, based on the classification results, select a segment representing an area associated with the keyword.

[0089] In one embodiment, the electronic device (2000) can determine an area associated with a keyword based on user input. For example, the electronic device (2000) can use instance segmentation to separate multiple instances (e.g., objects) within an image. The electronic device (2000) can display the segmentation results on a screen and receive a user input for selecting at least one of the segments displayed on the screen.

[0090] Meanwhile, the image segmentation method performed by the electronic device (2000) is not limited to the examples described above. For example, the electronic device (2000) may utilize a panoptic segmentation method, which is a combination of semantic segmentation and instance segmentation. Alternatively, the electronic device (2000) may select a segment representing an area related to a keyword based on a user input (e.g., rectangular selection, free selection (lasso selection), etc.) that designates an area of ​​an object corresponding to a keyword within a reference image.

[0091] In operation S250, the electronic device (2000) can obtain image feature information based on an area related to a keyword of a reference image.

[0092] In the present disclosure, data used to reflect personalized elements when generating an image may be referred to as "reference data." For example, the reference data may include text and / or images. Specifically, text included in the reference data may include, but is not limited to, a text description of the image, keywords, synonyms of keywords, similar words of keywords, etc. Furthermore, images included in the reference data may include, but are not limited to, a raw reference image, a region (e.g., a segment) related to a keyword in the reference image, etc. The reference data may correspond to various expressions representing the same / similar concepts. In one embodiment, the reference data may be included in data that may be referred to as "data for personalization."

[0093] In one embodiment, the electronic device (2000) may preprocess reference data to generate image feature information. The image feature information may include, but is not limited to, various data related to the features of the image, such as visual patterns, attributes, structures, or text corresponding to the image extracted from the image.

[0094] Image feature information can be used as input to a generative model. For example, image feature information can be obtained by going through certain preprocessing steps so that images and / or text of reference data can be used as input to a generative model.

[0095] In one embodiment, image feature information can be obtained by vector embedding, which is a preprocessing technique for converting data into vector representations. The electronic device (2000) can generate a reference embedding by converting reference data into a vector representation. A reference embedding refers to data converted from reference data into a vector representation. For example, the electronic device (2000) can convert an area related to a keyword in a reference image into data in a vector representation. Furthermore, for example, the electronic device (2000) can convert a text-image pair consisting of a keyword and a reference image into data in a vector representation. Furthermore, for example, the electronic device (2000) can convert a text-image pair consisting of a keyword and an area related to the keyword in a reference image into data in a vector representation. The electronic device (2000) can generate a reference embedding having a plurality of vector values ​​by converting a keyword and a reference image into a vector representation including numerical values ​​in a standardized form. The electronic device (2000) can perform tokenization, which divides data into smaller units, and convert each token into a unique vector value. Reference embeddings contain information about keywords and reference images, and can be used as input data when a generative model generates personalized images.

[0096] Reference embeddings can correspond to various expressions representing identical or similar concepts. Reference embeddings can be replaced with expressions such as "vector," "embedding," "vector embedding," "vector representation," and "vector encoding," and are not limited to the examples described above. In one embodiment, reference data may be included in data that may be referred to as "data for personalization."

[0097] In one embodiment, the electronic device (2000) may generate a reference embedding by encoding text-image pairs of keywords and reference images corresponding to the keywords using an encoder. The encoder may be trained to receive text and / or images as input and compress information in the text and / or images.

[0098] Meanwhile, the preprocessing method by which the electronic device (2000) acquires image feature information is not limited to vector embedding, which converts data into vector representations. For example, the electronic device (2000) can perform preprocessing such as tokenization and normalization on text, and can perform preprocessing such as normalization, resizing, cropping, and changing image properties (brightness, contrast, color, etc.) on images. In other words, the electronic device (2000) can acquire image feature information by processing reference data in various ways.

[0099] Meanwhile, since there are limitations in listing all the cases in which an electronic device (2000) acquires image feature information, the present disclosure will describe the use of reference embedding, which is vectorized reference data, as an example of image feature information. However, vector embedding is merely an example for explaining the technical implementation method of the present disclosure. The present disclosure can also be implemented using similar methods, such as the preprocessing methods listed in the examples described above, including using vector embedding.

[0100] In operation S260, the electronic device (2000) may transmit prompt and image feature information to the server (3000).

[0101] The server (3000) can operate a generative model. To operate the generative model, the server (3000) may be a device with high computing power capable of processing more calculations than the electronic device (2000). The server (3000) may perform training and / or inference of the generative model, which requires a large amount of calculations. The server (3000) may use the generative model to generate a personalized image and transmit the personalized image to the electronic device (2000).

[0102] In one embodiment, the generative model may be a distributed version after training is complete. The server (3000) can generate a "personalized image" that reflects the characteristics of an image unique to the individual user using the distributed generative model without retraining the generative model. For example, when the server (3000) generates a personalized image, the existing weights of the distributed generative model are used, and image feature information (e.g., reference embedding) is used as input data only in the inference operation using the generative model. Accordingly, even if the generative model has not learned the individualized characteristics of the user (e.g., has not learned the images owned by the individual user), it can generate a personalized image that includes the characteristics of a reference image, which is an image of the individual user.

[0103] In one embodiment, the generative model may generate a prompt embedding by encoding a prompt received from the electronic device (2000). Alternatively, the prompt embedding may be generated by the electronic device (2000) together with the reference embedding when it is generated in operation S250. The generative model generates a personalized image using the prompt embedding and the reference embedding.

[0104] The electronic device (2000) can provide various advantages by transmitting image feature information composed of preprocessed data to the server (3000) instead of transmitting reference data to the server (3000). For example, the electronic device (2000) can reduce the data processing amount of the server (3000) by transmitting a reference embedding, which is a vector representation of a reference image, to the server (3000). In addition, the electronic device (2000) can protect the user's privacy by transmitting only the reference embedding to the server (3000), thereby preventing the user's personal image stored only in the media storage of the electronic device (2000) from being shared, while allowing a personalized image to be generated.

[0105] In some embodiments, the electronic device (2000) may transmit reference data to the server (3000). In this case, the generation model may receive the reference data as input, perform preprocessing, and generate a personalized image. Alternatively, the generation model may receive preprocessed reference data as input and generate a personalized image.

[0106] In operation S270, the electronic device (2000) may receive a personalized image from the server (3000). There may be one or more personalized images.

[0107] The electronic device (2000) can display a personalized image received from the server (3000).

[0108] FIG. 3 illustrates the operation of an electronic device and a server according to one embodiment of the present disclosure.

[0109] Referring to FIG. 3, the electronic device (2000) may include at least a keyword extraction module (310), an image search module (320), an image segmentation module (330), and an encoder (340). The server (3000) may include an image generation module (350). The modules illustrated in FIG. 3 may be configurations implemented by at least one processor included in the electronic device (2000) executing a program or command stored in a memory included in the electronic device (2000). Accordingly, the operations described below as being performed by the modules of the electronic device (2000) may be performed by at least one processor included in the electronic device (2000).

[0110] The keyword extraction module (310) processes the prompt to extract one or more keywords. Once one or more keywords are extracted, all or part of the extracted keywords may be determined as personalized keywords. There may be one or more personalized keywords.

[0111] Keywords may be passed to an encoder (340) to be vectorized, passed to an image search module (320) to be used for searching images, or passed to an image segmentation module to be used for segmenting images. A more detailed description of an example of a keyword extraction module (310) will be further described in the description of FIG. 4.

[0112] The image search module (320) searches for one or more images corresponding to a keyword among images stored in the media storage (322). The image search module (320) can search for one or more images corresponding to the keyword using image metadata. When one or more images are searched, a reference image to be reflected in the personalized image can be determined or selected from among the searched images. There may be one or more reference images.

[0113] The reference image may be selected by the electronic device (2000) based on predetermined conditions, or may be selected based on user input. The reference image may be passed to the image segmentation module (330) for segmentation, or may be passed to the encoder (340) for vectorization. A more detailed description of an example of the image retrieval module (320) will be further described in the description of FIGS. 5A-5B .

[0114] The image segmentation module (330) segments and determines regions related to keywords within a reference image. There may be one or more regions related to keywords.

[0115] The region associated with a keyword may be determined based on the results of the electronic device (2000) analyzing the keyword and the reference image (e.g., semantic segmentation), and the region to be segmented may also be selected based on user input. The image segmentation module (330) may extract the region associated with the keyword to obtain a segment of the reference image. The segment of the reference image may be transmitted to the encoder (340) and vectorized. A more detailed description of the image segmentation module (330) will be further described in the description of FIG. 4.

[0116] Meanwhile, in one embodiment, the operation of the image segmentation module (330) may be omitted. In this case, the operation using the segment of the reference image may be replaced with an operation using the reference image.

[0117] The encoder (340) converts image and / or text data into a vector representation.

[0118] For example, the encoder (340) may generate a prompt embedding by converting the prompt into a vector representation. Alternatively, the prompt may be transmitted to the server (3000) in its original form and encoded there.

[0119] For example, the encoder (340) can generate a reference embedding by converting a text-image pair consisting of a keyword and a reference image into a vector representation.

[0120] For example, the encoder (340) can generate a reference embedding by converting a text-image pair consisting of segments of keywords and reference images into a vector representation.

[0121] The prompt (or prompt embedding) and reference embedding may be transmitted to the server (3000) and used by the server (3000) to generate a personalized image. A more detailed description of an example of an encoder (340) will be further described in the description of FIG. 7.

[0122] The image generation module (350) generates a personalized image. The personalized image may utilize a generative model. The generative model may be an artificial intelligence model that utilizes a mechanism (e.g., cross-attention) to process the correlation between prompts and images to generate an image based on a prompt (or prompt embedding).

[0123] In one embodiment, the generative model may be a deployed version after training is complete. In this case, the reference embedding received from the electronic device (2000) may be used only in the inference operation of the generative model. In other words, the generative model is an artificial intelligence model that takes a prompt (or prompt embedding) and a reference embedding as input and outputs a personalized image.

[0124] The server (3000) can use the image generation module (350) to generate a “personalized image” that reflects the characteristics of an image that only an individual user has, without retraining the generation model.

[0125] A more detailed description of an example of the image generation module (350) will be further described in the description of FIG. 8.

[0126] FIG. 4 is a diagram for explaining an operation of an electronic device extracting a keyword according to one embodiment of the present disclosure.

[0127] In one embodiment, the electronic device (2000) may display one or more extracted keywords (420). FIG. 4 illustrates a graphical interface (400) (hereinafter, referred to as the graphical interface (400)) that displays the result of keyword extraction by the electronic device (2000). The graphical interface (400) may include various elements. For example, the graphical interface (400) may include a prompt (410), one or more keywords (420), and various interface elements for guiding image personalization (e.g., text (430), buttons (440), etc.). However, some of the exemplary elements described above may be omitted, or other elements may be added.

[0128] In one embodiment, the graphical interface (400) may display a prompt (410) input for creating a personalized image. For example, a user-entered prompt (410) such as "Draw me a picture of a dog playing in a park" may be included in the graphical interface (400). The electronic device (2000) may display the prompt (410) on the graphical interface (400) to allow the user to confirm whether the prompt (410) has been entered correctly. The electronic device (2000) may confirm the prompt (410) based on user input confirming the prompt (410). Alternatively, the electronic device (2000) may change the prompt (410) based on user input that changes the prompt (410).

[0129] In one embodiment, the graphical interface (400) may display one or more extracted keywords (420). For example, the keywords (420) of “dog,” “park,” and “appearance” extracted from the prompt (410) may be included in the graphical interface (400).

[0130] The electronic device (2000) can parse the prompt to break down the text into multiple components and extract one or more keywords (420) from the multiple components. For example, the electronic device (2000) can perform text preprocessing to remove special characters, etc., from a sentence. The electronic device (2000) can create a list of words, such as nouns, adjectives, and verbs, and select at least some of the words. In this case, word frequency, predetermined word importance, etc. can be utilized, but are not limited thereto.

[0131] The electronic device (2000) can extract one or more keywords (420) from the prompt (410) using an NLP model. For example, the electronic device (2000) can analyze a sentence using a pre-trained language model. For example, the electronic device (2000) can tokenize the sentence and perform semantic analysis to determine the meaning, context, intent, etc. of words within the sentence. Based on the sentence analysis results, the electronic device (2000) can extract one or more keywords (420).

[0132] In one embodiment, the electronic device (2000) may determine a personalized keyword (422) from among one or more keywords (420) extracted from the prompt (410). The personalized keyword (422) may be all or part of one or more keywords (420), and thus, the personalized keyword (422) is a concept included in one or more keywords (420). The personalized keyword (422) may be automatically determined by the electronic device (2000) or manually determined based on user input.

[0133] In one embodiment, the electronic device (2000) may change one or more keywords (420) based on user input. For example, if the user selects an interface element (424) for changing keywords within the graphical interface (400), the word "playing," which was not previously extracted as a keyword, may be added to and displayed among one or more keywords (420).

[0134] In one embodiment, the graphical interface (400) may display various interface elements (e.g., text (430), buttons (440), etc.) to guide image personalization. For example, the graphical interface (400) may display text (430) asking the user which point he or she would like to focus on to create a personalized image (e.g., "What would you like to change?"). Alternatively, the graphical interface (400) may display a button (440) (e.g., "Create Image") to allow the user to proceed to the next step in image personalization.

[0135] In one embodiment, the electronic device (2000) may suggest keywords related to the prompt. For example, the electronic device (2000) may recommend a word highly related to the prompt by including it in one or more keywords (420), even if the word is not included in the prompt (410). Alternatively, the electronic device (2000) may recommend a keyword previously used by the user by including it in one or more keywords (420), based on the user's personalized image creation history.

[0136] FIG. 5A is a diagram for explaining an operation of an electronic device searching for an image according to one embodiment of the present disclosure.

[0137] In one embodiment, the electronic device (2000) can search for one or more images corresponding to a keyword among the user's images stored in the local media storage.

[0138] In one embodiment, images stored in a media storage may include metadata. The metadata of the image may include, but is not limited to, the title, description, shooting location, shooting date, person information, tags, etc. of the image. The image metadata may also include information about various objects included in the image. The information about the various objects included in the image metadata may be obtained by the electronic device (2000) classifying the image or detecting objects within the image and performing object recognition. The image metadata may be automatically generated when a photo is taken by the electronic device (2000). Alternatively, the image metadata may be generated by the electronic device (2000) by analyzing a stored image. Alternatively, the image metadata may be manually input by a user.

[0139] For the convenience of explaining the images stored in the media storage, the description will be made with reference to the screen (510) of the application where the images stored in the media storage are visually displayed. The screen (510) of the application may be displayed on the screen of the electronic device (2000).

[0140] In one embodiment, images stored in the media storage of the electronic device (2000) may be pre-organized for image retrieval. For example, the electronic device (2000) may group images stored in the media storage by subject through image clustering.

[0141] More specifically, for example, images stored in a media storage may be divided into multiple image groups (512) such as #people, #location, #my pet, #OOTD (Outfit of the Day), etc.

[0142] Additionally, each of the plurality of image groups (512) may be divided into subgroups. For example, referring to the person group (516), the person group (516) may include subgroups corresponding to each person (Chris, Joy, Tom, John). Each of the subgroups may include images of each person. Alternatively, referring to the location group (518), the location group (518) may include subgroups corresponding to each location (Seoul, Jeju Island, Busan, Paris). Each of the subgroups may include images of each location.

[0143] When the electronic device (2000) searches for an image corresponding to a keyword, the electronic device (2000) may utilize pre-organized information (e.g., image group information) based on metadata. However, this is not limited thereto, and the electronic device (2000) may search for an image corresponding to a keyword in various ways.

[0144] For example, the electronic device (2000) can search for an image corresponding to a keyword using raw metadata. The raw metadata may include metadata generated by the electronic device (2000) or added by a user. For example, class information acquired by the electronic device (2000) through image analysis (e.g., image captioning, image classification, object recognition, etc.) may be included in the metadata. Alternatively, information directly tagged by the user for the image may be included in the metadata.

[0145] In one embodiment, the electronic device (2000) may display one or more images (522) that have been retrieved. For example, the electronic device (2000) may display one or more images (522) corresponding to the keyword “puppy.”

[0146] The reference image (530) may be determined or selected from one or more searched images (522). For example, the electronic device (2000) may determine the reference image (530) based on a predetermined condition. For example, the electronic device (2000) may select, from among the searched images (522), images whose similarity with a keyword is greater than or equal to a threshold value as the reference image (530). However, the predetermined condition of the electronic device (2000) is not limited to the above-described example. Alternatively, the electronic device (2000) may receive a user input for selecting at least one image (522) displayed on the screen. The electronic device (2000) may determine an image corresponding to the user input as the reference image (530).

[0147] In one embodiment, there may be more than one reference image (530). If multiple reference images (530) are selected, the electronic device (2000) may apply a weight to each of the reference images (530) based on a user input.

[0148] FIG. 5b is a diagram for explaining an operation of an electronic device searching for an image according to one embodiment of the present disclosure.

[0149] The electronic device (2000) can use an image search module to search for one or more images corresponding to a keyword, internally and / or externally, within the electronic device (2000). In one embodiment, the electronic device (2000) can determine a database from which to search for images. For example, the electronic device (2000) can search for images in at least one of a media storage (540) and an external database (550) based on user input.

[0150] In one embodiment, the electronic device (2000) may search for one or more images corresponding to a keyword among images stored in a media storage (540), which is or includes an image database. This operation has been described above with respect to FIG. 5A, and thus, a repeated description thereof is omitted for brevity.

[0151] In one embodiment, the electronic device (2000) can search for one or more images corresponding to a keyword among images stored in an external database (550). For example, the electronic device (2000) can use a search engine to search for images matching the keyword and / or images matching synonyms, similar words, etc. of the keyword among various images shared on the web. The electronic device (2000) can search for one or more images corresponding to the keyword on a web page that supports image search, an image sharing platform, etc. The electronic device (2000) can provide the image search results to the user. For example, the electronic device (2000) can display the image search results on the screen.

[0152] The electronic device (2000) can determine or select a reference image from among one or more retrieved images. This operation has been described above in FIG. 5A, and thus, a repeated description thereof is omitted for brevity.

[0153] In one embodiment, the electronic device (2000) may utilize user information (560) when performing an image search. User information (560) may include various items representing information related to the user. User information may include, but is not limited to, profile information, preference and interest information, and history and activity record information.

[0154] Profile information may include basic information about the user. For example, a user profile may include, but is not limited to, name, gender, date of birth, residence, the user's current location, travel route, and profile photo. Profile information may be obtained through user input.

[0155] Preference and interest information may include various information related to a user's personal tastes and interests in various categories. In one embodiment, preference and interest information may be obtained through user input. In one embodiment, preference and interest information may be collected by the electronic device (2000). For example, the electronic device (2000) may collect and store information related to a user's preferences and interests, such as content for which the user clicks the "Like" button on an SNS service, emojis, emoticons, and words frequently used by the user in a messaging service, etc.

[0156] History and activity record information may include various information related to a user's use of the electronic device (2000). For example, history and activity record information may include, but is not limited to, search history, purchase history, service usage history, web visit history, application history, etc.

[0157] The electronic device (2000) may utilize user information (560) when searching for one or more images related to a keyword from a media storage (540) and / or an external database (550). The electronic device (2000) may refine and process data within the user information (560) into a form usable for image retrieval.

[0158] The electronic device (2000) can search for one or more images related to a keyword and filter out images that match or are similar to the user's preferences and interests. For example, the electronic device (2000) can filter out images that match or are similar to the user's preferences and interests based on keywords, content categories, colors, styles, etc. related to the user's interests. Accordingly, the image search results can include images that suit the user's preferences.

[0159] The electronic device (2000) can determine or select a reference image from among one or more retrieved images. This operation has been described above in FIG. 5A, and thus, a repeated description thereof is omitted for brevity.

[0160] FIG. 6 is a diagram for explaining an operation of an electronic device segmenting an image according to one embodiment of the present disclosure.

[0161] In one embodiment, the electronic device (2000) can segment an area associated with a keyword within a reference image (610).

[0162] A keyword-related region can be an area within an image that represents the visual characteristics of the keyword. For example, if the keyword is "object A," the keyword-related region refers to the area within the image that corresponds to "object A." Similarly, if the keyword is "puppy," the keyword-related region refers to the area within the image that corresponds to "puppy."

[0163] In one embodiment, the electronic device (2000) may perform image segmentation to divide an image into a plurality of regions based on objects, foregrounds, backgrounds, etc. included in the image from a reference image (610). For example, when image segmentation is performed, the reference image (610) may be divided into a first region (611) corresponding to an object (e.g., a person), a second region (612) corresponding to the object (e.g., a dog), and a third region (613) corresponding to a background (e.g., a forest).

[0164] The region associated with the keyword may be selected from among multiple segmented regions. In the example of FIG. 6, the second region (612) may be determined or selected as the region corresponding to the keyword. The electronic device (2000) may segment the region corresponding to the keyword from the reference image (610) and isolate only the segmented region. The segmented segment may be referred to as a "segment (620) of the reference image (610)."

[0165] In one embodiment, the electronic device (2000) can analyze a reference image (610) to determine an area associated with a keyword.

[0166] For example, the electronic device (2000) can separate multiple semantic elements within an image using semantic segmentation. The electronic device (2000) can classify the multiple semantic elements and select a segment representing an area related to a keyword based on the classification result. For example, the electronic device (2000) can divide the reference image (610) into a first area (611), a second area (612), and a third area (613). The electronic device (2000) can classify the first area (611) as a person, the second area (612) as a puppy, and the third area (613) as a forest. The electronic device (2000) can determine the second area (612) corresponding to the puppy as an area related to the keyword. Accordingly, a segment (620) of the reference image (610) corresponding to the second area (612) can be extracted.

[0167] In one embodiment, the electronic device (2000) can determine or select an area associated with a keyword based on user input.

[0168] For example, the electronic device (2000) can separate multiple instances (e.g., objects) within an image using instance segmentation. The electronic device (2000) can display the segmentation result on the screen and receive a user input for selecting at least one of the segments displayed on the screen. Specifically, for example, the electronic device (2000) can divide the reference image (610) into a first region (611), a second region (612), and a third region (613), and display the segmentation result on the screen. Based on the user input for selecting the second region (612), the electronic device (2000) can determine the second region (612) as a region related to a keyword. Accordingly, a segment (620) of the reference image (610) corresponding to the second region (612) can be extracted.

[0169] Meanwhile, the image segmentation method performed by the electronic device (2000) is not limited to the examples described above. For example, the electronic device (2000) may utilize a panoptic segmentation method, which is a combination of semantic segmentation and instance segmentation. Alternatively, the electronic device (2000) may select a segment (620) of the reference image (610) representing an area related to a keyword based on a user input (e.g., rectangular selection, free selection (lasso selection), etc.) that designates an area of ​​an object corresponding to a keyword within the reference image.

[0170] In one embodiment, the electronic device (2000) may store a keyword and a reference image (610) corresponding to the keyword as a text-image pair. In one embodiment, the electronic device (2000) may store a keyword and a segment (620) of the reference image (610) corresponding to the keyword as a text-image pair. The stored text-image pair may be encoded by an encoder.

[0171] FIG. 7 is a diagram illustrating an operation of an electronic device generating a vector representation according to one embodiment of the present disclosure.

[0172] In one embodiment, the electronic device (2000) can convert keywords and reference images (or segments of the reference images) into vector representations to generate reference embeddings (712).

[0173] In describing FIG. 7, for convenience of explanation, text-image pairs of keywords and reference images (or segments of reference images) will be described as being included in reference data (710). In one embodiment, text-image pairs of reference images and keywords may be expressed as {keyword, reference image}.

[0174] The electronic device (2000) can encode reference data (710) using an encoder (720). The encoder (720) can include a text encoder (722) and an image encoder (724). The encoder (720) can be trained to find relationships between text and images and generate a joint vector representation between the text and the image. The encoder (720) can be implemented using a neural network architecture capable of processing text and images, or through a modification of the neural network architecture. For example, the encoder (720) can be implemented based on, but not limited to, the Contrastive Language-Image Pretraining (CLIP) architecture.

[0175] In one embodiment, the electronic device (2000) can convert reference data (710) into a vector representation to generate a reference embedding (712), which is data for image personalization, and transmit the reference embedding (712) to the server (3000).

[0176] In one embodiment, the electronic device (2000) may convert the prompt (730) into a vector representation to generate a prompt embedding (732). The electronic device (2000) may transmit the prompt embedding (732) to the server (3000). Alternatively, after the prompt (730) is transmitted to the server (3000), the server (3000) may convert the prompt embedding (732) into the prompt embedding (732).

[0177] FIG. 8 is a diagram illustrating an operation of a server generating a personalized image according to one embodiment of the present disclosure.

[0178] In one embodiment, the server (3000) can generate a personalized image. The server (3000) can apply or provide data (e.g., reference embedding, prompt embedding) received from the electronic device (2000) as input data to a generation model (800) to generate the personalized image. The generation model (800) can include an image information generator (802) and an image decoder (804).

[0179] In one embodiment, the generative model (800) can process input data of various modalities according to various conditioning conditions. For example, the generative model (800) can receive text as input and generate an image from the text (Text-to-Image), or can receive an image as input and generate an image from the image (Image-to-Image). This drawing illustrates an example of a generative model (800) that processes a text prompt as input.

[0180] In one embodiment, the generative model (800) may employ a classifier-free guidance (CFG) method. For example, a condition for generating an image from text among the conditional options may be described. During the training process of the generative model (800), the training of the generative model (800) may proceed while randomly removing conditions in which text is given. Accordingly, the image generation of the generative model (800) may be guided by adjusting the trade-off between the conditional likelihood, which is the likelihood of generating an image when text is given, and the unconditional likelihood, which is the likelihood of generating an image without text. In the inference operation of the generative model (800) trained through the aforementioned process, a CFG scale s may be used as an input parameter. A higher CFG scale value may increase the likelihood that the generative model (800) will generate an image similar to the prompt, but the image quality may deteriorate. As a smaller value is input for the CFG scale, the quality of the image generated by the generative model (800) improves, but the similarity with the prompt may decrease.

[0181] In one embodiment, the generative model (800) may be deployed after training is completed. The training process of the generative model (800) may include, but is not limited to, a diffusion-reverse diffusion process (820), which is a diffusion process that adds noise to the original image step by step and a reverse diffusion process that denoises the noise from the noisy image to restore the original image.

[0182] In one embodiment, the diffusion-de-diffusion process (820) of the training process of the generative model (800) may be processed by the image information generator (802) of the generative model (800). In this case, the image information generator (802) may include a noise predictor for processing the de-diffusion process. The noise predictor may be implemented using a known neural network architecture or a modification of a known neural network architecture. For example, the noise predictor may be implemented based on the U-Net architecture, but is not limited thereto.

[0183] In the diffusion-de-diffusion process (820), noise is added stepwise to an image, and the noise is removed by predicting the amount of noise. In this case, during the training process of the generative model (800) that generates images from text, an attention mechanism (e.g., cross-attention) may be utilized to merge images with noise added stepwise and prompt embeddings that transform prompts (e.g., text) to find relationships between text and images.

[0184] In one embodiment, the reference embedding is only used in the inference operation of the generation model (800). The inference operation of the generation model (800) may be executed by the inference code (810). When the server (3000) executes the inference operation using the inference code (810), the reference embedding and the prompt embedding may be used as input data of the generation model (800).

[0185] The image information generator (802) of the generation model (800) can process input data (embedding) to generate a multidimensional information array (tensor) that the image decoder (804) uses to generate an image. The image decoder (804) can transform the tensor output from the image information generator (802) to output a personalized image.

[0186] In one embodiment, the server (3000) may apply weights to reference images based on user input. In this case, the input for selecting weights may be obtained from the user's electronic device (2000) and transmitted to the server, but is not limited thereto. When the server (3000) performs an inference operation using the generative model (800), the server may apply a CFG measure s for the prompt. p CFG scale s for and reference r can be received as an input parameter. In this case, s p The higher the value, the more personalized the image will be generated, similar to the prompt, s r The higher the value, the more likely the personalized image will be generated to resemble a reference (e.g., keyword, reference image).

[0187] In one embodiment, the server (3000) measures the CFG for the reference s r can receive multiple references. For example, references can be {(keyword 1, image 1), ..., (keyword n, image n)}. The server can receive CFG measures corresponding to each reference = {s r1 , ..., s rn} and perform an inference operation using the generation model (800), thereby adjusting the degree to which each of the multiple references is reflected in the personalized image.

[0188] Therefore, the server (3000) performs an inference operation using the generated model (800) distributed after training is completed, and can use the reference embedding as input data only in the inference operation.

[0189] The reference image used to create the reference embedding may be an image stored only in the user's local media storage, and thus may be data not used in training the deployed generation model (800). That is, the image personalization service provided by the server (3000) does not provide personalization by additionally training the distributed generation model (800) with the reference image owned by the user. Instead, the server (3000) uses the reference embedding in the inference operation of the generation model (800), thereby enabling the generation model (800) to generate a personalized image that includes the features of the reference image without retraining the generation model (800).

[0190] That is, the personalized image generated by the generation model (800) may include features of a reference image that were not used in training the generation model (800).

[0191] FIG. 9 is a diagram illustrating an example of an electronic device providing a personalized image according to one embodiment of the present disclosure.

[0192] In one embodiment, the electronic device (2000) may obtain a prompt. For example, the electronic device (2000) may obtain the prompt "Draw my puppy." Here, "my puppy" is explained assuming that the user intended to utter the word "draw my puppy."

[0193] The electronic device (2000) can extract keywords from the acquired prompt using the keyword extraction module (910). For example, the electronic device (2000) can extract "puppy" as a keyword. If only one keyword is extracted, the extracted keyword can be used as a personalized keyword. Alternatively, although not illustrated in FIG. 9 , the electronic device (2000) can also extract "Our" and "Puppy" as keywords. In this case, both "Our" and "Puppy" can be determined or selected as personalized keywords, only "Our" can be determined or selected as a personalized keyword, or only "Puppy" can be determined or selected as a personalized keyword.

[0194] The electronic device (2000) can use the image search module (920) to search for one or more images corresponding to a keyword among images stored in the media storage (922) of the electronic device (2000). In this case, image metadata can be used. For example, the electronic device (2000) can use the image metadata to provide puppy images stored in the media storage (922) as search results. In this case, the image search results can include an image of a puppy raised by the user, referred to as “our puppy,” and images of other puppies stored in the media storage (922). In this case, the electronic device (2000) can receive a user input in which the user checks the image search results and selects the puppy raised by the user as a reference image.

[0195] Alternatively, image metadata might include tags associated with "our dog." In this case, image search results might display images of a dog owned by a user who was referred to as "our dog."

[0196] Meanwhile, if a reference image is determined, the area related to the keyword within the reference image may be segmented separately.

[0197] The electronic device (2000) may generate a reference embedding and transmit it to the server (3000). The reference embedding may be a text-image pair converted into a vector representation using an encoder. For example, the reference embedding may be a keyword-reference image pair converted into a vector representation. Alternatively, for example, it may be a keyword-reference image segment pair converted into a vector representation.

[0198] The server (3000) can generate a personalized image using the image generation module (930). As a result of performing the operations according to the above-described example, the personalized image may be a new image depicting a puppy raised by the user. The server (3000) transmits the personalized image to the electronic device (2000), and the electronic device (2000) can display the personalized image.

[0199] FIG. 10A is a flowchart illustrating the operation of an electronic device and a server according to one embodiment of the present disclosure.

[0200] In one embodiment, the electronic device (2000) can interact with a server (3000) to generate a personalized image. For example, a system for generating a personalized image may include an electronic device (2000) and a server (3000), and the electronic device (2000) may act as a client.

[0201] In operation S1010, the electronic device (2000) obtains a prompt. Operation S1010 corresponds to operation S210 of FIG. 2, and therefore, a repeated description is omitted for brevity.

[0202] In operation S1020, the electronic device (2000) extracts a personalized keyword from the prompt. Operation S1020 corresponds to operation S220 of FIG. 2, and therefore, a repeated description is omitted for brevity.

[0203] In one embodiment, there may be multiple keywords for personalization. If multiple keywords are extracted, the electronic device (2000) may determine in operation S1025 whether images corresponding to all keywords have been retrieved. Based on the result of operation S1025, operations S1030 to S1036 may be repeatedly performed.

[0204] In operation S1030, the electronic device (2000) searches for an image corresponding to a keyword that has not yet been searched in the media storage. Operation S1030 corresponds to operation S230 of FIG. 2, and therefore, a repeated description is omitted for brevity.

[0205] In operation S1032, the electronic device (2000) selects an image corresponding to a keyword from among one or more searched images. The selected image may be used as a reference image. There may be more than one selected image corresponding to a single keyword. Operation S1032 corresponds to operation S230 of FIG. 2, and therefore, a repeated description thereof is omitted for brevity.

[0206] The electronic device (2000) may perform image segmentation on the selected image. For example, in operation S1033, the electronic device (2000) may determine whether to apply image segmentation based on a user input. If image segmentation is to be applied, operation S1034 may be performed. If image segmentation is not to be applied, operation S1036 may be performed.

[0207] In operation S1034, the electronic device (2000) may apply image segmentation and select a segment to be used for image personalization from among a plurality of segments. The selected segment may be an image of an area related to a keyword. Operation S1034 corresponds to operation S240 of FIG. 2 , and thus, for the sake of brevity, a repeated description thereof will be omitted.

[0208] In operation S1036, the electronic device (2000) may add a keyword and a selected image to the reference data. The selected image may be used as a reference image. The reference data may be a text-image pair consisting of a keyword and a reference image, or a text-image pair consisting of segments of a keyword and a reference image.

[0209] The electronic device (2000) may perform operation S1040 when one or more reference images corresponding to each of the plurality of keywords are determined for the plurality of keywords.

[0210] In operation S1040, the electronic device (2000) can generate a prompt embedding and a reference embedding. Operation S1040 corresponds to operation S250 of FIG. 2, and thus, a repeated description thereof is omitted for brevity.

[0211] The electronic device (2000) can transmit a prompt embedding (or, prompt) and a reference embedding to the server (3000).

[0212] In operation S1045, the server (3000) may generate an image. The server (3000) may generate one or more images using a generation model. The generation model may use a reference embedding as input data. Since the generated image includes features of a reference image stored in the user's media storage, the generated image may be referred to as a personalized image. The server (3000) may transmit the personalized image to the electronic device (2000).

[0213] In operation S1050, the electronic device (2000) can display one or more images received from the server (3000).

[0214] FIG. 10b is a flowchart illustrating additional operations of an electronic device and a server according to one embodiment of the present disclosure.

[0215] The electronic device (2000) can perform an action to regenerate a personalized image based on predefined conditions (e.g., create a new personalized image).

[0216] In one embodiment, the electronic device (2000) can regenerate a personalized image based on the similarity between an image (personalized image) received from the server (3000) and a reference image. The electronic device (2000) can compare the similarity between the personalized image and the reference image. The electronic device (2000) can use various algorithms for comparing the similarity between images. For example, the electronic device (2000) can calculate the similarity between the personalized image and the reference image through pixel-by-pixel comparison, histogram comparison, etc. Alternatively, the electronic device (2000) can extract features of the images using a neural network (e.g., CNN, etc.) that processes images and calculate the similarity between the personalized image and the reference image. If the similarity between the images is lower than a preset threshold value, the electronic device (2000) can transmit the prompt embedding (or the prompt) and the reference embedding to the server (3000) so that operation S1045 can be performed again. The electronic device (2000) can receive the regenerated image from the server (3000) and display it on the screen.

[0217] In one embodiment, the electronic device (2000) can regenerate a personalized image based on user input. In operation S1052, the electronic device (2000) can receive a user input requesting image regeneration. When the user input requesting image regeneration is received, the electronic device (2000) can transmit a prompt embedding (or a prompt) and a reference embedding to the server (3000), so that operation S1045 can be performed again. The electronic device (2000) can receive the regenerated image from the server (3000) and display it on the screen.

[0218] FIG. 11 is a diagram illustrating an operation of an electronic device according to one embodiment of the present disclosure to provide image feature information to a user.

[0219] In one embodiment, the electronic device (2000) can obtain image feature information. For example, the electronic device (2000) can convert reference data consisting of text-image pairs into a vector representation to generate a reference embedding for image generation. In this case, the reference embedding is transmitted to the server (3000), and the server (3000) generates an image using the reference embedding.

[0220] In one embodiment, the electronic device (2000) can store image feature information. The electronic device (2000) can store the image feature information, and if there is another personalized image generation request after the image feature information is stored, the electronic device (2000) can visualize and display the image feature information. For example, the electronic device (2000) can store the generated reference embedding. If there is another personalized image generation request after the reference embedding is stored, the electronic device (2000) can visualize and display the stored reference embedding. FIG. 11 illustrates an example graphical interface that allows the electronic device (2000) to receive a prompt from a user. The graphical interface may include various elements. For example, the graphical interface may include a prompt input field (1100) and recently used reference data (1110). However, some of the exemplary elements described above may be omitted, or other elements may be added, to the graphical interface that allows the prompt input.

[0221] In one embodiment, the electronic device (2000) may convert the reference embedding into reference data (1110) using a decoder that performs the reverse operation of the encoder. For example, the reference data may be a text-image pair consisting of a keyword and a reference image = {keyword, reference image}. For example, referring to the reference data (1110) illustrated in FIG. 11, the first reference data = {keyword: Chris, reference image: image containing Chris}. Similarly, the second reference data (1112) = {Joy, image containing Joy}, the third reference data = {Tom, image containing Tom}, and the fourth reference data (1114) = {My pet, image containing My pet}.

[0222] The electronic device (2000) can receive user input for selecting reference data. The user of the electronic device (2000) can utilize the reference data (1110) when inputting a prompt. For example, the user can use the reference data (1110) instead of inputting the sentence "Draw a picture of Joy playing with my pet in the park" as text or by voice input. For example, the electronic device (2000) can receive user input for selecting the second reference data (1112) and the fourth reference data (1114) from among the reference data (1110).

[0223] The electronic device (2000) can transmit a reference embedding corresponding to the reference data (1110) to the server (3000) based on a user input selecting the reference data (1110). The server (3000) can generate a personalized image using a generation model based on the reference embedding and transmit the image to the electronic device (2000).

[0224] The electronic device (2000) can omit a series of operations for generating a reference embedding by storing the reference embedding and reusing the stored reference. For example, the electronic device (2000) can omit all or part of the operations for generating the reference embedding, such as prompt acquisition, keyword extraction, image search, image segmentation, and vector conversion.

[0225] FIG. 12 is a diagram for explaining an operation of an electronic device generating or processing a prompt according to one embodiment of the present disclosure.

[0226] The prompt generation module (1200) may include a natural language processing model for performing at least one of generating, processing, and transforming a prompt.

[0227] In one embodiment, the electronic device (2000) may process the acquired prompt (1210) using the prompt generation module (1200). The electronic device (2000) may acquire the prompt (1210) and extract a keyword (1220) from the prompt (1210). For example, the electronic device (2000) may extract the keyword (1220) "puppy" from the prompt (1210) "draw my puppy."

[0228] The electronic device (2000) may search for one or more images (1230) corresponding to a keyword and select one or more reference images from among the searched one or more images (1230). For example, the electronic device (2000) may search for one or more images (1230) corresponding to a puppy from among images stored in the media storage of the electronic device (2000). In this case, the reference image may be determined from among the one or more images (1230).

[0229] The electronic device (2000) can extract a text description (1240) of the reference image from the reference image. The electronic device (2000) can obtain the text description (1240) of the image using various techniques for extracting text from the image (e.g., image captioning, etc.). For example, the electronic device (2000) can extract the text description (1240) "Samoyed with white fur" from the reference image. Alternatively, the electronic device (2000) can also extract the text description (1240) from one or more retrieved images (1230).

[0230] The electronic device (2000) may change the prompt (1210) based on the text description (1240) of the reference image. The new prompt (1212) may include the contents of the text description (1240).

[0231] In one embodiment, the electronic device (2000) may use the prompt generation module (1200) to generate a new prompt (1212). For example, the electronic device (2000) may extract a text description (1240) from one or more retrieved images (1230) and / or reference images, and generate a new prompt (1212) based on the text description. The new prompt (1212) may include the content of the text description (1240) and may be composed of text with different content than the original prompt (1210).

[0232] In one embodiment, the electronic device (2000) may process a text description (1240) as input data for generating a personalized image. For example, the electronic device (2000) may include the text description in the reference data. In this case, the reference data may be {keyword, reference image, text description (1240)}.

[0233] The electronic device (2000) can acquire image feature information using reference data including keywords, reference images, and text descriptions (1240). For example, the electronic device (2000) can convert the reference data into a vector representation to generate a reference embedding. The generated reference embedding can be transmitted to the server (3000).

[0234] In one embodiment, the server (3000) performs an inference operation using a generative model, and the CFG measure s for the text description (1240) t can receive as an input parameter. That is, the server (3000) can receive the CFG scale s for the prompt. p CFG scale s for and reference r , CFG scale s for text description (1240) t can be received as an input parameter. In this case, s p The higher the value, the more personalized the image will be generated, similar to the prompt, s r The larger the value, the more similar the personalized image will be to the reference (e.g., keyword, reference image), and s t The higher the value, the more personalized the image can be generated, similar to the text description.

[0235] FIG. 13 is a diagram illustrating an operation of an electronic device providing a personalized image according to one embodiment of the present disclosure.

[0236] In one embodiment, the electronic device (2000) can use inpainting to provide an inpainted image (1330) as a personalized image to a user. The inpainted image (1330) can be generated based on a reference image (1300), an original image (1310), and a mask (1320).

[0237] In one embodiment, the electronic device (2000) may obtain an original image (1310) and a mask (1320). For example, the electronic device (2000) may receive a user input for selecting the original image (1310) from the user. The original image (1310) may be stored in the local media storage of the electronic device (2000) or may be obtained by the user from an external path (e.g., a search result, etc.). In addition, the electronic device (2000) may receive an input for specifying a mask (1320) corresponding to the original image (1310) from the user.

[0238] The electronic device (2000) can determine a reference image (1300). Furthermore, the electronic device (2000) can generate a reference embedding using the reference image (1300). For example, the electronic device (2000) can perform all or part of prompt acquisition, keyword extraction, image search, image segmentation, and vector conversion. Since the specific descriptions of the operations for generating the reference embedding have been described above in the descriptions of the previous drawings, a repetitive description will be omitted.

[0239] The electronic device (2000) can transmit a reference image (1300) (or reference embedding), an original image (1310), and a mask (1320) to the server (3000). The server (3000) can apply the data received from the electronic device (2000) (e.g., a prompt or prompt embedding, the reference image (1300) or reference embedding, the original image (1310), the mask (1320)) as input data to a generation model to generate an inpainted image (1330), which is a personalized image. In this case, the generation model can generate the inpainted image (1330) using an image-to-image method. Specifically, the generative model can identify an area to be inpainted within the original image (1310) based on the original image (1310) and the mask (1320), and fill in pixels in the identified area so that the features of the reference image (1300) are included.

[0240] The server (3000) can transmit the inpainted image (1330) to the electronic device (2000).

[0241] Meanwhile, the term "original image" can correspond to various expressions that represent the same or similar concepts. For example, "original image" can correspond to expressions such as "initial image," "base image," and "base image."

[0242] FIG. 14A is a diagram illustrating an operation of an electronic device providing a personalized image according to one embodiment of the present disclosure.

[0243] In one embodiment, the electronic device (2000) may provide a personalized image based on an input image. For example, the electronic device (2000) may provide a user with a personalized image (generated image (1420)) in which features of a reference image (1410) are applied to the original image (1410) while maintaining the composition of the original image (1400). The generated image (1420) may be generated based on the original image (1400) and the reference image (1410).

[0244] In one embodiment, the electronic device (2000) may obtain an original image (1400). For example, the electronic device (2000) may receive a user input for selecting the original image (1400) from the user. The original image (1400) may be stored in the local media storage of the electronic device (2000) or may be obtained by the user from an external path (e.g., through a search, etc.).

[0245] The electronic device (2000) can determine a reference image (1410). Furthermore, the electronic device (2000) can generate a reference embedding using the reference image (1410). For example, the electronic device (2000) can perform all or part of prompt acquisition, keyword extraction, image search, image segmentation, and vector conversion. Since specific descriptions of the operations for generating the reference embedding have been described above in the descriptions of the previous drawings, a repetitive description will be omitted.

[0246] The electronic device (2000) can transmit the original image (1400) and the reference image (1410) (or, reference embedding) to the server (3000). The server (3000) can apply the data (prompt or prompt embedding, original image (1400), reference image (1410), or reference embedding) received from the electronic device (2000) as input data for a generation model to generate a generated image (1420), which is a personalized image. In this case, the generation model can generate the generated image (1420) using an image-to-image method. For example, the generation model can change the original image (1400) to include the features of the reference image (1410) while maintaining the composition of the original image (1400). Specifically, if the prompt is “Change the puppy’s face,” a generated image (1420) can be created in which the puppy’s face is changed to that of the puppy in the reference image (1410) while maintaining the appearance of the puppy in the original image (1400).

[0247] FIG. 14b is a diagram illustrating an operation of an electronic device providing a personalized image according to an embodiment of the present disclosure.

[0248] In one embodiment, the electronic device (2000) may provide an image generated based on an image input. In the description of FIG. 14b, the operation of generating a personalized image using the image-to-image method described in the description of FIG. 14a is further described.

[0249] In one embodiment, the electronic device (2000) may perform data processing operations to generate a personalized image using an image-to-image method. The electronic device (2000) may provide a new image (e.g., a personalized image) generated based on an input image (original image) and a reference image. In this case, the personalized image may be an original image in which the characteristics of the reference image are reflected.

[0250] In operation S1410, the electronic device (2000) may acquire an original image. The original image of operation S1410 may correspond to the original image (1400) described above in the description of FIG. 14A. The original image (1400) may be stored in the local media storage of the electronic device (2000) or may be acquired by the user from an external path (e.g., search results, etc.).

[0251] In operation S1420, the electronic device (2000) may determine a reference image. For example, the electronic device (2000) may determine the reference image based on a user input selecting a reference image. The reference image of operation S1420 may correspond to the reference image (1410) described above in the description of FIG. 14A.

[0252] In operation S1430, the electronic device (2000) may generate a reference embedding. The electronic device (2000) may perform all or part of prompt acquisition, keyword extraction, image search, image segmentation, and vector conversion to generate the reference embedding.

[0253] In one embodiment, the electronic device (2000) can generate a reference embedding by converting a reference image into a vector representation. The electronic device (2000) can convert the reference image into a vector representation, or perform image segmentation on the reference image and convert the obtained segments (e.g., objects, etc.) into a vector representation.

[0254] In one embodiment, the electronic device (2000) can generate a reference embedding by converting a text-image pair including a keyword and a reference image into a vector representation. The keyword can be obtained in various ways. For example, the electronic device (2000) can analyze the reference image. The electronic device (2000) can obtain text related to the reference image using image captioning, image classification, object recognition, etc., and extract the keyword from the text. For example, the electronic device (2000) can obtain a user input indicating a keyword corresponding to the reference image. For example, the electronic device (2000) can obtain a prompt. The electronic device (2000) can extract the keyword from the prompt. Meanwhile, when the electronic device (2000) obtains a keyword, image segmentation can be performed on the reference image to extract an area related to the keyword.

[0255] In one embodiment, the electronic device (2000) may transmit a prompt, an original image, and a reference embedding to a server (3000). The prompt transmitted to the server (3000) may be converted into a prompt embedding in a vector representation by the server (3000), and the original image transmitted to the server (3000) may be converted into an image embedding in a vector representation by the server (3000).

[0256] In one embodiment, the electronic device (2000) can convert the prompt into a vector representation to generate a prompt embedding and transmit it to the server (3000).

[0257] In one embodiment, the electronic device (2000) can convert an original image into a vector representation to generate an image embedding and transmit it to a server (3000).

[0258] In operation S1431, the server (3000) may generate an image. The server (3000) may generate one or more images using a generation model. The generation model may use a prompt (or prompt embedding), an original image (or image embedding), and a reference embedding as input data. The generated image may include features of the original image and the reference image. The generated image may be referred to as a personalized image. The server (3000) may transmit the personalized image to the electronic device (2000).

[0259] In operation S1440, the electronic device (2000) can display one or more images received from the server (3000).

[0260] FIG. 14c is a diagram illustrating an operation of an electronic device providing a personalized image according to one embodiment of the present disclosure.

[0261] In one embodiment, the electronic device (2000) can perform operations to generate a personalized image based on an image generated using a generative model.

[0262] In operation S1402, the server (3000) may generate a first image using a generation model. The first image may be an image generated using only a prompt (a first prompt). The server (3000) may obtain the first prompt. For example, the electronic device (2000) may obtain the first prompt and transmit it to the server (3000). The server (3000) may receive the first prompt from the electronic device (2000). As another example, the server (3000) may also receive the first prompt from another electronic device (e.g., another client device, such as a desktop PC) that is in communication with the server (3000).

[0263] The server (3000) can apply the first prompt as input data for the generation model and obtain the first image as output data. Specific data processing may be applied to the first prompt. For example, the first prompt may be converted into a prompt embedding, which is a vector representation.

[0264] The electronic device (2000) can receive a first image generated based on a first prompt from the server (3000). There can be one or more first images.

[0265] In operation S1406, the electronic device (2000) may display one or more first images received from the server (3000).

[0266] In operation S1422, the electronic device (2000) may determine a reference image from among one or more first images. For example, the electronic device (2000) may determine the selected first image as a reference image based on an input selecting one of the one or more first images.

[0267] In operation S1432, the electronic device (2000) can generate a reference embedding.

[0268] In one embodiment, the electronic device (2000) may obtain a second prompt. The second prompt may be another prompt different from the first prompt. For example, the second prompt may be a prompt input to generate a second image.

[0269] In one embodiment, the electronic device (2000) may generate a reference embedding by converting a text-image pair containing a keyword and a reference image into a vector representation. The keyword may be extracted from a second prompt or input by a user.

[0270] The electronic device (2000) may transmit the second prompt and reference embedding to the server (3000). The second prompt transmitted to the server (3000) may be converted into a vector representation by the server (3000). In one embodiment, the electronic device (2000) may convert the second prompt into a vector representation to generate a second prompt embedding and transmit it to the server (3000).

[0271] In operation S1434, the server (3000) may generate a second image. The server (3000) may generate one or more second images using a generation model. The generation model may use a second prompt (or a second prompt embedding) and a reference embedding as input data. The second image may include features of a reference image, which is an input image. Here, since the reference image is determined from one or more first images generated by the generation model, the second image may include features of the first image. The second image may be referred to as a personalized image. The server (3000) may transmit the personalized image to the electronic device (2000).

[0272] In operation S1442, the electronic device (2000) may display one or more second images received from the server (3000).

[0273] FIG. 14d is a diagram illustrating an operation of an electronic device providing a personalized image according to one embodiment of the present disclosure.

[0274] In one embodiment, the electronic device (2000) can perform operations to generate a personalized image using an image-to-image method based on an image generated using a generative model.

[0275] In operation S1404, the server (3000) may generate a first image using a generation model. The first image may be generated by the generation model based on the first prompt.

[0276] In operation S1408, the electronic device (2000) may display one or more first images received from the server (3000).

[0277] In operation S1412, the electronic device (2000) may determine an original image from among one or more first images. The original image of operation S1412 may correspond to the original image (1400) described above in the description of FIG. 14A.

[0278] In operation S1424, the electronic device (2000) may determine a reference image. For example, the electronic device (2000) may determine the reference image based on a user input selecting a reference image. The reference image of operation S1424 may correspond to the reference image (1410) described above in the description of FIG. 14A.

[0279] In operation S1434, the electronic device (2000) may generate a reference embedding. To generate the reference embedding, the electronic device (2000) may perform all or part of the second prompt acquisition, keyword extraction, image search, image segmentation, and vector conversion. The second prompt may be a prompt input to generate a second image. Operation S1434 may correspond to operation S1430 of FIG. 14B . Therefore, for the sake of brevity, a repeated description is omitted.

[0280] In operation S1436, the server (3000) may generate a second image. The server (3000) may generate one or more second images using a generation model. The generation model may use a prompt (or prompt embedding), an original image (or image embedding), and a reference embedding as input data. The second image may include features of the original image and the reference image. Here, since the original image is determined from one or more first images generated by the generation model, the second image may include features of the first image and the reference image. The second image may be referred to as a personalized image. The server (3000) may transmit the personalized image to the electronic device (2000).

[0281] In operation S1444, the electronic device (2000) may display one or more second images received from the server (3000).

[0282] FIG. 15 is a diagram illustrating an operation of an electronic device providing a personalized image according to one embodiment of the present disclosure.

[0283] In one embodiment, the electronic device (2000) can generate images depicting various aspects of the user at the user's request. For example, the electronic device (2000) can generate images depicting the user wearing a product. In describing FIG. 15, generating an image depicting the user wearing virtual clothing will be described as an example.

[0284] The electronic device (2000) can obtain an image (1500). For example, the electronic device (2000) can receive a user input for selecting the image (1500). The image (1500) may be stored in the local media storage (1522) of the electronic device (2000) or may be obtained by the user from an external path (e.g., search results, etc.).

[0285] The electronic device (2000) can generate a prompt corresponding to the image (1500) using the prompt generation module (1510). For example, the device (2000) can extract a text description of the image (1500) from the image (1500) and generate a prompt based on the extracted text. The prompt generated by the prompt generation module (1510) can be confirmed or changed based on user input. Alternatively, the prompt can be received directly from the user. Specifically, for example, when the image (1500) includes a product "black sleeveless shirt," the prompt can be "me wearing a black sleeveless shirt."

[0286] The electronic device (2000) can determine one or more personalized keywords from the prompt. For example, the electronic device (2000) can extract a first personalized keyword "me" and a second personalized keyword "black sleeveless shirt" from the prompt "me wearing a black sleeveless shirt."

[0287] The electronic device (2000) can use the image search module (1520) to search for an image corresponding to a personalized keyword among one or more images stored in the media storage (1522) of the electronic device (2000). Among the one or more images searched, a reference image can be determined. For example, an image containing a user, which is an image corresponding to the keyword "me," can be determined as the first reference image (1502).

[0288] In one embodiment, the electronic device (2000) can segment an area within an image (1500) using an image segmentation module (1530). Various methods can be applied for image segmentation, and since this operation has been described above, a repeated description will be omitted. As a result of the image segmentation, a segment corresponding to a keyword can be determined as a reference image. For example, a segment containing a black sleeveless shirt, which is an image corresponding to the keyword "black sleeveless shirt," can be determined as a second reference image (1504).

[0289] The electronic device (2000) can convert reference data = {keyword, reference image}, which consists of personalized keywords and reference images, into a vector representation. For example, the electronic device (2000) can convert first reference data = {me, image including the user} to generate a first reference embedding, and convert second reference data = {black sleeveless shirt, image including a black sleeveless shirt} to generate a second reference embedding.

[0290] The electronic device (2000) can transmit a prompt (or prompt embedding) and a reference embedding to the server (3000). For example, the electronic device (2000) can transmit the prompt "I am wearing a black sleeveless shirt," a first reference embedding, and a second reference embedding to the server (3000).

[0291] The server (3000) can generate a personalized image (1506) using the image generation module (1540). For example, the personalized image (1506) may include an image of the user wearing a black sleeveless shirt. The server (3000) can transmit the personalized image to the electronic device (2000).

[0292] The electronic device (2000) can display a personalized image received from the server (3000) on the screen.

[0293] FIG. 16 is a diagram for explaining an operation of an electronic device according to one embodiment of the present disclosure to provide information related to a personalized image.

[0294] In explaining Fig. 16, a situation in which a personalized image (1600) has already been created is described as an example. The personalized image (1600) is described as an example of a user wearing a product, as exemplified in the explanation of Fig. 15.

[0295] In one embodiment, the electronic device (2000) may provide information related to a personalized image (1600). For example, the electronic device (2000) may provide the user with information related to an object or product included in the personalized image (1600).

[0296] The electronic device (2000) can detect one or more objects (e.g., products) included in the personalized image (1600).

[0297] The electronic device (2000) can extract a segment (1612) corresponding to an object within a personalized image (1600) using an image segmentation module (1610). For example, the electronic device (2000) can detect the product "black sleeveless shirt" within the personalized image (1600). In addition, the electronic device (2000) can detect an object within the personalized image (1600) using various detection algorithms.

[0298] The electronic device (2000) can search for an object using the image search module (1620). For example, the electronic device (2000) can search for the detected product, "black sleeveless shirt," in the clothing database (1622). The clothing database (1622) may be a vector database or may include a vector database.

[0299] The electronic device (2000) may search for an object included in the personalized image (1600) and provide the user with image search results (1630) as information related to the personalized image (1600). For example, the electronic device (2000) may provide the user with search results for the product "black sleeveless top."

[0300] In one embodiment, the electronic device (2000) may also provide additional information (e.g., product name, brand, price, purchase link, etc.) of an object included in the image search results (1630) as information related to the personalized image (1600).

[0301] In one embodiment, the electronic device (2000) may generate another personalized image based on information related to the personalized image (1600). For example, the electronic device (2000) may receive a user input for selecting an image including a detected object and generate another personalized image using the selected image as a reference image. Specifically, the electronic device (2000) may receive a user input for selecting an image of another piece of clothing from the image search results (1630) and generate and provide an image depicting the user wearing the other piece of clothing. Since this operation has been described above in the description of FIG. 15, a repeated description thereof will be omitted.

[0302] FIG. 17 is a diagram illustrating an operation of an electronic device providing a personalized image according to one embodiment of the present disclosure.

[0303] In one embodiment, the electronic device (2000) may provide recommendations to the user in response to a user request. The recommendations provided to the user may be provided via a personalized image (1760) containing the recommended content.

[0304] The electronic device (2000) can acquire user input. The user input may include, but is not limited to, text input, voice input, etc. For example, the electronic device (2000) can receive user input indicating a request from the user to "recommend padding."

[0305] The electronic device (2000) can generate a prompt corresponding to user input using a prompt generation module (1710). The prompt generation module (1710) may include a natural language processing model for generating / processing / modifying the prompt. For example, the electronic device (2000) may generate the prompt "I am wearing padding."

[0306] The electronic device (2000) can extract one or more keywords from a prompt using the keyword extraction module (1720). For example, the electronic device (2000) can extract the first keyword "me" and the second keyword "padding" from the prompt "me wearing padding."

[0307] The electronic device (2000) can use the first image search module (1730) to search for images corresponding to a keyword among images stored in the media storage (1732) of the electronic device (2000). For example, the electronic device (2000) can select a first image including a user corresponding to the first keyword “me” among the images stored in the media storage (1732). The electronic device (2000) can select a second image including the second keyword “padding” among the images stored in the media storage (1732).

[0308] The electronic device (2000) can extract segments from an image using the image segmentation module (1735). For example, the electronic device (2000) can segment an area corresponding to a user from a first image and an area corresponding to padding from a second image.

[0309] The electronic device (2000) can use the second image search module (1740) to search for images similar to a segment of an image. For example, the electronic device (2000) can search for images similar to padding among data stored in the clothing DB (1742). The clothing DB (1742) may be a vector database or may include a vector database. Meanwhile, the first image search module (1730) and the second image search module (1740) may also be implemented as a single image search module.

[0310] The electronic device (2000) can determine a reference image. For example, the electronic device (2000) can determine a first image corresponding to the first keyword "me" as the first reference image. Furthermore, for example, the electronic device (2000) can determine an image searched using the second image search module (1740) as the second reference image for recommendation. As a result, text-image pairs, first reference data = {me, first reference image} and second reference data = {padding, second reference image}, are determined.

[0311] The electronic device (2000) can convert the first reference data and the second reference data into vector representations to generate a reference embedding. The electronic device (2000) can transmit the reference embedding to a server. The server (3000) can generate a personalized image (1760) using a generation model. In this case, the generation model can use the reference embedding as input data. The personalized image (1760) can include features of the reference image. Specifically, the personalized image (1760) can be an image of a user wearing padding retrieved from a clothing database (1742).

[0312] Meanwhile, the electronic device (2000) can provide the user with information related to the personalized image (1760) based on the generated personalized image. This operation has been described above in the description of FIG. 16, so a repeated description will be omitted.

[0313] FIG. 18a is a block diagram illustrating a configuration of an electronic device according to one embodiment of the present disclosure.

[0314] In one embodiment, the electronic device (2000) may include a communication interface (2100), a memory (2200), a processor (2300), and a display (2400).

[0315] The communication interface (2100) can perform data communication with other electronic devices under the control of the processor (2300).

[0316] The communication interface (2100) may include a communication circuit that can perform data communication between the electronic device (2000) and another electronic device (e.g., a server (3000)) using at least one of data communication methods including, for example, wired LAN, wireless LAN, Wi-Fi, Bluetooth, ZigBee, Wi-Fi Direct (WFD), infrared Data Association (IrDA), Bluetooth Low Energy (BLE), Near Field Communication (NFC), Wireless Broadband Internet (Wibro), World Interoperability for Microwave Access (WiMAX), Shared Wireless Access Protocol (SWAP), Wireless Gigabit Alliances (WiGig), and RF communication.

[0317] The communication interface (2100) can transmit and receive data to and from the server (3000) for providing personalized images. For example, the communication interface (2100) can transmit a reference embedding to the server (3000) and receive a personalized image from the server (3000).

[0318] The memory (2200) may store instructions, data structures, and program codes that can be read by the processor (2300). Operations performed by the processor (2300) may be implemented by executing instructions or codes of a program stored in the memory (2200).

[0319] The memory (2200) may include a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), and may include a non-volatile memory including at least one of a ROM (Read-Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), a PROM (Programmable Read-Only Memory), a magnetic memory, a magnetic disk, and an optical disk, and a volatile memory such as a RAM (Random Access Memory) or an SRAM (Static Random Access Memory).

[0320] The memory (2200) may store one or more instructions and / or programs that cause the electronic device (2000) to operate to provide personalized images. For example, the memory (2200) may store instructions and / or programs for implementing the functions of the keyword extraction module (2210), the image search module (2220), the image segmentation module (2230), and the encoder (2240). Meanwhile, the memory (2200) may further store instructions and / or programs for implementing the functions of the prompt generation module (not shown).

[0321] The processor (2300) can control the overall operations of the electronic device (2000). For example, the processor (2300) can control the overall operations of the electronic device (2000) to provide personalized images by executing one or more instructions of a program stored in the memory (2200). There may be one or more processors (2300).

[0322] The processor (2300) may be configured with at least one of, for example, a central processing unit, a microprocessor, a graphic processing unit, an application specific integrated circuits (ASICs), a digital signal processor (DSPs), a digital signal processing device (DSPDs), a programmable logic device (PLDs), a field programmable gate array (FPGAs), an application processor, a neural processing unit, or an artificial intelligence processor designed with a hardware structure specialized for processing an artificial intelligence model, but is not limited thereto.

[0323] The processor (2300) can extract one or more keywords from a prompt by executing a keyword extraction module (2210). Since the operations of the keyword extraction module (2210) have already been described in the description of the previous drawings, a repeated description will be omitted.

[0324] The processor (2300) executes the image search module (2220) to search for one or more images corresponding to a keyword. Among the one or more images searched, a reference image may be selected. Furthermore, a keyword-reference image pair may be stored as reference data. Since the operations of the image search module (2220) have already been described in the description of the previous drawings, a repetitive description will be omitted.

[0325] The processor (2300) may execute the image segmentation module (2230) to segment an area corresponding to a keyword within a reference image. In this case, a pair of keyword-reference image segments may be stored as reference data. Since the operations of the image segmentation module (2230) have already been described in the description of the previous drawings, a repetitive description will be omitted.

[0326] The processor (2240) can generate a reference embedding by converting reference data into a vector representation using the encoder (2240). Since the operations of the encoder (2240) have already been described in the description of the previous drawings, a repetitive description will be omitted.

[0327] Meanwhile, the modules stored in the aforementioned memory (2200) are provided for convenience of explanation and are not necessarily limited thereto. To implement the aforementioned embodiments, other modules may be added (e.g., a prompt generation module), and some modules may be omitted (e.g., an image segmentation module). Furthermore, a single module may be divided into multiple modules with distinct functions, and some of the aforementioned modules may be combined to be implemented as a single module.

[0328] The display (2400) can output a video signal to the screen of the electronic device (2000) under the control of the processor (2300). For example, the display (2400) can output a video signal processed in the process of the electronic device (2000) providing a personalized image, such as a prompt input result, an image search result, or a personalized image creation result, to the screen. The display (2400) can include a touch panel. The touch panel can include one or more touch sensors that detect a touch input. In one embodiment, the prompt can be a text input entered through the touch panel.

[0329] Meanwhile, although not illustrated in FIG. 18A, the electronic device (2000) may further include additional components to perform the operations described in the aforementioned embodiments. For example, the electronic device (2000) may further include a camera, a microphone, etc. The electronic device (2000) may store an image acquired using the camera in the memory (2200). The electronic device (2000) may receive a voice input indicating a prompt using the microphone.

[0330] FIG. 18b is a block diagram illustrating a configuration of an electronic device according to one embodiment of the present disclosure.

[0331] In one embodiment, the electronic device (2000) may include a communication interface (2100), a memory (2200), a processor (2300), and an input / output interface (2500). The communication interface (2100), the memory (2200), and the processor (2300) of FIG. 18B may correspond to the communication interface (2100), the memory (2200), and the processor (2300) of FIG. 18A. Therefore, for the sake of brevity, a repeated description is omitted.

[0332] The input / output interface (2500) can process input to the electronic device (2000) and output from the electronic device (2000). The input / output interface (2500) can provide a connection between the electronic device (2000) and an external device. The input / output interface (2500) can include, but is not limited to, a USB (Universal Serial Bus) port, an HDMI (High-Definition Multimedia Interface) port, a DP (Display Port) port, a VGA (Video Graphics Array) port, an RGB port, a DVI (Digital Visual Interface), an audio jack, and the like, for example. The electronic device (2000) can be connected to external devices such as a display, a camera, a microphone, a speaker, a keyboard, a mouse, a touch pad, and the like through the input / output interface (2500).

[0333] The electronic device (2000) can output a video signal using a display connected to the electronic device (2000) via an input / output interface (2500). For example, prompt input results, image search results, personalized image creation results, etc. can be displayed on the screen of the display connected to the electronic device (2000).

[0334] The electronic device (2000) can obtain input data using an input device connected to the electronic device (2000) via an input / output interface (2500). For example, the electronic device (2000) can receive text input indicating a prompt using a keyboard, mouse, touch pad, or the like connected to the electronic device (2000). For example, the electronic device (2000) can receive voice input indicating a prompt using a microphone connected to the electronic device (2000).

[0335] When a method according to an embodiment of the present disclosure includes multiple operations, the multiple operations may be performed by one processor or multiple processors. For example, when a first operation, a second operation, and a third operation are performed by a method according to an embodiment, the first operation, the second operation, and the third operation may all be performed by a first processor, or the first and second operations may be performed by a first processor (e.g., a general-purpose processor) and the third operation may be performed by a second processor (e.g., an AI-specific processor). Here, an AI-specific processor, which is an example of the second processor, may perform operations for training / inference of an AI model. However, the embodiments of the present disclosure are not limited thereto.

[0336] One or more processors according to the present disclosure may be implemented as a single-core processor or as a multi-core processor.

[0337] When a method according to one embodiment of the present disclosure includes multiple operations, the multiple operations may be performed by one core or may be performed by multiple cores included in one or more processors.

[0338] FIG. 19a is a block diagram illustrating a configuration of a server according to one embodiment of the present disclosure.

[0339] In one embodiment, the server (3000) may include a communication interface (3100), a memory (3200), and a processor (3300). The server (3000) may be a high-performance computing device, higher than the electronic device (2000), capable of processing complex operations and tasks using large amounts of data, such as training, inference, management, and distribution of generative models.

[0340] The communication interface (3100) can perform data communication with other electronic devices under the control of the processor (3300).

[0341] The communication interface (3100) may include a communication circuit that can perform data communication between the server (3000) and another electronic device (e.g., the electronic device (2000)) using at least one of data communication methods including, for example, wired LAN, wireless LAN, Wi-Fi, Bluetooth, ZigBee, Wi-Fi Direct (WFD), infrared Data Association (IrDA), Bluetooth Low Energy (BLE), Near Field Communication (NFC), Wireless Broadband Internet (Wibro), World Interoperability for Microwave Access (WiMAX), Shared Wireless Access Protocol (SWAP), Wireless Gigabit Alliances (WiGig), and RF communication.

[0342] The communication interface (3100) can transmit and receive data for providing a personalized image to and from the electronic device (2000) under the control of the processor (3300). For example, the server (3000) can receive a prompt (or prompt embedding) and a reference embedding from the electronic device (2000) via the communication interface (3100), and can also transmit a personalized image to the electronic device (2000).

[0343] The memory (3200) may store instructions, data structures, and program codes that can be read by the processor (3300). Operations performed by the processor (3300) may be implemented by executing instructions or codes of a program stored in the memory (3200).

[0344] The memory (3200) may include a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), and may include a non-volatile memory including at least one of a ROM (Read-Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), a PROM (Programmable Read-Only Memory), a magnetic memory, a magnetic disk, and an optical disk, and a volatile memory such as a RAM (Random Access Memory) or a SRAM (Static Random Access Memory).

[0345] The memory (3200) may store one or more instructions and / or programs that cause the server (3000) to operate to generate a personalized image. For example, the memory (3200) may store instructions and / or programs for implementing the functions of the image generation module (3210). The image generation module (3210) may include a generation model.

[0346] The processor (3300) can control the overall operations of the server (3000). For example, the processor (3300) can control the overall operations of the server (3000) to generate a personalized image by executing one or more instructions of a program stored in the memory (3200). There may be one or more processors (3300).

[0347] The processor (3300) may be configured as at least one of, for example, a Central Processing Unit, a microprocessor, a Graphic Processing Unit, Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), an Application Processor, a Neural Processing Unit, or an artificial intelligence processor designed with a hardware structure specialized for processing artificial intelligence models, but is not limited thereto.

[0348] The processor (3300) can generate a personalized image by executing the image generation module (3210). The processor (3300) can generate a personalized image using a generation model that uses reference embeddings as input data. Since the operations of the image generation module (3210) have already been described in the descriptions of the previous drawings, a repetitive description will be omitted.

[0349] FIG. 19b is a block diagram illustrating a configuration of a server according to one embodiment of the present disclosure.

[0350] In one embodiment, the operations performed by the electronic device (2000) and the server (3000) described above may be performed solely by the server (3000). The server (3000) may obtain a prompt from the user, obtain an image stored in the user's electronic device (2000) as a reference image, and generate a personalized image based on the reference image. In this case, the server (3000) may obtain access to the media storage of the user's electronic device (2000).

[0351] In explaining Fig. 19b, any content overlapping with that explained in Fig. 19a is omitted for brevity.

[0352] In one embodiment, the memory (3200) may store instructions and / or programs for implementing the functions of a keyword extraction module (3220), an image search module (3230), and an image segmentation module (3240). The image generation module (3210) may include a generation model. In addition, the memory (3200) may further store a prompt generation module (not shown).

[0353] The keyword extraction module (3220), the image search module (3230), the image segmentation module (3240), and the prompt generation module can be executed by the processor (3300). Since the descriptions related to the operations of each of the aforementioned modules have already been described in the descriptions of the previous drawings, a repeated description will be omitted.

[0354] The present disclosure relates to a method, electronic device, and server for generating and providing personalized images. Furthermore, the present disclosure relates to a method for generating personalized images that reflect unique image characteristics of a user, using a distributed generative model, without retraining the generative model. The technical challenges addressed by the present disclosure are not limited to those described above, and other technical challenges not mentioned herein will be readily apparent to those skilled in the art, based on the description herein.

[0355] According to one aspect of the present disclosure, a method for an electronic device to provide a personalized image may be provided.

[0356] The method may include a step of obtaining a prompt for image generation.

[0357] The method may include a step of extracting keywords from the prompt.

[0358] The method may include a step of searching for one or more images corresponding to the keyword among a plurality of images stored in the electronic device.

[0359] The method may include a step of selecting a reference image from among one or more of the retrieved images.

[0360] The above method may include a step of segmenting an area related to the keyword within the reference image.

[0361] The method may include a step of generating a reference embedding by converting a segment of the reference image into a vector representation.

[0362] The method may include a step of transmitting the prompt and the reference embedding to a server.

[0363] The method may include receiving, from the server, a personalized image generated using a generative model that receives the prompt and the reference embedding as inputs.

[0364] The above-mentioned generative model may be deployed after training is completed.

[0365] The above generation model may generate the personalized image by using the reference embedding only in an inference operation using the above generation model.

[0366] The personalized image may include features of the reference image that were not used in training the generative model.

[0367] The step of extracting the above keywords may include a step of displaying one or more of the above keywords extracted from the prompt.

[0368] The step of extracting the above keywords may include a step of determining a keyword for personalization from among the one or more keywords based on a user input.

[0369] The step of selecting the above reference image may include the step of displaying one or more searched images.

[0370] The step of selecting the reference image may include a step of determining one or more of the reference images based on a user input.

[0371] The step of selecting the above reference image may include the step of storing the keyword and the reference image corresponding to the keyword as a text-image pair.

[0372] The method may include applying a weight to the reference image based on user input.

[0373] The above method may include a step of storing the reference embedding.

[0374] The method may include a step of visualizing and displaying the stored reference embedding in response to another personalized image generation request after the reference embedding is stored.

[0375] The method may include a step of extracting a text description of the reference image from the reference image.

[0376] The method may include changing the prompt based on the text description of the reference image.

[0377] The step of generating the above reference embedding may include converting a text description of the reference image into a vector representation.

[0378] The method may include a step of detecting one or more products included in the personalized image.

[0379] The method may include a step of displaying information related to the one or more products.

[0380] The method may include a step of displaying the personalized image.

[0381] According to one aspect of the present disclosure, an electronic device providing a personalized image may be provided.

[0382] The electronic device may include a communication interface; a memory storing one or more instructions; and one or more processors executing the one or more instructions stored in the memory.

[0383] The one or more processors can obtain a prompt for image generation by executing the one or more instructions.

[0384] The one or more processors can extract keywords from the prompt by executing the one or more instructions.

[0385] The one or more processors can search for one or more images corresponding to the keyword among a plurality of images stored in the electronic device by executing the one or more instructions.

[0386] The one or more processors can select a reference image from among the one or more searched images by executing the one or more instructions.

[0387] The one or more processors can segment an area related to the keyword within the reference image by executing the one or more instructions.

[0388] The one or more processors can generate a reference embedding by converting a segment of the reference image into a vector representation by executing the one or more instructions.

[0389] The one or more processors can transmit the prompt and the reference embedding to the server through the communication interface by executing the one or more instructions.

[0390] The one or more processors can receive, by executing the one or more instructions, a personalized image generated using a generative model that receives the prompt and the reference embedding as input from the server through the communication interface.

[0391] The above-mentioned generative model may be deployed after training is completed.

[0392] The above generation model may generate the personalized image by using the reference embedding only in an inference operation using the above generation model.

[0393] The personalized image may include features of the reference image that were not used in training the generative model.

[0394] The one or more processors can display one or more keywords extracted from the prompt by executing the one or more instructions.

[0395] The one or more processors can determine a keyword for personalization from among the one or more keywords based on a user input by executing the one or more instructions.

[0396] The one or more processors can display one or more searched images by executing the one or more instructions.

[0397] The one or more processors can determine one or more of the reference images based on user input by executing the one or more instructions.

[0398] The one or more processors can store the keyword and the reference image corresponding to the keyword as a text-image pair by executing the one or more instructions.

[0399] The one or more processors may apply weights to the reference image based on user input by executing the one or more instructions.

[0400] The one or more processors can store the reference embedding by executing the one or more instructions.

[0401] The one or more processors can, by executing the one or more instructions, visualize and display the stored reference embedding in response to another personalized image generation request after the reference embedding is stored.

[0402] The one or more processors can extract a text description of the reference image from the reference image by executing the one or more instructions.

[0403] The one or more processors may, by executing the one or more instructions, change the prompt based on a text description of the reference image.

[0404] The one or more processors can convert a text description of the reference image into a vector representation by executing the one or more instructions.

[0405] The one or more processors can detect one or more products included in the personalized image by executing the one or more instructions.

[0406] The one or more processors can display information related to the one or more products by executing the one or more instructions.

[0407] The one or more processors can display the personalized image by executing the one or more instructions.

[0408] According to one aspect of the present disclosure, a method for a server to provide a personalized image may be provided.

[0409] The method may include a step of obtaining a prompt for image generation.

[0410] The method may include a step of extracting keywords from the prompt.

[0411] The method may include a step of searching for one or more reference images corresponding to the keyword among a plurality of images stored in the electronic device.

[0412] The above method may include a step of segmenting an area related to the keyword within the reference image.

[0413] The method may include a step of obtaining image feature information based on an area related to the keyword of the reference image.

[0414] The method may include a step of generating a personalized image using a generative model.

[0415] The above-mentioned generative model may be an artificial intelligence model that receives the prompt and the image feature information as input and outputs the personalized image.

[0416] According to one aspect of the present disclosure, a server for generating personalized images may be provided.

[0417] The server may include a communication interface; a memory storing one or more instructions; and one or more processors executing the one or more instructions stored in the memory.

[0418] The one or more processors can obtain a prompt for image generation by executing the one or more instructions.

[0419] The one or more processors can extract keywords from the prompt by executing the one or more instructions.

[0420] The one or more processors can search for one or more reference images corresponding to the keyword among a plurality of images stored in the electronic device by executing the one or more instructions.

[0421] The one or more processors can segment an area related to the keyword within the reference image by executing the one or more instructions.

[0422] The one or more processors can obtain image feature information based on an area related to the keyword of the reference image by executing the one or more instructions.

[0423] The one or more processors can generate a personalized image using the generation model by executing the one or more instructions.

[0424] The above-mentioned generative model may be an artificial intelligence model that receives the prompt and the image feature information as input and outputs the personalized image.

[0425] Meanwhile, embodiments of the present disclosure may also be implemented in the form of a recording medium containing computer-executable instructions, such as program modules, executed by a computer. Computer-readable media may be any available media that can be accessed by a computer, and include both volatile and nonvolatile media, removable and non-removable media. Furthermore, computer-readable media may include computer storage media and communication media. Computer storage media include both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data. Communication media may typically include computer-readable instructions, data structures, or other data in a modulated data signal, such as program modules.

[0426] Additionally, a computer-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory storage medium" simply means a tangible device that does not contain signals (e.g., electromagnetic waves). This term does not distinguish between cases where data is permanently stored in the storage medium and cases where data is temporarily stored. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0427] According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0428] The above description of the present disclosure is provided for illustrative purposes only, and those skilled in the art will readily appreciate that modifications to other specific forms can be made without altering the technical spirit or essential features of the present disclosure. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. For example, components described as being single may be implemented in a distributed manner, and similarly, components described as being distributed may be implemented in a combined manner.

[0429] The scope of the present disclosure is indicated by the claims described below rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present disclosure.

Claims

1. A method for providing a personalized image using an electronic device, Step 1: Obtain a prompt for image generation; A step of extracting keywords from the above prompt; A step of searching for one or more reference images corresponding to the keyword among a plurality of images stored in the electronic device; A step of segmenting an area related to the keyword within each reference image of the one or more reference images; A step of obtaining image feature information based on an area related to the above keyword; a step of transmitting the above prompt and the above image feature information to the server; and A method comprising the step of receiving a personalized image generated by providing the prompt and the image feature information as inputs to a generation model from the server.

2. In paragraph 1, The step of obtaining the above image feature information is: A step of obtaining a text-image pair including the keyword and an area corresponding to the keyword of the reference image; and A method comprising the step of converting the above text-image pair into a vector representation to generate a reference embedding.

3. In any one of paragraphs 1 and 2, The above generative model is distributed after being trained, and generates the personalized image by using the image feature information only in the inference operation using the generative model. A method wherein the personalized image includes features of the reference image that were not used for training the generative model.

4. In any one of paragraphs 1 to 3, The steps for extracting the above keywords are: a step of displaying one or more of the keywords extracted from the above prompt; and A method comprising the step of determining a keyword for personalization from among said one or more keywords based on a user input.

5. In any one of paragraphs 1 to 4, The steps for searching the above reference images are: A step of searching for one or more images corresponding to the above keywords and retrieving the one or more images; a step of displaying one or more images; and A method comprising the step of determining one or more of the reference images based on user input.

6. In any one of paragraphs 1 to 5, The above method, a step of storing the above image feature information; and A method further comprising the step of visualizing and displaying the stored image feature information in response to another personalized image generation request after the image feature information is stored.

7. In any one of paragraphs 1 to 6, The above method, A step of extracting a text description of the reference image from the reference image; and A method further comprising the step of changing the prompt based on the text description of the reference image.

8. In an electronic device for generating a personalized image, communication interface; Memory that stores one or more instructions; and comprising one or more processors executing one or more instructions stored in said memory; The one or more processors cause the electronic device to execute the one or more instructions, Get a prompt to create an image, Extract keywords from the above prompts, Searching for one or more reference images corresponding to the keyword among a plurality of images stored in the electronic device, Segmenting an area related to the keyword within each reference image of the one or more reference images, Obtain image feature information based on the area related to the above keyword, Through the above communication interface, the above prompt and the above image feature information are transmitted to the server, An electronic device that receives a personalized image generated by providing the prompt and the image feature information as inputs to a generation model from the server through the communication interface.

9. In paragraph 8, The one or more processors cause the electronic device to execute the one or more instructions, Obtaining a text-image pair including the keyword and an area corresponding to the keyword in the reference image, An electronic device for converting the above text-image pair into a vector representation to generate a reference embedding.

10. In any one of paragraphs 8 to 9, The above generative model is distributed after being trained, and generates the personalized image by using the image feature information only in the inference operation using the generative model. An electronic device, wherein the personalized image includes features of the reference image that were not used for training the generative model.

11. In any one of paragraphs 8 to 10, The one or more processors cause the electronic device to execute the one or more instructions, Display one or more of the above keywords extracted from the above prompt, An electronic device that determines a keyword for personalization from among one or more keywords based on user input.

12. In any one of paragraphs 8 to 11, The one or more processors cause the electronic device to execute the one or more instructions, Search for one or more images corresponding to the above keywords, and retrieve the one or more images; Displaying one or more of the above images, An electronic device that determines one or more of the reference images based on user input.

13. In any one of paragraphs 8 to 12, The one or more processors cause the electronic device to execute the one or more instructions, Store the above image feature information, An electronic device that visualizes and displays the stored image feature information in response to another personalized image generation request after the image feature information has been stored.

14. In any one of paragraphs 8 to 13, The one or more processors cause the electronic device to execute the one or more instructions, Extracting a text description of the reference image from the reference image, An electronic device that changes the prompt based on a text description of the reference image.

15. A computer-readable recording medium having recorded thereon a program for executing the method of any one of clauses 1 to 7 on a computer.

Images