Apparatus, method and program for generating enhanced output content by iteration

Abstract

A method for iteratively generating improved output content is provided. The method includes: receiving natural language input; obtaining user intent information based on the natural language input using a natural language understanding (NLU) model; setting a target region in base content based on a first user input; determining input content based on the user intent information or a second user input; generating output content related to the base content based on the input content, the target region, and the user intent information using a neural network (NN) model; generating captions for the output content using an image captioning model; calculating the similarity between the natural language input text and the generated output content; and iteratively generating the output content based on the similarity.

Description

Technical Field

[0001] This disclosure relates to an artificial intelligence (AI) system and its applications for mimicking the functions of the human brain, such as cognition and judgment, by utilizing machine learning algorithms. More specifically, this disclosure relates to improving output content through the iterative generation of AI. Background Technology

[0002] Artificial intelligence (AI) systems refer to computer systems that enable machines to become intelligent through learning and autonomous decision-making, unlike existing rule-based intelligent systems. AI systems can improve their recognition rates and understand user preferences more accurately through experience. Therefore, existing rule-based intelligent systems are gradually being replaced by deep learning-based AI systems.

[0003] AI technologies can include machine learning (deep learning) and element technologies that use machine learning.

[0004] Machine learning can refer to algorithmic techniques used to autonomously classify / learn features of input data, while element technology is the technique used to simulate the functions of the human brain (such as cognition and decision-making) using machine learning algorithms, and includes technical fields such as language understanding, visual understanding, reasoning / prediction, knowledge representation, and motion control.

[0005] For example, various technical fields where AI technology can be applied are as follows: Language understanding refers to technologies used to recognize human language / characters in applications / processes, and includes natural language processing, machine translation, dialogue systems, question answering, speech recognition / synthesis, etc. Visual understanding refers to technologies used to identify and process objects in the same way as performed by the human visual system, and includes object recognition, object tracking, image retrieval, person recognition, scene understanding, spatial understanding, image enhancement, etc. Reasoning / prediction refers to technologies used to judge information and logically infer and predict new information, and includes knowledge / probability-based perturbation, optimization prediction, preference-based planning, suggestions, etc. Knowledge representation refers to technologies used to automatically process information about human experience as knowledge data, and includes knowledge construction (data generation / classification), knowledge management (data utilization), etc. Motion control refers to technologies used to control the autonomous driving of vehicles and the movement of robots, and includes motion control (navigation, collision avoidance, and driving), manipulation control (action control), etc.

[0006] The above information is presented as background information only to aid in understanding this disclosure. No determination or assertion is made as to whether any of the above content can be considered prior art applicable to this disclosure. Summary of the Invention

[0007] Solution to the problem

[0008] The aspects of this disclosure are intended to at least address the aforementioned problems and / or drawbacks, and to provide at least the advantages described below. Therefore, one aspect of this disclosure is to provide an apparatus and method for improving output content through iterative generation using AI.

[0009] Other aspects will be set forth in part in the description which follows, and will also be apparent in part from the description, or may be learned by practice of the presented embodiments.

[0010] According to the embodiments, content that conforms to the user's intent can be generated.

[0011] According to the embodiments, the process of generating content can be improved.

[0012] According to one aspect of this disclosure, an apparatus is provided for iteratively generating improved output content. The apparatus includes: a memory storing instructions; and at least one processor configured to execute instructions to: receive natural language input; obtain user intent information based on the natural language input using a natural language understanding (NLU) model; set a target region in base content based on a first user input; determine input content based on the user intent information or a second user input; generate output content related to the base content based on the input content, the target region, and the user intent information using a neural network (NN) model; generate captions for the output content using an image captioning model; calculate the similarity between the natural language input text and the generated output content; and iteratively generate the output content based on the similarity.

[0013] In this embodiment, the base content, input content, and output content are images, and the output content is generated by compositing the input content onto a target region of the base content.

[0014] In this embodiment, the basic content includes multiple regions, and the target region includes a region selected from the multiple regions by a first user input.

[0015] In this embodiment, an automatic speech recognition (ASR) model is used to convert speech input into text that is natural language input.

[0016] In this embodiment, the input content is determined based on content information included in the user intent information.

[0017] In this embodiment, the input content is determined from multiple pieces of content corresponding to the content information.

[0018] In this embodiment, the attributes of multiple pieces of content are different from each other.

[0019] In an embodiment, the attributes of the input content include at least one of posture, facial expression, makeup, hair, clothing, or accessories, and the attributes of the input content are determined based on content attribute information included in the user intent information.

[0020] In this embodiment, the NN model is associated with a Generative Adversarial Network (GAN) model, and the output is generated by the generator of the GAN model.

[0021] In this embodiment, the probability distribution of the output content corresponds to the probability distribution of the actual content.

[0022] In this embodiment, the probability distribution of the base content included in the output content approximates the probability distribution of the actual content.

[0023] In this embodiment, the NN model is associated with a generative adversarial network (GAN) model, and when the similarity does not meet a predetermined condition, the discriminator of the GAN model will identify the output content as fake content.

[0024] In this embodiment, the output content is a first output content, and the processor is further configured to: when the similarity does not meet a predetermined condition, execute instructions to: generate a second output content different from the first output content by using an NN model based on the input content, the target region, and user intent information.

[0025] In this embodiment, the input content is a first input content, and the output content is a first output content. The processor is also configured to: when the similarity does not meet a predetermined condition, execute instructions to: when the similarity does not meet the predetermined condition, determine a second input content that is different from the first input content, and generate a second output content that is different from the first output content based on the second input content and the target region by using an NN model.

[0026] In this embodiment, the processor is also configured to execute instructions to: receive user feedback on portions of the output content, and modify portions of the output content using a neural network model.

[0027] In this embodiment, the basic content includes the application's workspace, and the input content includes the work objects located in the workspace.

[0028] In this embodiment, the output includes animations related to the work object, and the animations are generated based on the work object, user intent information, and the application's application programming interface (API).

[0029] In this embodiment, the subtitles used for outputting content include subtitles used for animation.

[0030] In this embodiment, the NLU model, NN model, and image captioning model are stored in memory.

[0031] According to another aspect of this disclosure, a method for iteratively generating improved output content is provided. The method includes: receiving natural language input; obtaining user intent information based on the natural language input using a natural language understanding (NLU) model; setting a target region in base content based on a first user input; determining input content based on the user intent information or a second user input; generating output content related to the base content based on the input content, the target region, and the user intent information using a neural network (NN) model; generating captions for the output content using an image captioning model; calculating the similarity between the natural language input text and the generated output content; and iteratively generating the output content based on the similarity.

[0032] According to another aspect of this disclosure, a computer-readable storage medium is provided. The computer-readable storage medium includes instructions that, when executed by at least one processor, cause at least one processor to: receive natural language input; obtain user intent information based on the natural language input using a natural language understanding (NLU) model; set a target region in base content based on a first user input; determine input content based on the user intent information or a second user input; generate output content related to the base content based on the input content, the target region, and the user intent information using a neural network (NN) model; generate captions for the output content using an image captioning model; calculate the similarity between the natural language input text and the generated output content; and iteratively generate the output content based on the similarity.

[0033] Other aspects, advantages, and key features of this disclosure will become apparent to those skilled in the art from the following detailed description of various embodiments of the disclosure taken in conjunction with the accompanying drawings. Attached Figure Description

[0034] The above and other aspects, features, and advantages of specific embodiments of this disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings, wherein:

[0035] Figure 1 This is a diagram illustrating the iterative generation of content according to embodiments of this disclosure;

[0036] Figure 2 A block diagram of an electronic device according to an embodiment of the present disclosure is shown;

[0037] Figure 3 A flowchart illustrating a method according to an embodiment of this disclosure is shown;

[0038] Figure 4 These are diagrams illustrating the generated example content according to embodiments of this disclosure;

[0039] Figure 5 This is a diagram illustrating a method for obtaining user intent information according to embodiments of the present disclosure;

[0040] Figure 6a This is a flowchart of a method for determining input content according to an embodiment of the present disclosure;

[0041] Figure 6b An example graphical user interface (GUI) for selecting input content is shown according to an embodiment of the present disclosure;

[0042] Figure 7a This is a diagram illustrating a generative adversarial network (GAN) according to embodiments of the present disclosure;

[0043] Figure 7b These are figures illustrating a method of using subtitles according to embodiments of this disclosure;

[0044] Figure 8 This is a diagram illustrating a method for changing the attributes of input content according to an embodiment of the present disclosure;

[0045] Figure 9 This is a diagram illustrating a method for generating content in a GAN according to embodiments of the present disclosure;

[0046] Figure 10 This is a diagram illustrating a method for generating content in a GAN according to embodiments of the present disclosure;

[0047] Figure 11 This is a diagram illustrating a method of using user feedback according to an embodiment of this disclosure;

[0048] Figure 12 An example GUI for receiving user feedback according to an embodiment of this disclosure is shown;

[0049] Figure 13 This is a diagram illustrating the content generated in the workspace of an application according to embodiments of the present disclosure;

[0050] Figure 14 This is a diagram showing the content of the workspace adaptively generated for illustration and application according to embodiments of this disclosure;

[0051] Figure 15 This is a diagram illustrating a method for generating animations according to embodiments of the present disclosure;

[0052] Figure 16 These are diagrams illustrating the generated example animations according to embodiments of this disclosure; and

[0053] Figure 17 An example electronic device according to an embodiment of this disclosure is shown.

[0054] The same reference numerals are used to denote the same elements throughout the figure. Detailed Implementation

[0055] The following description with reference to the accompanying drawings is intended to aid in a full understanding of the various embodiments of this disclosure as defined by the claims and their equivalents. This includes various specific details used to aid understanding, but these details should be considered exemplary only. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the various embodiments described herein without departing from the scope and spirit of this disclosure. Furthermore, descriptions of well-known functions and structures may be omitted for clarity and conciseness.

[0056] The terms and words used in the following description and claims are not limited to their literal meaning, but are used solely by the inventors to enable a clear and consistent understanding of this disclosure. Therefore, it will be clearly understood by those skilled in the art that the following description of various embodiments of this disclosure is for illustrative purposes only and not intended to limit the disclosure as defined by the appended claims and their equivalents.

[0057] It should be understood that, unless the context clearly specifies otherwise, the singular forms of “a,” “an,” and “the” include plural references. Thus, for example, a reference to “a component surface” includes a reference to one or more such surfaces.

[0058] It should be understood that the terms "comprising," "including," and "having" are inclusive and thus specify the presence of the described features, quantities, operations, components, units, or combinations thereof, but do not preclude the presence or addition of one or more other features, quantities, operations, components, units, or combinations thereof. Specifically, numbers should be understood as examples used to illustrate clear points and should not be construed as limiting the embodiments by the quantities stated.

[0059] "Content" can be any type of data that can be input into an electronic device, generated by an electronic device, or output at an electronic device. For example, content can be images, vector images, videos, animations, backgrounds, workspaces, work objects, audio, text, vibrations, etc., or combinations thereof. In this document, terms such as base content, input content, output content, reconstructed content, target content, dummy content, and real content are used to distinguish each type of content mentioned in each operation of the method according to the embodiments, and their meanings can be readily understood by those skilled in the art based on the context. For example, base content can refer to content that is the object of analysis, such as editing, modifying, or compositing. Base content can be the workspace of an application. For example, base content can be a document in the workspace of a document editing application, a slide in the workspace of a presentation editing application, a spreadsheet in the workspace of a spreadsheet editing application, a user-created mode in a game application, or a drawing document in a drawing document in a drawing application. Meanwhile, terms referring to content can refer to the same type of content (e.g., images), but are not limited to this. Terms referring to content can also refer to different types of content. For example, base content can be a workspace, input content can be an image, and output content can be an animation of an image.

[0060] "User input" refers to any type of input provided by a user and received at an electronic device, and is not limited to input from a specific user. User input may be associated with one or more coordinates, but is not limited to this. For example, user input can be audio input, voice input, text input, or a combination thereof. Coordinate-associated input can be touch input, click input, gesture input, etc.

[0061] "Natural language input" refers to input received at an electronic device in the form of language used by people in daily life, and can be voice input, text input, or a combination thereof.

[0062] Figure 1 This is a diagram illustrating the iterative generation of content according to embodiments of the present disclosure.

[0063] refer to Figure 1 The electronic device 2000 can generate output content 112 based on the user's natural language input and user input. Output content 112 can be generated by combining input content with base content 100. Base content 100, input content, and output content can be, but are not limited to, images. Specific methods for generating output content 112 will be discussed later. Figure 7a and Figure 7b Please provide an explanation.

[0064] In this embodiment, the input content used to generate output content 112 can be determined based on the user's natural language input. For example, refer to... Figure 1 An image of a cat, or a cat resting, can be determined as input content based on natural language input expressing "painting a cat resting here." The input content can be determined from multiple pieces of content stored in the electronic device 2000, or from images obtained by searching the internet. The method for determining input content based on natural language input will be referenced... Figure 5 Please provide an explanation.

[0065] refer to Figure 1 Output content 112 can be generated in the target region 102 of the base content 100. Output content 112 can be generated by composited input content onto the target region 102 of the base content 100. The target region 102 refers to the region on which the input image of the base content 100 is composited. The target region 102 can be the entire region or a portion of the base content 100. The target region 102 of the base content 110 can include the output content generated after compositing. According to an embodiment, the efficiency of the compositing process can be improved by compositing the input content into the target region 102 of the base content 100 because the number of pixels used for compositing is reduced compared to compositing the input content into the entire region of the base content 100.

[0066] The target region 102 may correspond to the bounding box of an object detected or located in the base content 100, such as a table, chair, or stool. The base content 100 may include multiple regions, such as multiple bounding boxes each containing an object. The target region 102 can be selected from multiple bounding boxes by user input. The size and shape of the target region 102 can be adjusted by user input (such as drag input). The target region 102 may have a predetermined size and shape.

[0067] refer to Figure 1 The system can generate subtitles for the generated output content 112. The subtitles for the generated output content are text for the output content 112 and can be generated using an image captioning model. The subtitles can be text describing the output content 112. In an embodiment, the similarity between the text for natural language input and the output content 112 can be calculated. The generated output content 112 can be displayed on the electronic device 2000 based on the similarity. In an embodiment, the process of generating output content 112 can be iterative when the similarity does not meet a specific condition. For example, the process of generating output content 112 can be iteratively synthesized by combining another input content into the target region 102 of the base content 100. For example, the process of generating output content 112 can be iteratively synthesized by combining the same input content into the target region 102 of the base content 100, which will be discussed later by reference. Figure 7a and Figure 7b Please provide an explanation.

[0068] According to an embodiment, the process of generating output content 112 can be iteratively based on the similarity between the natural language input text and the subtitles used for output content 112, so that the generated output content 112 can conform to the user's intent.

[0069] Furthermore, the various operations described in this disclosure (such as interpreting the user's natural language input, generating output content 112, generating captions for the output content 112, and calculating the similarity between the natural language input text and the captions) can be performed by an artificial intelligence (AI) model. The AI ​​model can be referred to as a neural network model. The AI ​​model can include multiple neural network layers. Each neural network layer can have multiple weight values ​​and can perform various neural network calculations by performing arithmetic operations on the calculation results in the previous layer and the multiple weight values ​​in the current layer. The multiple weights in each neural network layer can be optimized through the results of training the AI ​​model. For example, the multiple weights can be updated to reduce or minimize the loss or cost values ​​acquired by the AI ​​model during the training process. Artificial neural networks can include, for example, but not limited to, deep neural networks (DNNs), and can include, for example, but not limited to, convolutional neural networks (CNNs), DNNs, recurrent neural networks (RNNs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent DNNs (BRDNNs), deep Q-networks (DQNs), etc., but are not limited thereto.

[0070] Figure 2 A block diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0071] Electronic device 200 may include, but is not limited to: smartphones, tablet PCs, PCs, smart TVs (TVs), mobile phones, personal digital assistants (PDAs), laptops, media players, microservers, global positioning systems (GPS), e-book terminals, digital broadcasting terminals, navigation devices, telephone booths, MP3 players, gateway devices, digital cameras, home appliances, and other mobile or non-mobile computing devices. Furthermore, electronic device 200 may be a wearable device with communication and data processing capabilities, such as, but not limited to, watches, glasses, hairbands, and bracelets. However, embodiments of this disclosure are not limited to these, and electronic device 200 may include any means capable of generating output content. Electronic device 200 may be an end-user device, but is not limited to this.

[0072] refer to Figure 2 The electronic device 200 may include a user intent acquisition unit 210, a target area setting unit 220, an input content determination unit 230, an output content generation unit 240, a subtitle generation unit 250, and a similarity calculation unit 260. The electronic device 200 may be composed of a comparison unit... Figure 2The components shown may be implemented with more or fewer components. In this disclosure, modules of electronic device 200 are named to explicitly describe their operation, which is performed by modules within electronic device 200. Therefore, it should be understood that these operations are performed according to embodiments and should not be construed as limiting the role or function of the modules. For example, an operation described herein as being performed by a specific module of electronic device 200 may be performed by another module of electronic device 200 or other modules thereof, and an operation described herein as being performed through interaction or interaction processing between modules of electronic device 200 may be performed by a single module of electronic device 200. Furthermore, operations described herein as being performed by electronic device 200 may be performed at another device or together with another device to provide substantially the same functionality.

[0073] The units of electronic device 200 can be implemented through software modules, and these software modules can be executed by at least one processor to provide the functionality of the units. Simultaneously, the software modules can be stored as a set of instructions in memory, and the instructions stored in memory can be executed by at least one processor. The units of electronic device 200 can also be implemented through hardware modules or a combination of hardware and software modules. Electronic device 200 may include memory and a processor.

[0074] Each unit of the electronic device 200 will be further referenced Figure 3 Please provide an explanation.

[0075] Figure 3 A flowchart illustrating a method according to an embodiment of this disclosure is shown.

[0076] refer to Figure 3 In operation S310, user intent information can be obtained based on the received natural language input. The user intent acquisition unit 210 of the electronic device 200 can obtain user intent information based on the received natural language input.

[0077] User intent information can represent a user's intent as interpreted by electronic device 200 from the user's natural language input (or its text). User intent information can include action information about actions that can be performed at electronic device 200. Action information can be inferred from verbs in the text of the natural language input. For example, generated content can be identified as action information from the verb "paint" in the natural language input expressing "painting the cat resting here." The word representing the generated content is not limited to "painting" and can be of various kinds. Electronic device 200 can perform the action represented by the action information. For example, electronic device 200 can generate output content in response to action information representing the generated content identified from the user intent information. In embodiments, user intent information can be obtained using an AI model, which will be discussed later via reference. Figure 5 Please provide an explanation.

[0078] In operation S320, a target region can be set in the base image based on user input. The target region setting unit 220 of the electronic device 200 can set the target region in the base content based on user input. The target region can be the entire area or a portion of the base content. Furthermore, the target region can be an area that can be adjusted in the base content through user input. The target region can correspond to a bounding box identified in the base content. In this embodiment, the identification of the bounding box can be performed by an AI model.

[0079] In this embodiment, user input and natural language input can be received simultaneously at the electronic device. For example, when a user touches basic content displayed on the electronic device 200 while speaking, the touch input and spoken natural language input can be received at the electronic device simultaneously.

[0080] In operation S330, input content can be determined. The input content determination unit 230 of the electronic device 200 can determine the input content. In an embodiment, the input content can be determined based on user intent information obtained in operation S320. For example, when the user intent information includes content information, the input content can be determined based on the content information. The content information can be inferred from nouns in the text input in natural language. For example, the noun "cat" can be identified as content information from natural language input expressing "painting a cat resting here". In an embodiment, the name of an entity can be identified as content information using Named Entity Recognition (NER). The electronic device 200 can determine an image of a cat as input content in response to identifying "cat" as content information from the text input in natural language. In an embodiment, the content information can be inferred from the object of the text input in natural language. For example, the object "cat resting" of the verb "painting" can be identified from natural language input expressing "painting a cat resting here", and the electronic device 200 can determine an image of a cat as input content. The input content can be determined from multiple pieces of content (such as images) stored in the electronic device 200, or from multiple pieces of content obtained by searching the Internet. The content information can represent the object of analysis to be input into the AI ​​model to perform actions represented by the action information.

[0081] In this embodiment, the input content can be determined by user input. For example, content selected by the user at electronic device 200 can be determined as the input content. Specific methods for determining the input content can be found later in the references. Figure 6a and Figure 6b Please provide an explanation.

[0082] Meanwhile, according to an embodiment, user intent information can be used to set a target region in the base content. User intent information may include region information. Expressions representing locations or positions (such as "here," "there," "everywhere," "somewhere," prepositions and postpositions indicating location, and any words connected to prepositions and postpositions indicating location) can be identified as region information from text input in natural language. In an embodiment, the target region can be set as the region represented by the region information for a location selected in the base content via user input. When a bounding box is selected in the base content via user input, the region represented by the region information can be set as the target region for the selected bounding box. In an embodiment, a target region can be set in the base content based on user intent information.

[0083] In this embodiment, user intent information may include content attribute information. For example, when content information corresponding to "cat" is identified from a natural language input tag expressing "painting a cat here for a rest," the word "rest" modifying the content information can be identified as content attribute information representing an attribute of the content. This will be discussed later by referring to... Figure 8 , Figure 9 and Figure 10 Please provide an explanation.

[0084] In operation S340, output content can be generated based on input content, target area, and user intent information. The output content can be generated based on the base content. The output content generation unit 240 of the electronic device 200 can generate output content based on input content, target area, and user intent information. Output content refers to content generated through the user's natural language input, and may include content output and displayed to the user by the electronic device 200, as well as content that has been generated but not displayed to the user. In this embodiment, the output content can be generated using an AI model, which will be referred to later. Figure 7a and 7b Please provide an explanation.

[0085] In operation S350, subtitles for the output content can be generated. The subtitle generation unit 250 of the electronic device 200 can generate subtitles for the output content. In this embodiment, the subtitles for the output content can be generated using an AI model. For example, subtitles for the output content can be generated using an image subtitle model. For example, subtitles for animations of the output content can be generated using a speech subtitle model. Various imaging subtitle models and video subtitle models are well known to those skilled in the art, therefore, redundant descriptions are omitted.

[0086] In operation S360, the similarity between the natural language input text and the subtitles can be calculated. The similarity calculation unit 260 of the electronic device 200 can calculate the similarity between the natural language input text and the subtitles. The natural language input text can be determined in operation S310 during the process of obtaining user intent information from the natural language input.

[0087] In this embodiment, the similarity between the natural language input text and the captions can be calculated using an AI model. The similarity can be vector similarity. The natural language input text and captions can be encoded using semantic vectors to generate corresponding vectors, and the similarity between these vectors can be calculated.

[0088] In operation S370, it can be determined whether the similarity meets the conditions. In one embodiment, the condition can be met when the similarity exceeds a preset threshold. In another embodiment, the condition can be met when the similarity is within a preset range.

[0089] If the similarity condition is met in operation S370, it can be determined that the generated output content conforms to the user's intent. If the similarity condition is not met in operation S370, it can be determined that the generated output content does not conform to the user's intent.

[0090] In this embodiment, when the generated output content does not conform to the user's intent, that is, when the similarity does not meet the condition, the process of generating output content can be iterated in order to generate output content that conforms to the user's intent. For example, operations S340, S350, S360, and S370 can be iterated.

[0091] In an embodiment, when the generated output content does not conform to the user's intent, that is, when the similarity does not meet the condition, another input content different from the input content selected in operation S330 can be determined. Then, operations S340, S350, S360 and S370 can iterate with the newly determined other input content in order to generate output content that conforms to the user's intent.

[0092] According to an embodiment, output content that aligns with the user's intent can be generated by comparing the user's natural language input with captions used for the output content. Furthermore, the quality of the output content can be improved through iterative generation.

[0093] Figure 4 These are diagrams illustrating the generated example content according to embodiments of this disclosure.

[0094] In this embodiment, the target area set in the basic content is... Figure 1The target area is shown as a rectangle, but is not limited to this; it can be a circle, ellipse, triangle, pentagon, polygon, or any shape.

[0095] refer to Figure 4 Based on the natural language input expressing "put Amy's face here" and the target region 412a set by the user input, the input content "Amy's face" can be synthesized into the target region 412a to generate output content. The generated output content is then synthesized into a base content 410a, which can be provided to the user. The input content can be obtained by searching the internet.

[0096] In this embodiment, the input content can be obtained from images stored on an electronic device or cloud server using an AI model. Images captured by the user can be processed on the electronic device or cloud server using a face detection AI model, and the detected faces can be named directly by the user or named using the user's browsing history on social networking services (SNS). Therefore, a face named "Amy" can be identified as the input content from the detected faces.

[0097] In this embodiment, the output content is generated by synthesizing the input content into the target region 412a of the base content 410a. Then, captions for the generated output content (e.g., “Amy’s face among friends”) can be generated. The captions for the generated output content can be compared with the text of the natural language input, and the comparison can be used to determine whether to iteratively generate the output content.

[0098] In this embodiment, the bounding box including the face can be identified from the base content 410a using an AI model, and the target region 412a can be set based on the identified bounding box. When natural language input expressing "replace Bell's face with Amy" is received at the electronic device, and Bell's face is detected in the base content 410a, Bell's face can be set as the target region 412a.

[0099] refer to Figure 4The input content 402b, such as "cat" or "jumping cat," can be synthesized into a target region based on natural language input expressing "put the jumping cat here" to generate output content. In an embodiment, the input content 402b can be determined from the base content 400b using an AI model. For example, when the content information derived from the natural language input corresponds to an object detected in the base content 400b, the detected object can be determined as the input content 402b. In an embodiment, the attributes of the input content 402b detected in the base content 400b can be changed based on user intent information. The user intent information may include content attribute information, and the content attribute information may be related to posture. Furthermore, a posture animation corresponding to the posture represented by the content attribute information can be obtained. The posture animation of the posture of the content attribute information can be obtained from a posture animation library. The posture animation library can be generated by detecting key points from the body of an animal or human using an AI model.

[0100] In this embodiment, the input content 402b can be a gesture animation obtained based on user intent information. The output content can be generated using an AI model based on objects and gesture animations within a target area. When the base content 400b is video, each frame of the base content 400b can be modified based on the gesture animation to generate the output content, and then the base content 410b, including the output content, can be provided to the user. The size and shape of the target area can be adjusted according to the gesture animation.

[0101] In this embodiment, the output content is generated by compositing the input content 402b into a target area of ​​the base content. Then, subtitles for the generated output content (e.g., "Jumping Russian Blue Cat") can be generated. The subtitles for the generated output content can be compared with the received natural language input text, and the comparison can be used to determine whether to iteratively generate the output content. AI models (such as video captioning models) can be used to generate the subtitles for the output content.

[0102] In this embodiment, the bounding box of the object can be identified from the base content 400b using an AI model, and the target region can be set based on the identified bounding box. The input content 402b can be determined from either the bounding box of the base content 400b or the target region.

[0103] refer to Figure 4 When natural language input expressing "painting a black and white sun here" is received at an electronic device, the input content of "sun" or "black and white sun" can be synthesized into target area 412c to generate output content. Then, the base content synthesized into the output content can be provided to the user.

[0104] In this embodiment, the output content can be generated using an AI model to have the same form as the base content 410c. For example, when the base content 410c is a cartoon-type image, the output content can also be generated as an image with the same cartoon type. The cartoon-type image can be a vector image. The output content can be a vector image converted from the image. In this embodiment, the output content can be generated based on commands supported by a vector drawing program. The output content can be generated based on the program's application programming interface (API). The method of generating output content based on the program's API will be described later through reference. Figure 15 and Figure 16 This will be explained in more detail. According to the embodiments, the output content can be generated on various platforms and is not limited to image compositing.

[0105] In this embodiment, the output content is generated by synthesizing the input content into the target region 412c of the base content 410a. Then, captions (e.g., "black hole") for the generated output content can be generated. The captions for the generated output content can be compared with the natural language input text, and a determination is made based on the comparison to iteratively generate the output content. The process of generating the output content can iterate when the similarity between the captions of the output content and the user's natural language input text does not meet a specific condition. Iterative generation of the output content can continue until the output content is generated as captions with text similar to the natural language input "black and white sun".

[0106] refer to Figure 4 When a natural language input expressing "make a door here" is received at an electronic device, the input content for "door" can be synthesized into target area 412d to generate output content matching base content 410d. The base content 410d, including the generated output content, can then be provided to the user. In an embodiment, the output content can be generated on the same platform as the base content 410d using an AI model. That is, the output content can be generated in a way that is limited to the platform to which the base content 410d belongs. For example, when the platform of the base content 410d is a specific game, the output content can be generated in a way supported by the game. The game can be a sandbox game or a game creation tool. In an embodiment, the output content can be generated based on commands supported by the game. The output content can be generated based on the game's application programming interface (API). According to an embodiment, the output content can be generated on various platforms and is not limited to image synthesis.

[0107] Figure 5 This is a diagram illustrating a method for obtaining user intent information according to embodiments of the present disclosure.

[0108] In this embodiment, user intent information can be obtained based on the user's natural language input.

[0109] refer to Figure 5 The user intent acquisition unit 210 of the electronic device may include an automatic speech recognition (ASR) model 212 and a natural language understanding (NLU) model 214.

[0110] refer to Figure 5 When speech input is received as natural language input, the speech of the natural language input can be converted into text by ASR model 212. The converted text can then be processed by NLU model 214 to obtain user intent information. The speech input can be received by the microphone of an electronic device. The text converted by ASR model 212 can be compared with captions generated for a later output, and as a result of the comparison, the similarity between the natural language input text and the captions used for the generated captions can be calculated.

[0111] In one embodiment, the user intent acquisition unit 210 may include a spoken language understanding (SLU) model, which can process voice input received through a microphone to obtain user intent information. In another embodiment, natural language input may be transmitted from another device.

[0112] In this embodiment, when text input is received as natural language input, the NLU model 214 can process the text input to obtain user intent information. The text input can be received by an input interface of an electronic device, such as a keyboard, touchscreen, keypad, etc. In this embodiment, the text input can be transmitted from another device. The text input can be compared with captions generated for a later output, and as a result of the comparison, the similarity between the natural language input text and the captions used for the generated captions can be calculated.

[0113] When natural language input text is fed into the NLU model 214, the NLU model 214 can analyze the text to obtain user intent information. The NLU model 214 can obtain action information, content information, content attribute information, and region information, and the obtained information can be used to generate output content. In addition to the described information, user intent information can also be obtained according to various categories to enable electronic devices to understand the user's intent.

[0114] Figure 6a This is a flowchart of a method for determining input content according to an embodiment of the present disclosure.

[0115] Figure 6b An example graphical user interface (GUI) for selecting input content is shown according to an embodiment of the present disclosure.

[0116] Reference Figure 6a and 6b , Figure 3Operation S330 can be performed differently depending on whether user input for selecting input content is received. Operation S330 can be performed after receiving user input and natural language input for selecting target region 602 in base content 600, but is not limited to this. For example, the input content can be determined before receiving user input and natural language input for selecting target region 602 in base content 600.

[0117] In operation S331, it can be determined whether user input selecting input content has been received. When it is determined in operation S331 that user input selecting input content has been received, in operation S332, the content selected by the user input can be determined as the input content.

[0118] For example, refer to Figure 6b The GUI 603, used to select input content, can be displayed on the screen of an electronic device, and the content can be selected by the user from the GUI 603 as input content.

[0119] If it is determined in operation S331 that no user input for selecting input content has been received, in operation S333, it can be determined whether the content information is included in the user intent information. If it is determined in operation S333 that the content information is included in the user intent information, in operation S334, the input content can be determined based on the content information. For example, refer to... Figure 6b When natural language input expressing "draw a cat to rest here" is received, "cat" can be identified as content information. Then, the electronic device can identify "cat" as content information in response to the text input from the natural language input, and determine the cat image as the input content. In an embodiment, the input content can be determined based on the content information with the user's confirmation. For example, found content can be displayed based on the content information, and then, when the user agrees to use the found content as input content, the found content can be determined as the input content. When multiple pieces of content are found based on the content information, a GUI 603 for selecting the input content from the multiple pieces of content can be displayed on the electronic device to allow the user to select the input content.

[0120] When it is determined in operation S333 that the content information is not included in the user intent information, multiple pieces of content can be displayed in operation S335. That is, a GUI 603 for selecting input content from multiple pieces of content can be displayed on the electronic device, and the input content can then be determined from the multiple pieces of content based on user input. The GUI 603 for determining the input content can be displayed when the content information is not obtained based on natural language input, and when the content information is not included in the user intent information.

[0121] Figure 7aThis is a diagram illustrating a Generative Adversarial Network (GAN) according to embodiments of the present disclosure.

[0122] The output content generation unit 240 may include an AI model for generating output content.

[0123] refer to Figure 7a The output content generation unit 240 may include a generative model of the GAN model, namely, a generator 242. The output content generation unit 240 may also include a discriminative model of the GAN model, namely, a discriminator 244. The generator 242 and discriminator 244 of the GAN model can be trained adversarially against each other. The discriminator 244 can be trained based on real and fake content to determine whether the output content generated by the generator 242 is real or fake. The generator 242 can generate output content that can be determined as real content by the discriminator 244 based on the input data. The generator 242 and discriminator 244 can be trained until an equilibrium point is reached. When the two networks reach an equilibrium point, the loss function of the GAN can be appropriately adjusted to adequately mimic the data probability distribution of real content.

[0124] In operation S742, generator 242 can generate output content based on the target region of the input content and the base content. The probability distribution of the output content generated by generator 242 that has reached or is close to the equilibrium point can approximate the probability distribution of the real content. Generator 242 can generate output content whose probability distribution approximates the probability distribution of the real content, thereby enabling the output content discriminator 244 to determine it as real content.

[0125] In operation S744, discriminator 244 can distinguish between real and fake content for a target area. In operation S745, discriminator 244 can distinguish between real and fake content for the entire area of ​​the basic content.

[0126] In operation S780, generator 242 can be trained based on the discrimination results of discriminator 244. Discriminator 244 is trained to distinguish between real and fake content for the target region and the entire region, and generator 242 is trained to generate output content that can be determined by discriminator 244 to be real content for the target region and the entire region. Therefore, the quality of the generated output content can be improved.

[0127] refer to Figure 4The probability that the output content generated by generator 242 is identified as real content by discriminator 244 can be approximated as 0.5, but is not limited to this. That is, the data probability distribution of the output content generated by generator 242 can approximate the data probability distribution of the real content. The real content used as a reference for discrimination can have the same form as the base content, or be supported by the same platform as the base content. In other words, real content collected on the same platform as the base content, or real content with the same form as the base content, can be used to train discriminator 244.

[0128] Figure 7b This is a diagram illustrating a method of using subtitles according to an embodiment of this disclosure.

[0129] In an embodiment, the discriminator 244 can be further trained based on the similarity between the natural language input text and the captions used for the content. After the generator 242 and the discriminator 244 of the GAN reach an equilibrium point, the probability that the output content generated by the generator 242 in operation S742 is identified as fake content by the discriminator 244 can be approximately 0.5, but is not limited to this.

[0130] refer to Figure 7b The discriminator 244 can determine whether the output content is real or fake based on the similarity between the natural language input text and the subtitles used for the output content. Specifically, the subtitles can be generated by the subtitle generation unit for the output content generated by the generator 242, and the similarity between the generated subtitles and the user's natural language input text can be calculated by the similarity calculation unit. When it is determined in operation S746 that the similarity does not meet a predetermined condition, the discriminator 244 can determine that the generated output content is fake. When it is determined in operation S746 that the similarity meets a predetermined condition, the discriminator 244 can determine that the generated output content is real. In this embodiment, the discriminator 244 can be further trained based on the similarity between the natural language input text and the subtitles of the output content. Furthermore, in operation S780, the generator 242 is also trained to generate output content that can be determined as real by the discriminator 244. The generator 242 can iteratively generate the output content, which can improve the quality of the output content.

[0131] Figure 8 This is a diagram illustrating a method for changing the attributes of input content according to an embodiment of the present disclosure.

[0132] refer to Figure 8A picture of the parents can be selected as input content 804 via user input 1. A target area 802 can be set in the base content 800 based on user input 2. When the electronic device receives natural language input expressing "let them stand here," it can obtain user intent information based on the natural language input. In the natural language input, "let" can correspond to action information. In the natural language input, "they" can correspond to content information. In the natural language input, "stand" modifying the content information may be identified as content attribute information representing the attributes of the content.

[0133] In this embodiment, output content 814 can be generated based on content attribute information. Output content 814 can be generated by compositing input content into a target area 802 of base content 800. The attributes of output content 814 can be the same as those represented by the content attribute information. The attributes of output content 814 can be changed based on the attributes of input content 804. Content attributes may include, but are not limited to, the content's pose, facial expression, makeup, hair, clothing, accessories, or style. Content attributes may include, but are not limited to, the pose, facial expression, makeup, hair, clothing, accessories, or style of objects within the content. For example, refer to... Figure 8 The attributes of input content 804 (such as posture) can be a sitting posture, and the attributes of output content 814 can be a standing posture. Methods for changing the attributes of input content 804 will be explained later in the reference. Figure 9 and Figure 10 Please provide an explanation.

[0134] In an embodiment, output content 814 can be generated to match base content 800. Output content 814 can be generated to have the same attributes as base content 800 or objects within base content 800 by referencing a target region 802 or the entire region of base content 800. For example, referencing... Figure 8 When all objects in the base content 800 wear white clothes, the output content 814, in which the objects also wear white clothes, can be generated. That is, even if objects in the input content 804 wear different colored clothes, the output content 814, where the objects wear white clothes, can be generated, and the base content 810 and / or its target region 812 can include the output content 814. For example, see reference... Figure 8 When all objects in the base content 800 are standing, output content 814, in which the objects are also standing, can be generated. That is, even if the objects in the input content 804 are sitting on the floor, output content 814, in which the objects are standing, can be generated, and the base content 810 and / or its target area 812 can include output content 814. A method to make output content 814 have the same properties as the base content 810 will be provided by referencing... Figure 9 and Figure 10Please provide an explanation.

[0135] Figure 9 This is a diagram illustrating a method for generating content from a GAN, according to an embodiment of this disclosure.

[0136] In this embodiment, content attribute information can be obtained from natural language expressions that modify the content information. Content attribute information can be related to attributes in the input content that the user wants to change. For example, when the input content includes a person, attributes such as the person's posture, facial expressions, makeup, hair, clothing, accessories, or style that the user wants to change can be identified as content attribute information.

[0137] refer to Figure 9 The output content generation unit may include two generators, namely, a first generator 242a and a second generator 242b. Furthermore, the output content generation unit may include two discriminators, namely, a first discriminator 244a and a second discriminator 244b.

[0138] In operation S942a, the first generator 242a can generate output content with attribute "B" based on input content with attribute "A". That is, the first generator 242a can generate fake content with attribute "B" based on real content with attribute "A". The first generator 242a can be trained to generate output content with attribute "B".

[0139] In operation S942b, the second generator 242b can reconstruct content with attribute "A" based on the generated output content with attribute "B". That is, the second generator 242b can reconstruct the original content with attribute "A" from spurious content with attribute "B". The first generator 242a can be trained to generate output content with attribute "A". The first generator 242a can generate output content by changing only the attribute of the input content from attribute "A" to attribute "B", while changes to other attributes are suppressed, so that the second generator 242b can reconstruct content similar to the original input content with attribute "A". According to the embodiment, generating content unrelated to the original content can be suppressed by reconstructing the original content. When the reconstructed content approximates the input content, the loss value can be reduced, which can be interpreted as the cyclic consistency between the input and output content being maintained.

[0140] Meanwhile, in operation S944a, the first discriminator 244a can determine whether the output content with the "B" attribute generated by the first generator 242a is real content or fake content. The first discriminator 244a can be trained using real content with the "B" attribute to distinguish between real and fake content. In operation S980a, the first generator 242a can be trained based on the discrimination result of the first discriminator 244a.

[0141] In operation S944b, the second discriminator 244b can determine whether the content reconstructed by the second generator 242b with the attribute "A" is real content or fake content. The second discriminator 244b can be trained using real content with the attribute "A" to distinguish between real and fake content. In operation 980b, the second generator 242b can be trained based on the discrimination result of the second discriminator 244b.

[0142] According to an embodiment, when the attribute of the input content is changed from attribute "A" to attribute "B", the consistency of other attributes can be maintained. According to an embodiment, the attribute of the input content can be changed from attribute "A" to attribute "B" without marking the attributes of the content.

[0143] Figure 10 This is a diagram illustrating a method for generating content from a GAN, according to an embodiment of this disclosure.

[0144] refer to Figure 10 The output content generation unit may include a generator 242 and a discriminator 244.

[0145] In operation S1042, generator 242 can generate output content based on attribute labels and input content. Attribute labels can represent labels of any attribute that can be classified by discriminator 244. That is, generator 242 can receive attribute labels and input content to generate output content. Generator 242 can be trained to generate output content that can be identified as real content by discriminator 244 and classified into the received attribute labels.

[0146] In operation S1043, generator 242 can reconstruct content based on the same attribute labels and output content as those in operation S1042. That is, generator 242 can receive initial attribute labels and generated output content to reconstruct content. Generator 242 can modify the input content only to the extent that the generated output content can be classified as attribute labels, so that generator 242 can reconstruct content close to the initial content (input content). The input content is modified to the extent that the output content can be classified as attribute labels, and output content classified as a specific attribute can be generated. Because the reconstructed content approximates the input content, the loss value of the output content can be reduced.

[0147] In operation S1044, discriminator 244 can determine whether the output content generated by generator 242 is real or fake content. When the output content is determined to be real content, in operation S1045, discriminator 244 can classify the attributes of the output content. Discriminator 244 can receive real and fake content to distinguish between real and fake content. Discriminator 244 can classify the attributes of the output content determined to be real content to train discriminator 244.

[0148] In operation S1080, generator 242 can be trained based on the discrimination result of discriminator 244.

[0149] According to an embodiment, when the attribute of the input content is changed from attribute "A" to attribute "B", the consistency of other attributes can be maintained. Figure 10 To illustrate this, we can use an example that only changes one attribute, but the number of attributes of the input that can be changed can be increased based on the number of attributes that the discriminator 244 is configured to classify.

[0150] Figure 11 This is a diagram illustrating a method for using user feedback according to an embodiment of this disclosure.

[0151] refer to Figure 11 The electronic device can receive user feedback regarding the base content 1110 into which the output content is synthesized, the output content, or a region of the output content corresponding to the target region 1112 of the input content. In an embodiment, the process of generating the output content can be iterative when user feedback is received. For example, if the similarity between the captions of the output content and the natural language input does not meet a specific condition, the process of generating the output content can be iterative when negative user feedback is received.

[0152] According to an example, user feedback can be reflected in the AI ​​model to provide users with more customized results.

[0153] Figure 12 An example GUI for receiving user feedback is shown according to an embodiment of this disclosure.

[0154] refer to Figure 12 The electronic device can receive user feedback on the base content 1210 into which the output content is synthesized, the output content, or a region 1212 of the output content corresponding to a target region of the input content. In an embodiment, when user feedback is received, a GUI 1213 for changing the attributes of objects in the input content can be displayed. For example, when negative user feedback is received on the attributes of the output content, a GUI for changing the attributes of the output content can be displayed so that the user can select attributes that are favorable to him or her.

[0155] Figure 13 This is a diagram illustrating the content generated in the workspace of an application according to an embodiment of the present disclosure.

[0156] refer to Figure 13 The content of the black cloud is determined as input content 1304, and natural language input expressing "put this cloud here" can be received at the electronic device. In this embodiment, when user input is received in target areas 1312a, 1312b, 1312c, and 1312d, the electronic device can generate output content in target areas 1312a, 1312b, 1312c, and 1312d. The output content can be generated using an AI model based on the underlying content. Figure 13 As shown, when workspace 1310 includes another object such as a white circle, output content can be generated for workspace 1310 that includes the white circle object. The relationship between the output content and workspace 1310 can be obtained based on user intent information determined from natural language input. For example, by referencing the white cloud in workspace 1310 based on the expression "put here," a white cloud can be generated as multiple output contents (work objects). That is, output content with a different color than the initial input content 1304 can be generated.

[0157] Figure 14 This is a diagram showing the content of the workspace adaptively generated for illustration and application according to embodiments of this disclosure.

[0158] refer to Figure 14 The silhouette of a woman holding a bag is identified as input content 1404, and natural language input expressing "have her hold the arrow like on the left" can be received at the electronic device. When user input is received in target area 1412, the electronic device can generate output content in target area 1412 by using an AI model.

[0159] refer to Figure 14 When workspace 1410 includes another object such as a silhouette of a man holding an arrow, output content can be generated for workspace 1410 that includes other objects. For example, a silhouette of a man holding an arrow in workspace 1410, and a woman holding an arrow like the man, can be generated as output content based on the expression reference of "holding an arrow like on the left".

[0160] Figure 15 This is a diagram illustrating a method for generating animations according to embodiments of the present disclosure.

[0161] According to an embodiment, when the base content is the application's workspace, the output content can be generated based on the application's API. The application's API can call various commands related to animation. Animation-related commands may include, but are not limited to, commands for setting animation motion parameters (such as appear, highlight, hide, move), the start and end points of motion parameters (such as where to appear, where to hide, where to move), the movement direction of motion parameters, the movement timing of motion parameters, the movement speed of motion parameters, etc.

[0162] refer to Figure 15 The output can be an animation related to the input 1504. The input 1504 can be a working object.

[0163] Animations related to input content 1504 can be generated using the aforementioned AI models (such as GAN models). The generator of a GAN model can receive input content 1504, the application's API, and user intent information, and then combine the commands provided by the application's API to generate animations related to input content 1504, which can be played (animate) according to the user's intent.

[0164] For example, when input content 1504 is selected in workspace 1500 and natural language input expressing "apply an animation to count this number" is received at the electronic device, the animation can be generated from the "applied animation" identifier as action information. The animation associated with input content 1504 can be generated in response to the action information indicating the generation of the animation. The target area can be, but is not limited to, the area in workspace 1500 where input content 1504 is located. Animations that conform to the user's intent can be inferred from content attribute information. For example, "the number" can be identified as content information in the natural language input, and "count" modifying the content information can be identified as content attribute information. The animation can be generated based on input content 1504, the applied API, and the content attribute information of "count".

[0165] An electronic device can generate captions for the content 1514 used to apply animation. The captions describing the animation of content 1514 can be generated using a video captioning model. The animation generation process can iterate based on the similarity between the generated captions and the text input in natural language. Therefore, animations that conform to the user's intent can be generated. According to an embodiment, the animation can be applied to input content 1504 using natural language input, even if the user does not know the exact command of the application. The animation of input content 1504 can function in the presentation mode of workspace 1510.

[0166] Figure 16 This is a diagram illustrating an example animation generated according to an embodiment of the present disclosure.

[0167] refer to Figure 16 The base content can be the application's workspace 1600, and the input content 1604 can be a work object. For example, a work object can be created using tools supported by the application.

[0168] When a work object is selected in workspace 1600 and natural language input expressing "apply an animation to make the graphic grow horizontally" is received at the electronic device, the animation can be generated from the "applied animation" identifier as action information. Animations related to the input content 1604 can be generated in response to the action information indicating the generation of the animation. Animations that match the user's intent can be inferred from content attribute information. For example, "the graphic" can be identified as content information in the natural language input, and "horizontal growth," which modifies the content information, can be identified as content attribute information. The animation can be generated based on the input content 1604, the applied API, and the content attribute information of "horizontal growth."

[0169] Figure 17 An example electronic device according to an embodiment of this disclosure is shown.

[0170] refer to Figure 17 An electronic device 2000 or software running on an electronic device 2000 may perform the operations of the methods described or illustrated herein to provide the functions described or illustrated herein. An electronic device 2000 may refer to one or more electronic devices 2000. One or more electronic devices 2000 may perform the operations of the methods described or illustrated herein without substantial spatial or temporal limitations. One or more electronic devices 2000 may perform the operations of the methods described or illustrated herein in real time or in batch processing.

[0171] Electronic device 2000 may include, but is not limited to, smartphones, tablet PCs, PCs, smart TVs, mobile phones, personal digital assistants (PDAs), laptop computers, media players, microservers, global positioning systems (GPS), e-book terminals, digital broadcasting terminals, navigation devices, telephone booths, MP3 players, gateway devices, digital cameras, home appliances, and other mobile or non-mobile computing devices. Furthermore, electronic device 2000 may be a wearable device with communication and data processing functions, such as, but not limited to, watches, glasses, hairbands, and bracelets. However, embodiments of this disclosure are not limited thereto, and electronic device 2000 may include any means capable of generating output content. Electronic device 2000 may be an end-user device.

[0172] Electronic device 2000 may include processor 2300 and memory 2700. Software modules of electronic device 2000 (such as program modules) may be stored as sets of instructions in memory 2700, which can be executed by the processor to perform corresponding functions. Electronic device 2000 can... Figure 17 The components shown can be implemented with more or fewer components.

[0173] For example, electronic device 2000 may include input interface 2100, output interface 2200, processor 2300, communication interface 2500, microphone 2600, and memory 2700.

[0174] The processor 2300 of the electronic device may include, but is not limited to, an AI processor for generating trained network models. The AI ​​processor may be implemented as a separate chip from the processor 2300. The processor 2300 may perform each operation of the methods described or illustrated herein.

[0175] The input interface 2100 of the electronic device can receive user input. The input interface 2100 may include units for interacting with the user. For example, the user interface 1100 may be a keypad, a dome switch, a touchpad (e.g., contact capacitive type, pressure resistance film type, infrared detection type, surface acoustic wave propagation type, overall strain type, piezoelectric effect type, etc.), a haptic wheel, and a haptic switch, but is not limited to these.

[0176] The output interface 2200 of the electronic device can output information to be provided to the user, such as audio signals, video signals, or vibration signals. The output interface 2200 may include, but is not limited to, a display, a sound output interface, and a vibration motor.

[0177] The audio output interface can output audio data received from the communication interface 2500 or stored in the memory 2700. Furthermore, the audio output interface can output audio signals related to functions performed by the electronic device 2000 (e.g., call signal reception audio, information reception audio, notification audio, etc.). The audio output interface 2200 may include a speaker, buzzer, etc.

[0178] A vibration motor can output a vibration signal. For example, a vibration motor can output a vibration signal based on outputting audio or video data. A vibration motor can also output a vibration signal in response to receiving touch input.

[0179] Output interface 2200 can provide a dialog interface to the user. The dialog interface can be provided in the form of a message window or a chat window, but is not limited to these.

[0180] The communication interface 2500 of the electronic device 2000 may include one or more components that allow the electronic device 2000 to communicate with another device (such as a server). For example, the communication interface 2500 may include a short-range communication interface, a mobile communication interface, and a broadcast receiver for communicating with a network.

[0181] The network may include, for example, but not limited to, a local area network (LAN), a wide area network (WAN), a value-added network (VAN), a mobile radio communication network, a satellite communication network, and any combination thereof. Furthermore, the network may include, in a comprehensive sense, configurations to enable... Figure 1 The data communication network shown enables smooth communication between network entities, and includes, for example, but not limited to, wired internet, wireless internet, mobile wireless communication networks, etc. Examples of wireless communication technologies may include, but are not limited to, wireless LAN (WLAN) or Wi-Fi, Bluetooth, Bluetooth Low Energy (BLE), ZigBee, Wi-Fi Direct (WFD), ultra-wideband (UWB), infrared communication (Infrared Data Association (IrDA)), near field communication (NFC), etc.

[0182] The mobile communication interface can transmit wireless signals via a mobile communication network to at least one of a base station, an external terminal, and a server. Wireless signals may include voice call signals, video call signals, or any type of data used to transmit text / multimedia messages.

[0183] A broadcast receiver can receive broadcast signals and / or broadcast-related information from external sources via a broadcast channel. Broadcast channels can include satellite channels, terrestrial channels, etc.

[0184] Microphone 2600 can receive and process sound signals from an external source to convert them into electronic sound data. For example, microphone 2600 can receive sound signals from an external device or speaker. Microphone 2600 can employ any of a variety of noise reduction algorithms to reduce noise that occurs when receiving sound signals from an external source.

[0185] The memory 2700 can store programs for processing and control, as well as input or output data. The programs stored in the memory 2700 can be categorized into multiple modules.

[0186] According to this disclosure, functions related to artificial intelligence (AI) can be operated via processor 2300 and memory 2700. Processor 2300 may include one or more processors. The one or more processors 2300 may include, but are not limited to, general-purpose processors (such as central processing units (CPUs)), application processors (APs), digital signal processors (DSPs), dedicated graphics processors (such as graphics processing units (GPUs) or vision processing units (VPUs)), dedicated AI processors (such as neural processing units (NPUs)). The one or more processors 2300 can control input data to be processed according to predefined operating rules or AI models stored in memory 2700. When the one or more processors 2300 are dedicated AI processors 2300, the dedicated AI processors 2300 can be designed with hardware architectures specifically designed to process a particular AI model.

[0187] Predefined operating rules or AI models can be created via a training process. For example, a predefined operating rule or AI model can be set to perform desired characteristics (or objectives) created by training a base AI model using a learning algorithm that utilizes a large amount of training data. The training process can be performed by a device used to perform the AI ​​or by a separate server and / or system. Examples of learning algorithms may include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, and reinforcement learning, but embodiments of this disclosure are not limited thereto.

[0188] The method can be performed by hardware, software, or a combination of hardware and software according to the embodiments. When the embodiments are implemented using software, one or more programs (software, software modules) may be used. The program may be included in a computer-readable recording medium, but is not limited thereto. The program may be included in a computer program product. A computer-readable recording medium storing the program may be included in a computer program product. The program may be configured to be executable by one or more processors in an electronic device. The one or more processors may include instructions that, when executed by the one or more processors in the electronic device, cause the electronic device to perform the method according to the embodiments.

[0189] Where appropriate, a computer-readable recording medium may be volatile, non-volatile, or a combination of volatile and non-volatile. A computer-readable recording medium may include computer storage media and communication media. A computer-readable recording medium may be any available medium accessible by a computer and includes both volatile and non-volatile media, as well as both removable and non-removable media. Furthermore, a computer-readable recording medium may include computer storage media and communication media. Computer storage media includes both volatile and non-volatile media, as well as both removable and non-removable media, implemented by any method or technology to store information, such as computer-readable instructions, data structures, program modules, or other data. Communication media typically contain computer-readable instructions, data structures, or program modules, and include any information transmission medium. Embodiments of this disclosure may be implemented by a computer-readable recording medium or computer program product on which computer-executable instructions (such as program modules executed by a computer) are recorded. Multiple computer-readable recording media may be distributed across multiple computer systems connected via a network, and data (such as instructions and code) stored in the distributed recording media may be executed by at least one processor.

[0190] The functionality of the various elements shown in the diagram can be provided not only by using hardware capable of executing the relevant software but also by using dedicated hardware. When functionality is provided by a processor, it can be provided by a dedicated processor, a single shared processor, or multiple separate processors that can be partially shared.

[0191] The terms “processor” or “controller” should not be interpreted as referring only to hardware that can execute software, and may include digital signal processor hardware, read-only memory, random access memory, and volatile storage devices for storing software.

[0192] In this disclosure, terms such as “-unit” or “-module” should be understood as a unit that performs at least one function or operation. A component referred to as a “-unit” or “-module” can be implemented as hardware, software, or a combination of hardware and software.

[0193] When expressions such as "at least one" precede a list of elements, they modify the entire list and not any individual elements within it. For example, the expression "at least one of a, b, and c" should be understood as including only a, only b, only c, including both a and b, including both a and c, including both b and c, or including all of a, b, and c.

[0194] The operations of all methods described herein may be performed in an appropriate order unless otherwise indicated herein or clearly contradicted by the context, and are not limited to the indicated order. The use of any and all examples or language provided herein (e.g., “such as”) is solely for the purpose of clearly describing embodiments and does not constitute a limitation on the embodiments unless otherwise stated. For example, each function may be performed in a distributed or combined manner.

[0195] While this disclosure has been shown and described with reference to various embodiments thereof, those skilled in the art will understand that various changes in form and detail may be made in this disclosure without departing from the spirit and scope of this disclosure as defined by the appended claims and their equivalents.

Claims

1. An apparatus for generating improved output content through iteration, the apparatus comprising: Memory, which stores instructions; and At least one processor is configured to execute instructions to: It can receive natural language input while displaying basic content; User intent information is obtained based on natural language input by using a natural language understanding model; The target area in the base content is set based on the first user input, which includes touch input applied to the base content. The first input content is determined based on user intent information or second user input. By using a neural network model, a first output content related to the basic content is generated based on the first input content, the target region, and the user intent information; Subtitles for the first output content are generated using an image captioning model; Calculate the similarity between the natural language input text and the generated subtitles for the first output content; as well as The final output content is generated iteratively based on the similarity. When the similarity does not meet the predetermined conditions, a second output content different from the first output content is generated by using the neural network model based on the first input content, the target region, and the user intent information.

2. The device according to claim 1, in, The basic content, the first input content, and the first output content are images, and The first output content is generated by compositing the first input content into the target area of ​​the base content.

3. The device according to claim 1, in, The basic content includes multiple areas, and The target area includes the area selected from multiple areas by the first user input.

4. The device according to claim 1, in, Natural language input includes language input, and This involves using an Automatic Speech Recognition (ASR) model to convert speech input into text that is natural language input.

5. The device according to claim 1, wherein, The first input content is determined based on the content information included in the user intent information.

6. The device according to claim 5, wherein, The first input content is further determined from multiple pieces of content corresponding to the content information, and the attributes of the multiple pieces of content are different from each other.

7. The device according to claim 1, in, The attributes of the first input content include at least one of posture, facial expression, makeup, hair, clothing, or accessories, and The attributes of the first input content are determined based on the content attribute information included in the user intent information.

8. The device according to claim 1, wherein, The neural network model mentioned is related to the generative adversarial network model. The first output content is generated by the generator of the generative adversarial network model, and When the similarity does not meet the predetermined conditions, the discriminator of the generative adversarial network model will identify the first output content as fake content.

9. The device according to claim 1, in, At least one processor is also configured to: when the similarity does not meet a predetermined condition, execute instructions to: Determine a second input that is different from the first input; as well as When the similarity does not meet the predetermined conditions, the neural network model is used to generate a second output content that is different from the first output content based on the second input content and the target region.

10. The device according to claim 1, wherein, At least one processor is also configured to execute instructions to: Receive user feedback on a portion of the first output content, and The first output content is modified by using the neural network model.

11. The device according to claim 1, in, The basic content includes the application's workspace, and The first input includes the work objects located in the workspace.

12. The device according to claim 11, in, The first output includes animations related to the work object. The animation is generated based on the working object, user intent information, and the application's application programming interface (API). The subtitles used for the first output content include subtitles used for animation.

13. A method for iteratively generating improved output content, the method comprising: It can receive natural language input while displaying basic content; User intent information is obtained based on natural language input by using a Natural Language Understanding (NLU) model; The target area in the base content is set based on the first user input, which includes touch input applied to the base content. The first input content is determined based on user intent information or second user input. By using a neural network (NN) model, a first output content related to the basic content is generated based on the first input content, the target region, and the user intent information; Subtitles for the first output content are generated using an image captioning model; Calculate the similarity between the natural language input text and the generated subtitles for the first output content; as well as The final output content is generated iteratively based on the similarity. When the similarity does not meet the predetermined conditions, a second output content different from the first output content is generated by using the neural network model based on the first input content, the target region, and the user intent information.

14. A computer-readable storage medium comprising instructions that, when executed by at least one processor, cause at least one processor to: It can receive natural language input while displaying basic content; User intent information is obtained based on natural language input by using a Natural Language Understanding (NLU) model; The target area in the base content is set based on the first user input, which includes touch input applied to the base content. The first input content is determined based on user intent information or second user input. By using a neural network (NN) model, a first output content related to the basic content is generated based on the first input content, the target region, and the user intent information; Subtitles for the first output content are generated using an image captioning model; Calculate the similarity between the natural language input text and the generated subtitles for the first output content; as well as The final output content is generated iteratively based on the similarity. When the similarity does not meet the predetermined conditions, a second output content different from the first output content is generated by using the neural network model based on the first input content, the target region, and the user intent information.

Images