Apparatus, method, and program for generating enhanced output content by iteration
Through the artificial intelligence system, NLU, NN and GAN models are used to solve the shortcomings of existing intelligent systems in understanding and generating user intentions, and high-quality iterative content generation is achieved to ensure that the output content is consistent with the user intention.
Patent Information
- Application Number
- CN202411027872.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2019-12-04
- Filing Date
- 2020-12-04
- Publication Date
- 2024-11-29
AI Technical Summary
Existing rules-based intelligent systems are difficult to accurately understand and generate output content that meets user intentions, especially when dealing with complex and variable user input, and lack effective iterative generation capabilities.
Using artificial intelligence systems, through natural language understanding (NLU) models and neural network (NN) models, combined with the generation adversarial network (GAN) models, receive natural language input, determine user intentions, generate and iterate the output content until the user intention is met Similarity conditions to input content.
It realizes the generation of output content that meets user's intentions, improves the accuracy and quality of content generation, and can iterate when the similarity conditions are not met, ensuring the authenticity and user satisfaction of the output content.
Smart Images

Figure CN119045698A_ABST
Abstract
Description
[0001] This application is a divisional application of the invention patent application with the application date of December 4, 2020, application number: 202080084570.7, and invention name: "Apparatus, method and program for iteratively generating enhanced output content". Technical Field
[0002] The present disclosure relates to an artificial intelligence (AI) system and its application for mimicking the functions of the human brain (such as cognition and judgment) by utilizing a machine learning algorithm. More specifically, the present disclosure relates to improving output content by iterative generation using AI. Background Art
[0003] Artificial intelligence (AI) systems can refer to computer systems that enable machines to become smart through learning and autonomous decision-making, which is different from existing rule-based intelligent systems. AI systems can improve their recognition rate and can understand user preferences more accurately through experience. Therefore, existing rule-based intelligent systems are gradually being replaced by AI systems based on deep learning.
[0004] AI technologies can include machine learning (deep learning) and element technologies that use machine learning.
[0005] Machine learning may refer to algorithmic technology used to autonomously classify / learn features of input data, and element technology is technology used to simulate the functions of the human brain (such as cognition and decision-making) using machine learning algorithms, and includes technical fields such as language understanding, visual understanding, reasoning / prediction, knowledge representation, motion control, etc.
[0006] For example, various technical fields to which AI technology can be applied are as follows. Language understanding refers to technology for recognizing human language / characters for application / processing, and includes natural language processing, machine translation, dialogue systems, question-answering, speech recognition / synthesis, etc. Visual understanding refers to technology for recognizing and processing objects in the same way as performed by the human visual system, and includes object recognition, object tracking, image retrieval, character recognition, scene understanding, spatial understanding, image enhancement, etc. Reasoning / prediction refers to technology for judging information and logically inferring and predicting new information, and includes knowledge / probability-based interference, optimization prediction, preference-based planning, suggestions, etc. Knowledge representation refers to technology for automatically processing information about human experience as knowledge data, and includes knowledge construction (data generation / classification), knowledge management (data utilization), etc. Motion control refers to technology for controlling vehicle autonomous driving and robot motion, and includes motion control (navigation, collision avoidance and driving), manipulation control (action control), etc.
[0007] The above information is presented as background information only to assist in understanding the present disclosure. No determination has been made, and no assertion has been made, as to whether any of the above may be applicable prior art to the present disclosure. Summary of the invention
[0008] Solution to the problem
[0009] Aspects of the present disclosure are to address at least the above-mentioned problems and / or disadvantages and to provide at least the advantages described below. Therefore, one aspect of the present disclosure is to provide an apparatus and method for improving output content by iterative generation using AI.
[0010] Additional aspects will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the presented embodiments.
[0011] According to the embodiment, content that meets the user's intention can be generated.
[0012] According to an embodiment, a process of generating content may be improved.
[0013] According to one aspect of the present disclosure, a device for improving output content by iterative generation is provided. The device includes: a memory storing instructions; and at least one processor configured to execute instructions to: receive natural language input; obtain user intent information based on the natural language input by using a natural language understanding (NLU) model; set a target area in the base content based on the first user input; determine the input content based on the user intent information or the second user input; generate output content related to the base content based on the input content, the target area and the user intent information by using a neural network (NN) model; generate captions for the output content by using an image caption model; calculate the similarity between the text of the natural language input and the generated output content; and iteratively generate the output content based on the similarity.
[0014] In an embodiment, the base content, the input content, and the output content are images, and the output content is generated by compositing the input content into a target area of the base content.
[0015] In an embodiment, the base content includes a plurality of regions, and the target region includes a region selected from among the plurality of regions by a first user input.
[0016] In an embodiment, the speech input is converted to text of the natural language input by using an automatic speech recognition (ASR) model.
[0017] In an embodiment, the input content is determined based on content information included in the user intention information.
[0018] In an embodiment, the input content is determined from a plurality of pieces of content corresponding to the content information.
[0019] In an embodiment, attributes of the plurality of pieces of content are different from each other.
[0020] In an embodiment, the attribute of the input content includes at least one of gesture, facial expression, makeup, hair, clothing, or accessories, and the attribute of the input content is determined based on content attribute information included in the user intention information.
[0021] In an embodiment, the NN model is associated with a generative adversarial network (GAN) model, and the output content is generated by a generator of the GAN model.
[0022] In an embodiment, the probability distribution of the output content corresponds to the probability distribution of the real content.
[0023] In an embodiment, the probability distribution of the base content included in the output content approximates the probability distribution of the real content.
[0024] In an embodiment, the NN model is associated with a generative adversarial network (GAN) model, and when the similarity does not satisfy a predetermined condition, the discriminator of the GAN model identifies the output content as false content.
[0025] In an embodiment, the output content is a first output content, and the processor is further configured to: when the similarity does not satisfy a predetermined condition, execute instructions to: generate a second output content different from the first output content based on the input content, the target area and the user intention information by using the NN model.
[0026] In an embodiment, the input content is a first input content, and the output content is a first output content, and the processor is further configured to: when the similarity does not satisfy a predetermined condition, execute instructions to: when the similarity does not satisfy the predetermined condition, determine a second input content different from the first input content, and generate a second output content different from the first output content based on the second input content and the target area by using an NN model.
[0027] In an embodiment, the processor is further configured to execute the instructions to: receive user feedback for a portion of the output content, and modify the portion of the output content by using the NN model.
[0028] In an embodiment, the base content includes a workspace of an application, and the input content includes a work object located in the workspace.
[0029] In an embodiment, the output content includes an animation related to the work object, and the animation is generated based on the work object, user intent information, and an application programming interface (API) of the application.
[0030] In an embodiment, the subtitles for the output content include subtitles for the animation.
[0031] In an embodiment, the NLU model, the NN model, and the image captioning model are stored in a memory.
[0032] According to another aspect of the present disclosure, a method for improving output content by iterative generation is provided. The method includes: receiving natural language input; obtaining user intent information based on the natural language input by using a natural language understanding (NLU) model; setting a target area in the base content based on the first user input; determining the input content based on the user intent information or the second user input; generating output content related to the base content based on the input content, the target area and the user intent information by using a neural network (NN) model; generating subtitles for the output content by using an image subtitle model; calculating the similarity between the text of the natural language input and the generated output content; and iteratively generating the output content based on the similarity.
[0033] According to another aspect of the present disclosure, a computer-readable storage medium is provided. The computer-readable storage medium includes instructions, which, when executed by at least one processor, cause the at least one processor to: receive natural language input; obtain user intent information based on the natural language input by using a natural language understanding (NLU) model; set a target area in the base content based on the first user input; determine the input content based on the user intent information or the second user input; generate output content related to the base content based on the input content, the target area and the user intent information by using a neural network (NN) model; generate subtitles for the output content by using an image subtitle model; calculate the similarity between the text of the natural language input and the generated output content; and iteratively generate the output content based on the similarity.
[0034] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing instructions is provided, which, when executed by at least one processor of a device, causes the device to: present base content; receive user input for selecting a target area of the base content; present an indication corresponding to the selected target area of the base content on the base content; receive a natural language input for generating output content; and present, in the target area of the base content, modified base content in which the base content is modified to include the output content, wherein the output content is generated based on the base content, the target area, and the natural language input, and wherein the output content is generated by using an artificial intelligence (AI) model.
[0035] According to another aspect of the present disclosure, a method for modifying content performed by a device is provided, the method comprising: presenting base content; receiving user input for selecting a target area of the base content; presenting an indication corresponding to the selected target area of the base content on the base content; receiving a natural language input for generating output content; and presenting, in the target area of the base content, modified base content in which the base content is modified to include the output content, wherein the output content is generated based on the base content, the target area, and the natural language input, and wherein the output content is generated by using an artificial intelligence (AI) model.
[0036] According to another aspect of the present disclosure, a device for modifying content is provided, the device comprising: at least one processor; and a memory storing instructions, which, when executed by the at least one processor, causes the device to: present base content; receive user input for selecting a target area of the base content; present an indication corresponding to the selected target area of the base content on the base content; receive a natural language input for generating output content; and, in the target area of the base content, present the modified base content in which the base content is modified to include the output content, wherein the output content is generated based on the base content, the target area, and the natural language input, and wherein the output content is generated by using an artificial intelligence (AI) model.
[0037] Other aspects, advantages, and salient features of the disclosure will become apparent to those skilled in the art from the following detailed description, which, taken in conjunction with the annexed drawings, discloses various embodiments of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings, in which:
[0039] Figure 1 is a diagram for schematically illustrating iteratively generating content according to an embodiment of the present disclosure;
[0040] Figure 2 A block diagram of an electronic device according to an embodiment of the present disclosure is shown;
[0041] Figure 3 A flowchart showing a method according to an embodiment of the present disclosure;
[0042] Figure 4 is a diagram for illustrating example content generated according to an embodiment of the present disclosure;
[0043] Figure 5 is a diagram for explaining a method for obtaining user intention information according to an embodiment of the present disclosure;
[0044] Figure 6a is a flowchart of a method for determining input content according to an embodiment of the present disclosure;
[0045] Figure 6b An example graphical user interface (GUI) for selecting input content according to an embodiment of the present disclosure is shown;
[0046] Figure 7a is a diagram for illustrating a generative adversarial network (GAN) according to an embodiment of the present disclosure;
[0047] Figure 7b is a diagram for explaining a method of using subtitles according to an embodiment of the present disclosure;
[0048] Figure 8 is a diagram for explaining a method for changing an attribute of input content according to an embodiment of the present disclosure;
[0049] Fig. 9 is a diagram for explaining a method for generating content in a GAN according to an embodiment of the present disclosure;
[0050] Fig.10 is a diagram for explaining a method for generating content in a GAN according to an embodiment of the present disclosure;
[0051] Fig.11 is a diagram for explaining a method of using user feedback according to an embodiment of the present disclosure;
[0052] Fig.12 An example GUI for receiving user feedback according to an embodiment of the present disclosure is shown;
[0053] Fig.13 is a diagram for explaining content generated in a workspace of an application according to an embodiment of the present disclosure;
[0054] Fig.14 is a diagram for explaining content adaptively generated with a workspace of an application according to an embodiment of the present disclosure;
[0055] Fig.15 is a diagram for explaining a method for generating an animation of content according to an embodiment of the present disclosure;
[0056] Fig.16 is a diagram for illustrating a generated example animation according to an embodiment of the present disclosure; and
[0057] Fig.17 An example electronic device according to an embodiment of the present disclosure is shown.
[0058] The same reference numbers are used throughout the drawings to denote the same elements. DETAILED DESCRIPTION
[0059] The following description with reference to the accompanying drawings is intended to assist in a comprehensive understanding of the various embodiments of the present disclosure as defined by the claims and their equivalents. This includes various specific details to assist in understanding, but these details should be considered as merely exemplary. Therefore, it will be appreciated by those of ordinary skill in the art that various changes and modifications may be made to the various embodiments described herein without departing from the scope and spirit of the present disclosure. In addition, for clarity and conciseness, descriptions of well-known functions and structures may be omitted.
[0060] The terms and words used in the following description and claims are not limited to the bibliographical meanings, but are merely used by the inventor to enable a clear and consistent understanding of the present disclosure. Therefore, it should be clearly understood by those skilled in the art that the following description of various embodiments of the present disclosure is for illustrative purposes only and is not intended to limit the present disclosure defined by the attached claims and their equivalents.
[0061] It should be understood that the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. Thus, for example, reference to "a component surface" includes reference to one or more of such surfaces.
[0062] It should be understood that the terms "include", "comprises", and "have" are inclusive, thus specifying the presence of the described features, quantities, operations, components, units, or combinations thereof, but not excluding the presence or addition of one or more other features, quantities, operations, components, units, or combinations thereof. Specifically, numbers should be understood as examples for clarity of expression, and should not be understood as limiting the embodiments by the stated quantities.
[0063] "Content" can be any type of data that can be input into an electronic device, generated by an electronic device, or output at an electronic device. For example, content can be an image, a vector image, a video, an animation, a background, a workspace, a work object, an audio, a text, a vibration, etc., or a combination thereof. In this article, terms such as base content, input content, output content, reconstruction content, target content, false content, and real content are used to distinguish each content mentioned in each operation of the method according to the embodiment, and their meanings can be easily understood by those skilled in the art based on the context. For example, base content can refer to the content of the analysis object (subject) as editing, modification, synthesis, etc. The base content can be the workspace of an application. For example, the base content can be a document as a workspace of a document editing application, a slide as a workspace of a presentation editing application, a spreadsheet as a workspace of a spreadsheet editing application, a user creation mode in a game application, and a drawing document in a drawing document in a drawing application. At the same time, the term referring to content can refer to the same type of content (e.g., image), but is not limited thereto. The term referring to content can refer to different types of content. For example, the base content can be a workspace, the input content can be an image, and the output content can be an animation of an image.
[0064] "User input" refers to any type of input provided by a user and received at an electronic device, and is not limited to input from a specific user. The user input may be associated with one or more coordinates, but is not limited thereto. For example, the user input may be an audio input, a voice input, a text input, or a combination thereof. The input associated with the coordinates may be a touch input, a click input, a gesture input, etc.
[0065] “Natural language input” refers to input received at an electronic device in the form of a language used by people in daily life, and may be voice input, text input, or a combination thereof.
[0066] Figure 1 is a diagram for schematically illustrating iteratively generating content according to an embodiment of the present disclosure.
[0067] refer to Figure 1 , the electronic device 2000 can generate output content 112 based on the user's natural language input and the user input. The output content 112 can be generated by synthesizing the input content onto the basic content 100. The basic content 100, the input content, and the output content can be images, but are not limited thereto. A specific method for generating the output content 112 can be referred to later. Figure 7a and Figure 7b Provide explanation.
[0068] In an embodiment, the input content used to generate the output content 112 can be determined based on the user's natural language input. Figure 1 , an image of a cat or a resting cat can be determined as input content based on a natural language input expressing "drawing a cat resting here". The input content can be determined from a plurality of pieces of content stored in the electronic device 2000, or from an image obtained by searching the Internet. The method of determining input content based on natural language input will be described in detail with reference to Figure 5 Provide explanation.
[0069] refer to Figure 1 , the output content 112 may be generated in the target area 102 of the base content 100. The output content 112 may be generated by synthesizing the input content onto the target area 102 of the base content 100. The target area 102 refers to an area on which the input image of the base content 100 is synthesized. The target area 102 may be the entire area or a partial area of the base content 100. The target area 102 of the base content 110 may include the output content generated after synthesis. According to an embodiment, the efficiency of the synthesis process may be improved by synthesizing the input content into the target area 102 of the base content 100 because the number of pixels used for synthesis may be reduced compared to when the input content is synthesized into the entire area of the base content 100.
[0070] The target area 102 may correspond to a bounding box of an object detected or located in the base content 100, such as a table, a chair, or a stool. The base content 100 may include multiple areas, such as multiple bounding boxes including objects respectively. The target area 102 may be selected from the multiple bounding boxes by user input. The size and shape of the target area 102 may be adjusted by user input (such as a drag input). The target area 102 may have a predetermined size and shape.
[0071] refer to Figure 1 , subtitles for the generated output content 112 may be generated. The subtitles for the generated output content are text for the output content 112, and may be generated by using an image subtitle model. Subtitles may be text for describing the output content 112. In an embodiment, the similarity between the text for natural language input and the output content 112 may be calculated. The generated output content 112 may be displayed on the electronic device 2000 based on the similarity. In an embodiment, when the similarity does not satisfy a specific condition, the process of generating the output content 112 may be iterated. For example, the process of generating the output content 112 may be iterated by synthesizing another input content into the target area 102 of the base content 100. For example, the process of generating the output content 112 may be iterated by synthesizing the same input content into the target area 102 of the base content 100, which will be described later by reference Figure 7a and Figure 7b Provide explanation.
[0072] According to an embodiment, the process of generating the output content 112 may be iterated based on the similarity between the text of the natural language input and the subtitles for the output content 112 , so that the generated output content 112 may conform to the user's intention.
[0073] Meanwhile, various operations described in the present disclosure (such as interpretation of the user's natural language input, generation of output content 112, generation of subtitles for output content 112, calculation of similarity between text and subtitles of natural language input) may be performed by an artificial intelligence (AI) model. The AI model may be referred to as a neural network model. The AI model may include multiple neural network layers. Each neural network layer may have multiple weight values, and various neural network calculations may be performed by performing arithmetic operations on the calculation results in the previous layer and the multiple weight values in the current layer. The multiple weights in each neural network layer may be optimized by the results of training the AI model. For example, multiple weights may be updated to reduce or minimize the loss or cost value obtained by the AI model during the training process. The artificial neural network may include, for example, but not limited to, a deep neural network (DNN), and may include, for example, but not limited to, a convolutional neural network (CNN), a DNN, a recursive neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recursive DNN (BRDNN), a deep Q network (DQN), etc., but is not limited thereto.
[0074] Figure 2 A block diagram of an electronic device according to an embodiment of the present disclosure is shown.
[0075] The electronic device 200 may include, but is not limited to, a smart phone, a tablet personal computer (PC), a PC, a smart television (TV), a mobile phone, a personal digital assistant (PDA), a laptop computer, a media player, a micro server, a global positioning system (GPS), an e-book terminal, a digital broadcast terminal, a navigation device, a telephone booth, an MP3 player, a gateway device, a digital camera, a home appliance, and other mobile or non-mobile computing devices. In addition, the electronic device 200 may be a wearable device with a communication function and a data processing function, such as, but not limited to, a watch, glasses, a hairband, a bracelet, etc. However, the embodiments of the present disclosure are not limited thereto, and the electronic device 200 may include any device capable of generating output content. The electronic device 200 may be an end-user device, but is not limited thereto.
[0076] refer to Figure 2 The electronic device 200 may include a user intention obtaining unit 210, a target area setting unit 220, an input content determining unit 230, an output content generating unit 240, a subtitle generating unit 250, and a similarity calculating unit 260. The electronic device 200 may be composed of a comparison Figure 2The components shown may be implemented with more or fewer components. In the present disclosure, the modules of the electronic device 200 are named to clearly illustrate their operations, which are performed by the modules in the electronic device 200. Therefore, it should be understood that these operations are performed according to the embodiments and should not be interpreted as limiting the role or function of the modules. For example, the operations described herein as being performed by a specific module of the electronic device 200 may be performed by another module or other modules of the electronic device 200, and the operations described herein as being performed by the interaction between the modules of the electronic device 200 or their interactive processing may be performed by a module of the electronic device 200. In addition, the operations described herein as being performed by the electronic device 200 may be performed at another device or performed together with another device to provide substantially the same functionality.
[0077] The units of the electronic device 200 may be implemented by software modules, and the software modules may be executed by at least one processor to provide the functions of the units. Meanwhile, the software modules may be stored in a memory as a set of instructions, and the instructions stored in the memory may be executed by at least one processor. The units of the electronic device 200 may be implemented by hardware modules or a combination of hardware modules and software modules. The electronic device 200 may include a memory and a processor.
[0078] Each unit of the electronic device 200 will be further referred to as Figure 3 Provide explanation.
[0079] Figure 3 A flow chart of a method according to an embodiment of the present disclosure is shown.
[0080] refer to Figure 3 In operation S310, user intention information may be obtained based on the received natural language input. The user intention obtaining unit 210 of the electronic device 200 may obtain user intention information based on the received natural language input.
[0081] The user intent information may represent the user's intent as interpreted by the electronic device 200 from the user's natural language input (or its text). The user intent information may include action information about actions that can be performed at the electronic device 200. The action information may be inferred from the verbs in the text of the natural language input. For example, the generated content may be identified as the action information of the verb "painting" from the natural language input expressing "painting cat rests here". The word representing the generated content is not limited to "painting" and may be various. The electronic device 200 may perform the action represented by the action information. For example, the electronic device 200 may generate output content in response to identifying the action information representing the generated content from the user intent information. In an embodiment, the user intent information may be obtained by using an AI model, which will be described later by reference. Figure 5 Provide explanation.
[0082] In operation S320, a target area may be set in the base image based on user input. The target area setting unit 220 of the electronic device 200 may set the target area in the base content based on user input. The target area may be the entire area or a partial area of the base content. In addition, the target area may be an area that can be adjusted in the base content by user input. The target area may correspond to a bounding box identified in the base content. In an embodiment, the recognition of the bounding box may be performed by an AI model.
[0083] In an embodiment, user input and natural language input may be received together at the electronic device. For example, when the user touches the underlying content displayed in the electronic device 200 while speaking, the user input of the touch and the natural language input of the speech may be received at the electronic device at the same time.
[0084] In operation S330, the input content may be determined. The input content determination unit 230 of the electronic device 200 may determine the input content. In an embodiment, the input content may be determined based on the user intent information obtained in operation S320. For example, when the user intent information includes content information, the input content may be determined based on the content information. The content information may be inferred from nouns in the text input by natural language. For example, the noun "cat" may be identified as content information from the natural language input expressing "painting cats rest here". In an embodiment, the name of the entity may be identified as content information by using named entity recognition (NER). The electronic device 200 may identify "cat" as content information in response to the text input from the natural language, and determine the image of the cat as the input content. In an embodiment, the content information may be inferred from the object of the text input by the natural language. For example, the object "cat rest" of the verb "painting" may be identified from the natural language input expressing "painting cats rest here", and the electronic device 200 may determine the image of the cat as the input content. The input content may be determined from a plurality of pieces of content (such as images) stored in the electronic device 200, or from a plurality of pieces of content obtained by searching the Internet. The content information may represent an analysis object to be input to the AI model to perform the action represented by the action information.
[0085] In an embodiment, the input content may be determined by user input. For example, the content selected by the user at the electronic device 200 may be determined as the input content. The specific method for determining the input content may be described later by referring to Figure 6a and Figure 6b Provide explanation.
[0086] Meanwhile, according to an embodiment, user intent information may be used to set a target area in the base content. User intent information may include area information. Expressions representing places or locations (such as "here", "there", "everywhere", "somewhere", prepositions and postpositions representing locations, and any words connected to prepositions and postpositions representing locations) may be identified as area information from text input in a natural language. In an embodiment, the target area may be set to an area represented by the area information for a location selected in the base content by user input. When a bounding box is selected in the base content by user input, the area represented by the area information may be set to a target area for the selected bounding box. In an embodiment, the target area may be set in the base content based on the user intent information.
[0087] In an embodiment, the user intention information may include content attribute information. For example, when the content information corresponding to "cat" is identified from the natural language input expressing "drawing cats take a break here", the word "break" that modifies the content information may be identified as content attribute information indicating the attribute of the content, which will be described later by reference. Figure 8 , Fig. 9 and Fig.10 Provide explanation.
[0088] In operation S340, output content may be generated based on the input content, the target area, and the user intent information. The output content may be generated for the base content. The output content generation unit 240 of the electronic device 200 may generate output content based on the input content, the target area, and the user intent information. The output content refers to content generated by the user's natural language input, and may include content output by the electronic device 200 and displayed to the user, and may also include content that has been generated but not displayed to the user. In an embodiment, the output content may be generated by using an AI model, which will be referred to later. Figure 7a and 7b Provide explanation.
[0089] In operation S350, subtitles for outputting content may be generated. The subtitle generation unit 250 of the electronic device 200 may generate subtitles for outputting content. In an embodiment, subtitles for outputting content may be generated using an AI model. For example, subtitles for outputting content may be generated using an image subtitle model. For example, subtitles for animations of outputting content may be generated using a voice subtitle model. Various imaging subtitle models and video subtitle models are already well known to those skilled in the art, and therefore, redundant descriptions are omitted.
[0090] In operation S360, the similarity between the natural language input text and the subtitles may be calculated. The similarity calculation unit 260 of the electronic device 200 may calculate the similarity between the natural language input text and the subtitles. The natural language input text may be determined in the process of obtaining user intent information from the natural language input in operation S310.
[0091] In an embodiment, the similarity between the text and subtitles of the natural language input can be calculated by using an AI model. The similarity can be a vector similarity. The text and subtitles of the natural language input can be encoded using semantic vectors to generate corresponding vectors, and the similarity between the vectors can be calculated.
[0092] In operation S370, it may be determined whether the similarity satisfies a condition. In an embodiment, the condition may be satisfied when the similarity exceeds a preset threshold. In an embodiment, the condition may be satisfied when the similarity is within a preset range.
[0093] When the similarity satisfies the condition in operation S370, it can be determined that the generated output content meets the user's intention. When the similarity does not meet the condition in operation S370, it can be determined that the generated output content does not meet the user's intention.
[0094] In an embodiment, when the generated output content does not meet the user's intention, that is, when the similarity does not meet the condition, the process of generating the output content can be iterated to generate output content that meets the user's intention. For example, operations S340, S350, S360, and S370 can be iterated.
[0095] In an embodiment, when the generated output content does not meet the user's intention, that is, when the similarity does not meet the condition, another input content different from the input content selected in operation S330 can be determined, and then, operation S340, operation S350, operation S360 and operation S370 can be iterated with the newly determined other input content to generate output content that meets the user's intention.
[0096] According to an embodiment, output content that meets the user's intention can be generated by comparing the user's natural language input with the subtitles for the output content. In addition, the quality of the output content can be improved by iteratively generating the output content.
[0097] Figure 4 is a diagram for illustrating example content generated according to an embodiment of the present disclosure
[0098] In an embodiment, the target area set in the base content is Figure 1, but not limited thereto, the target area may be circular, elliptical, triangular, pentagonal, polygonal, or have any shape.
[0099] refer to Figure 4 , based on the natural language input expressing "put Amy's face here" and the target area 412a set by the user input, the input content of "Amy's face" can be synthesized into the target area 412a to generate output content, and then the basic content 410a including the synthesized output content into which the generated output content is synthesized can be provided to the user. The input content can be obtained by searching the Internet.
[0100] In an embodiment, the input content may be obtained from an image stored in an electronic device or a cloud server by using an AI model. The image captured by the user may be processed in the electronic device or cloud server by using a face detection AI model, and the detected face may be named directly by the user, or by using the browsing history of the user's social networking service (SNS). Therefore, a face named "Amy" may be determined as input content from the detected face.
[0101] In an embodiment, output content is generated by synthesizing the input content into the target area 412a of the base content 410a, and then, a subtitle for the generated output content (e.g., "Amy's face among friends") can be generated. The subtitle for the generated output content can be compared with the text of the natural language input, and it can be determined whether to iteratively generate the output content based on the comparison.
[0102] In an embodiment, a bounding box including a face may be identified from the base content 410a using an AI model, and a target area 412a may be set based on the identified bounding box. When a natural language input expressing "replace Bell's face with Amy" is received at the electronic device, and Bell's face is detected in the base content 410a, Bell's face may be set as the target area 412a.
[0103] refer to Figure 4, the input content 402b of "cat" or "jumping cat" can be synthesized into a target area based on the natural language input expressing "put the jumping cat here" to generate output content. In an embodiment, the input content 402b can be determined from the base content 400b by using an AI model. For example, when the content information derived from the natural language input corresponds to an object detected in the base content 400b, the detected object can be determined as the input content 402b. In an embodiment, the attributes of the input content 402b detected in the base content 400b can be changed based on user intent information. The user intent information may include content attribute information, and the content attribute information may be related to the gesture. In addition, a gesture animation corresponding to the gesture represented by the content attribute information can be obtained. The gesture animation of the gesture of the content attribute information can be obtained from a gesture animation library. The gesture animation library can be generated by detecting key points from the body of an animal or a person using an AI model.
[0104] In an embodiment, the input content 402b may be a gesture animation obtained based on user intent information. The output content may be generated based on the object and gesture animation in the target area using an AI model. When the base content 400b is a video, each frame of the base content 400b may be modified based on the gesture animation to generate the output content, and then the base content 410b including the output content may be provided to the user. The size and shape of the target area may be adjusted based on the gesture animation.
[0105] In an embodiment, the output content is generated by synthesizing the input content 402b into the target area of the base content, and then, subtitles for the generated output content (e.g., "Jumping Russian Blue Cat") can be generated. The subtitles for the generated output content can be compared with the text of the received natural language input, and it can be determined based on the comparison whether to iteratively generate the output content. An AI model (such as a video subtitle model) can be used to generate subtitles for the output content.
[0106] In an embodiment, a bounding box including an object may be identified from the base content 400b by using an AI model, and a target area may be set based on the identified bounding box. The input content 402b may be determined from a bounding box or a target area of the base content 400b.
[0107] refer to Figure 4 , when natural language input expressing “draw a black and white sun here” is received at an electronic device, input content of “sun” or “black and white sun” may be synthesized into the target area 412c to generate output content, and then the base content into which the output content is synthesized may be provided to the user.
[0108] In an embodiment, the output content may be generated to have the same form as the base content 410c by using an AI model. For example, when the base content 410c is a cartoon-type image, the output content may also be generated to have the same cartoon-type image. The cartoon-type image may be a vector image. The output content may be a vector image converted from an image. In an embodiment, the output content may be generated based on a command supported by a vector drawing program. The output content may be generated based on an application programming interface (API) of the program. The method of generating output content based on the program's API will be described later by reference. Fig.15 and Fig.16 According to the embodiment, the output content can be generated on various platforms and is not limited to image synthesis.
[0109] In an embodiment, the output content is generated by synthesizing the input content into the target area 412c of the base content 410a, and then, a subtitle (e.g., "black hole") for the generated output content can be generated. The subtitles for the generated output content can be compared with the text of the natural language input, and based on the comparison, it is determined whether to iteratively generate the output content. When the similarity between the subtitles of the output content and the text of the natural language input of the user does not meet a specific condition, the process of generating the output content can be iterated. Iterative generation of the output content can continue until the output content is generated as a subtitle with text similar to the natural language input "black and white sun".
[0110] refer to Figure 4 , when a natural language input expressing "make a door here" is received at an electronic device, the input content of "door" can be synthesized into the target area 412d to generate output content that matches the basic content 410d, and then the basic content 410d including the generated output content can be provided to the user. In an embodiment, the output content can be generated on the same platform as the basic content 410d by using an AI model. That is, the output content can be generated to be limited to the platform to which the basic content 410d belongs. For example, when the platform of the basic content 410d is a specific game, the output content can be generated in a manner supported by the game. The game can be a sandbox game or a game creation tool. In an embodiment, the output content can be generated based on commands supported by the game. The output content can be generated based on the application programming interface (API) of the game. According to an embodiment, the output content can be generated on various platforms and is not limited to image synthesis.
[0111] Figure 5 is a diagram for explaining a method for obtaining user intention information according to an embodiment of the present disclosure.
[0112] In an embodiment, user intent information may be obtained based on a user's natural language input.
[0113] refer to Figure 5 The user intention obtaining unit 210 of the electronic device may include an automatic speech recognition (ASR) model 212 and a natural language understanding (NLU) model 214.
[0114] refer to Figure 5 , when a speech input is received as a natural language input, the speech of the natural speech input may be converted into text by the ASR model 212, and then the converted text may be processed by the NLU model 214 to obtain user intent information. The speech input may be received by a microphone of the electronic device. The text converted by the ASR model 212 may be compared with the subtitles generated for the output generated later, and as a result of the comparison, the similarity between the text of the natural language input and the subtitles for the generated subtitles may be calculated.
[0115] In an embodiment, the user intent obtaining unit 210 may include a spoken language understanding (SLU) model, and the speech input received through the microphone may be processed by the SLU model to obtain user intent information. In an embodiment, the natural language input may be transmitted from another device.
[0116] In an embodiment, when a text input is received as a natural language input, the NLU model 214 may process the text of the text input to obtain user intent information. The text input may be received by an input interface of an electronic device, such as a keyboard, a touch screen, a keypad, etc. In an embodiment, the text input may be transmitted from another device. The text of the text input may be compared with the subtitles generated for the output generated later, and as a result of the comparison, the similarity between the text of the natural language input and the subtitles for the generated subtitles may be calculated.
[0117] When the text of the natural language input is input to the NLU model 214, the NLU model 214 can analyze the text to obtain user intent information. The NLU model 214 can obtain action information, content information, content attribute information, and region information, etc., and the obtained information can be used to generate output content. In addition to the described information, the user intent information can also be obtained according to various categories so that the electronic device understands the user's intent.
[0118] Figure 6a is a flowchart of a method for determining input content according to an embodiment of the present disclosure.
[0119] Figure 6b An example graphical user interface (GUI) for selecting input content according to an embodiment of the present disclosure is shown.
[0120] Reference Figure 6a and 6b , Figure 3Operation S330 of may be performed differently based on whether a user input for selecting input content is received. Operation S330 may be performed after receiving a user input and a natural language input for selecting the target area 602 in the base content 600, but is not limited thereto. For example, the input content may be determined before receiving a user input and a natural language input for selecting the target area 602 in the base content 600.
[0121] In operation S331, it may be determined whether a user input selecting input content is received. When it is determined in operation S331 that a user input selecting input content is received, in operation S332, content selected by the user input may be determined as input content.
[0122] For example, refer to Figure 6b , a GUI 603 for selecting input content may be displayed on a display of the electronic device, and content may be selected from the GUI 603 as input content by a user input.
[0123] When it is determined in operation S331 that no user input for selecting input content is received, in operation S333, it may be determined whether the content information is included in the user intention information. When it is determined in operation S333 that the content information is included in the user intention information, in operation S334, the input content may be determined based on the content information. For example, referring to Figure 6b , when a natural language input expressing "pause drawing cat here" is received, "cat" can be identified as content information, and then the electronic device can identify "cat" as content information in response to the text input from the natural language, and determine the cat image as input content. In an embodiment, the input content can be determined based on the content information with the confirmation of the user. For example, the found content can be displayed based on the content information, and then, when the user agrees to use the found content as input content, the found content can be determined as input content. When multiple pieces of content are found based on the content information, a GUI 603 for selecting input content from the multiple pieces of content can be displayed at the electronic device to allow the user to select the input content.
[0124] When it is determined in operation S333 that the content information is not included in the user intention information, in operation S335, multiple pieces of content may be displayed. That is, a GUI 603 for selecting input content among multiple pieces of content may be displayed at the electronic device, and then the input content may be determined from the multiple pieces of content based on the user input. When the content information is not obtained based on the natural language input, and when the content information is not included in the user intention information, a GUI 603 for determining the input content may be displayed.
[0125] Figure 7aIt is a diagram for illustrating a generative adversarial network (GAN) according to an embodiment of the present disclosure.
[0126] The output content generating unit 240 may include an AI model for generating output content.
[0127] refer to Figure 7a , the output content generation unit 240 may include a generation model of the GAN model, that is, a generator 242. The output content generation unit 240 may also include a discriminative model of the GAN model, that is, a discriminator 244. The generator 242 and the discriminator 244 of the GAN model may be trained adversarially with each other. The discriminator 244 may be trained based on real content and false content to determine whether the output content generated by the generator 242 is real content or false content. The generator 242 may generate output content that may be determined as real content by the discriminator 244 based on input data. The generator 242 and the discriminator 244 may be trained until a balance point is reached. When the two networks reach a balance point, the loss function of the GAN may be appropriately corrected so as to sufficiently mimic the data probability distribution of the real content.
[0128] In operation S742, the generator 242 may generate output content based on the target area of the input content and the base content. The probability distribution of the output content generated by the generator 242 that has reached or is close to the equilibrium point may be close to the probability distribution of the real content. The generator 242 may generate output content whose probability distribution is close to the probability distribution of the real content, so that the output content is determined as the real content by the discriminator 244.
[0129] In operation S744, the discriminator 244 may discriminate true content and false content with respect to the target area. In operation S745, the discriminator 244 may discriminate true content and false content with respect to the entire area of the base content.
[0130] In operation S780, the generator 242 may be trained based on the discrimination result of the discriminator 244. The discriminator 244 is trained to discriminate between true content and false content for the target area and the entire area, and the generator 242 is trained to generate output content that may be determined as true content for the target area and the entire area by the discriminator 244. Therefore, the quality of the generated output content may be improved.
[0131] refer to Figure 4, the probability that the output content generated by the generator 242 is determined by the discriminator 244 to be real content can be approximately 0.5, but is not limited to this. That is, the data probability distribution of the output content generated by the generator 242 can be approximately the data probability distribution of the real content. The real content used as a reference for discrimination can have the same form as the basic content, or be supported by the same platform as the basic content. That is, real content collected on the same platform as the basic content or real content in the same form as the basic content can be used to train the discriminator 244.
[0132] Figure 7b is a diagram for explaining a method of using subtitles according to an embodiment of the present disclosure.
[0133] In an embodiment, the discriminator 244 may be further trained based on the similarity between the text of the natural language input and the subtitles for the content. After the generator 242 and the discriminator 244 of the GAN reach an equilibrium point, the probability that the output content generated by the generator 242 is determined by the discriminator 244 as false content in operation S742 may be approximately 0.5, but is not limited thereto.
[0134] refer to Figure 7b , the discriminator 244 can determine whether the output content is real content or false content based on the similarity between the text of the natural language input and the subtitles for the output content. Specifically, the subtitles can be generated by the subtitle generation unit for the output content generated by the generator 242, and the similarity between the generated subtitles and the text of the user's natural language input can be calculated by the similarity calculation unit. When it is determined in operation S746 that the similarity does not meet the predetermined condition, the discriminator 244 can determine that the generated output content is false content. When it is determined in operation S746 that the similarity meets the predetermined condition, the discriminator 244 can determine that the generated output content is real content. In an embodiment, the discriminator 244 can be further trained based on the similarity between the text of the natural language input and the subtitles of the output content. In addition, in operation S780, the generator 242 is also trained to generate output content that can be determined as real content by the discriminator 244, and the generator 242 can iteratively generate the process of output content, which can improve the quality of the output content.
[0135] Figure 8 is a diagram for explaining a method of changing an attribute of input content according to an embodiment of the present disclosure.
[0136] refer to Figure 8, a picture of parents may be selected as input content 804 through user input 1. A target area 802 may be set in the base content 800 based on user input 2. When a natural language input expressing "let them stand here" is received at an electronic device, the electronic device may obtain user intention information based on the natural language input. In the natural language input, "let" may correspond to action information. In the natural language input, "they" may correspond to content information. In the natural language input, "stand" that modifies the content information may be identified as content attribute information representing an attribute of the content.
[0137] In an embodiment, output content 814 may be generated based on content attribute information. Output content 814 may be generated by synthesizing input content into target area 802 of base content 800. The attributes of output content 814 may be the same as the attributes represented by the content attribute information. The attributes of output content 814 may be changed according to the attributes of input content 804. The attributes of the content may include, but are not limited to, the posture, facial expression, makeup, hair, clothing, accessories, or style of the content. The attributes of the content may include, but are not limited to, the posture, facial expression, makeup, hair, clothing, accessories, or style of the object in the content. For example, referring to Figure 8 , the attribute (such as posture) of the input content 804 may be a sitting posture, and the attribute of the output content 814 may be a standing posture. The method of changing the attribute of the input content 804 will be described later by reference to Fig. 9 and Fig.10 Provide explanation.
[0138] In an embodiment, the output content 814 may be generated to match the base content 800. The output content 814 may be generated to have the same attributes as the base content 800 or an object in the base content 800 by referring to the target area 802 or the entire area of the base content 800. For example, referring to Figure 8 , when all objects in the base content 800 wear white clothes, output content 814 in which the objects also wear white clothes may be generated. That is, the objects in the input content 804 wear clothes of different colors, but the output content 814 in which the objects wear white clothes may be generated, and the base content 810 and / or its target area 812 may include the output content 814. For example, referring to Figure 8 , when all objects in the base content 800 stand, output content 814 in which the objects also stand can be generated. That is, the objects in the input content 804 sit on the floor, but the output content 814 in which the objects stand can be generated, and the base content 810 and / or its target area 812 can include the output content 814. The method of making the output content 814 have the same attributes as the base content 810 will be described by reference. Fig. 9 and Fig.10Provide explanation.
[0139] Fig. 9 is a diagram for illustrating a method for generating content by GAN according to an embodiment of the present disclosure.
[0140] In an embodiment, the content attribute information may be obtained from a natural language expression that modifies the content information. The content attribute information may be related to an attribute in the input content that the user wants to change. For example, when the input content includes a person, an attribute among the person's posture, facial expression, makeup, hair, clothing, accessories, or style that the user wants to change may be identified as the content attribute information.
[0141] refer to Fig. 9 , the output content generation unit may include two generators, namely, a first generator 242a and a second generator 242b. In addition, the output content generation unit may include two discriminators, namely, a first discriminator 244a and a second discriminator 244b.
[0142] In operation S942a, the first generator 242a may generate output content having a "B" attribute based on input content having an "A" attribute. That is, the first generator 242a may generate false content having a "B" attribute based on real content having an "A" attribute. The first generator 242a may be trained to generate output content having a "B" attribute.
[0143] In operation S942b, the second generator 242b can reconstruct the content with the "A" attribute based on the generated output content with the "B" attribute. That is, the second generator 242b can reconstruct the original content with the "A" attribute from the false content with the "B" attribute. The first generator 242a can be trained to generate output content with the "A" attribute. The first generator 242a can only change the attribute of the input content from the "A" attribute to the "B" attribute to generate the output content, while the change of other attributes is suppressed so that the second generator 242b can reconstruct the content similar to the original input content with the "A" attribute. According to an embodiment, the generation of content that is unrelated to the original content can be suppressed by reconstructing the original content. When the reconstructed content is similar to the input content, the loss value can be reduced, which can be interpreted as the cycle consistency between the input content and the output content is maintained.
[0144] Meanwhile, in operation S944a, it may be determined by the first discriminator 244a whether the output content with the "B" attribute generated by the first generator 242a is real content or false content. The first discriminator 244a may be trained by using real content with the "B" attribute to discriminate real content from false content. In operation S980a, the first generator 242a may be trained based on the discrimination result of the first discriminator 244a.
[0145] In operation S944b, the content with the "A" attribute reconstructed by the second generator 242b may be discriminated by the second discriminator 244b as real content or false content. The second discriminator 244b may be trained by using real content with the "A" attribute to discriminate real content from false content. In operation 980b, the second generator 242b may be trained based on the discrimination result of the second discriminator 244b.
[0146] According to an embodiment, when the attribute of the input content is changed from the "A" attribute to the "B" attribute, the consistency of other attributes can be maintained. According to an embodiment, the attribute of the input content can be changed from the "A" attribute to the "B" attribute without marking the attribute of the content.
[0147] Fig.10 is a diagram for illustrating a method for generating content by GAN according to an embodiment of the present disclosure.
[0148] refer to Fig.10 , the output content generation unit may include a generator 242 and a discriminator 244 .
[0149] In operation S1042, the generator 242 may generate output content based on the attribute label and the input content. The attribute label may represent a label of any attribute that may be classified by the discriminator 244. That is, the generator 242 may receive the attribute label and the input content to generate output content. The generator 242 may be trained to generate output content that may be determined as true content by the discriminator 244 and classified into the received attribute label.
[0150] In operation S1043, generator 242 can reconstruct content based on the attribute tag and output content that are the same as the attribute tag in operation S1042. That is, generator 242 can receive the initial attribute tag and the generated output content to reconstruct content. Generator 242 can modify the input content only within the degree that the generated output content can be classified as the attribute tag, so that generator 242 can reconstruct content close to the initial content (input content). The input content is modified within the degree that the output content can be classified as the attribute tag, and the output content classified as a specific attribute can be generated. Since the reconstructed content is similar to the input content, the loss value of the output content can be reduced.
[0151] In operation S1044, the output content generated by the generator 242 may be discriminated by the discriminator 244 as true content or false content. When the output content is determined to be true content, in operation S1045, the discriminator 244 may classify the attributes of the output content. The discriminator 244 may receive true content and false content to discriminate true content or false content. The discriminator 244 may classify the attributes of the output content determined to be true content to train the discriminator 244.
[0152] In operation S1080 , the generator 242 may be trained based on the discrimination result of the discriminator 244 .
[0153] According to an embodiment, when the attribute of the input content is changed from the 'A' attribute to the 'B' attribute, the consistency of other attributes may be maintained. Fig.10 An example of changing only one attribute is used for explanation, but the number of attributes of the input content that can be changed may increase according to the number of attributes that the discriminator 244 is configured to classify.
[0154] Fig.11 is a diagram for explaining a method of using user feedback according to an embodiment of the present disclosure.
[0155] refer to Fig.11 , the electronic device may receive user feedback for the base content 1110 into which the output content is synthesized, the output content, or an area of the output content corresponding to the target area 1112 of the input content. In an embodiment, the process of generating the output content may be iterated when user feedback is received. For example, similar to the case where the similarity between the subtitles of the output content and the natural language input does not meet a certain condition, the process of generating the output content may be iterated when negative user feedback is received.
[0156] According to an embodiment, user feedback may be reflected on the AI model to provide more customized results to the user.
[0157] Fig.12 An example GUI for receiving user feedback according to an embodiment of the present disclosure is shown.
[0158] refer to Fig.12 , the electronic device may receive user feedback on the base content 1210 into which the output content is synthesized, the output content, or an area 1212 of the output content corresponding to the target area of the input content. In an embodiment, when the user feedback is received, a GUI 1213 for changing the properties of the object in the input content may be displayed. For example, when negative user feedback on the properties of the output content is received, a GUI for changing the properties of the output content may be displayed so that the user can select the properties that are favorable to him or her.
[0159] Fig.13 is a diagram for explaining content generated in a workspace of an application according to an embodiment of the present disclosure.
[0160] refer to Fig.13 , the content of the black cloud is determined as input content 1304, and a natural language input expressing "put this cloud here" may be received at the electronic device. In an embodiment, when user input on target area 1312a, target area 1312b, target area 1312c, and target area 1312d is received, the electronic device may generate output content in target area 1312a, target area 1312b, target area 1312c, and target area 1312d. The output content may be generated by using an AI model for the basic content. Fig.13 As shown, when the workspace 1310 includes another object such as a white circle, output content can be generated for the workspace 1310 including the white circle object. The relationship between the output content and the workspace 1310 can be obtained based on the user intention information determined from the natural language input. For example, by referring to the white cloud in the workspace 1310 based on the expression "drop here", the white cloud can be generated as a plurality of output content (work objects). That is, output content having a different color from the initial input content 1304 can be generated.
[0161] Fig.14 is a diagram for explaining content adaptively generated with a workspace of an application according to an embodiment of the present disclosure.
[0162] refer to Fig.14 , the content of the silhouette of the woman holding the bag is determined as input content 1404, and a natural language input expressing "let her hold the arrow like the left one" may be received at the electronic device. When receiving the user input on the target area 1412, the electronic device may generate output content in the target area 1412 by using the AI model.
[0163] refer to Fig.14 , when the workspace 1410 includes another object such as a silhouette of a man holding an arrow, output content may be generated for the workspace 1410 including the other object. For example, by referring to the man holding an arrow in the workspace 1410 based on the expression “holding an arrow like the one on the left”, a silhouette of a woman holding an arrow like a man may be generated as output content.
[0164] Fig.15 is a diagram for explaining a method of generating an animation of content according to an embodiment of the present disclosure.
[0165] According to an embodiment, when the base content is a workspace of an application, the output content may be generated based on an API of the application. The API of the application may call various commands related to the animation. The commands related to the animation may include commands for setting the action parameters of the animation (such as appear, highlight, hide, move), the starting point and end point of the action parameters (such as where to appear, where to hide, where to move), the moving direction of the action parameters, the moving timing of the action parameters, the moving speed of the action parameters, etc., but are not limited thereto.
[0166] refer to Fig.15 , the output content may be an animation related to the input content 1504. The input content 1504 may be a work object.
[0167] The animation associated with the input content 1504 can be generated by using the above-mentioned AI model (such as a GAN model). The generator of the GAN model can receive the input content 1504, the application's API, and the user's intent information, and then generate the animation associated with the input content 1504 in combination with the command provided by the application's API, and the animation can be played (animate) according to the user's intent.
[0168] For example, when input content 1504 is selected in workspace 1500, and a natural language input expressing "apply the animation of the number count" is received at the electronic device, the generated animation can be identified as action information from "apply animation". An animation related to input content 1504 can be generated in response to the action information identifying the generated animation. The target area can be the area where the input content 1504 in workspace 1500 is located, but is not limited to this. The animation that conforms to the user's intention can be inferred from the content attribute information. For example, "the number" can be identified as content information in the natural language input, and the "count" that modifies the content information can be identified as content attribute information. The animation can be generated based on the input content 1504, the API of the application, and the content attribute information of "count".
[0169] The electronic device can generate subtitles for the content 1514 of the application animation. The subtitles for describing the animation of the content 1514 can be generated by using a video subtitle model. The process of generating the animation can be iterated based on the similarity between the generated subtitles and the text of the natural language input. Therefore, an animation that meets the user's intention can be generated. According to an embodiment, the animation can be applied to the input content 1504 by using natural language input, even if the user does not know the exact command of the application. The animation of the input content 1504 can work in the presentation mode of the workspace 1510.
[0170] Fig.16 is a diagram for explaining a generated example animation according to an embodiment of the present disclosure.
[0171] refer to Fig.16 , the base content may be a workspace 1600 of an application, and the input content 1604 may be a work object. For example, the work object may be created by a tool supported by the application.
[0172] When a work object is selected in the workspace 1600 and a natural language input expressing "apply the animation of the horizontal growth of the graphic" is received at the electronic device, the generated animation can be identified as the action information from the "apply animation". An animation related to the input content 1604 can be generated in response to the action information identifying the generated animation. An animation that conforms to the user's intention can be inferred from the content attribute information. For example, "the graphic" can be identified as the content information in the natural language input, and the "horizontal growth" that modifies the content information can be identified as the content attribute information. The animation can be generated based on the input content 1604, the API of the application, and the content attribute information of "horizontal growth".
[0173] Fig.17 An example electronic device according to an embodiment of the present disclosure is shown.
[0174] refer to Fig.17 , the electronic device 2000 or the software running on the electronic device 2000 can perform the operations of the methods described or illustrated herein to provide the functions described or illustrated herein. The electronic device 2000 can refer to one or more electronic devices 2000. One or more electronic devices 2000 can perform the operations of the methods described or illustrated herein without substantial spatial or temporal limitations. One or more electronic devices 2000 can perform the operations of the methods described or illustrated herein in real time or in a batch processing manner.
[0175] The electronic device 2000 may include, but is not limited to, a smart phone, a tablet PC, a PC, a smart TV, a mobile phone, a personal digital assistant (PDA), a laptop computer, a media player, a micro server, a global positioning system (GPS), an e-book terminal, a digital broadcast terminal, a navigation device, a telephone booth, an MP3 player, a gateway device, a digital camera, a home appliance, and other mobile or non-mobile computing devices. In addition, the electronic device 2000 may be a wearable device with a communication function and a data processing function, such as, but not limited to, a watch, glasses, a hair band, a bracelet, etc. However, the embodiments of the present disclosure are not limited thereto, and the electronic device 2000 may include any device capable of generating output content. The electronic device 2000 may be an end-user device.
[0176] The electronic device 2000 may include a processor 2300 and a memory 2700. A software module (such as a program module) of the electronic device 2000 may be stored in the memory 2700 as a set of instructions, and the instructions may be executed by the processor to perform corresponding functions. Fig.17 The components shown may be implemented with more or fewer components.
[0177] For example, the electronic device 2000 may include an input interface 2100 , an output interface 2200 , a processor 2300 , a communication interface 2500 , a microphone 2600 , and a memory 2700 .
[0178] The processor 2300 of the electronic device may include an AI processor for generating a trained network model, but is not limited thereto. The AI processor may be implemented as a chip separate from the processor 2300. The processor 2300 may perform each operation of the method described or illustrated herein.
[0179] The input interface 2100 of the electronic device can receive user input. The input interface 2100 may include a unit for interacting with the user. For example, the user interface 1100 may be a keypad, a dome switch, a touch pad (e.g., a contact electrostatic capacitance type, a pressure resistance film type, an infrared detection type, a surface acoustic wave propagation type, an integral strain type, a piezoelectric effect type, etc.), a jog wheel, and a jog switch, but is not limited thereto.
[0180] The output interface 2200 of the electronic device may output information to be provided to the user, such as an audio signal, a video signal, or a vibration signal. The output interface 2200 may include a display, a sound output interface, a vibration motor, but is not limited thereto.
[0181] The sound output interface may output audio data received from the communication interface 2500 or stored in the memory 2700. In addition, the sound output interface may output sound signals (e.g., call signal reception sound, information reception sound, notification sound, etc.) related to the functions performed by the electronic device 2000. The sound output interface 2200 may include a speaker, a buzzer, etc.
[0182] The vibration motor may output a vibration signal. For example, the vibration motor may output a vibration signal based on outputting audio or video data. The vibration motor may output a vibration signal in response to receiving a touch input.
[0183] The output interface 2200 may provide a dialog interface to the user. The dialog interface may be provided in the form of a message window or a chat window, but is not limited thereto.
[0184] The communication interface 2500 of the electronic device 2000 may include one or more components that allow the electronic device 2000 to communicate with another device such as a server. For example, the communication interface 2500 may include a short-range communication interface, a mobile communication interface, and a broadcast receiver to communicate with a network.
[0185] The network may include, for example but not limited to, a local area network (LAN), a wide area network (WAN), a value-added network (VAN), a mobile radio communication network, a satellite communication network, etc., and any combination thereof. In addition, the network may include a comprehensive configuration to enable Figure 1 The network entities shown are data communication networks capable of smooth communication, and include, for example but not limited to, wired Internet, wireless Internet, mobile wireless communication networks, etc. Examples of wireless communication technologies may include, but are not limited to, wireless LAN (WLAN) or Wi-Fi, Bluetooth, Bluetooth Low Energy (BLE), ZigBee, Wi-Fi Direct (WFD), Ultra Wideband (UWB), infrared communication (Infrared Data Association (IrDA)), near field communication (NFC), etc.
[0186] The mobile communication interface may transmit wireless signals with at least one of a base station, an external terminal and a server via a mobile communication network. The wireless signals may include voice call signals, video call signals or any type of data for transmitting text / multimedia messages.
[0187] The broadcast receiver may receive a broadcast signal and / or broadcast-related information from the outside via a broadcast channel. The broadcast channel may include a satellite channel, a terrestrial channel, and the like.
[0188] Microphone 2600 can receive and process sound signals from the outside to convert them into electronic sound data. For example, microphone 2600 can receive sound signals from an external device or a speaker. Microphone 2600 can use any of a variety of noise reduction algorithms to reduce noise that occurs when receiving sound signals from the outside.
[0189] The memory 2700 may store programs for processing and control, and store input data or output data. The programs stored in the memory 2700 may be classified into a plurality of modules.
[0190] According to the present disclosure, functions related to artificial intelligence (AI) may be operated via the processor 2300 and the memory 2700. The processor 2300 may include one or more processors 2300. The one or more processors 2300 may include a general-purpose processor (such as a central processing unit (CPU)), an application processor (AP), a digital signal processor (DSP), a dedicated graphics processor (such as a graphics processing unit (GPU) or a visual processing unit (VPU)), a dedicated AI processor (such as a neural processing unit (NPU)), etc., but are not limited thereto. The one or more processors 2300 may control input data to be processed according to a predefined operating rule or AI model stored in the memory 2700. When the one or more processors 2300 are dedicated AI processors 2300, the dedicated AI processor 2300 may be designed with a hardware structure specifically for processing a specific AI model.
[0191] The predefined operating rules or AI models may be created via a training process. For example, the predefined operating rules or AI models may be set to execute a desired characteristic (or purpose) created by training a basic AI model with a learning algorithm that utilizes a large amount of training data. The training process may be performed by a device for executing AI or a separate server and / or system. Examples of learning algorithms may include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, and reinforcement learning, but the disclosed embodiments are not limited thereto.
[0192] The method may be performed by hardware, software, or a combination of hardware and software according to an embodiment. When the embodiment is implemented using software, one or more programs (software, software modules) may be used. The program may be included in a computer-readable recording medium, but is not limited thereto. The program may be included in a computer program product. A computer-readable recording medium storing the program may be included in a computer program product. The program may be configured to be executable by one or more processors in an electronic device. One or more processors may include instructions, which, when executed by one or more processors in an electronic device, cause the electronic device to perform the method according to an embodiment.
[0193] Where appropriate, the computer-readable recording medium may be volatile, nonvolatile, or a combination of volatile and nonvolatile. The computer-readable recording medium may include a computer storage medium and a communication medium. The computer-readable recording medium may be any available medium accessible by a computer, and includes both volatile and nonvolatile media and both removable and non-removable media. In addition, the computer-readable recording medium may include a computer storage medium and a communication medium. The computer storage medium includes both volatile and nonvolatile media and both removable and non-removable media, which are implemented by any method or technology to store information, such as computer-readable instructions, data structures, program modules or other data. The communication medium generally contains computer-readable instructions, data structures or program modules, and includes any information transmission medium. The disclosed embodiment may be implemented by a computer-readable recording medium or a computer program product having a computer-executable instruction (such as a program module executed by a computer) recorded thereon. A plurality of computer-readable recording media may be distributed in (a plurality of) computer systems connected via a network, and the data (such as instructions and codes) stored in the distributed recording medium may be executed by at least one processor.
[0194] The functions of the various elements shown in the figures may be provided by using not only hardware capable of executing the relevant software but also dedicated hardware. When the functions are provided by a processor, the functions may be provided by a dedicated processor, a single shared processor or a plurality of separate processors of which parts may be shared.
[0195] The term "processor" or "controller" should not be construed to refer only to hardware that can execute software, and may include digital signal processor hardware, read-only memory, random access memory, and volatile storage devices for storing software.
[0196] In the present disclosure, terms such as "-unit" or "-module" should be understood as a unit that processes at least one function or operation. A component called "-unit" or "-module" may be implemented as hardware, software, or a combination of hardware and software.
[0197] Expressions such as "at least one" when preceding a list of elements modify the entire list of elements and do not modify the individual elements in the list. For example, the expression "at least one of a, b, and c" should be understood to include only a, only b, only c, including a and b, including a and c, including b and c, or including all of a, b, and c.
[0198] The operations of all methods described herein may be performed in an appropriate order, unless otherwise indicated herein or clearly contradictory to the context, and are not limited to the order indicated. The use of any and all examples or language provided herein (e.g., "such as") is only used to clearly describe the embodiments and does not constitute a limitation on the embodiments, unless otherwise stated. For example, each function may be performed in a distributed manner or in a combined manner.
[0199] While the disclosure has been shown and described with reference to various embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the disclosure as defined by the following claims and their equivalents.
Claims
1. A non-transitory computer-readable storage medium storing instructions that, when executed by at least one processor of a device, cause the device to: Present basic content; receiving user input for selecting a target area of the base content; presenting, on the base content, an indication corresponding to the selected target area of the base content; receiving natural language input for generating output content; as well as presenting, in a target area of the base content, modified base content in which the base content is modified to include the output content, wherein the output content is generated based on the base content, the target area, and the natural language input, Wherein, the output content is generated by using an artificial intelligence AI model.
2. The non-transitory computer-readable storage medium according to claim 1, in, The base content and the modified base content are images.
3. The non-transitory computer-readable storage medium according to claim 1, in, The target area is a partial area of the basic content.
4. The non-transitory computer-readable storage medium according to claim 1, in, The target region corresponds to an object detected in the base content.
5. The non-transitory computer-readable storage medium according to claim 1, in, At least one of the size or shape of the target area is user adjustable.
6. The non-transitory computer-readable storage medium of claim 1, in, The natural language input includes at least one of voice input or text input, and Wherein, in the case where the natural language input includes the voice input, the voice input is converted into text by using an automatic speech recognition (ASR) model.
7. The non-transitory computer-readable storage medium according to claim 1, in, The output content is generated based on the input content corresponding to the natural language input, and Wherein, the input content corresponding to the natural language input is obtained by using user intention information obtained based on the natural language input.
8. The non-transitory computer-readable storage medium according to claim 7, in, At least one of the input content or the user intention information is obtained by using the AI model.
9. The non-transitory computer-readable storage medium of claim 1, in, The output content is generated by synthesizing input content corresponding to the natural language input into a target area of the base content.
10. The non-transitory computer-readable storage medium of claim 1, in, identifying an object from the underlying content by using the AI model, and Wherein, the output content is generated based on the object by using the AI model.
11. The non-transitory computer-readable storage medium of claim 1, in, When the instructions are executed by the at least one processor, the device is further caused to: A user interface is presented for selecting among a plurality of contents, each content corresponding to the natural language input.
12. The non-transitory computer-readable storage medium of claim 1, in, When the instructions are executed by the at least one processor, the device is further caused to: In a target area of the base content, a transition is made from presenting the base content with the indication presented thereon to presenting modified base content in which the base content is modified to include the output content.
13. The non-transitory computer-readable storage medium of claim 1, in, The output content is generated to match the base content.
14. A method for modifying content executed by a device, the method include: Present basic content; receiving user input for selecting a target area of the base content; presenting, on the base content, an indication corresponding to the selected target area of the base content; receiving natural language input for generating output content; as well as presenting, in a target area of the base content, modified base content in which the base content is modified to include the output content, wherein the output content is generated based on the base content, the target area, and the natural language input, Wherein, the output content is generated by using an artificial intelligence AI model.
15. A device for modifying content, the device include: at least one processor; as well as a memory storing instructions that, when executed by the at least one processor, cause the apparatus to: Present basic content; receiving user input for selecting a target area of the base content; presenting, on the base content, an indication corresponding to the selected target area of the base content; receiving natural language input for generating output content; as well as presenting, in a target area of the base content, modified base content in which the base content is modified to include the output content, wherein the output content is generated based on the base content, the target area, and the natural language input, Wherein, the output content is generated by using an artificial intelligence AI model.
