Electronic device and method for providing summary data related to utterance of user

Abstract

An electronic device and a method for providing summary data related to an utterance of a user are provided. The method by which an electronic device generates summary data comprises the operations of: acquiring media data related to utterances from a plurality of users; acquiring a text script related to the utterances on the basis of the media data; acquiring text summary data from the text script by using a first artificial intelligence model; identifying a main keyword and a main person related to the text script; acquiring an image of the main person corresponding to text in the text summary data from the media data; and using a second artificial intelligence model to acquire image summary data on the basis of the text summary data and the acquired image.

Description

Electronic device and method for providing summary data related to a user's speech

[0001] The present disclosure relates to an electronic device and method for providing summary data related to a user's speech.

[0002] Recently, artificial intelligence systems that achieve human-level intelligence are being utilized in various fields. Unlike existing rule-based smart systems, AI systems are machines that learn and make decisions on their own. As AI systems become more used, their recognition rates improve and their ability to understand user preferences more accurately is increasing. As a result, existing rule-based smart systems are gradually being replaced by deep learning-based AI systems.

[0003] Artificial intelligence technology consists of machine learning (e.g., deep learning) and elemental technologies that utilize machine learning.

[0004] Machine learning is an algorithm technology that classifies / learns the characteristics of input data on its own, and element technology is a technology that imitates the functions of the human brain, such as cognition and judgment, by utilizing machine learning algorithms such as deep learning, and is composed of technical fields such as linguistic understanding, visual understanding, inference / prediction, knowledge representation, and motion control.

[0005] The various fields in which artificial intelligence technology is applied are as follows. Linguistic understanding refers to the technology that recognizes, applies, and processes human language / text, including natural language processing, machine translation, dialogue systems, question-answering, and speech recognition / synthesis. Visual understanding refers to the technology that recognizes and processes objects similar to human vision, including object recognition, object tracking, image search, person recognition, scene understanding, spatial understanding, and image enhancement. Inference prediction refers to the technology that logically infers and predicts information by judging it, including knowledge / probability-based inference, optimization prediction, preference-based planning, and recommendations. Knowledge representation refers to the technology that automatically processes human experience information into knowledge data, including knowledge construction (data creation / classification) and knowledge management (data utilization). Motion control refers to the technology that controls the movement of autonomous vehicles and robots, including movement control (navigation, collision, driving), and manipulation control (behavior control).

[0006] Meanwhile, technologies that facilitate user-to-user conversations through video calls, video conferencing, voice calls, or chat are advancing. Furthermore, recent electronic devices or programs can summarize and provide data using models acquired through artificial intelligence learning.

[0007] Accordingly, there is a growing need to provide a function that effectively summarizes data related to a user's speech using an artificial intelligence model.

[0008] A method for an electronic device to generate summary data related to a user's utterance according to one embodiment may include: executing an application that provides a conversation service between a plurality of users; obtaining media data related to utterances from the plurality of users output through the application; obtaining a text script related to the utterances based on the media data; obtaining text summary data from the text script using a first artificial intelligence model trained for text summarization; identifying a main keyword and a main person related to the text script; obtaining an image of the main person corresponding to a text in the text summary data from the media data; obtaining image summary data based on the text summary data and the obtained image using a second artificial intelligence model trained for generation of image summary data, wherein the image summary data includes a virtual image representing the main person generated by the second artificial intelligence model; and outputting summary data including the text summary data and the image summary data.

[0009] According to one embodiment, an electronic device (1000) that generates summary data related to a user's utterance includes: a display (2060); a memory (2030) that stores commands; And one or more processors (2020); wherein the instructions, when executed by the one or more processors, cause the electronic device to: execute an application that provides a conversation service between a plurality of users, obtain media data related to utterances from the plurality of users output through the application, obtain text scripts related to the utterances based on the media data, obtain text summary data from the text script using a first artificial intelligence model trained for text summarization, identify main keywords and main characters related to the text script, obtain an image of the main character corresponding to text in the text summary data from the media data, and obtain image summary data based on the text summary data and the obtained image using a second artificial intelligence model trained for generation of image summary data, wherein the image summary data includes a virtual image representing the main character generated by the second artificial intelligence model, and output summary data including the text summary data and the image summary data.

[0010] According to one embodiment, a computer-readable recording medium having recorded thereon a program for executing a method, the method comprising: executing an application that provides a conversation service between a plurality of users; obtaining media data related to utterances from the plurality of users output through the application; obtaining text scripts related to the utterances based on the media data; obtaining text summary data from the text script using a first artificial intelligence model trained for text summarization; identifying main keywords and main characters related to the text script; obtaining an image of the main character corresponding to text in the text summary data from the media data; obtaining image summary data based on the text summary data and the obtained image using a second artificial intelligence model trained for generating image summary data, wherein the image summary data includes a virtual image representing the main character generated by the second artificial intelligence model; and outputting summary data including the text summary data and the image summary data.

[0011] According to one embodiment, a method for an electronic device to generate summary data related to a user's utterance includes: obtaining media data related to utterances from a plurality of users; obtaining a text script related to the utterances based on the media data; obtaining text summary data from the text script using a first artificial intelligence model trained for text summarization; identifying whether an image of a main character corresponding to a text in the text summary data is obtained from the media data; and, if the image of the main character is not identified, obtaining image summary data based on the text summary data and the obtained image using a second artificial intelligence model trained for image generation; wherein the image summary data may include a virtual image representing the main character generated by the second artificial intelligence model.

[0012] FIG. 1 is a diagram illustrating an overview of an electronic device according to one embodiment of the present invention that generates summary data summarizing content regarding conversations between users.

[0013] FIG. 2 is a flowchart of a method for generating summary data by an electronic device according to one embodiment.

[0014] FIG. 3 is a flowchart of a method for an electronic device to obtain a text script for a conversation between users according to one embodiment.

[0015] FIG. 4 is a diagram illustrating an example of a GUI for setting key characters and key keywords in an electronic device according to one embodiment.

[0016] FIG. 5 is a diagram illustrating an example of a first artificial intelligence model generating text summary data according to one embodiment.

[0017] FIG. 6 is a flowchart of a method for generating image summary data by an electronic device according to one embodiment.

[0018] FIG. 7 is a diagram illustrating examples of input data and output data of a second artificial intelligence model according to one embodiment.

[0019] FIG. 8 is a diagram illustrating examples of input data and output data of a second artificial intelligence model according to one embodiment.

[0020] FIG. 9 is a diagram illustrating examples of input data and output data of a second artificial intelligence model according to one embodiment.

[0021] FIG. 10 is a drawing showing an example of a GUI displayed on a screen of an electronic device in the process of generating summary data according to one embodiment.

[0022] FIG. 11 is a drawing showing an example of a GUI displayed on a screen of an electronic device in the process of generating summary data according to one embodiment.

[0023] FIG. 12 is a drawing showing an example of a GUI displayed on a screen of an electronic device in the process of generating summary data according to one embodiment.

[0024] FIG. 13a is a drawing showing an example of an execution screen of an application providing video conferencing according to one embodiment.

[0025] FIG. 13b is a diagram illustrating an example of an execution screen of an application that provides conversation between users according to one embodiment.

[0026] FIG. 14 is a diagram illustrating an overview of an electronic device according to one embodiment that summarizes content input from an external input device together with video data in relation to video data.

[0027] FIG. 15 is a flowchart of a method for an electronic device according to one embodiment to summarize content including content input from an external input device and user speech.

[0028] FIG. 16 is a flowchart of a method for an electronic device according to one embodiment to summarize content including content input from an external input device and user speech.

[0029] FIG. 17 is a diagram illustrating an example of generating summary data including input content from an external input device according to one embodiment.

[0030] FIG. 18 is a diagram illustrating an example of a process for generating summary data through an artificial intelligence model according to one embodiment.

[0031] FIG. 19 is a block diagram of an electronic device within a network environment according to various embodiments.

[0032] Figure 20 is a block diagram of an electronic device according to one embodiment.

[0033] FIG. 21 is a diagram illustrating a system including a generative artificial intelligence model according to one embodiment.

[0034] Below, embodiments of the present disclosure are described in detail with reference to the attached drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In addition, for the purpose of clearly explaining the present disclosure in the drawings, parts irrelevant to the description are omitted, and similar parts are designated with similar reference numerals throughout the specification.

[0035] The terms used in this disclosure are described as currently common terms, taking into account the functions mentioned herein. However, these terms may mean various other terms depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Therefore, the terms used in this disclosure should not be interpreted solely based on their names, but rather based on the meanings of the terms and the overall content of this disclosure.

[0036] Additionally, while terms such as first, second, etc. may be used to describe various components, the components should not be limited by these terms. These terms are used to distinguish one component from another.

[0037] Throughout the specification, when a part is said to be "connected" to another part, this includes not only the cases where the parts are "directly connected" but also the cases where the parts are "electrically connected" with other elements intervening. Furthermore, when a part is said to "include" a component, this does not exclude other components, but rather includes other components, unless otherwise stated.

[0038] The phrases “in one embodiment” and the like appearing in various places throughout this disclosure do not necessarily all refer to the same embodiment.

[0039] An embodiment of the present disclosure may be represented by functional block configurations and various processing steps. Some or all of these functional blocks may be implemented by various hardware and / or software configurations that perform specific functions. For example, the functional blocks of the present disclosure may be implemented by one or more microprocessors or by circuit configurations for a given function. Furthermore, for example, the functional blocks of the present disclosure may be implemented in various programming or scripting languages. The functional blocks may be implemented by algorithms that execute on one or more processors. Furthermore, the present disclosure may employ conventional techniques for electronic configuration, signal processing, and / or data processing. Terms such as "mechanism," "element," "means," and "configuration" may be used broadly and are not limited to mechanical and physical configurations.

[0040] Additionally, the connecting lines or connecting members between components depicted in the drawings are merely exemplary representations of functional connections and / or physical or circuit connections. In an actual device, connections between components may be represented by various functional connections, physical connections, or circuit connections that may be replaced or added.

[0041] The present disclosure will be described in detail with reference to the attached drawings below.

[0042] FIG. 1 is a diagram illustrating an overview of an electronic device according to one embodiment of the present invention that generates summary data summarizing content regarding conversations between users.

[0043] Referring to FIG. 1, an electronic device (1000) according to one embodiment can generate summary data summarizing a conversation between users. The conversation between users may include, but is not limited to, at least one of a video conference, a video call, a voice call, or text messaging, for example.

[0044] According to one embodiment, an electronic device (1000) may acquire media data (10) regarding a conversation between users and generate a text script (11) representing the content of the conversation between users by analyzing the acquired media data. The media data may include text data, voice data, and image data (e.g., still images and / or moving images).

[0045] According to one embodiment, the electronic device (1000) may input a text script (11) into at least one artificial intelligence model (13) trained to generate summary data. In this case, the electronic device (1000) may input an image (13) necessary for generating image summary data into at least one artificial intelligence model (13). For example, the electronic device (1000) may input an image extracted from media data related to a conversation and / or a photo of a key person among users having a conversation into at least one artificial intelligence model (13). In addition, the electronic device (1000) may input, for example, information regarding key keywords related to the conversation into at least one artificial intelligence model (13).

[0046] According to one embodiment, the electronic device (1000) can obtain text summary data (14) and image summary data (15) from at least one artificial intelligence model (13). The at least one artificial intelligence model (13) can generate summary data by reflecting weights regarding key characters and / or key keywords. According to one embodiment, the image summary data (15) can include a portion of an image extracted from media data related to a conversation and / or an image generated by at least one artificial intelligence model (13). Thereafter, the electronic device (1000) can output summary data (16) including the text summary data (14) and the image summary data (15).

[0047] An electronic device (1000) according to one embodiment may be, but is not limited to, a smartphone, a tablet PC, a PC, a smart TV, a mobile phone, a personal digital assistant (PDA), a laptop, a media player, a microserver, a global positioning system (GPS) device, an e-book reader, a digital broadcasting terminal, a navigation device, a kiosk, an MP3 player, a digital camera, a home appliance, or any other mobile or non-mobile computing device. In addition, the electronic device (100) may be a wearable device, such as a watch, glasses, a hair band, or a ring, having communication and data processing capabilities. However, the electronic device (100) is not limited thereto, and may include any type of device capable of displaying a message acquired through a network.

[0048] FIG. 2 is a flowchart of a method for generating summary data by an electronic device according to one embodiment.

[0049] In operation 210, the electronic device (1000) may obtain media data representing a conversation between users. According to one embodiment, the electronic device (1000) may obtain media data related to a conversation between users. According to one embodiment, a user of the electronic device (1000) may converse with a user of another electronic device, and the electronic device (1000) may obtain media data received from the other electronic device and media data generated in the electronic device (1000). For example, the electronic device (1000) may provide at least one function among video conferencing, video calling, voice calling, and text messaging, and the electronic device (1000) may obtain media data generated during a conversation between users through such functions. The media data may include, but is not limited to, text data, voice data, and image data (e.g., still images and / or moving images). For example, the electronic device (1000) may provide a function for video conferencing, and the electronic device (1000) may obtain image data (e.g., still images and / or moving images) captured by users participating in the video conference, text data input by users, or voice data recorded by users' speech.

[0050] In operation 220, the electronic device (1000) may obtain a text script of a conversation from media data. According to one embodiment, the electronic device (1000) may obtain text to be included in the text script of the conversation by inputting at least a portion of the media data into an artificial intelligence model. According to one embodiment, the electronic device (1000) may extract text from voice data recorded during a conversation between users and extract text from video data captured to derive a conversation between users.

[0051] According to one embodiment, the electronic device (1000) may generate a text script representing the content of a conversation between users using text input during a conversation between users, text extracted from voice data, and text extracted from video data. The texts within the text script may be arranged in the order of the users' speech times corresponding to the texts.

[0052] The method by which the electronic device (1000) generates a text script regarding a conversation between users will be described in more detail in FIG. 3, which will be described later.

[0053] In operation 230, the electronic device (1000) can identify key figures and / or key keywords for generating summary data.

[0054] An electronic device (1000) according to one embodiment may determine key persons and / or key keywords to be used in summarizing media data generated regarding a conversation between users. According to one embodiment, the electronic device (1000) may provide a graphical user interface (GUI) for selecting key persons and / or key keywords, and may determine the key persons and / or key keywords based on a user input to the GUI. For example, after media data regarding a video conference between users is generated, the electronic device (1000) may display a list including recommended key persons and / or recommended key keywords. Furthermore, based on a user input selecting one or more of the recommended key persons and / or key keywords in the list, the electronic device (1000) may identify the key persons and / or key keywords.

[0055] According to one embodiment, the electronic device (1000) may determine recommended key figures and / or recommended key keywords based on analysis results of media data regarding conversations between users. For example, the electronic device (1000) may identify users' positions and conversation topics from the media data, and determine recommended key figures and / or recommended key keywords based on the users' positions and conversation topics.

[0056] According to one embodiment, the electronic device (1000) may determine recommended key people and / or recommended key keywords based on the profile information of users participating in the conversation and the conversation topics entered into the electronic device (1000). For example, if a video conferencing function is provided through an application of the electronic device (1000), the electronic device (1000) may determine recommended key people and / or recommended key keywords based on the profile information of users participating in the video conference and the conversation topics entered for the opened video conference.

[0057] According to one embodiment, the electronic device (1000) can determine key figures and / or key keywords for generating summary data based on analysis results of media data without user input via a GUI.

[0058] According to one embodiment, the electronic device (1000) can set weights for identified key figures and / or key keywords. The electronic device (1000) can then reflect the weights for key figures and / or key keywords in generating summary data. For example, text corresponding to key figures' utterances and text related to key keywords in utterances can be given a high weight in generating summary data.

[0059] In operation 240, the electronic device (1000) may obtain text summary data for a text script using a first artificial intelligence model. According to an embodiment, the electronic device (1000) may input a text script into a first artificial intelligence model trained for text summarization, and obtain text summary data output from the first artificial intelligence model. In this case, the electronic device (1000) may apply information about key figures and / or key keywords to the first artificial intelligence model together with the text script. The text script may be preprocessed into a format suitable for the first artificial intelligence model before being input to the first artificial intelligence model. The first artificial intelligence model may extract important sentences from the text script, understand the meaning of the text script, and generate new sentences based on the extracted sentences to generate the text summary data. The first artificial intelligence model may be an artificial intelligence model based on natural language interpretation, and may include, for example, at least one of BERT (bidirectional encoder representations from transformers), GPT (generative pre-trained transformer), T5 (text-to-text transfer transformer), or BART (bidirectional and auto-regressive transformers).

[0060] According to one embodiment, the electronic device (1000) may select a first artificial intelligence model suitable for the type of text script from among a plurality of first artificial intelligence models. Furthermore, the electronic device (1000) may request a summary of the text script from the selected first artificial intelligence model.

[0061] In operation 250, the electronic device (1000) can obtain an image related to a person included in text summary data from media data.

[0062] According to one embodiment, the electronic device (1000) can identify a person corresponding to a sentence included in text summary data and obtain an image of the identified person. For example, when text summary data is generated, an identification value of a person corresponding to a sentence in the text summary data may be tagged. In this case, the electronic device (1000) can identify the identification value of the person tagged to the sentence in the text summary data.

[0063] According to one embodiment, the electronic device (1000) may obtain images (e.g., still images and / or videos) related to a person included in text summary data from media data including the user's conversation content and / or separate profile information. For example, an image including the person's face may be obtained from an image captured during the user's conversation. For example, an image of a person who uttered an utterance corresponding to a sentence in the text summary data may be extracted from video data captured for a video conference. Furthermore, for example, an image of a person who uttered an utterance corresponding to a sentence in the text summary data may be extracted from profile information of users registered in an application that provides the user's conversation. Furthermore, for example, an image of a person who uttered an utterance corresponding to a sentence in the text summary data may be extracted from a separate storage device (e.g., another electronic device, a server) that stores profile information of users participating in the conversation.

[0064] In operation 260, the electronic device (1000) can obtain text summary data and image summary data based on the obtained image using the second artificial intelligence model.

[0065] According to one embodiment, the electronic device (1000) may generate image summary data reflecting the summary of the conversation based on the text summary data and the acquired images. The image summary data may include, for example, a plurality of still images reflecting the summary of the conversation. The image summary data may include, for example, at least one video reflecting the summary of the conversation.

[0066] According to one embodiment, the electronic device (1000) may input text summary data and an image of a person to a second artificial intelligence model trained to generate or synthesize an image related to the content of the text based on the text or based on the text and the image. According to one embodiment, the electronic device (1000) may obtain image summary data related to an image of a person corresponding to an utterance in the text summary data.

[0067] The method by which the electronic device (1000) obtains image summary data using the second artificial intelligence model will be described in more detail in FIG. 6, which will be described later.

[0068] In operation 270, the electronic device (1000) may obtain summary data including text summary data and image summary data. According to one embodiment, the electronic device (1000) may display a GUI for displaying the summary data including the text summary data and the image summary data on the screen of the electronic device (1000). Depending on a user input through the GUI, the text summary data and / or the image summary data in the summary data may be displayed on the screen of the electronic device (1000).

[0069] According to one embodiment, the first artificial intelligence model and the second artificial intelligence model may be the same artificial intelligence model. If the second artificial intelligence model is the same as the first artificial intelligence model, the electronic device (1000) may generate text summary data and image summary data using a single artificial intelligence model. In this case, for example, the electronic device (1000) may input at least one of a text script, media data including the user's conversation content, profile information of a person, or an image of a person related to the user's conversation content into the artificial intelligence model, and obtain text summary data and image summary data from the artificial intelligence model.

[0070] FIG. 3 is a flowchart of a method for an electronic device to obtain a text script for a conversation between users according to one embodiment.

[0071] Actions 310 to 350 may correspond to actions 210 and 220 of FIG. 2.

[0072] In operation 310, the electronic device (1000) can extract voice data from video data captured during a conversation between users. The electronic device (1000) can extract voice data from video data captured of a user speaking during a conversation between users.

[0073] In operation 320, the electronic device (1000) can obtain voice data that records voices spoken by users during a conversation between users. The electronic device (1000) can record voices spoken by users using a microphone of the electronic device (1000).

[0074] In operation 330, the electronic device (1000) may apply voice data representing a conversation between users to an artificial intelligence model trained for voice recognition. According to one embodiment, the electronic device (1000) may input the voice data acquired in operations 310 and 320 into the artificial intelligence model trained for voice recognition. According to one embodiment, the electronic device (1000) may preprocess the voice data and input the preprocessed voice data into the artificial intelligence model. For example, the electronic device (1000) may remove noise, such as background noise and echo, included in the voice data and adjust the signal size of the voice data from which the noise has been removed by normalizing it. In addition, for example, the electronic device (1000) may convert the voice data into a sampling rate within a preset range.

[0075] An AI model according to one embodiment can input speech data and convert the speech data into text. The AI ​​model for obtaining text from speech data may include, but is not limited to, an acoustic model that converts audio signals into phonemes, a language model that understands context by converting phoneme sequences into word sequences, and a decoder that combines the outputs of the acoustic model and the language model to generate text.

[0076] According to one embodiment, the electronic device (1000) may, for example, remove unnecessary text from text output from an artificial intelligence model, recognize boundaries between sentences, and check the grammar of the text. Alternatively, for example, the text output from the artificial intelligence model may be text that has undergone removal of unnecessary text, recognition of boundaries between sentences, and grammar checks.

[0077] According to one embodiment, the electronic device (1000) may tag each sentence in the text obtained from the artificial intelligence model with an identification value of a person (e.g., a speaker) corresponding to each sentence. Alternatively, for example, the text output from the artificial intelligence model may be text that includes sentences to which the speaker is tagged.

[0078] According to one embodiment, based on the profile information of the people participating in the conversation, the sentences in the text may be tagged with the identification value of the person. For example, the profile information of the person may include the person's affiliation ID, email address, name, position (vice-president, director, team leader), role (chief technology officer, chief financial officer), nickname, etc. In addition, for example, the profile information of the person may include the person's voice sample and image sample that are pre-stored in the electronic device (1000). According to one embodiment, voice data and the profile information of the person may be input into the artificial intelligence model, and if the voice of the person in the profile information and the voice in the voice data are compared and it is confirmed that the voice in the voice data is the voice of the CFO (chief financial officer), CF0 may be tagged to the text converted from the CFO's voice.

[0079] According to one embodiment, a text script may be tagged with time information corresponding to sentences.

[0080] Although the above description describes the character identification value and time information being tagged for sentences contained within a text script, this is not limited thereto. For example, text contained within a text script may include sentences corresponding to a character's utterance, the character's identification value, and time information.

[0081] In operation 340, the electronic device (1000) may receive a text input for a conversation between users. According to one embodiment, the electronic device (1000) may receive a user's text input entered into an application that provides a conversation between users. For example, the electronic device (1000) may receive a text input entered into the video conferencing application from a user while running a video conferencing application and providing a video conference. For example, the electronic device (1000) may receive the text of a message entered by a user through a message application.

[0082] In operation 350, the electronic device (1000) may obtain a text script representing a conversation between users. According to one embodiment, the electronic device (1000) may generate a text script using the text obtained in operation 330 and the text obtained in operation 340. According to one embodiment, the electronic device (1000) may generate a text script by arranging sentences in the text obtained in operation 330 and sentences in the text obtained in operation 340 in chronological order.

[0083] FIG. 4 is a diagram illustrating an example of a GUI for setting key characters and key keywords in an electronic device according to one embodiment.

[0084] The GUI of FIG. 4 can be displayed by the electronic device (1000) in FIG. 230.

[0085] Referring to FIG. 4, an electronic device (1000) according to one embodiment may display a GUI for setting key persons and key keywords for generating summary data. The electronic device (1000) may display a GUI for setting key persons and key keywords prior to generating summary data for a conversation.

[0086] In one embodiment, the GUI may include a list of key figures (42) and a list of key keywords (44) for generating summary data.

[0087] According to one embodiment, the list of key characters (42) may include identification values ​​of characters selected by the user from among characters corresponding to sentences in the text script. For example, the list of key characters (42) may include “Hosea” and “Swan.” Once the key characters are determined, the electronic device (1000) may set a high weight for sentences corresponding to the key characters when generating summary data. In addition, for example, the key characters in the list of key characters (42) may be set based on the profile information of the characters. For example, the profile information of the character may include the character’s affiliation ID, email address, name, title (Vice-President, Director, Team Leader), role (Chief Technology Officer, Chief Financial Officer), nickname, etc. In addition, for example, the profile information of the character may include the character’s voice sample and image sample that are pre-stored in the electronic device (1000). In this case, the electronic device (1000) may include selected key characters in the list of key characters (42) based on the profile information of the characters corresponding to sentences included in the text script. For example, key characters may be set based on at least one of the characters' ranks or their roles in the conversation.

[0088] In one embodiment, the list of main keywords (44) may include keywords related to the topic of the conversation. For example, the list of main keywords (44) may include "new," "current status," and "plan." Once the main keywords are determined, the electronic device (1000) may assign a high weight to sentences related to the main keywords when generating summary data. For example, the main keywords in the list of main keywords (42) may be determined based on the title or topic of a conversation window opened for conversation. Additionally, for example, the main keywords in the list of main keywords (42) may include keywords selected by the user from among keywords recommended to the user based on the interpretation of the text script.

[0089] For example, the primary keyword may be a keyword related to the content of the conversation. If image data can be expressed as text, keywords expressed from the image data can also be used as primary keywords. For example, if a conversation between users is a meeting, primary keywords can be recommended or determined from the meeting title, the meeting agenda, and the documents to be used at the meeting. For example, if the meeting title is "SW Platform Upgrade Plan Report," the primary keyword may be the meeting title. When summarizing a text transcript obtained from a conversation between users, texts related to the meeting title may be given higher weighting among the text transcripts. Additionally, for example, since the title of the meeting is highly related to the CTO's work, texts corresponding to the CTO's utterances among the attendees may be given higher weighting.

[0090] FIG. 5 is a diagram illustrating an example of a first artificial intelligence model generating text summary data according to one embodiment.

[0091] Referring to FIG. 5, according to one embodiment, a text script and information regarding key characters and key keywords may be input into a first artificial intelligence model (50), and text summary data may be output from the first artificial intelligence model (50). Furthermore, for example, sentences within the text script may be tagged with an identification value of the person who uttered the sentence. Furthermore, for example, information regarding key characters and key keywords may include an identification value of the key characters and values ​​of key keywords.

[0092] In one embodiment, the text script may be preprocessed into a format suitable for the first artificial intelligence model (50) before being input to the first artificial intelligence model (50). For example, the electronic device (1000) may remove unnecessary symbols, numbers, and spaces from the text within the text script, and segment the text into words and / or subwords.

[0093] According to one embodiment, the first artificial intelligence model (50) may extract important sentences from a text script, for example, to generate text summary data. In addition, the first artificial intelligence model (50) may understand the meaning of the text script and generate new sentences based on the understanding result, for example, to generate text summary data. The first artificial intelligence model (50) may be an artificial intelligence model based on natural language interpretation, and may include, for example, at least one of BERT (bidirectional encoder representations from transformers), GPT (generative pre-trained transformer), T5 (text-to-text transfer transformer), or BART (bidirectional and auto-regressive transformers). In this case, depending on the type of the first artificial intelligence model (50), an input prompt in a supported format may be input to the first artificial intelligence model (50). For example, when the first artificial intelligence model (50) is a generative artificial intelligence model, the input prompt may include, but is not limited to, a prompt in a natural language format. In this case, the input prompt may include information regarding how the first artificial intelligence model (50) will generate text summary data. For example, the input prompt may include, but is not limited to, elements for determining the topic, style, and / or length of the text summary data generated from the first artificial intelligence model (50).

[0094] According to one embodiment, sentences in the text summary data output from the first artificial intelligence model (50) may be tagged with an identification value of a person (speaker) corresponding to the sentence.

[0095] According to one embodiment, the calculation of the first artificial intelligence model (50) may be performed by at least one of a central processing unit (CPU), a graphical processing unit (GPU), or a neutral processing unit (NPU) of the electronic device (1000), but is not limited thereto. For example, the calculation of the first artificial intelligence model (50) may also be performed by another electronic device and / or a server. In this case, data for the calculation of the first artificial intelligence model (50) may be provided from the electronic device (1000) to the other electronic device and / or the server.

[0096] FIG. 6 is a flowchart of a method for generating image summary data by an electronic device according to one embodiment.

[0097] Actions 610 to 660 of FIG. 6 may correspond to actions 250 and 260 of FIG. 2.

[0098] In operation 610, the electronic device (1000) may identify a person associated with a text within the text summary data. According to one embodiment, the electronic device (1000) may identify identification values ​​of persons corresponding to text within the text summary data in order to obtain images of persons associated with the text within the text summary data. For example, the electronic device (1000) may identify identification values ​​of persons tagged with texts within the text summary data.

[0099] In operation 620, the electronic device (1000) may obtain an image of an identified person from media data. According to one embodiment, the electronic device (1000) may obtain an image of a person related to text in text summary data from images (e.g., still images and moving images) captured during a conversation between users.

[0100] According to one embodiment, the electronic device (1000) can recognize a person in a captured image by inputting the captured image (e.g., still image and video) taken during a conversation into an artificial intelligence model (e.g., vision model) trained for object recognition. The artificial intelligence model trained for object recognition may include, but is not limited to, an artificial intelligence model based on convolutional neural networks (CNN) and an artificial intelligence model based on region-based convolutional neural networks (R-CNN).

[0101] According to one embodiment, in order to recognize a person in a captured image, images of the people stored in the electronic device (1000) and / or images of the people registered in a server (not shown) by the user of the electronic device (1000) may be used. For example, the artificial intelligence model may identify the names of the people in the captured image by comparing the images of the people identified in the captured image with images of the people stored or registered. In addition, for example, the electronic device (1000) may identify the identification values ​​of the people in the captured image by comparing the identification values ​​and images of the people stored in the contacts of the electronic device (1000) with the people in the captured image. For example, the electronic device (1000) may identify the identification values ​​of the people in the captured image based on the profile information of the people who participated in the conversation service of an application that provides conversation.

[0102] In addition, the electronic device (1000) may obtain an image of a person corresponding to text in the text summary data from the captured image based on the output of the artificial intelligence model. For example, the electronic device (1000) may crop or extract an image of a person related to the text summary data from the captured image based on output data (e.g., identification values ​​of people in the captured image) obtained from the artificial intelligence model. However, the output data of the artificial intelligence model is not limited thereto. For example, when an image and / or identification value of a person related to the captured image and the text summary data are input to the artificial intelligence model, the artificial intelligence model may extract and output an image of a person related to the text summary data from the captured image.

[0103] In operation 630, the electronic device (1000) can determine whether an image of a person was acquired from media data. According to one embodiment, the electronic device (1000) can identify whether an image of a person corresponding to text in the text summary data was acquired from a photographed image taken during a conversation between users.

[0104] If the image of the person is not obtained from the media data, in operation 640, the electronic device (1000) may input an input prompt based on the text summary data to the second artificial intelligence model. According to one embodiment, the second artificial intelligence model may be an artificial intelligence model trained to generate image summary data. In this case, the electronic device (1000) may provide the second artificial intelligence model with an input prompt requesting the generation of the image summary data and the text summary data. In addition, the electronic device (1000) may additionally provide voice data related to the text summary data to the second artificial intelligence model. The second artificial intelligence model may generate image summary data corresponding to the text summary data by generating a virtual image that reflects at least one of the attributes of a sentence in the text summary data, the profile of a person corresponding to the sentence, or the attributes of a voice corresponding to the sentence. The image summary data may include, for example, an image of a virtual person speaking the text in the text summary data. The image summary data may include, for example, a synthesized image that makes it appear as if the virtual person is actually speaking. The image summary data may include, for example, a generated still image and / or a generated video.

[0105] For example, image summary data may include video data generated from sentences within text summary data. In this case, the voice within the video data may be a voice converted from text within the text summary data. Additionally, the image within the video data may be a virtual image that includes a virtual person speaking the text within the text summary data. In this case, for example, a voice may be generated that reflects at least one of the following attributes: the attributes of the sentence within the text summary data, the profile of the person corresponding to the sentence, or the attributes of the voice corresponding to the sentence.

[0106] For example, by analyzing speech corresponding to sentences within text summary data, image summary data can be generated that includes an AI character such as an avatar speaking based on the user's speech characteristics such as intonation, speech style, speed, and gender.

[0107] In one embodiment, image summary data can be generated that excludes, for example, the user's jokes, unorganized words, and interjections. For example, unimportant parts of a user's speech, such as "Um... what I'm trying to say is...", can be removed, and image summary data can be generated that includes an image of a person speaking a complete sentence, focusing on the content of the summarized sentence.

[0108] In one embodiment, the second AI model may be a generative AI model. In this case, the input prompt may include, but is not limited to, a natural language prompt. For example, the input prompt may include information regarding how the second AI model generates image summary data. For example, the input prompt may include, but is not limited to, elements for determining the style and / or length of the image summary data generated by the second AI model.

[0109] If the image of the person is obtained from media data, in operation 650, the electronic device (1000) may input an input prompt based on the text summary data and the image of the person to the second artificial intelligence model. According to one embodiment, the second artificial intelligence model may be an artificial intelligence model trained to generate image summary data. The electronic device (1000) may provide the second artificial intelligence model with an input prompt requesting the generation of image summary data, the text summary data, and the image of the person. In this case, the electronic device (1000) may additionally provide voice data related to the text summary data to the second artificial intelligence model. The second artificial intelligence model may generate image summary data corresponding to the text summary data by generating an image that reflects at least one of the attributes of a sentence in the text summary data, the profile of a person corresponding to the sentence, the attributes of a voice corresponding to the sentence, or the image of a person corresponding to the sentence. The image summary data may include, for example, an image of a person uttering text in the text summary data and / or a virtual image of the person. The image summary data may include, for example, a synthesized image that appears as if the person is actually uttering the text. To generate a virtual image of a person uttering sentences within the text summary data, a second AI model may use an actual image of the person input. The image summary data may include, for example, still images and / or videos.

[0110] For example, the image summary data may include video data generated from sentences within the text summary data. In this case, the voice within the video data may be voice converted from text within the text summary data. Additionally, the image within the video data may be an image generated based on an image of a person input into the second artificial intelligence model. The virtual image within the video data may be generated by modifying an actual image of the person input into the second artificial intelligence model, and thus, the image within the video data may be an image that includes an actual person uttering the text within the text summary data.

[0111] In one embodiment, image summary data can be generated that excludes, for example, the user's jokes, unorganized words, and interjections. For example, unimportant parts of a user's speech, such as "Um... what I'm trying to say is...", can be removed, and image summary data can be generated that includes an image of a person speaking a complete sentence, focusing on the content of the summarized sentence.

[0112] In one embodiment, the second AI model may be a generative AI model. In this case, the input prompt may include, but is not limited to, a natural language prompt. For example, the input prompt may include information regarding how the second AI model generates image summary data. For example, the input prompt may include, but is not limited to, elements for determining the style and / or length of the image summary data generated by the second AI model.

[0113] In operation 660, the electronic device (1000) can obtain image summary data. The electronic device (1000) can receive image summary data output from the second artificial intelligence model from the second artificial intelligence model.

[0114] FIG. 7 is a diagram illustrating examples of input data and output data of a second artificial intelligence model according to one embodiment.

[0115] Referring to FIG. 7, according to one embodiment, when an image of a person related to text summary data is not acquired from media data acquired during a conversation, the text summary data may be input to the second artificial intelligence model (70), and image summary data may be output from the second artificial intelligence model (70). For example, the electronic device (1000) may input an input prompt requesting generation of image summary data and text summary data to the second artificial intelligence model (70). According to one embodiment, voice data of a person corresponding to a sentence in the text summary data may be additionally input to the second artificial intelligence model (70). According to one embodiment, data input to the second artificial intelligence model (70) may be preprocessed into a format suitable for the second artificial intelligence model (70) before being input to the second artificial intelligence model (70).

[0116] In one embodiment, the second artificial intelligence model (70) can generate an image reflecting the attributes of a person associated with a sentence in the text summary data. For example, the second artificial intelligence model (70) can operate by combining natural language processing (NLP) and computer vision technologies. For example, the second artificial intelligence model (70) can include a model for analyzing the meaning or attributes of text, a model for analyzing the attributes of speech, and a model for generating an image based on text.

[0117] According to one embodiment, the second artificial intelligence model (70) may generate image summary data corresponding to the text summary data by generating an image that reflects at least one of the attributes of a sentence in the text summary data, the profile of a person corresponding to the sentence, or the attributes of a voice corresponding to the sentence. The image summary data may include, for example, an image of a virtual person speaking the text in the text summary data. The image summary data may include, for example, an image synthesized as if the virtual person were actually speaking. The image summary data may include, for example, a generated still image and / or a generated video. For example, by analyzing the voice corresponding to a sentence in the text summary data, image summary data including an image of an AI person, such as an avatar, speaking based on the voice characteristics of the user, such as intonation, speech pattern, speed, and gender, may be generated.

[0118] According to one embodiment, the calculation of the second artificial intelligence model (70) may be performed by at least one of a central processing unit (CPU), a graphical processing unit (GPU), or a neutral processing unit (NPU) of the electronic device (1000), but is not limited thereto. For example, the calculation of the second artificial intelligence model (70) may also be performed by another electronic device and / or a server. In this case, data for the calculation of the second artificial intelligence model (70) may be provided from the electronic device (1000) to the other electronic device and / or the server.

[0119] FIG. 8 is a diagram illustrating examples of input data and output data of a second artificial intelligence model according to one embodiment.

[0120] Referring to FIG. 8, according to one embodiment, when an image of a person related to text summary data is acquired from media data acquired during a conversation, the text summary data and the image of the person may be input to a second artificial intelligence model (80), and image summary data may be output from the second artificial intelligence model (80). For example, the electronic device (1000) may input an input prompt requesting generation of image summary data, the text summary data, and the image of the person to the second artificial intelligence model (80). According to one embodiment, voice data of the person corresponding to a sentence in the text summary data may be additionally input to the second artificial intelligence model (80). According to one embodiment, data input to the second artificial intelligence model (80) may be preprocessed into a format suitable for the second artificial intelligence model (80) before being input to the second artificial intelligence model (80).

[0121] In one embodiment, the second artificial intelligence model (80) can generate an image reflecting the attributes of a person related to a sentence in the text summary data. For example, the second artificial intelligence model (80) can operate by combining natural language processing (NLP) and computer vision technologies. For example, the second artificial intelligence model (80) can include a model for analyzing the meaning or attributes of text, a model for analyzing the attributes of an image, a model for analyzing the attributes of speech, and a model for generating an image based on text and images.

[0122] According to one embodiment, the second artificial intelligence model (80) may generate image summary data corresponding to the text summary data by generating an image that reflects at least one of the attributes of a sentence in the text summary data, the profile of a person corresponding to the sentence, the attributes of a person image corresponding to the sentence, or the attributes of a voice corresponding to the sentence. The image summary data may include, for example, an image of an actual person speaking the text in the text summary data. For example, the image in the image summary data may be generated by modifying an image of an actual person input to the second artificial intelligence model (80), and thus, the image in the image summary data may be an image that includes an image of an actual person speaking the text in the text summary data.

[0123] According to one embodiment, the calculation of the second artificial intelligence model (80) may be performed by at least one of a central processing unit (CPU), a graphical processing unit (GPU), or a neutral processing unit (NPU) of the electronic device (1000), but is not limited thereto. For example, the calculation of the second artificial intelligence model (80) may also be performed by another electronic device and / or a server. In this case, data for the calculation of the second artificial intelligence model (80) may be provided from the electronic device (1000) to the other electronic device and / or the server.

[0124] FIG. 9 is a diagram illustrating examples of input data and output data of a second artificial intelligence model according to one embodiment.

[0125] Referring to FIG. 9, according to one embodiment, text summary data and an image of a person may be input into a second artificial intelligence model (80), and summary data including text summary data and image summary data may be output from the second artificial intelligence model (80).

[0126] Summary data including text summary data and image summary data according to one embodiment may include a GUI displaying the text summary data and the image summary data. For example, sentences within the text summary data and images within the image summary data may be synchronized, and the GUI of the summary data may be configured to display the sentences within the synchronized text summary data and the images within the image summary data together.

[0127] According to one embodiment, the first artificial intelligence model and the second artificial intelligence model may be the same artificial intelligence model. If the second artificial intelligence model is the same as the first artificial intelligence model, the electronic device (1000) may generate text summary data and image summary data using a single artificial intelligence model. In this case, the artificial intelligence model that generates the text summary data and the image summary data may be, for example, an artificial intelligence model that provides comprehensive functions including natural language interpretation, voice analysis, image analysis, and image generation. In addition, for example, the electronic device (1000) may input at least one of a text script, a main character, a main keyword, media data including the user's conversation content, the character's profile information, and an image of the character related to the user's conversation content into the artificial intelligence model, and may obtain the text summary data and the image summary data from the artificial intelligence model.

[0128] FIG. 10 is a drawing showing an example of a GUI displayed on a screen of an electronic device in the process of generating summary data according to one embodiment.

[0129] Referring to the identification number 100 of FIG. 10, a text script representing the content of a conversation between users may be displayed on the display of the electronic device (1000). For example, if Jane, Swan, and Hosea participate in the conversation, sentences corresponding to the voices spoken by Jane, Swan, and Hosea may be included in the text script. Additionally, the identification values ​​of the speakers who uttered the sentences, Jane, Swan, and Hosea, may be displayed corresponding to the sentences in the text script. Furthermore, the sentences in the text script may be arranged in chronological order.

[0130] Referring to identification number 102, a GUI for setting key figures and key keywords for generating summary data may be displayed on the display of the electronic device (1000). The GUI for setting key figures and key keywords may include, for example, an object (102-1) for receiving a user input for setting key figures, an object (102-2) for receiving a user input for setting key keywords, and an object (102-3) for receiving a user input for requesting generation of summary data. In addition, in identification number 110, for example, Hosea and Swan may be set as key figures, and “new,” “current status,” and “plan” may be set as key keywords.

[0131] Referring to the identification number 104, when a main character and a main keyword are set and an object (102-3) is selected, summary data (104) may be displayed on the screen of the electronic device (1000). The summary data (104) may include text summary data (104-1) and image summary data (104-2). The text summary data (104-1) may include, for example, sentences corresponding to voices spoken by Hosea in relation to the main keyword and sentences corresponding to voices spoken by Swan in relation to the main keyword. In addition, the image summary data (104-2) may be video data including images of Hosea and Swan speaking sentences of the text summary data.

[0132] FIG. 11 is a drawing showing an example of a GUI displayed on a screen of an electronic device in the process of generating summary data according to one embodiment.

[0133] In Figure 11, for example, one main character and one main keyword can be set.

[0134] Referring to the identification number 110 of FIG. 11, a text script representing the content of a conversation between users may be displayed on the display of the electronic device (1000). For example, if Jane, Swan, and Hosea participate in the conversation, sentences corresponding to the voices spoken by Jane, Swan, and Hosea may be included in the text script. Additionally, the identification values ​​of the speakers who uttered the sentences, Jane, Swan, and Hosea, may be displayed corresponding to the sentences in the text script. Furthermore, the sentences in the text script may be arranged in chronological order.

[0135] Referring to identification number 112, a GUI for setting key figures and key keywords for generating summary data may be displayed on the display of the electronic device (1000). The GUI for setting key figures and key keywords may include, for example, an object (112-1) for receiving a user input for setting key figures, an object (112-2) for receiving a user input for setting key keywords, and an object (112-3) for receiving a user input for requesting generation of summary data. In addition, in identification number 110, for example, Swan may be set as a key figure, and “strategy” may be set as a key keyword.

[0136] Referring to the identification number 114, when a main character and a main keyword are set and an object (112-3) is selected, summary data (114) may be displayed on the screen of the electronic device (1000). The summary data (114) may include text summary data (114-1) and image summary data (114-2). The text summary data (114-1) may include, for example, sentences corresponding to speech uttered by Swan in relation to the main keyword. In addition, the image summary data (114-2) may be video data including an image of Swan uttering sentences of the text summary data.

[0137] FIG. 12 is a drawing showing an example of a GUI displayed on a screen of an electronic device in the process of generating summary data according to one embodiment.

[0138] In Fig. 12, multiple media data can be selected to generate a text script.

[0139] Referring to the identification number 120, a GUI for user settings for generating summary data may be displayed on the display of the electronic device (1000). The GUI for user settings may include, for example, objects (120-1, 120-2, 120-3) for selecting media data to be used for generating a text script, an object (120-4) for receiving user input for setting a main character, and an object (120-5) for receiving user input for requesting generation of summary data. Although not illustrated in FIG. 12, the GUI for user settings may include, for example, an object for receiving user input for setting a main keyword.

[0140] For example, objects (120-1, 120-2, 120-3) for selecting media data to be used for generating a text script may include an object (120-1) for using media data generated during a call between users (e.g., voice call, video call) for generating a text script, an object (120-2) for using mail exchanged between users for generating a text script, and an object (120-3) for using media data input through chat between users for generating a text script.

[0141] For example, media data generated during user conversations (e.g., voice calls, video calls) may include voice data and video data. For example, emails exchanged between users may include text data and image data contained within the email. For example, media data input through chat between users may include text data, voice data, and image data entered by the users.

[0142] Referring to identification number 122, when at least one of objects (120-1, 120-2, 120-3) is selected by a user, a text script may be generated based on media data corresponding to the selected objects (120-1, 120-2, 120-3). For example, sentences representing utterances of Gin, Jane, Sun, and Chun may be included in the text script. In addition, identification values ​​of Gin, Jane, Sun, and Chun corresponding to the sentences may be displayed corresponding to the sentences in the text script. In addition, the sentences in the text script may be arranged in chronological order.

[0143] Referring to identification number 120, Gin is set as the main character and an object (120-5) requesting the generation of summary data can be selected by the user.

[0144] Referring to identification number 124, when a main character is set and an object (120-5) is selected, summary data (124) may be displayed on the screen of the electronic device (1000). The summary data (124) may include text summary data (124-1) and image summary data (124-2). The text summary data (124-1) may include, for example, sentences corresponding to Gin's speech. Additionally, the image summary data (124-2) may be video data including an image of Gin uttering sentences of the text summary data.

[0145] FIG. 13a is a drawing showing an example of an execution screen of an application providing video conferencing according to one embodiment.

[0146] Referring to FIG. 13a, an execution screen (130) of an application providing video conferencing according to one embodiment may be displayed on the display of an electronic device (1000). Images of people participating in the video conference may be displayed in real time on the execution screen (130) of the application providing video conferencing.

[0147] For example, an execution screen (130) of an application providing video conferencing according to one embodiment may include an object (132) for receiving a user input requesting summary data of the video conference.

[0148] For example, as an object (132) is selected by a user, the electronic device (1000) can generate summary data about the video conference based on images and speech of people participating in the video conference.

[0149] FIG. 13b is a diagram illustrating an example of an execution screen of an application that provides conversation between users according to one embodiment.

[0150] Referring to FIG. 13b, an execution screen (135) of an application providing conversations between users according to one embodiment may be displayed on the display of an electronic device (1000). Images of people participating in the conversation may be displayed in real time on the execution screen (135) of the application providing conversations between users.

[0151] An execution screen (135) of an application that provides conversations between users according to one embodiment may include objects (136) that provide various functions related to conversations between users.

[0152] An execution screen (135) of an application providing conversations between users according to one embodiment may include an object (137) for receiving a user input requesting summary data of the conversation. For example, when the object (137) is selected by the user, the electronic device (1000) may generate summary data (138) regarding the conversation based on images and speech of the individuals participating in the conversation. The generated summary data (138) may be displayed within the execution screen (135) of the application providing conversations between users.

[0153] FIG. 14 is a diagram illustrating an overview of an electronic device according to one embodiment that summarizes content input from an external input device together with video data in relation to video data.

[0154] Referring to FIG. 14, an electronic device (1000) according to one embodiment can play video data (142) and, during playback of the video data, can generate handwritten content (144) based on input from a separate external input device (3000). The external input device (3000) may include, for example, a stylus pen, but is not limited thereto.

[0155] An electronic device (1000) according to one embodiment can generate summary data by summarizing video data (142) and handwritten content (144) together.

[0156] FIG. 15 is a flowchart of a method for an electronic device according to one embodiment to summarize content including content input from an external input device and user speech.

[0157] In operation 1510, the electronic device (1000) may receive an input from an external input device (3000). According to one embodiment, the electronic device (1000) may receive an input from the external input device (3000) related to the content being played while playing the content. For example, the content may include, but is not limited to, video lecture content, video for video calls between users, and video for video conferences between users. For example, the input from the external input device (3000) may include, but is not limited to, a handwriting input from a stylus pen.

[0158] In operation 1520, the electronic device (1000) can identify input content input from an external input device (3000). According to one embodiment, the electronic device (1000) can obtain input content generated from an input from the external input device (3000). For example, the electronic device (1000) can obtain handwritten content based on a handwritten input from a stylus pen. In this case, the electronic device (1000) can map the handwritten content to the playback time of the content being played based on the input time of the handwritten input.

[0159] In operation 1520, the electronic device (1000) can identify a portion corresponding to the input content from the summary data. According to one embodiment, the electronic device (1000) can generate summary data of the content. The electronic device (1000) can generate summary data of the content, for example, as shown in FIGS. 1 to 14 . In addition, for example, the electronic device (1000) can identify a portion corresponding to the input content from among texts included in the summary data of the content. For example, the electronic device (1000) can identify the playback time of a portion corresponding to the input content from video data included in the summary data of the content.

[0160] In operation 1530, the electronic device (1000) may merge the input content identified in the summary data. According to one embodiment, the electronic device (1000) may merge the input content in the summary data so that when a portion of text included in the summary data of the content corresponding to the input content is displayed, the input content is displayed together. For example, the electronic device (1000) may merge the input content in the summary data so that when a portion of video data included in the summary data of the content corresponding to the input content is displayed, the input content is displayed together.

[0161] Although the above description describes that the electronic device (1000) merges input content from an external input device (3000) into the generated summary data, the present invention is not limited thereto. For example, the electronic device (1000) may input input content from an external input device (3000) into at least one artificial intelligence model used to generate the summary data, thereby generating summary data that includes at least a portion of the input content.

[0162] FIG. 16 is a flowchart of a method for an electronic device according to one embodiment to summarize content including content input from an external input device and user speech.

[0163] Referring to FIG. 16, according to one embodiment, when text summary data is generated based on content including a user's speech, an image of a person related to the text summary data is acquired from the content including the user's speech, and input content input from an external input device (3000) is acquired, the text summary data, the image of the person, and the input content input from the external input device (3000) may be input to a second artificial intelligence model (160). In addition, summary data including text summary data, image summary data, and content summary data may be output from the second artificial intelligence model (160). The content summary data may be data generated by summarizing input content from the external input device (3000).

[0164] For example, the electronic device (1000) may input an input prompt requesting the generation of summary data, text summary data, an image of a person, and input content into the second artificial intelligence model (160). According to one embodiment, voice data of a person corresponding to a sentence in the text summary data may be additionally input into the second artificial intelligence model (160). According to one embodiment, data input into the second artificial intelligence model (160) may be preprocessed into a format suitable for the second artificial intelligence model (160) before being input into the second artificial intelligence model (160).

[0165] According to one embodiment, the second artificial intelligence model (160) can generate an image that reflects the attributes of a person related to a sentence in the text summary data. For example, the second artificial intelligence model (160) can operate by combining natural language processing (NLP) and computer vision technology. For example, the second artificial intelligence model (160) can include a model for analyzing the meaning or attributes of a text, a model for analyzing the attributes of an image, a model for analyzing the attributes of a voice, and a model for generating an image based on text and an image. According to one embodiment, the second artificial intelligence model (160) can generate image summary data corresponding to the text summary data by generating an image that reflects at least one of the attributes of a sentence in the text summary data, the profile of a person corresponding to the sentence, the attributes of a person image corresponding to the sentence, or the attributes of a voice corresponding to the sentence. The image summary data can include, for example, an image of an actual person speaking the text in the text summary data. For example, an image within the image summary data may be generated by transforming an image of a real person input into the second artificial intelligence model (160), and thus, the image within the image summary data may be an image that includes an image of a real person uttering text within the text summary data.

[0166] According to one embodiment, content summary data may be generated by summarizing input content from an external input device (3000). For example, if the input content is handwritten content, content summary data may be generated by removing portions of the handwritten content that are irrelevant to the text summary data. Furthermore, the content summary data may be mapped to text summary data and image summary data. For example, if the content summary data is summary data of handwritten content, the content summary data of the handwritten content may be mapped to a portion of the text summary data and a portion of the image summary data corresponding to the time at which the handwritten content is generated by input using a stylus pen.

[0167] According to one embodiment, the calculation of the second artificial intelligence model (160) may be performed by at least one of a central processing unit (CPU), a graphical processing unit (GPU), or a neutral processing unit (NPU) of the electronic device (1000), but is not limited thereto. For example, the calculation of the second artificial intelligence model (1600) may also be performed by another electronic device and / or a server. In this case, data for the calculation of the second artificial intelligence model (160) may be provided from the electronic device (1000) to the other electronic device and / or the server.

[0168] FIG. 17 is a diagram illustrating an example of generating summary data including input content from an external input device according to one embodiment.

[0169] Referring to the identification number 1710 of FIG. 17, the electronic device (1000) may display a GUI for setting the external input device (3000) in relation to the setting for generating summary data. For example, the electronic device (1000) may display an object (1711) for searching for an external input device (3000) connectable to the electronic device (1000). When the object (1711) is selected by the user, the electronic device (1000) may search for the external input device (3000). For example, the electronic device (1000) may search for a stylus pen. In addition, for example, the electronic device (1000) may display an object (1713) for setting the input from the external input device (3000). For example, the electronic device (1000) may determine to use handwritten content based on input from a stylus pen to generate summary data based on user input to the object (1712).

[0170] Referring to identification number 1720 of FIG. 17, the electronic device (1000) can display summary data including text summary data (1721) and content summary data (1722). The text summary data (1721) may be summary data of text generated from content including a user's speech. The text summary data may be text generated by summarizing a text script generated from a lecture video content, for example. In addition, the content summary data (1722) may be handwriting summary data generated by summarizing handwriting input input from a stylus pen, for example. The content summary data (1722) may be displayed together with a corresponding portion of the text summary data.

[0171] FIG. 18 is a diagram illustrating an example of a process for generating summary data through an artificial intelligence model according to one embodiment.

[0172] Referring to FIG. 18, an electronic device (1000) according to one embodiment may obtain a text script (180-2) from voice data (180-1) uttered by users through a conversation between users using voice recognition technology. The electronic device (1000) may identify the type of the text script (180-2) using a classifier (180-3) and select at least one artificial intelligence model suitable for the type from among a plurality of artificial intelligence models (180-4). In addition, the text script (180-2) may be input into the at least one selected artificial intelligence model. In this case, the electronic device (1000) may input information (180-12) on users participating in the conversation and information (180-13) regarding main keywords into the at least one selected artificial intelligence model. Accordingly, text summary data (180-5) may be generated from the at least one selected artificial intelligence model.

[0173] According to one embodiment, the electronic device (1000) may apply images (180-10) captured by users during a conversation between users (e.g., still images and / or videos), information (180-12) about users participating in the conversation, and information (180-13) about main keywords to an artificial intelligence model (180-14) for generating summary image data among a plurality of artificial intelligence models (180-4). Accordingly, image summary data (180-15) may be generated from the artificial intelligence model (180-14).

[0174] According to one embodiment, summary data (180-20) may be generated by merging text summary data (180-5) and image summary data (180-15).

[0175] According to one embodiment, a method for an electronic device (1000) to generate summary data related to a user's utterance may include: executing an application that provides a conversation service between a plurality of users; obtaining media data related to utterances from the plurality of users output through the application (210); obtaining a text script related to the utterances based on the media data (220); obtaining text summary data from the text script using a first artificial intelligence model trained for text summarization (240); identifying a main keyword and a main person related to the text script (230); obtaining an image of the main person corresponding to a text in the text summary data from the media data (250); obtaining image summary data based on the text summary data and the obtained image using a second artificial intelligence model trained for generation of image summary data, wherein the image summary data includes a virtual image representing the main person generated by the second artificial intelligence model (260); and outputting summary data including the text summary data and the image summary data (270).

[0176] According to one embodiment, the operation of obtaining a text script related to the utterances from the media data may include an operation of obtaining the text script by converting voice data of the users input from the users while the conversation service is provided into text data.

[0177] According to one embodiment, the method wherein the voice data is extracted from video data output from the electronic device while the conversation service is being provided.

[0178] According to one embodiment, the operation of obtaining a text script related to the utterances from the media data may generate the text script based on text input from the users while the conversation service is being provided.

[0179] In one embodiment, the first artificial intelligence model may include a generative artificial intelligence model trained to summarize the text script.

[0180] According to one embodiment, the operation of obtaining the text summary data from the text script using the first artificial intelligence model may include: an operation of generating a first input prompt based on the main keyword, the main character, and the text script; and an operation of obtaining the text summary data by applying the first input prompt to the first artificial intelligence model.

[0181] According to one embodiment, the method may further include an operation of selecting the first artificial intelligence model corresponding to the identified category from among a plurality of artificial intelligence models by identifying the category of the text script.

[0182] According to one embodiment, the method further includes an operation of providing a GUI (graphical user interface) for setting the main keyword and the main person through the execution screen of the application; and the operation of identifying the main keyword and the main person can identify the main keyword and the main person based on a user input to the GUI.

[0183] According to one embodiment, the second artificial intelligence model includes a generative artificial intelligence model trained to generate the image summary data, and the operation of obtaining image summary data based on the text summary data and the acquired image may include: an operation of generating a second input prompt based on the text summary data, the main keyword, the main person, and the acquired image; and an operation of obtaining the image summary data by applying the second input prompt to the second artificial intelligence model.

[0184] According to one embodiment, the image summary data may include a virtual image representing the main character generated by the second artificial intelligence model based on an image of the main character obtained from the media data and text corresponding to the main character in the text summary data.

[0185] According to one embodiment, the method further comprises: obtaining input content input by an external input device in relation to the media data; wherein the summary data may include at least a portion of the input content merged with the text summary data and the image summary data.

[0186] FIG. 19 is a block diagram of an electronic device (1901) within a network environment (1900) according to various embodiments. Referring to FIG. 19 , in the network environment (1900), the electronic device (1901) may communicate with the electronic device (1902) via a first network (1998) (e.g., a short-range wireless communication network), or may communicate with the electronic device (1904) or a server (1908) via a second network (1999) (e.g., a long-range wireless communication network). In one embodiment, the electronic device (1901) may communicate with the electronic device (1904) via the server (1908). According to one embodiment, the electronic device (1901) may include a processor (1920), a memory (1930), an input module (1950), an audio output module (1955), a display module (1960), an audio module (1970), a sensor module (1976), an interface (1977), a connection terminal (1978), a haptic module (1979), a camera module (1980), a power management module (1988), a battery (1989), a communication module (1990), a subscriber identification module (1996), or an antenna module (1997). In some embodiments, the electronic device (1901) may omit at least one of these components (e.g., the connection terminal (1978)), or may have one or more other components added. In some embodiments, some of these components (e.g., sensor module (1976), camera module (1980), or antenna module (1997)) may be integrated into a single component (e.g., display module (1960)).

[0187] The processor (1920) may control at least one other component (e.g., a hardware or software component) of the electronic device (1901) connected to the processor (1920) by executing, for example, software (e.g., a program (1940)), and may perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (1920) may store commands or data received from other components (e.g., a sensor module (1976) or a communication module (1990)) in a volatile memory (1932), process the commands or data stored in the volatile memory (1932), and store result data in a non-volatile memory (1934). According to one embodiment, the processor (1920) may include a main processor (1921) (e.g., a central processing unit or an application processor) or a secondary processor (1923) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (1921). For example, when the electronic device (1901) includes the main processor (1921) and the secondary processor (1923), the secondary processor (1923) may be configured to use less power than the main processor (1921) or to be specialized for a given function. The secondary processor (1923) may be implemented separately from the main processor (1921) or as a part thereof.

[0188] The auxiliary processor (1923) may control at least a portion of functions or states associated with at least one component (e.g., a display module (1960), a sensor module (1976), or a communication module (1990)) of the electronic device (1901), for example, on behalf of the main processor (1921) while the main processor (1921) is in an inactive (e.g., sleep) state, or together with the main processor (1921) while the main processor (1921) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (1923) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (1980) or a communication module (1990)). In one embodiment, the auxiliary processor (1923) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (1901) where the artificial intelligence is performed, or can be performed through a separate server (e.g., server (1908)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0189] The memory (1930) can store various data used by at least one component (e.g., the processor (1920) or the sensor module (1976)) of the electronic device (1901). The data can include, for example, software (e.g., the program (1940)) and input data or output data for commands related thereto. The memory (1930) can include volatile memory (1932) or non-volatile memory (1934).

[0190] The program (1940) may be stored as software in memory (1930) and may include, for example, an operating system (1942), middleware (1944), or an application (1946).

[0191] The input module (1950) can receive commands or data to be used in a component of the electronic device (1901) (e.g., a processor (1920)) from an external source (e.g., a user) of the electronic device (1901). The input module (1950) can include, for example, a microphone, a mouse, a keyboard, keys (e.g., buttons), or a digital pen (e.g., a stylus pen).

[0192] The audio output module (1955) can output audio signals to the outside of the electronic device (1901). The audio output module (1955) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0193] The display module (1960) can visually provide information to an external party (e.g., a user) of the electronic device (1901). The display module (1960) may include, for example, a display, a holographic device, or a projector, and a control circuit for controlling the device. In one embodiment, the display module (1960) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0194] The audio module (1970) can convert sound into an electrical signal, or vice versa. According to one embodiment, the audio module (1970) can acquire sound through the input module (1950), output sound through the sound output module (1955), or an external electronic device (e.g., electronic device (1902)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (1901).

[0195] The sensor module (1976) can detect the operating status (e.g., power or temperature) of the electronic device (1901) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (1976) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0196] The interface (1977) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (1901) with an external electronic device (e.g., the electronic device (1902)). In one embodiment, the interface (1977) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0197] The connection terminal (1978) may include a connector through which the electronic device (1901) may be physically connected to an external electronic device (e.g., the electronic device (1902)). In one embodiment, the connection terminal (1978) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0198] A haptic module (1979) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. In one embodiment, the haptic module (1979) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0199] The camera module (1980) can capture still images and moving images. In one embodiment, the camera module (1980) may include one or more lenses, image sensors, image signal processors, or flashes.

[0200] The power management module (1988) can manage the power supplied to the electronic device (1901). According to one embodiment, the power management module (1988) can be implemented as at least a part of, for example, a power management integrated circuit (PMIC).

[0201] A battery (1989) may power at least one component of the electronic device (1901). In one embodiment, the battery (1989) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0202] The communication module (1990) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (1901) and an external electronic device (e.g., electronic device (1902), electronic device (1904), or server (1908)), and the performance of communication through the established communication channel. The communication module (1990) may operate independently from the processor (1920) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (1990) may include a wireless communication module (1992) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (1994) (e.g., a local area network (LAN) communication module, or a power line communication module). Any of these communication modules may communicate with an external electronic device (1904) via a first network (1998) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (1999) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a local area network or a wide area network)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (1992) may use subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (1996) to identify or authenticate the electronic device (1901) within a communication network such as the first network (1998) or the second network (1999).

[0203] The wireless communication module (1992) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimizing terminal power and connecting multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (1992) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (1992) may support various technologies for securing performance in high frequency bands, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (1992) may support various requirements specified in the electronic device (1901), an external electronic device (e.g., the electronic device (1904)), or a network system (e.g., the second network (1999)). According to one embodiment, the wireless communication module (1992) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0204] The antenna module (1997) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (1997) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (1997) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (1998) or the second network (1999), may be selected from the plurality of antennas by, for example, the communication module (1990). A signal or power may be transmitted or received between the communication module (1990) and the external electronic device via the at least one selected antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (1997).

[0205] According to various embodiments, the antenna module (1997) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high frequency band.

[0206] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0207] According to one embodiment, commands or data may be transmitted or received between the electronic device (1901) and an external electronic device (1904) via a server (1908) connected to a second network (1999). Each of the external electronic devices (1902 or 1904) may be the same or a different type of device as the electronic device (1901). According to one embodiment, all or part of the operations executed in the electronic device (1901) may be executed in one or more of the external electronic devices (1902, 1904, or 1908). For example, when the electronic device (1901) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (1901) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or additional functions or services related to the request, and transmit the result of the execution to the electronic device (1901). The electronic device (1901) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (1901) may provide an ultra-low latency service using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device (1904) may include an Internet of Things (IoT) device. The server (1908) may be an intelligent server utilizing machine learning and / or a neural network.In one embodiment, an external electronic device (1904) or server (1908) may be included within the second network (1999). The electronic device (1901) may be applied to intelligent services (e.g., smart homes, smart cities, smart cars, or healthcare) based on 5G communication technology and IoT-related technology.

[0208] An electronic device (1901) according to one embodiment may correspond to the electronic device (1000) of FIGS. 1 to 18, and the electronic device (1901) may perform the operations of the electronic device (1000) of FIGS. 1 to 18.

[0209] Figure 20 is a block diagram of an electronic device according to one embodiment.

[0210] The electronic device (1000) of FIG. 20 may correspond to the electronic device (1901) of FIG. 19.

[0211] Referring to FIG. 20, the electronic device (1000) may include an input module (2050), a camera module (2080), a display module (2060), a memory (2030), and a processor (2020).

[0212] The input module (2050) can receive commands or data to be used in a component of the electronic device (1000) (e.g., a processor (2020)) from an external source (e.g., a user) of the electronic device (1000). The input module (2050) can receive input from, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0213] The camera module (2080) can capture still images and videos. In one embodiment, the camera module (2080) may include one or more lenses, image sensors, image signal processors, or flashes.

[0214] The display module (2060) can visually provide information to an external party (e.g., a user) of the electronic device (1000). The display module (2060) may include, for example, a display, a holographic device, or a projector, and a control circuit for controlling the device. In one embodiment, the display module (2060) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0215] The memory (2030) can store various data used by at least one component of the electronic device (1000) (e.g., the processor (2020) or a sensor module (not shown)). The data can include, for example, input data or output data for software and commands related thereto. The memory (2030) can include volatile memory or non-volatile memory. The memory (2030) can include at least one first artificial intelligence model (2030-1) and at least one second artificial intelligence model (2030-2). However, the present invention is not limited thereto, and the memory (2030) can also include other artificial intelligence models.

[0216] The processor (2020) may, for example, execute software to control at least one other component (e.g., hardware or software component) of the electronic device (1000) connected to the processor (2020) and perform various data processing or calculations. According to one embodiment, the processor (2020) may include a main processor (e.g., a central processing unit or an application processor) or an auxiliary processor (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that may operate independently or together therewith.

[0217] According to one embodiment, the processor (2020) can control the operations of the electronic device (1000) in FIGS. 1 to 18.

[0218] According to one embodiment, the processor (2020) may obtain media data representing a conversation between users. According to one embodiment, the processor (2020) may obtain media data related to a conversation between users. According to one embodiment, a user of the electronic device (1000) may converse with a user of another electronic device, and the processor (2020) may obtain media data received from the other electronic device and media data generated in the electronic device (1000). For example, the processor (2020) may provide at least one function among video conferencing, video calling, voice calling, and text messaging, and the processor (2020) may obtain media data generated during a conversation between users through such functions. The media data may include, but is not limited to, text data, voice data, and image data (e.g., still images and / or moving images). For example, the electronic device (1000) may provide a function for video conferencing, and the processor (2020) may obtain image data (e.g., still images and / or moving images) captured by users participating in the video conference, text data input by users, or voice data recorded by users' speech.

[0219] In one embodiment, the processor (2020) can obtain a text transcript of a conversation from media data. In one embodiment, the processor (2020) can obtain text to be included in the text transcript of the conversation by inputting at least a portion of the media data into an artificial intelligence model. In one embodiment, the processor (2020) can extract text from voice data recorded during a conversation between users and extract text from video data captured during the conversation between users.

[0220] According to one embodiment, the processor (2020) may generate a text script representing the content of a conversation between users using text input during a conversation between users, text extracted from voice data, and text extracted from video data. The texts within the text script may be arranged in the order of the users' speech times corresponding to the texts.

[0221] According to one embodiment, the processor (2020) may identify key figures and / or key keywords for generating summary data.

[0222] In one embodiment, a processor (2020) may determine key persons and / or key keywords to be used in summarizing media data generated regarding conversations between users. In one embodiment, the processor (2020) may provide a graphical user interface (GUI) for selecting key persons and / or key keywords, and may determine the key persons and / or key keywords based on user input to the GUI. For example, after media data regarding a video conference between users is generated, the processor (2020) may display a list including recommended key persons and / or recommended key keywords. Furthermore, based on a user input selecting one or more of the recommended key persons and / or key keywords within the list, the processor (2020) may identify the key persons and / or key keywords.

[0223] In one embodiment, the processor (2020) may determine recommended key figures and / or recommended key keywords based on analysis results of media data regarding conversations between users. For example, the processor (2020) may identify users' job titles and conversation topics from the media data, and determine recommended key figures and / or recommended key keywords based on the users' job titles and conversation topics.

[0224] According to one embodiment, the processor (2020) may determine recommended key people and / or recommended key keywords based on the profile information of users participating in the conversation and the conversation topics entered into the electronic device (1000). For example, if a video conferencing function is provided through an application of the electronic device (1000), the processor (2020) may determine recommended key people and / or recommended key keywords based on the profile information of users participating in the video conference and the conversation topics entered for the opened video conference.

[0225] According to one embodiment, the processor (2020) can determine key figures and / or key keywords for generating summary data based on the analysis results of media data without user input via the GUI.

[0226] In one embodiment, the processor (2020) may set weights for identified key figures and / or key keywords. The processor (2020) may then reflect the weights for key figures and / or key keywords in generating summary data. For example, text corresponding to key figures' utterances or text related to key keywords may be given a higher weight in generating summary data.

[0227] According to one embodiment, the processor (2020) may obtain text summary data for a text script using the first artificial intelligence model (2030-1). The processor (2020) according to one embodiment may input a text script into the first artificial intelligence model (2030-1) trained for text summarization, and obtain text summary data output from the first artificial intelligence model (2030-1). In this case, the processor (2020) may apply information about key people and / or key keywords to the first artificial intelligence model (2030-1) together with the text script. The text script may be preprocessed into a format suitable for the first artificial intelligence model (2030-1) before being input to the first artificial intelligence model (2030-1). The first artificial intelligence model (2030-1) may extract important sentences from the text script, understand the meaning of the text script, and generate new sentences based on the extracted sentences to generate the text summary data. The first artificial intelligence model (2030-1) may be an artificial intelligence model based on natural language interpretation, and may include, for example, at least one of BERT (bidirectional encoder representations from transformers), GPT (generative pre-trained transformer), T5 (text-to-text transfer transformer), or BART (bidirectional and auto-regressive transformers).

[0228] According to one embodiment, the processor (2020) may select a first artificial intelligence model (2030-1) suitable for the type of text script from among a plurality of first artificial intelligence models (2030-1). In addition, the processor (2020) may request a summary of the text script from the selected first artificial intelligence model (2030-1).

[0229] According to one embodiment, the processor (2020) can obtain an image related to a person included in text summary data from the media data.

[0230] According to one embodiment, the processor (2020) can identify a person corresponding to a sentence included in the text summary data and obtain an image of the identified person. For example, when the text summary data is generated, an identification value of the person corresponding to the sentence in the text summary data may be tagged. In this case, the processor (2020) can identify the identification value of the person tagged to the sentence in the text summary data.

[0231] According to one embodiment, the processor (2020) may obtain images (e.g., still images and / or videos) related to a person included in the text summary data from media data including the user's conversation content and / or separate profile information. For example, an image including the person's face may be obtained from an image captured during the user's conversation. For example, an image of a person who uttered an utterance corresponding to a sentence in the text summary data may be extracted from video data captured for a video conference. Furthermore, for example, an image of a person who uttered an utterance corresponding to a sentence in the text summary data may be extracted from the profile information of users registered in an application that provides the user's conversation. Furthermore, for example, an image of a person who uttered an utterance corresponding to a sentence in the text summary data may be extracted from a separate storage device (e.g., another electronic device, a server) that stores the profile information of users participating in the conversation.

[0232] According to one embodiment, the processor (2020) may use the second artificial intelligence model (2030-2) to obtain text summary data and image summary data based on the obtained image.

[0233] According to one embodiment, the processor (2020) may generate image summary data reflecting the summary of the conversation based on the text summary data and the acquired images. The image summary data may include, for example, multiple still images reflecting the summary of the conversation. The image summary data may include, for example, at least one video reflecting the summary of the conversation.

[0234] In one embodiment, the processor (2020) may input text summary data and an image of a person to a second artificial intelligence model (2030-2) trained to generate or synthesize an image related to the content of the text based on the text or based on the text and the image. In one embodiment, the processor (2020) may obtain image summary data related to an image of a person corresponding to an utterance in the text summary data.

[0235] According to one embodiment, the processor (2020) may obtain summary data including text summary data and image summary data. According to one embodiment, the processor (2020) may display a GUI for displaying the summary data including the text summary data and the image summary data on the screen of the electronic device (1000). Depending on a user input through the GUI, the text summary data and / or the image summary data in the summary data may be displayed on the screen of the electronic device (1000).

[0236] FIG. 21 is a diagram illustrating a system including a generative artificial intelligence model according to one embodiment.

[0237] Referring to FIG. 21, the User Query / Response Interface (2310) can receive user input. The user input may be in the form of natural language, images, and / or videos. Additionally, context information may also be transmitted when the user input is transmitted. Context information may include various additional information at the time of user input. For example, information about the application currently being used by the user or information about the user's location. Furthermore, the user input may be in a mixed form of the aforementioned natural language, images, sounds, and context information. Furthermore, the user input may also be in a non-natural language form, such as selecting a menu. The User Query / Response Interface (2310) can output the results of a generative artificial intelligence system to the user. The output may be in the form of natural language or specific content, and may also be provided in the form of an action requested by the user. The User Query Interface can output the results of a generative artificial intelligence system to the user. The output may be in the form of natural language or specific content, and may also be provided in the form of an action requested by the user.

[0238] The AI ​​framework (2320) can receive user input and coordinate and control each component necessary to perform the user's intention based on the user's query.

[0239] User input received from the User Query / Response Interface (2310) can be transmitted to the Prompt design component (2321). The Prompt design component (2321) can be used to generate a prompt suitable for inputting the user input into a Large Language Model (LLM) or a large multimodal model (LMM). The Prompt design component (2321) can be an AI component that uses a machine learning algorithm or a neural network to develop better prompts over time. The Prompt design component (2321) can access a knowledge component (e.g., knowledge repositories (2340)) containing user preference data, a prompt library, and prompt examples based on the user input to generate a prompt, and transmit the generated prompt to the LLM or LMM.

[0240] The API / Plug-in management component (2323) can communicate with external information when there is a request for additional information when passing user input as input to a generative model. The API / Plug-in management component (2323) can establish a channel for communicating with the outside of the AI ​​Interface through the API, and can enable access to various data sources (e.g., knowledge repositories (2340)) through the established channel. In addition, if the API / Plug-in management component (2323) needs to perform an action that performs the user input as a final result rather than an intermediate result in an application or service, it can request the action to the application / service component (2330) through the API. Information obtained from an external source can be used to generate a prompt in the prompt design component (2321) together with the user input, or can be passed as input to the generative model.

[0241] The Refiner component (e.g., the output modification component (2325)) can fine-tune the output from a generative model. For example, the Refiner component can verify that the content generated by the LLM and / or LMM is not irrelevant, biased, or harmful. Furthermore, the Refiner component can determine the degree to which the output matches the user's desired result and, if necessary, perform additional processing. The Refiner component can also configure and provide hints to the user to avoid undesirable output.

[0242] Generative AI Model (2350) can generally refer to an artificial intelligence neural network that creates new types of data based on user input information. Generative AI Model (2350) can include an image-generating model and / or a language-generating model. Representative models for generating images include a generative adversarial network (GAN) and a variational autoencoder (VAE), and examples include a diffusion-based generative model that uses a VAE and a transformer structure. A language-generating model is a model trained to statistically output the most appropriate output value based on input values, and representative examples include models such as CHAT-GPT 3 and CHAT-GPT 4. In addition, there is also an LMM that can recognize various types of data input, such as text, images, and voice, and generate new data corresponding to them.

[0243] According to one embodiment, the electronic device (1000) of FIGS. 1 to 20 may be configured to include at least a portion of the User Query / Response Interface (2310), the AI ​​framework (2320), the application / service component (2330), the knowledge repositories (2340), or the Generative AI Model (2350) of FIG. 21. According to one embodiment, at least a portion of the User Query / Response Interface (2310), the AI ​​framework (2320), the application / service component (2330), the knowledge repositories (2340), or the Generative AI Model (2350) of FIG. 21 may be included in another electronic device (e.g., another user's electronic device and / or a server).

[0244] According to one embodiment, an electronic device (1000) that generates summary data related to a user's utterance includes: a display (2060); a memory (2030) that stores commands; And one or more processors (2020); wherein the instructions, when executed by the one or more processors, cause the electronic device to: execute an application that provides a conversation service between a plurality of users, obtain media data related to utterances from the plurality of users output through the application, obtain text scripts related to the utterances based on the media data, obtain text summary data from the text script using a first artificial intelligence model trained for text summarization, identify main keywords and main characters related to the text script, obtain an image of the main character corresponding to text in the text summary data from the media data, and obtain image summary data based on the text summary data and the obtained image using a second artificial intelligence model trained for generation of image summary data, wherein the image summary data includes a virtual image representing the main character generated by the second artificial intelligence model, and output summary data including the text summary data and the image summary data.

[0245] According to one embodiment, the instructions, when executed by the one or more processors, may cause the electronic device to: obtain the text script by converting voice data of the users input from the users into text data while the conversation service is provided.

[0246] According to one embodiment, the voice data may be extracted from video data output from the electronic device while the conversation service is being provided.

[0247] According to one embodiment, the instructions, when executed by the one or more processors, may cause the electronic device to: generate the text script based on text input from the users while the conversation service is being provided.

[0248] According to one embodiment, the first artificial intelligence model comprises a generative artificial intelligence model trained for summarizing the text script, and the instructions, when executed by the one or more processors, cause the electronic device to: generate a first input prompt based on the main keyword, the main character, and the text script, and apply the first input prompt to the first artificial intelligence model, thereby obtaining the text summary data.

[0249] According to one embodiment, the instructions, when executed by the one or more processors, may cause the electronic device to: identify a category of the text script, thereby selecting the first artificial intelligence model corresponding to the identified category from among a plurality of artificial intelligence models.

[0250] According to one embodiment, the instructions, when executed by the one or more processors, may cause the electronic device to: provide a graphical user interface (GUI) for setting the main keyword and the main person through an execution screen of the application, and identify the main keyword and the main person based on a user input to the GUI.

[0251] According to one embodiment, the second artificial intelligence model comprises a generative artificial intelligence model trained to generate the image summary data, and the instructions, when executed by the one or more processors, cause the electronic device to: generate a second input prompt based on the text summary data, the main keyword, the main person, and the acquired image, and apply the second input prompt to the second artificial intelligence model, thereby obtaining the image summary data.

[0252] According to one embodiment, the image summary data may include a virtual image representing the main character generated by the second artificial intelligence model based on an image of the main character obtained from the media data and text corresponding to the main character in the text summary data.

[0253] According to one embodiment, a computer-readable recording medium having recorded thereon a program for executing a method, the method comprising: executing an application that provides a conversation service between a plurality of users; obtaining media data related to utterances from the plurality of users output through the application; obtaining text scripts related to the utterances based on the media data; obtaining text summary data from the text script using a first artificial intelligence model trained for text summarization; identifying main keywords and main characters related to the text script; obtaining an image of the main character corresponding to text in the text summary data from the media data; obtaining image summary data based on the text summary data and the obtained image using a second artificial intelligence model trained for generating image summary data, wherein the image summary data includes a virtual image representing the main character generated by the second artificial intelligence model; and outputting summary data including the text summary data and the image summary data.

[0254] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0255] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0256] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0257] Various embodiments of the present document may be implemented as software (e.g., a program (1940)) including one or more instructions stored in a storage medium (e.g., an internal memory (1936) or an external memory (1938)) readable by a machine (e.g., an electronic device (1901)). For example, a processor (e.g., a processor (1920)) of the machine (e.g., an electronic device (1901)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0258] According to one embodiment, the method according to various embodiments disclosed in this document may be provided as a computer program product. The computer program product may be traded between sellers and buyers as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or may be provided through an application store (e.g., Play Store). TM ) or directly between two user devices (e.g., smart phones), online distribution (e.g., downloading or uploading). In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily created in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0259] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

In a method for generating summary data related to a user's speech, an electronic device An action that runs an application that provides a conversation service between multiple users; An operation of obtaining media data related to utterances from the plurality of users output through the application; An operation of obtaining a text script related to the utterances based on the above media data; An operation of obtaining text summary data from the text script using a first artificial intelligence model trained for text summarization; The action of identifying key words and key characters related to the above text script; An operation of obtaining an image of the main character corresponding to the text in the text summary data from the media data; An operation of obtaining image summary data based on the text summary data and the obtained image using a second artificial intelligence model trained to generate image summary data, wherein the image summary data includes a virtual image representing the main character generated by the second artificial intelligence model; and An operation of outputting summary data including the above text summary data and the above image summary data; A method comprising: In the first paragraph, The operation of obtaining text scripts related to the utterances from the above media data is as follows: A method comprising an action of obtaining the text script by converting voice data input from the users into text data while the conversation service is provided. In the first paragraph, The operation of obtaining text scripts related to the utterances from the above media data is as follows: A method for generating the text script based on text input from the users while the conversation service is provided. In the first paragraph, The first artificial intelligence model includes a generative artificial intelligence model trained to summarize the text script, The operation of obtaining the text summary data from the text script using the first artificial intelligence model is as follows: An action of generating a first input prompt based on the main keyword, the main character, and the text script; and An operation of obtaining the text summary data by applying the first input prompt to the first artificial intelligence model; A method comprising: In the first paragraph, An action of providing a GUI (graphical user interface) for setting the main keyword and the main character through the execution screen of the above application; Including more, A method wherein the operation of identifying the main keyword and the main person identifies the main keyword and the main person based on a user input to the GUI. In the first paragraph, The second artificial intelligence model includes a generative artificial intelligence model trained to generate the image summary data, The operation of obtaining image summary data based on the above text summary data and the above obtained image is as follows: An operation of generating a second input prompt based on the text summary data, the main keyword, the main character, and the acquired image; and An operation of obtaining the image summary data by applying the second input prompt to the second artificial intelligence model; A method comprising: In an electronic device that generates summary data related to a user's speech, display; memory that stores commands; and One or more processors; Includes, The above instructions, when executed by the one or more processors, cause the electronic device to: Running an application that provides a conversation service between multiple users, Obtain media data related to utterances from the plurality of users output through the above application, Obtaining text scripts related to the utterances based on the above media data, Using the first artificial intelligence model trained for text summarization, text summary data is obtained from the text script, Identify key keywords and key figures related to the above text script, Obtaining an image of the main character corresponding to the text in the text summary data from the above media data, Using a second artificial intelligence model trained to generate image summary data, image summary data based on the text summary data and the acquired image is obtained, wherein the image summary data includes a virtual image representing the main character generated by the second artificial intelligence model, An electronic device that outputs summary data including the above text summary data and the above image summary data. In paragraph 7, The above instructions, when executed by the one or more processors, cause the electronic device to: An electronic device that obtains the text script by converting voice data input from the users into text data while the conversation service is provided. In paragraph 8, An electronic device wherein the above voice data is extracted from video data output from the electronic device while the conversation service is being provided. In paragraph 7, The above instructions, when executed by the one or more processors, cause the electronic device to: An electronic device that generates the text script based on text input from the users while the conversation service is being provided. In paragraph 7, The first artificial intelligence model includes a generative artificial intelligence model trained to summarize the text script, The above instructions, when executed by the one or more processors, cause the electronic device to: Generate a first input prompt based on the above main keywords, the above main characters, and the above text script, An electronic device that obtains the text summary data by applying the first input prompt to the first artificial intelligence model. In Article 11, The above instructions, when executed by the one or more processors, cause the electronic device to: An electronic device that identifies a category of the text script and selects the first artificial intelligence model corresponding to the identified category from among a plurality of artificial intelligence models. In paragraph 7, The above instructions, when executed by the one or more processors, cause the electronic device to: Provides a GUI (graphical user interface) for setting the main keywords and main characters through the execution screen of the above application, An electronic device that identifies the main keyword and the main person based on user input to the GUI. In paragraph 7, The second artificial intelligence model includes a generative artificial intelligence model trained to generate the image summary data, The above instructions, when executed by the one or more processors, cause the electronic device to: Generate a second input prompt based on the above text summary data, the main keywords, the main characters, and the acquired images, An electronic device that obtains the image summary data by applying the second input prompt to the second artificial intelligence model. In Article 14, An electronic device, wherein the image summary data includes a virtual image representing the main character generated by the second artificial intelligence model based on an image of the main character obtained from the media data and text corresponding to the main character in the text summary data.

Images