Electronic device for displaying visual representation of document and its operation method

Abstract

Disclosed are an electronic device for displaying a visual representation of a document and an operation method thereof. The electronic device may comprise a memory including instructions. The electronic device may comprise a processor for executing the instructions. The instructions, when executed by the processor, may cause the electronic device to receive an input for generating a visual representation of a document. The instructions, when executed by the processor, may cause the electronic device to, if a condition for generation of the visual representation corresponds to a first condition, generate a first visual representation corresponding to the visual representation. The instructions, when executed by the processor, may cause the electronic device to display the first visual representation on a display module in association with the document. The instructions, when executed by the processor, may cause the electronic device to, if the condition for generation of the visual representation corresponds to a second condition, transmit a request for generation of a second visual representation corresponding to the visual representation to a server. The instructions, when executed by the processor, may cause the electronic device to display the second visual representation, which is received from the server in response to the request, on the display module in association with the document.

Description

Electronic device for displaying a visual representation of a document and method of operating the same

[0001] An electronic device for displaying a visual representation of a document and a method of operating the same are disclosed.

[0002] Electronic devices can provide various digital information (e.g., text content, images, etc.). The digital information can be provided from a server or other electronic device via a communication interface, or from a storage device functionally connected to the electronic device.

[0003] For example, generative artificial intelligence (AI) models can be used to generate diverse content. Content creation using generative AI can be primarily based on natural language processing and machine learning technologies. Efforts are underway to utilize these generative AI models in various fields to provide diverse user experiences.

[0004] The background technology described above is possessed or acquired during the process of deriving the present disclosure, and cannot necessarily be said to be a publicly known technology disclosed to the general public prior to the filing of the present disclosure.

[0005] An electronic device according to one embodiment of the present disclosure may include a memory including instructions. The electronic device may include a processor that executes the instructions. The instructions, when executed by the processor, may cause the electronic device to receive an input for generating a visual representation of a document. The instructions, when executed by the processor, may cause the electronic device to generate a first visual representation corresponding to the visual representation if a condition for generating the visual representation satisfies a first condition. The instructions, when executed by the processor, may cause the electronic device to display the first visual representation on a display module in association with the document. The instructions, when executed by the processor, may cause the electronic device to transmit a request for generating a second visual representation corresponding to the visual representation to a server if a condition for generating the visual representation satisfies a second condition. The instructions, when executed by the processor, may cause the electronic device to display the second visual representation received from the server in response to the request on the display module in association with the document. The first visual representation may include at least some of the characters included in the document. The second visual representation may include an indicator indicating a classification of the document.

[0006] An electronic device according to one embodiment of the present disclosure may include a memory including instructions. The electronic device may include a processor that executes the instructions. The instructions, when executed by the processor, may cause the electronic device to receive an input for generating a visual representation of a document. The instructions, when executed by the processor, may cause the electronic device to initiate generation of a first visual representation if the number of characters included in the document is less than or equal to a threshold value. The instructions, when executed by the processor, may cause the electronic device to initiate generation of a second visual representation if the number of characters included in the document exceeds the threshold value. The instructions, when executed by the processor, may cause the electronic device to display the first visual representation or the second visual representation on a display module in association with the document. The first visual representation may include at least some of the characters included in the document. The second visual representation may include at least one of a summary, a classification, and a background of the document generated based on at least some of the characters.

[0007] According to one embodiment of the present disclosure, a method of operating an electronic device may include receiving an input for generating a visual representation for a document. The method of operating the electronic device may include generating a first visual representation corresponding to the visual representation if a condition for generating the visual representation satisfies a first condition. The method of operating the electronic device may include displaying the first visual representation on a display module in association with the document. The method of operating the electronic device may include transmitting a request for generating a second visual representation corresponding to the visual representation to a server if the condition for generating the visual representation satisfies a second condition. The method of operating the electronic device may include displaying the second visual representation received from the server in response to the request on the display module in association with the document. The first visual representation may include at least a portion of characters included in the document. The second visual representation may include an indicator indicating a classification of the document.

[0008] An operating method of an electronic device according to one embodiment of the present disclosure may include an operation of receiving an input for generating a visual representation for a document. The operating method of the electronic device may include an operation of initiating generation of a first visual representation if the number of characters included in the document is less than or equal to a threshold value. The operating method of the electronic device may include an operation of initiating generation of a second visual representation if the number of characters included in the document exceeds the threshold value. The operating method of the electronic device may include an operation of displaying the first visual representation or the second visual representation on a display module in association with the document. The first visual representation may include at least some of the characters. The second visual representation may include at least one of a summary, a classification, and a background for the document generated based on at least some of the characters.

[0009] An operating method of an electronic device according to one embodiment of the present disclosure may include an operation of requesting the generation of a visual representation of a file based on information related to the size of content included in the file. The operating method may include an operation of displaying the visual representation, including a summary and classification of the content, in association with the file. The operating method may include an operation of providing the content based on a selection input regarding the visual representation received from a user.

[0010] A computer-readable recording medium according to one embodiment of the present disclosure may include one or more computer programs including commands for executing any one of the above-described methods.

[0011] FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments.

[0012] FIG. 2 is a drawing for explaining an example of a visual representation of a document according to one embodiment of the present disclosure.

[0013] FIG. 3 is a diagram illustrating the start of generation of a visual representation according to one embodiment of the present disclosure.

[0014] FIGS. 4 and 5 are diagrams illustrating a method for generating a first visual representation according to one embodiment of the present disclosure.

[0015] FIGS. 6 to 8 are diagrams illustrating a method for generating a second visual representation according to one embodiment of the present disclosure.

[0016] FIG. 9 is a drawing for explaining a second visual representation according to one embodiment of the present disclosure.

[0017] FIG. 10 is a drawing for explaining a second visual representation according to classification according to one embodiment of the present disclosure.

[0018] FIGS. 11 to 13 are drawings for explaining a second visual expression using multimedia content according to one embodiment of the present disclosure.

[0019] FIG. 14 is a drawing for explaining modification of a visual representation according to one embodiment of the present disclosure.

[0020] FIG. 15 and FIG. 16 are drawings for explaining the generation of a visual representation according to one embodiment of the present disclosure.

[0021] FIG. 17 is a diagram illustrating simultaneous generation of multiple visual representations according to one embodiment of the present disclosure.

[0022] FIG. 18 is a drawing for explaining editing of a second visual representation according to one embodiment of the present disclosure.

[0023] FIG. 19 is a diagram illustrating the generation of a representative visual representation according to one embodiment of the present disclosure.

[0024] FIG. 20 is a drawing for explaining an operation method of an electronic device according to one embodiment of the present disclosure.

[0025] FIG. 21 is a block diagram of a generative artificial intelligence system according to various embodiments.

[0026] Hereinafter, embodiments will be described in detail with reference to the attached drawings. In the description with reference to the attached drawings, identical components are assigned the same reference numerals regardless of the drawing numbers, and redundant descriptions thereof will be omitted.

[0027] FIG. 1 is a block diagram of an electronic device (101) within a network environment (100) according to various embodiments. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with the electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of the electronic device (104) or the server (108) via a second network (199) (e.g., a long-range wireless communication network). In one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0028] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (121). For example, when the electronic device (101) includes the main processor (121) and the auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a given function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0029] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0030] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0031] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0032] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0033] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0034] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. In one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0035] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0036] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0037] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0038] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0039] The haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0040] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0041] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0042] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0043] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0044] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0045] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas, for example, by the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device via the at least one selected antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0046] According to various embodiments, the antenna module (197) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0047] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0048] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0049] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0050] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0051] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0052] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0053] According to one embodiment, the method according to various embodiments disclosed in this document may be provided as a computer program product. The computer program product may be traded between sellers and buyers as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or may be provided through an application store (e.g., Play Store). TM ) or directly between two user devices (e.g., smart phones), online distribution (e.g., downloading or uploading). In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily created in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0054] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0055] An electronic device (e.g., the electronic device (101) of FIG. 1) may determine whether a document satisfies a first condition or a second condition upon receiving an input for generating a visual representation of the document. The electronic device may display different visual representations in relation to the document depending on whether the document satisfies the first condition or the second condition. The document may include various types of documents created through first-party applications or third-party applications, such as word, spreadsheet, slide show, e-mail, message, and memo, as well as note contents stored through a note application. According to one embodiment, the document may also include a chat room of a social network service (SNS) and a web page of a web browser. In the present disclosure, note contents may be used for convenience of explanation, but it will be apparent to those skilled in the art that the embodiments of note contents may also be applied to other types of documents described above.

[0056] In this disclosure, a visual representation may be an item intended to provide users with basic information within a document without executing the document. A visual representation may include concepts corresponding to a document cover, as well as a thumbnail, icon, window menu, or widget.

[0057] The embodiments described in this disclosure can be applied not only to content in the form of documents, but also to multimedia content corresponding to audio, video, or photos.

[0058] FIG. 2 is a drawing for explaining an example of a visual representation of a document according to one embodiment of the present disclosure.

[0059] Referring to FIG. 2, a basic visual representation (200) for a document is illustrated. The basic visual representation (200) is a visual representation that is generated by default for a document and may simply display some of the first contents of the document.

[0060] The electronic device can generate a first visual representation (210) when a document satisfies a first condition. The electronic device can display the first visual representation (210) on a display module (e.g., the display module (160) of FIG. 1) in association with the document. In other words, the electronic device can generate the first visual representation (210) in an on-device format without using a separate device (e.g., a server (e.g., the server (108) of FIG. 1)) when the document satisfies the first condition. A method for generating the first visual representation (210) and displaying it in association with the document will be further described with reference to FIGS. 4 and 5.

[0061] If a document satisfies the second condition, the electronic device may display a second visual representation (220) on the display module in association with the document. The second visual representation (220) may be a different type of visual representation than the first visual representation (210). For example, the first visual representation (210) and the second visual representation (220) may differ in the types of information they contain, the arrangement of the information they contain, the data used for generation, the devices used for generation, etc. If the document satisfies the second condition, the electronic device may request a separate device (e.g., a server) to generate the second visual representation (220). A method of requesting the generation of the second visual representation (220) and displaying it in association with the document will be further described with reference to FIGS. 6 to 8.

[0062] A server that has been requested to generate a second visual representation (220) may analyze a document using an artificial intelligence model (e.g., a generative AI model) to output at least one of the title, content summary, and classification of the document, and generate a second visual representation (220) based on at least one of the title, content, and classification of the document.

[0063] According to one embodiment, the second visual representation (220) may be generated using an artificial intelligence model (e.g., a generative AI model) included in the electronic device. For example, the first visual representation (210) may be generated by a first processor (e.g., a main processor (121)). For example, the second visual representation (220) may be generated by a second processor (e.g., a processor specialized in processing artificial intelligence models or a subprocessor (123)). According to one embodiment, one processor (e.g., the main processor (121) or the subprocessor (123)) may generate both different types of first visual representations (210) and second visual representations (220) depending on conditions.

[0064] Generative AI models can be trained to generate new content using input data or information. Generative AI models can operate based on technologies such as deep learning, machine learning, language models, and prompts.

[0065] Generative AI models can primarily be trained using deep learning algorithms. Generative AI models utilize deep learning to learn complex patterns in data or information and then generate new data based on these patterns.

[0066] Generative AI models can generate text using a language model. A language model can be trained to understand context and generate meaningful text.

[0067] A generative AI model can receive a prompt as input and generate data based on the prompt. The prompt may be a text input that includes guidance or instructions for the content the user wishes to generate. The prompt may provide guidance or instructions to the generative AI model to achieve a desired result, and the generative AI model can generate content based on the guidance or instructions. The prompt may include at least one of an instruction, context, input, output, and demonstration, thereby directing the generative AI model to perform a precise task. The instruction may explicitly instruct the generative AI model on the task to be performed. The prompt may provide the generative AI model with initial input or context necessary for performing the task. The prompt may provide a format or description of the output to be generated by the generative AI model.

[0068] According to one embodiment, the conditions for generating a visual representation may vary.

[0069] For example, the electronic device may determine whether to initiate generation of the first visual representation (210) or the second visual representation (220) based on the number of pages. For example, the electronic device may determine to initiate generation of the second visual representation (220) if the number of pages in the document is two or more, and may determine to initiate generation of the first visual representation (210) if the number of pages in the document is less than two.

[0070] For example, the electronic device may determine whether to initiate generation of a first visual representation (210) or a second visual representation (220) based on the number of characters included in the document. For example, the electronic device may set a condition to generate the first visual representation (210) if the number of characters included in the document is less than or equal to a threshold value (e.g., 100 characters), and to generate the second visual representation (220) if the number of characters included in the document exceeds the threshold value (e.g., 100 characters).

[0071] For example, the electronic device may determine whether to launch the first visual representation (210) or the second visual representation (220) based on the memory capacity occupied by the document. For example, if the document includes multimedia content, the electronic device may determine to launch the second visual representation (220).

[0072] For example, the electronic device may determine to initiate a first visual representation (210) or a second visual representation (220) based on the importance of the content contained in the document.

[0073] For example, the electronic device may determine whether to initiate the first visual representation (210) or the second visual representation (220) based on the communication status between the electronic device and the server or the load level of the electronic device. For example, even if the condition for generating the visual representation satisfies the second condition, if the communication status between the electronic device and the server is poor, the electronic device may determine to initiate the first visual representation (210). For example, even if the condition for generating the visual representation satisfies the first condition, if the load level of the electronic device is high, the electronic device may determine to initiate the second visual representation (220).

[0074] In one embodiment, the target of visual representation generation may be a file containing at least one of audio, video, and photos in addition to a document. For example, file size, playback time, whether it is a favorite, and the number of views may serve as conditions for generating the visual representation. Furthermore, if the target of visual representation generation is video or audio, the inclusion of a user's or a specific person's voice may serve as a condition for generating the visual representation. Furthermore, if the target of visual representation generation is video or a photo, the inclusion of a user's or a specific person's face may serve as a condition for generating the visual representation. For example, documents, audio, video, or photos may be stored as files in the memory of a server or electronic device.

[0075] Below, we describe how to receive input for generating a visual representation of a document.

[0076] FIG. 3 is a diagram illustrating the start of generation of a visual representation according to one embodiment of the present disclosure.

[0077] Referring to FIG. 3, a screen (300) is disclosed showing at least a portion of a list of documents displaying documents in a basic visual representation (e.g., the basic visual representation (200) of FIG. 2) and a screen (310) showing at least a portion of a document being executed.

[0078] An electronic device (e.g., electronic device (101) of FIG. 1) can analyze a document and determine whether the document satisfies a first condition or a second condition. If the document satisfies the second condition, the electronic device can display an icon (305) and / or an icon (315). In other words, on the screen (300), the electronic device can display an icon (305) along with a basic visual representation if the document satisfies the second condition. On the screen (310), the electronic device can generate an icon (315) to the right of the title of a document that satisfies the second condition if the document satisfies the second condition. The electronic device can receive input from a user for generating a visual representation (e.g., selection of an icon (305) or an icon (315)). The locations where the icons are generated as described above are merely examples, and the present disclosure is not limited thereto.

[0079] If the document satisfies the first condition, the electronic device may receive input for generating a visual representation in a method other than the icon (305) or the icon (315). If the document satisfies the first condition, the electronic device may receive input for generating a visual representation in various ways.

[0080] Below, we will describe how to generate the first visual representation when the document satisfies the first condition.

[0081] FIGS. 4 and 5 are diagrams illustrating a method for generating a first visual representation according to one embodiment of the present disclosure.

[0082] In the following embodiments, the operations may be performed sequentially, but are not necessarily performed sequentially. For example, the order of the operations may be changed, and at least two operations may be performed in parallel. Operations (410) to (430) may be performed by an electronic device (e.g., the electronic device (101) of FIG. 1). When instructions stored in a memory (e.g., the memory (130) of FIG. 1) of the electronic device are executed by a processor (e.g., the processor (120) of FIG. 1), the operations below may be performed by the electronic device. For example, the processor (e.g., the processor (120) of FIG. 1) of the electronic device may correspond to multiple processors that collectively perform multiple operations by dividing them among the processors.

[0083] In operation (410), the electronic device may receive input for generating a visual representation of a document.

[0084] In operation (420), the electronic device can generate a first visual representation corresponding to the visual representation if the condition for generating the visual representation satisfies the first condition.

[0085] The first condition may be whether the number of letters included in the document is less than or equal to a threshold (e.g., 100). For example, in the case of a compound character such as Korean, a letter may mean a single letter composed of a consonant and a vowel, and in the case of a Latin-based character such as English, it may mean a single alphabet or its corresponding unit. For example, the condition for generating a visual representation for a document may be the number of words, phrases, clauses, sentences, or pages in addition to the number of letters. For example, the condition for generating a visual representation may include conditions related to the source, provider, generation device, generation time, or storage location of the document. If the condition for generating a visual representation satisfies the first condition, the electronic device may generate a first visual representation corresponding to the visual representation on its own (e.g., the first visual representation (210) of FIG. 2). In other words, the electronic device may generate the first visual representation in an on-device format without requesting generation of the first visual representation from a separate device.

[0086] In operation (430), the electronic device may display a first visual representation in association with a document on a display module (e.g., display module (160) of FIG. 1).

[0087] The electronic device may display a preview before applying the first visual representation.

[0088] Referring to FIG. 5 , an interface for describing the generation of an exemplary first visual representation of the present disclosure is illustrated. Referring to FIG. 5 , a screen (500) of an electronic device (e.g., the electronic device (101) of FIG. 1 ) is illustrated, showing at least a portion of a list of documents displaying documents as a basic visual representation (501) (e.g., the basic visual representation (200) of FIG. 2 ). The description of the generation of the first visual representation (520) described below may not necessarily be applicable when the screen (500) displays the basic visual representation (501). For example, it will be apparent to those skilled in the art that the description of the generation of the first visual representation (520) described below may also be applicable when the screen (500) displays titles, file names, or icons of documents.

[0089] An electronic device can receive input for generating a visual representation of a document in various ways. For example, if a user input (505) for a basic visual representation (501) (i.e., selection of an arbitrary area within the basic visual representation (501)) exceeds a threshold time (i.e., long press), the electronic device can provide a pop-up window (510) providing various options, and receive input for generating a visual representation through a user input for the pop-up window (510) (e.g., selection of a cover creation button (513)). For example, if the electronic device receives a user input for the settings button (503), the electronic device can access a cover creation menu in response thereto. The user input for generating the first visual representation (520) described above can be equally applied when generating a second visual representation.

[0090] If a condition for generating a visual representation satisfies a first condition, the electronic device can generate a first visual representation (520) corresponding to the visual representation (e.g., the first visual representation (210) of FIG. 2). The electronic device can arbitrarily select a background of the first visual representation (520). The electronic device can add at least some of the sentences of the document to the first visual representation (520). For example, the electronic device can display the first sentence among the characters of the document on the first visual representation (520). As an example, the electronic device can display characters corresponding to a preset maximum number of characters (e.g., 20 characters) or less in the first sentence of the document on the first visual representation (520).

[0091] The electronic device may display a first visual representation (520) in association with a document. The electronic device may generate the first visual representation (520) and display a preview of the first visual representation (520) in association with the document. For example, the electronic device may display a preview of the first visual representation (520) and inquire whether to replace the default visual representation (501) with the first visual representation (520).

[0092] When the electronic device receives a user input (523) (e.g., selecting a Done button) to replace the default visual representation (501) with a first visual representation (520), the electronic device may display a document as the first visual representation (520). For example, the electronic device may display the document as the first visual representation (520) on a screen (530) that displays at least a portion of a list of documents. When the electronic device receives a user input for the first visual representation (520), the electronic device may execute a document associated with the first visual representation (520). When the electronic device receives a user input for the first visual representation (520), the electronic device may display the document associated with the first visual representation (520) on a display module (e.g., the display module (160) of FIG. 1).

[0093] Although not shown in FIG. 5, in one embodiment, when the electronic device receives user input requesting a modification to the first visual representation (520), the electronic device may provide the user with an interface that provides the user with a modification to the first visual representation (520).

[0094] Here, we will describe a method for generating a second visual representation (e.g., the second visual representation (220) of FIG. 2) when a document satisfies the second condition.

[0095] FIGS. 6 to 8 are diagrams illustrating a method for generating a second visual representation according to one embodiment of the present disclosure.

[0096] In the following embodiments, the operations may be performed sequentially, but are not necessarily performed sequentially. For example, the order of the operations may be changed, and at least two operations may be performed in parallel. When instructions stored in the memory of the electronic device (e.g., the memory (130) of FIG. 1) are executed by the processor (e.g., the processor (120) of FIG. 1), operations (610) to (630) and operation (650) may be performed by the electronic device (601) (e.g., the electronic device (101) of FIG. 1).

[0097] In operation (610), the electronic device (601) may receive input for generating a visual representation of a document. The method by which the electronic device (601) receives input for generating a visual representation of a document has been described above with reference to FIG. 3 and will therefore be omitted. Furthermore, the method by which the electronic device (601) receives user input for generating a visual representation of a document has been described above with reference to FIG. 5 and will therefore be omitted.

[0098] In operation (620), the electronic device (601) can extract feature points from the document if the conditions for generating a visual representation meet the second condition.

[0099] A second condition may be whether the number of characters contained in the document exceeds a threshold (e.g., 100). In other words, if the number of characters contained in the document is 101 or more, the second condition may be met for generating a visual representation.

[0100] The electronic device (601) may extract feature points from a document if the conditions for generating a visual representation meet the second condition. According to one embodiment, the electronic device (601) may generate a cover using only the contents included in the document without extracting feature points if the conditions for generating a visual representation meet the second condition.

[0101] According to one embodiment, a document may include various indicators (or identifiers, marks), such as asterisks, underlines, boldface, highlights, and special characters. The document may include keywords and / or tags of the document. In other words, the document may include at least some of the indicators, keywords, and tags that the author of the document intends to further emphasize the contents of the document. The electronic device (601) may extract features based on at least some of the indicators, keywords, and tags in order to use them as input data for an artificial intelligence model (e.g., a generative AI model). The electronic device (601) may determine whether the document includes at least some of the indicators, keywords, and tags. The electronic device may extract features by determining a higher weight for characters indicated by at least some of the indicators, keywords, and tags than for characters that do not.

[0102] According to one embodiment, if the document does not include at least some of the indicators, keywords, and tags, the electronic device can extract features from the document alone without setting separate weights for specific characters.

[0103] The electronic device (601) can extract features from a document in various ways. For example, when a document is input, the electronic device (601) can extract features using an artificial intelligence model trained to extract features that are characteristic of the document. For example, the document can be transmitted to the artificial intelligence model in the form of a file. The identification of the features in the document can also be performed by the artificial intelligence model used to generate the second visual representation. However, this is merely an example, and the present disclosure is not limited thereto.

[0104] In operation (630), the electronic device (601) may generate a prompt requesting the server to generate a second visual representation (e.g., the second visual representation (220) of FIG. 2) based on the feature points. The prompt will be described later in FIG. 7.

[0105] The electronic device (601) can communicate with a server (608) (e.g., server (108) of FIG. 1) via a network (e.g., second network (199) of FIG. 1). The electronic device (601) can transmit a prompt to the server (608) via the network. In one embodiment, the electronic device (601) can transmit the prompt to an artificial intelligence model included in the electronic device (601). In one embodiment, the electronic device (601) can transmit other content (e.g., an image to be used as a background, reference information related to the document (e.g., another document, a web page address, an email)) to another device (e.g., the server of FIG. 1) or the artificial intelligence model within the electronic device, together with or separately from the content of the document, to be used in generating the second visual representation.

[0106] According to one embodiment, the electronic device may transmit to the server (608) a prompt containing at least some of the characters included in the document and indicator information. At least some of the characters may be the first 500 characters of the document. However, this is merely an example and the present disclosure is not limited thereto. The indicator information may be information indicating characters emphasized using various indicators, such as asterisks, underlines, boldface, highlights, and special characters.

[0107] In one embodiment, an electronic device may request the server to generate a second visual representation of a document and simultaneously request modification of the document. For example, the electronic device may request the server to correct typos, duplicate expressions, paragraph breaks, formatting, etc. contained in the document. The electronic device may request the server to process the generation of the second visual representation and the modification of the document in parallel or sequentially. Upon receiving the document request, the server may utilize an artificial intelligence model to perform the modification of the document. For example, if there is an incomplete sentence, the server may utilize the artificial intelligence model to complete the sentence. For example, if it is determined that adding an image is appropriate, the server may generate the image based on a prompt and add it to the document.

[0108] In operation (640), the server (608) may generate a second visual representation based on the prompt. The server (608) may generate the second visual representation using the generative AI model described above in FIG. 2 based on the prompt.

[0109] In one embodiment, the server (608) may receive a prompt including at least some characters included in a document and indicator information. The server (608) may identify the significance of the content included in at least some of the characters through an artificial intelligence model and generate a second visual representation including a summary generated based on the indicator information. In this case, the artificial intelligence model may be a model trained to generate the second visual representation when inputting characters and indicator information.

[0110] The server (608) can transmit the second visual representation to the electronic device (601) over a network.

[0111] In operation (650), the electronic device (601) may display a second visual representation in association with the document on a display module (e.g., the display module (160) of FIG. 1).

[0112] The electronic device (601) may display a preview of the document before applying the second visual representation to the document.

[0113] According to one embodiment, the electronic device (601) can group documents determined to have similar categories and titles and provide a single visual representation. For example, if lecture notes are written in multiple notes, the electronic device (601) can provide a guide to the user that determines the similarity of categories and titles between the notes and group them into a single visual representation. Through the guide, the electronic device (601) can merge (i.e., group) multiple notes into a single note and provide a visual representation of the merged note.

[0114] According to one embodiment, the electronic device (601) may provide a single representative visual representation for documents determined to have similar classifications and titles. For example, if lecture notes are divided into multiple notes, the electronic device (601) may determine the similarity in classifications and titles between the notes and provide a single representative visual representation to the user. Through this single representative visual representation, the user can access documents with similar classifications and titles.

[0115] In one embodiment, upon receiving an input for generating a visual representation for a specific document, the electronic device (601) may request generation of a second visual representation by utilizing documents similar or related to the document. For example, the electronic device (601) may utilize documents related to the document, such as the time of creation, location of creation, similarity of content, or connectivity of content. For example, 200 characters of document 1 and 200 characters of document 2, which are related to document 1, may be transmitted together to a server, and the server may generate a common visual representation for document 1 and document 2. In this case, the common visual representation may be applied to document 1 and document 2.

[0116] Additionally, according to one embodiment, the electronic device (601) may request the server to generate a second visual representation using documents related to a specific document even if the specific document satisfies the first condition. For example, even if document 1 satisfies the first condition with 90 characters, the electronic device (601) may determine that document 1 satisfies the second condition by adding 50 characters of document 2, which is related to document 1, to document 1. The electronic device (601) may request the server to generate a second visual representation using characters extracted from document 1 and document 2.

[0117] Additionally, according to one embodiment, the electronic device (601) may request the server to generate a visual representation using the contents of files (e.g., image files, video files, audio files) as well as documents associated with a particular document.

[0118] Referring to FIG. 7, an exemplary prompt (700) of the present disclosure is illustrated. Referring to FIG. 7, the prompt may include a description (710), an example sentence (720), and an input sentence (730).

[0119] A prompt may be a command requesting the server to obtain a desired second visual representation (e.g., the second visual representation (220) of FIG. 2), including a description (710), an example (720), and an input (730).

[0120] An electronic device (e.g., the electronic device (101) of FIG. 1 and the electronic device (601) of FIG. 6) may provide a description of a second visual representation desired to be generated via a description (710). For example, the electronic device may clearly and explicitly provide characteristics of the second visual representation desired to be generated via the description (710).

[0121] The electronic device can explain the conditions for generating a title through the description (710). For example, the electronic device can explain through the description (710) that a title should be generated accurately and concisely as a topic that represents all contents of the input text. The electronic device can explain through the description (710) that a condition for generating a summary should be generated. For example, the electronic device can explain through the description (710) that a summary should be generated concisely, including at least one sentence and all contents of the input text. The electronic device can explain through the description (710) that a condition for determining a classification of a document should be explained. For example, the electronic device can explain through the description (710) that a classification of a document should be determined from a classification list based on at least some of the characters included in the document (i.e., the input text (730)). The categorized list may include to-do items, meetings, lecture notes, information, shopping list, schedule, travel, diary, fitness, account book, cooking recipe, and newspaper. However, these are examples only and the present disclosure is not limited thereto.

[0122] The example sentence (720) may include a description of how the second visual representation is constructed. The example sentence (720) may include a description of how the second visual representation is constructed to generate the second visual representation. In other words, the example sentence (720) may include an output example of how the second visual representation is constructed. The example sentence (720) may include an input example to explain which input data is inputted and the output example is output. An AI model in a server (e.g., server (108) of FIG. 1 and server (608) of FIG. 6) may refer to the example sentence (720) to more accurately generate the second visual representation. The example included in the example sentence (720) may include one or two or more examples.

[0123] The input text (730) may include at least some characters included in the document to obtain a second visual representation. An AI model on the server may generate a second visual representation by inputting at least some characters described in the input text (730).

[0124] In one embodiment, the maximum number of characters transmitted to the server may be determined to be less than or equal to a threshold value (e.g., 500 characters). The electronic device may extract characters included in the document up to the threshold value and add them to the input text (730) to transmit them to the server. The electronic device may extract characters up to the threshold value, centered on a portion selected by the user or determined by the electronic device to be the main content.

[0125] According to one embodiment, a document may include characters in one or more formats. For example, the document may include at least one of a typed format, an attached format (e.g., a PDF file and an image file), and a handwritten format. The typed format may refer to a format typed using an input device such as a keyboard or a touch keyboard on a touchscreen. The attached format may refer to a format in which a heterogeneous file format, such as a PDF file, is pasted into the document. The handwritten format may be a format handwritten by a user, for example, using a finger or a touch pen.

[0126] In cases such as the above, where a document is composed of at least two types of formats, including typed, attached, and handwritten, the electronic device may select the characters to be transmitted to the server based on a predefined priority order. The predefined priority order may be typed, attached, and handwritten.

[0127] For example, if a document contains 500 characters in typed form, 500 characters in attached form (i.e., PDF format), and 100 characters in handwritten form, the electronic device can only transmit the typed form to the server (assuming that the maximum number of characters that can be transmitted to the server is 500). As another example, if a document contains 200 characters in typed form, 500 characters in PDF format, and 300 characters in handwritten form, the electronic device can transmit 200 characters in text format and 300 characters in PDF format to the server (assuming that the maximum number of characters that can be transmitted to the server is 500).

[0128] Additionally, for example, if a document contains 200 characters in typed format, 500 characters in PDF format, and 300 characters in handwritten format, the electronic device can check whether the number of characters in typed format is greater than 500 (assuming that the maximum number of characters transmitted to the server is 500). If the number of characters in typed format is less than 500, the electronic device can check whether the number of characters in PDF format is greater than 500. If the number of characters in PDF format is greater than 500, the electronic device can extract 500 characters from the PDF format and transmit them to the server.

[0129] In the embodiments described above, if the document includes characters or image files in handwritten format, the electronic device can convert the handwritten format or image files into typed format through a character recognition function, determine the number of characters, and transmit them to the server.

[0130] In the embodiments described above, characters can be extracted based on feature points.

[0131] According to one embodiment, the electronic device may further add a command (735) to the input statement (730) that causes the electronic device to generate components (e.g., title, summary, and classification) of the second visual representation in a specific language. For example, the electronic device may convert {source_language} in the command statement (735) into a language set by the electronic device to generate components of the second visual representation in the language set by the electronic device. For example, the electronic device may generate components of the second visual representation by directly inputting a generation language, such as "Classification must be English." For example, the command statement (735) may specify which language (e.g., Korean, English) to generate for each of the title, summary, and classification included in the second visual representation. For example, the type of language included in the second visual representation may also be specified by an artificial intelligence model. For example, the above command (735) may specify what type of language style (e.g., polite, conversational, regional dialect, phrase, clause, sentence) to generate for the title, summary, or classification included in the second visual representation.

[0132] In one embodiment, the electronic device may indicate to the server that the content described in the description (710) is important by reiterating it in the input text (730). This may enhance the accuracy of the generated second visual representation.

[0133] Referring to FIG. 8, an interface is illustrated for describing the generation of an exemplary second visual representation (820) of the present disclosure (e.g., the second visual representation (220) of FIG. 2). Referring to FIG. 8, a screen (800) of an electronic device (e.g., the electronic device (101) of FIG. 1 and the electronic device (601) of FIG. 6) is illustrated showing at least a portion of a list of documents being displayed as a basic visual representation (801) (e.g., the basic visual representation (200) of FIG. 2 and the basic visual representation (501) of FIG. 5).

[0134] Electronic devices can receive input for creating a visual representation of a document in various ways. For example, the electronic device can receive input for creating a visual representation through the icon (305) or icon (315) of FIG. 3. However, for the sake of explanation, in FIG. 8, it is assumed that input for creating a visual representation is received through user input (505) or a settings button (503), as in FIG. 5.

[0135] The electronic device may receive input for generating a visual representation, and if a condition for generating the visual representation satisfies a second condition, the electronic device may generate a prompt (e.g., prompt (700) of FIG. 7) requesting generation of a second visual representation (820). The electronic device may transmit the prompt to a server (e.g., server (108) of FIG. 1 and server (608) of FIG. 6).

[0136] The electronic device may display a screen (810) including an indicator (811) indicating that a second visual representation (820) is being generated and a descriptive text while receiving the second visual representation (820) from the server in response to transmitting the prompt. A description of the second visual representation (820) will be described later in FIG. 9.

[0137] The electronic device may display a second visual representation (820) in association with a document. The electronic device may display a preview of the second visual representation (820) in association with the document. For example, the electronic device may display a preview of the second visual representation (820) and inquire whether to replace the default visual representation (801) with the second visual representation (820).

[0138] The electronic device may display a document in the second visual representation (820) upon receiving user input (823) (e.g., selecting a Done button) to replace the primary visual representation (801) with a second visual representation (820). For example, the electronic device may display the document in the second visual representation (820) on a screen (830) displaying at least a portion of a list of documents.

[0139] In one embodiment, the first or second visual representation may be displayed to reflect the user's preferences. For example, a favorite icon for the first or second visual representation may be displayed on or adjacent to the corresponding visual representation. For example, the first or second visual representation may be displayed in a different size than other visual representations on the screen based on conditions (e.g., user preferences, importance of content contained in the document, sharing history, viewing time, or number of views).

[0140] FIG. 9 is a drawing for explaining a second visual representation according to one embodiment of the present disclosure.

[0141] Referring to FIG. 9, a second visual representation (900) (e.g., the second visual representation (220) of FIG. 2 and the second visual representation (820) of FIG. 8) is illustrated.

[0142] The second visual representation (900) may be different from the first visual representation (e.g., the first visual representation (210) of FIG. 2 and the first visual representation (520) of FIG. 5) generated by an electronic device (e.g., the electronic device (101) of FIG. 1 and the electronic device (601) of FIG. 6) without using a server (e.g., the server (108) of FIG. 1 and the server (608) of FIG. 6).

[0143] The first visual representation may include a randomly selected background and at least some of the sentences of the document. However, the second visual representation (900) may include at least one of a title (910), a summary (920), a background (930), and an icon (940). The title (910), the summary (920), and the background (930) may be generated by an artificial intelligence model (e.g., a generative AI model) on a server (e.g., the server (108) of FIG. 1 and the server (608) of FIG. 6) based on a prompt (e.g., the prompt (700) of FIG. 7). The background (930) may include a background color and a background style (e.g., a design including the presence or absence of a pattern, the type of pattern, or the arrangement of components). As an example, the title (910), the summary (920), or the background (930) of the second visual representation may include a moving graphic (e.g., a text animation, a video). For example, if the document involves moving images, or if the document falls into a specific category (e.g., travel, fitness), the secondary visual representation may include moving graphics as a background.

[0144] A document can be classified into one of several categories pre-determined by the server's artificial intelligence model. These categories may include "to do," "meeting," "lecture notes," "information," "shopping list," "schedule," "travel," "diary," "fitness," "account book," "cooking recipe," and "newspaper." However, these are merely examples and the present disclosure is not limited thereto.

[0145] Each category can be matched with an icon corresponding to that category. In other words, the icon (940) can represent the category of the document determined based on the prompt. The icon (940) can be matched with the category of the document determined based on the prompt. For example, if the category of the document is lecture notes, an icon (940) matching the lecture notes can be displayed.

[0146] According to one embodiment, the classification for the document may be indicated on or adjacent to the second visual representation (900) by an indication or indicator including at least one of an icon (940) and a character.

[0147] In one embodiment, a classification for a document may function as a button that receives user input. When a user selects a classification, a user interface may be provided that allows the user to view documents with the same classification. In one embodiment, when a user selects a classification, a user interface may be provided that allows the user to change the classification.

[0148] The server's artificial intelligence model can generate a title (910) based on the summary (920) and classification of the document. For example, if the summary (920) includes content about the Precambrian Era and the classification is determined as lecture notes, the server's artificial intelligence model can determine the title as "Precambrian Era Lecture Notes." However, according to one embodiment, if sentences or phrases determined to be titles already exist within the document (i.e., the input text (e.g., the input text (730) of FIG. 7), these may be determined as the title (910). For example, the title of the document may be generated using characters included in the document.

[0149] According to one embodiment, the second visual representation (900) may further include buttons for playing and / or viewing files and / or multimedia content contained in the document. According to one embodiment, the second visual representation (900) may further include information related to the document's sharing history, number of views, number of pages, whether it has been designated as a favorite, preferences, importance, security settings, the last page viewed, the last date viewed, etc.

[0150] The background (930) may be determined to a different color depending on the classification. For example, if the classification for a document is "Travel," the background of the second visual representation generated for that document may be determined to be blue. For example, if the classification for a document is "Recipe," the background of the second visual representation generated for that document may be determined to be green. Alternatively, the background (930) may be determined to a color determined by an artificial intelligence model based on the summary and classification.

[0151] FIG. 10 is a drawing for explaining a second visual representation according to classification according to one embodiment of the present disclosure.

[0152] Depending on the classification of the document, the layout of the second visual representation (e.g., the second visual representation (220) of FIG. 2, the second visual representation (820) of FIG. 8, and the second visual representation (900) of FIG. 9) may be different.

[0153] For example, if the server's artificial intelligence model (e.g., a generative AI model) determines that the document is classified as a travel document, a second visual representation (1000) may be generated that includes a title at the top, a summary at the bottom, and an icon between the top and bottom. For example, if the server's artificial intelligence model determines that the document is classified as a diary, a second visual representation (1010) may be generated that includes only a summary, without an icon or title.

[0154] Below, we will explain cases where multimedia content is included in a document.

[0155] FIGS. 11 to 13 are drawings for explaining a second visual expression using multimedia content according to one embodiment of the present disclosure.

[0156] Referring to FIG. 11, a flowchart is illustrated to explain a method of generating a background of a second visual representation (e.g., background (930) of FIG. 9) by using multimedia content when it exists in a document.

[0157] In the following embodiments, the operations may be performed sequentially, but are not necessarily performed sequentially. For example, the order of the operations may be changed, and at least two operations may be performed in parallel. Operations (1110) to (1140) may be performed by an electronic device (e.g., the electronic device (101) of FIG. 1 and the electronic device (601) of FIG. 6). When instructions stored in a memory of the electronic device (e.g., the memory (130) of FIG. 1) are executed by a processor (e.g., the processor (120) of FIG. 1), the following operations may be performed by the electronic device.

[0158] In operation (1110), the electronic device may receive input for generating a visual representation of a document. The description of receiving input for generating a visual representation is omitted as described above.

[0159] In operation (1120), the electronic device may determine whether the document includes multimedia content. The multimedia content may include video, photos, audio, and other content other than text. The electronic device may use the multimedia content to generate at least one of a title, summary, and classification.

[0160] In operation (1130), the electronic device can determine whether the multimedia content can be used to create a visual representation.

[0161] According to one embodiment, an electronic device may generate a visual representation of operations (1110) to (1130) based on multimedia content associated with a document. The multimedia content associated with a document may be content that is not included in the document but is transmitted along with the document. For example, the multimedia content associated with a document may be an image included in another document. For example, the multimedia content associated with a document may be content that is accessible to the electronic device through a web browser and is related to at least one of various attributes of the document, such as the subject matter of the document.

[0162] Electronic devices can determine whether multimedia content can be used as a background for visual presentations. At this time, the electronic device can determine whether the format of the multimedia content is supported. For example, if the format of the multimedia content is GIF, the electronic device can determine that the multimedia content cannot be used to create a visual presentation.

[0163] The electronic device may generate a prompt (e.g., prompt (700) of FIG. 7) containing information about the multimedia content, if the multimedia content can be used to create a visual representation.

[0164] An artificial intelligence model (e.g., a generative AI model) of a server (e.g., a server (108) of FIG. 1 and a server (608) of FIG. 6) can generate a second visual representation (e.g., a second visual representation (220) of FIG. 2, a second visual representation (820) of FIG. 8, a second visual representation (900) of FIG. 9, a second visual representation (1000) of FIG. 10, and a second visual representation (1010) of FIG. 10) based on a prompt. The artificial intelligence model of the server can generate a second visual representation including a background generated based on multimedia content using the prompt. For example, a conversation recognized from voice recording content can be reflected in a title, summary, or classification. For example, an object (e.g., a person, an animal, an object, a place), a genre, an artist, a user preference, etc. recognized from video, photo, or music content can be reflected in a title, summary, or classification. For example, the content of metadata included in the multimedia content can be reflected in a title, summary, or classification.

[0165] In operation (1140), the electronic device may display a second visual representation including a background generated based on multimedia content on a display module (e.g., display module (160) of FIG. 1) in association with a document. That is, the electronic device may display the second visual representation as a preview.

[0166] Referring to FIG. 12, a second visual representation (1210) generated based on an image (1200) included in or associated with a document according to one embodiment of the present disclosure (e.g., the second visual representation (220) of FIG. 2 , the second visual representation (820) of FIG. 8 , the second visual representation (900) of FIG. 9 , the second visual representation (1000) of FIG. 10 , and the second visual representation (1010) of FIG. 10 ) is illustrated. An image associated with a document may be an image that is not included in the document but is transmitted along with the document. For example, an image associated with a document may be an image included in another document.

[0167] The second visual representation (1210) may include a background (e.g., background (930) of FIG. 9) generated based on the image (1200).

[0168] According to one embodiment, the background may be a cropped or resized image of the image (1200). According to one embodiment, the background may include at least one of the objects included in the image (1200). To this end, a model for extracting objects from the image (1200) may be used. According to one embodiment, the background may include a color determined as a representative color of the image (1200) (e.g., a color included in the image at the largest proportion, a color mixed with at least a portion of the colors included in the image, a color of a main object of the image (e.g., a person, an animal)) as a background color.

[0169] Referring to FIG. 13, a second visual representation (1310) generated based on an image (1300) included in a document or transmitted together with a document according to one embodiment of the present disclosure (e.g., the second visual representation (220) of FIG. 2, the second visual representation (820) of FIG. 8, the second visual representation (900) of FIG. 9, the second visual representation (1000) of FIG. 10, the second visual representation (1010) of FIG. 10, and the second visual representation (1210) of FIG. 12) is illustrated.

[0170] An electronic device (e.g., the electronic device (101) of FIG. 1 and the electronic device (601) of FIG. 6) may select a representative image from among multiple images included in a document. For example, the electronic device may select the first image among the multiple images as the representative image. For example, the electronic device may select an image among the multiple images that best represents the content of the document (e.g., an image that represents content included in a topic, summary, etc., or an image displayed or attached adjacent to content included in a topic, summary, etc.) as the representative image.

[0171] In one embodiment, the image selected as the representative image of the document may be resized or cropped and applied to at least a portion of the background of the second visual representation (1310). For example, at least a portion of the representative image of the document may be displayed as the background at a location where the title of the document (e.g., "Trip to Hawaii") is displayed. For example, the location of the title may be determined so as not to obscure a main object included in the representative image.

[0172] In one embodiment, at least a portion of the background of the second visual representation (1310) (e.g., an area where a summary of a document is displayed) may include a representative color of the image (1300) (e.g., a color included in the most area, a color of an object related to the subject, etc.). Alternatively, in one embodiment, the background of the second visual representation (1310) may be an image (e.g., a still image, a moving image) generated by an artificial intelligence model based on the image (1300) (i.e., multimedia content), summary, and classification.

[0173] Below we will explain the modifications to the generated visual representation.

[0174]

[0175] FIG. 14 is a drawing for explaining modification of a visual representation according to one embodiment of the present disclosure.

[0176] It will be apparent to those skilled in the art that the following description can equally apply to the first visual representation. However, for convenience of explanation, the second visual representation will be used as the basis for the description.

[0177] Referring to FIG. 14, a preview of a second visual representation (1400) generated by an artificial intelligence model is illustrated. An electronic device (e.g., the electronic device (101) of FIG. 1 and the electronic device (601) of FIG. 6) may provide a preview of the second visual representation (1400) upon receiving the second visual representation (1400) from a server as described in FIG. 8. The electronic device may provide an edit button together with the preview of the second visual representation (1400). Upon receiving a user input (1401) (e.g., selecting the edit button) to provide an interface for editing the second visual representation (1400), the electronic device may provide the user with an interface (1410) for editing.

[0178] The interface (1410) may provide a second visual representation (1400), a list (1413) providing selections for categories, and a list (1415) providing selections for styles. The list (1415) providing selections for styles may include a list (1417) for selections of background colors and a list (1419) for selections of background styles. In the list (1419), the background style may be determined based on the classification.

[0179] The icon (1403) displayed first when generating the second visual representation (1400) may be an icon matching the classification determined to have the highest similarity. In other words, the icon (1403) may represent a classification for a document. The sorting order of icons corresponding to classifications included in the list (1413) may be provided according to the similarity order for each classification provided by an artificial intelligence model (e.g., a generative AI model) when generating the second visual representation (1400). For example, if the similarity of lecture notes is the highest at 0.4 and the similarity of information is the second highest at 0.3, the icon for lecture notes may be placed first in the list (1413) and the icon for information may be placed next.

[0180] When a selection is input for any one of the icons displayed in the list (1413), the electronic device may change the icon (1403) included in the second visual representation (1400) to the icon selected from the list (1413) and display it. According to one embodiment, when a selection is input for any one of the icons displayed in the list (1413), the electronic device may request a server (e.g., server (108) of FIG. 1 and server (608) of FIG. 6) to regenerate the second visual representation (1400) in a type corresponding to the selected icon. The server may regenerate the second visual representation in response to the request and transmit it to the electronic device. The electronic device may display the regenerated second visual representation on the interface (1410).

[0181] According to one embodiment, when displaying the interface (1410), the electronic device may request the server to generate second visual representations (1430) in advance for categories having a similarity level greater than or equal to a threshold similarity. Alternatively, when displaying the interface (1410), the electronic device may request the server to generate second visual representations (1430) in advance for a specific number of categories having a high similarity level. The server may, upon request, generate second visual representations (1430) in advance for categories having a similarity level greater than or equal to a threshold similarity or for a specific number of categories having a high similarity level and provide the second visual representations (1430) to the electronic device. By securing the second visual representations (1430) in advance, when a selection is input for any one of the icons displayed in the list (1413), the electronic device may immediately display the second visual representation corresponding to the selected icon without delay.

[0182] FIG. 15 and FIG. 16 are drawings for explaining the generation of a visual representation according to one embodiment of the present disclosure.

[0183] According to one embodiment, an electronic device (e.g., electronic device (101) of FIG. 1 and electronic device (601) of FIG. 6) may provide a separate interface (1513) that provides for generation of a visual representation when a condition for generation of a visual representation satisfies a first condition.

[0184] An electronic device may display an edit icon (1501) on a screen (1500) when a document satisfies the first condition. The screen (1500) may be an editing screen for a title of a document. However, the screen (1500) is merely an example, and the present disclosure is not limited thereto. For example, the screen (1500) may be an editing screen for a document, and the electronic device may provide a separate interface (1513) that provides generation of a visual representation when receiving a specific input through the editing screen for the document.

[0185] The electronic device may provide a separate interface (1513) that provides generation of a visual representation upon receiving user input (e.g., selecting an edit icon (1501)). In this case, the option to generate the visual representation may be set to OFF by default. Accordingly, the interface (1513) may be in an inactive state even when displayed on the screen (1510). For example, the interface (1513) may be in a dim state on the screen (1510).

[0186] Upon receiving user input (1515) (e.g., selection of an option button) to provide an option to create a visual representation, the electronic device may display a screen (1520) with an interface (1513) activated. The interface (1513) may include at least one of a title area (1521), a summary area (1523), a category selection button (1525), a list (1527) providing a choice of styles (e.g., list (1415) of FIG. 14), and a create visual representation button (1529).

[0187] If a title previously created by a user or included in a document exists, the electronic device may display the title in the title area (1521). The electronic device may provide a text input mode in which a title can be entered or modified when the title area (1521) is selected by the user.

[0188] The electronic device may provide a text input mode in which a summary area (1523) can be selected by a user to enter or edit a summary.

[0189] The electronic device may provide a classification of "none" by default. The electronic device may provide a list of classifications when the classification selection button (1525) is selected by the user.

[0190] The electronic device may provide a list (1527) providing choices for background colors and background styles of the visual representation's cover.

[0191] An electronic device may provide a visual representation creation button (1529). If the state of the interface (1513) does not satisfy the creation conditions for a visual representation, the electronic device may disable the visual representation creation button (1529). For example, if the state of the interface (1513) does not satisfy the creation conditions for a visual representation, the electronic device may display the visual representation creation button (1529) in a dim state. If the user selects the visual representation creation button (1529) while the visual representation creation button (1529) is disabled, the electronic device may output an error message. In one embodiment, if a document does not satisfy the creation conditions for a visual representation, the electronic device may not display the visual representation creation button (1529).

[0192] The electronic device can activate the visual representation generation button (1529) when the state of the interface (1513) satisfies the visual representation generation condition.

[0193] The condition for generating a visual representation may be the completion of selection or input of at least one of the title area (1521), the summary area (1523), the category selection button (1525), and the list (1527). For example, if the condition for generating a visual representation is the completion of summary input through the summary area (1523) and the selection of a category through the category selection button (1525), the electronic device may not activate the visual representation generation button (1529) simply by completing the title input through the title area (1521).

[0194] The electronic device may display an indicator (1530) indicating that a visual representation is being generated when the activated visual representation generation button (1529) is selected. The electronic device may further display text indicating that a visual representation is being generated along with the indicator (1530).

[0195] The electronic device may provide a preview of the visual representation (1540) once the generation of the visual representation is complete. The electronic device may provide a preview of the visual representation (1540) through the interface (1513). Additionally, the electronic device may provide modifications to the visual representation (1540) through the interface (1513).

[0196] According to one embodiment, the visual representation (1540) generated by the method described above in FIG. 15 may be generated through a user's selection and input and may be different from the automatically provided first visual representation described in FIGS. 1 to 14 (e.g., the first visual representation (210) of FIG. 2 and the first visual representation (520) of FIG. 5) and the second visual representation (e.g., the second visual representation (220) of FIG. 2, the second visual representation (820) of FIG. 8, the second visual representation (900) of FIG. 9, the second visual representation (1000) of FIG. 10, the second visual representation (1010) of FIG. 10, the second visual representation (1210) of FIG. 12, and the second visual representation (1310) of FIG. 13).

[0197] Referring to FIG. 16, an electronic device (e.g., the electronic device (101) of FIG. 1 and the electronic device (601) of FIG. 6) may display an icon (1601) (e.g., the icon (305) of FIG. 3 and the icon (315) of FIG. 3) on a screen (1600) when a document satisfies the second condition. The screen (1600) may be an editing screen for a title for a document. However, the screen (1600) is merely an example and the present disclosure is not limited thereto.

[0198] The icon (1601) relates to automatic generation of visual representations and may have a different shape than the edit icon (1501) of FIG. 15.

[0199] When the electronic device receives a user input (e.g., selection of an icon (1601)), the electronic device may receive a second visual representation (1613) (e.g., the second visual representation (220) of FIG. 2, the second visual representation (820) of FIG. 8, the second visual representation (900) of FIG. 9, the second visual representation (1000) of FIG. 10, the second visual representation (1010) of FIG. 10, the second visual representation (1210) of FIG. 12, and the second visual representation (1310) of FIG. 13) according to the method described above in FIGS. 6 to 14, and display a preview of the second visual representation (1613) on the screen (1610).

[0200] The electronic device may display a screen (1620) (e.g., screen (1520) of FIG. 15) with an interface (1623) (e.g., interface (1513) of FIG. 15) activated when user input (e.g., selection of a second visual representation (1613)) is received. The electronic device may provide modifications to the second visual representation (1613) via the interface (1623).

[0201] FIG. 17 is a diagram illustrating simultaneous generation of multiple visual representations according to one embodiment of the present disclosure.

[0202] Referring to FIG. 17, a screen (1700) is shown showing at least a portion of a document list displaying documents.

[0203] An electronic device (e.g., the electronic device (101) of FIG. 1 and the electronic device (601) of FIG. 6) can simultaneously generate visual representations for up to N documents. N may be a natural number. For convenience of explanation, N will be assumed to be 6 below. However, the present disclosure is not limited thereto.

[0204] Six basic visual representations (1702, 1703, 1704, 1705, 1706, 1707) can be selected on the screen (1700). That is, six documents can be selected. When six basic visual representations (1702, 1703, 1704, 1705, 1706, 1707) are selected, the electronic device can display a visual representation generation option (1701). If more than six basic visual representations are selected, the electronic device can hide the visual representation generation option (1701). In one embodiment, the electronic device can display the visual representation generation option (1701) when it receives a long press input for at least a portion of the screen (1700) or an input for a "See More" button.

[0205] When the Create Visual Representation option (1701) is selected, the electronic device may display an indicator (1710) (e.g., indicator (1530) of FIG. 15) indicating that a visual representation is being created. The electronic device may further display text indicating that a visual representation is being created along with the indicator (1710).

[0206] When the generation of the visual representation is completed, the electronic device may provide a preview of the visual representations (e.g., the second visual representation (220 of FIG. 2, the second visual representation (820) of FIG. 8, the second visual representation (900) of FIG. 9, the second visual representation (1000) of FIG. 10, the second visual representation (1010) of FIG. 10, the second visual representation (1210) of FIG. 12, the second visual representation (1310) of FIG. 13, and the second visual representation (1613) of FIG. 16)) through the screen (1720). A visual representation may not be generated for a document that does not satisfy the second condition. For example, a basic visual representation (1721) (e.g., the basic visual representation (200) of FIG. 2, the basic visual representation (501) of FIG. 5, and the basic visual representation (801) of FIG. 8)) may be provided as a preview for a document that does not satisfy the second condition. The second condition is omitted from the description given above in Fig. 6. According to one embodiment, a first visual representation (e.g., the first visual representation (210) of Fig. 2 and the first visual representation (520) of Fig. 5) may be generated for a document that does not satisfy the second condition.

[0207] In one embodiment, if a document is filtered by a filtering model, a visual representation may not be generated even if the document satisfies the second condition. The server or electronic device may further include a filtering model. The filtering model may be a model trained to filter documents containing sensitive content. Sensitive content may include sexual content, derogatory content, toxic content, and violent content. For example, sensitive content may include specific words, sentences, photos, videos, or audio whose use is restricted (e.g., prohibited) in relation to the generation of a visual representation. The filtering model may perform filtering on documents for which the generation of a visual representation is requested. A visual representation may not be generated for a filtered document.

[0208] Ultimately, the visual representations provided as previews through the screen (1720) can correspond to the second visual representation.

[0209] The electronic device may receive a user input (1723) (e.g., a check mark) to disable the visual representation (i.e., the second visual representation). In response to receiving the user input, the electronic device may disable the visual representation for the document. The electronic device may disable the visual representation for the document and revert the document's cover to its state before the visual representation was created. Alternatively, the electronic device may disable the visual representation for the document and display the document in its default visual representation (1730).

[0210] FIG. 18 is a drawing for explaining editing of a second visual representation according to one embodiment of the present disclosure.

[0211] Referring to FIG. 18, a screen (1800) of an electronic device (e.g., the electronic device (101) of FIG. 1 and the electronic device (601) of FIG. 6) is illustrated showing at least a portion of a list of documents being displayed, one of a plurality of documents being displayed as a second visual representation (1803) (e.g., the second visual representation (220) of FIG. 2, the second visual representation (820) of FIG. 8, the second visual representation (900) of FIG. 9, the second visual representation (1000) of FIG. 10, the second visual representation (1010) of FIG. 10, the second visual representation (1210) of FIG. 12, the second visual representation (1310) of FIG. 13, and the second visual representation (1613) of FIG. 16). Upon receiving a user input (1801) selecting a second visual representation (1803), the electronic device can display a state in which the second visual representation (1803) is selected on the screen (1800).

[0212] An electronic device may receive user input for a More button. Upon receiving user input for the More button, the electronic device may display a screen (1810) including a pop-up window (1811) (e.g., pop-up window (510) of FIG. 5) providing various options. The pop-up window (1811) may include a cover editing menu. The electronic device may receive user input (1813) for the cover editing menu.

[0213] Upon receiving user input (1813), the electronic device may display a separate interface (1821) (e.g., interface (1513) of FIG. 15) on the screen (1820) that provides editing of a second visual representation. The interface (1821) may include a second visual representation (1803) selected on the screen (1800), a list providing a selection of styles (e.g., list (1415) of FIG. 14 and list (1527) of FIG. 15), and a create cover button (1823) (e.g., create visual representation button (1529) of FIG. 15).

[0214] A list providing selections for styles may include a list for background color selection (e.g., list (1417) of FIG. 14) and a list for background style selection (e.g., list (1419) of FIG. 14). The electronic device may receive selection inputs for background colors and background styles via the interface (1821). The electronic device may receive selection inputs for background colors and / or background styles different from the second visual representation (1803) via the interface (1821), and upon receiving selection inputs for the create cover button (1823), may generate a new second visual representation (1841). For example, upon receiving selection inputs for the create cover button (1823), the electronic device or a server (e.g., server (108) of FIG. 1 and server (608) of FIG. 6) may generate a new second visual representation (1841).

[0215] The electronic device may display an indicator (1831) (e.g., indicator (1710) of FIG. 7) on the screen (1830) indicating that a new second visual representation (1841) is being generated while the new second visual representation (1841) is being generated. The electronic device may deactivate the create cover button (1823) along with the display of the indicator (1831). For example, the electronic device may display the create cover button (1823) in a dim state.

[0216] Once the generation of the new second visual representation (1841) is complete, the electronic device may display the second visual representation (1841) on an interface (1821) of the screen (1840). The new second visual representation (1841) may include a background color and / or background style selected on the screen (1820).

[0217] FIG. 19 is a diagram illustrating the generation of a representative visual representation according to one embodiment of the present disclosure.

[0218] An electronic device (e.g., the electronic device (101) of FIG. 1 and the electronic device (601) of FIG. 6) can group documents determined to have similar classifications and titles and provide a single visual representation. In other words, the electronic device can determine the similarity of classifications and titles between documents and generate a representative visual representation (1921) for documents determined to have similar classifications and titles.

[0219] Referring to screen (1900), basic visual representations (1901) for multiple documents with similar categories and titles are illustrated. For example, the basic visual representations (1901) may correspond to multiple documents in which lecture content is divided and written. In screen (1900), the basic visual representations (1901) may be selected.

[0220] An electronic device may receive user input for a More button on a screen (1900). Upon receiving user input for the More button, the electronic device may display a pop-up window (e.g., pop-up window (510) of FIG. 5 and pop-up window (1811) of FIG. 18) providing various options. The pop-up window may include a representative visual representation generation menu. Upon receiving user input (1902) for the representative visual representation generation menu, the electronic device may generate a representative visual representation (1921).

[0221] When the electronic device receives user input (1902), it may display an indicator (1911) on the screen (1910) indicating that a visual representation is being generated (e.g., indicator (1710) of FIG. 7 and indicator (1831) of FIG. 18).

[0222] The electronic device may display a screen (1920) upon completion of generating the visual representation. The screen (1920) may include a representative visual representation (1921) as well as secondary visual representations (1922) corresponding to documents selected on the screen (1900).

[0223] In one embodiment, the electronic device may display a button for regenerating a visual representation (e.g., a visual representation generation button (1923)) adjacent to or overlapping the generated visual representation. The electronic device may regenerate the visual representation and provide the regenerated visual representation based on a user input to the button for regenerating the visual representation.

[0224] FIG. 20 is a drawing for explaining an operation method of an electronic device according to one embodiment of the present disclosure.

[0225] In the following embodiments, the operations may be performed sequentially, but are not necessarily performed sequentially. For example, the order of the operations may be changed, and at least two operations may be performed in parallel. Operations (2010) to (2030) may be performed by an electronic device (e.g., the electronic device (101) of FIG. 1 and the electronic device (601) of FIG. 6). When instructions stored in a memory (e.g., the memory (130) of FIG. 1) of the electronic device are executed by a processor (e.g., the processor (120) of FIG. 1), the operations below may be performed by the electronic device. For example, the processor (e.g., the processor (120) of FIG. 1) of the electronic device may correspond to multiple processors that collectively perform multiple operations by dividing them among the processors.

[0226] In operation (2010), the electronic device may request generation of a visual representation for a file based on information relating to the size of content contained in the file.

[0227] According to one embodiment, information related to the size of the content may include the number of characters contained in the content. The electronic device can generate a visual representation (i.e., a first visual representation (e.g., a first visual representation (210) of FIG. 2 and a first visual representation (520) of FIG. 5)) including at least a part of the characters without a request to the server if the number of characters is less than or equal to a threshold value (i.e., a first condition). If the number of characters exceeds the threshold value, the electronic device can generate a visual representation (i.e., a second visual representation (e.g., a second visual representation (220) of FIG. 2, a second visual representation (820) of FIG. 8, a second visual representation (900) of FIG. 9, a second visual representation (1000) of FIG. 10, a second visual representation (1010) of FIG. 10, a second visual representation (1210) of FIG. 12, a second visual representation (1310) of FIG. 13, a second visual representation (1613) of FIG. 16, and a second visual representation (1710) of FIG. 17) for an artificial intelligence computing device. You can request the creation of a second visual representation of 18 (1803)).

[0228] In one embodiment, information related to the size of the content may be the amount of space the content occupies in the memory of the electronic device (e.g., memory (130) of FIG. 1 ). The electronic device may request the server to generate a visual representation if the amount of space the content occupies in the memory of the electronic device exceeds a threshold. In other words, the electronic device may request the server to generate a second visual representation.

[0229] The electronic device may transmit a prompt (e.g., prompt (700) of FIG. 7) to the AI ​​computing device, which may include content contained in the document and other content associated with the document. The other content associated with the document may include content contained in other documents, as well as content stored on the electronic device and content accessible to the electronic device via a web browser.

[0230] According to one embodiment, the electronic device may transmit a prompt to the artificial intelligence computing device as at least part of the request, the prompt including conditions associated with the generation of at least some of the characters and visual representations of the characters.

[0231] In operation (2020), the electronic device may display a visual representation associated with the file, including a summary and classification of the file.

[0232] In operation (2030), the electronic device may provide content based on a selection input regarding a visual representation received from the user.

[0233] Since the contents described above in FIGS. 1 to 19 can be equally applied to actions (2010) to (2030), detailed descriptions are omitted.

[0234] FIG. 21 is a block diagram of a generative artificial intelligence system according to various embodiments.

[0235] The generative artificial intelligence system (2100) illustrated in FIG. 21 may be an artificial intelligence model that generates a second visual representation (e.g., the second visual representation (220) of FIG. 2, the second visual representation (820) of FIG. 8, the second visual representation (900) of FIG. 9, the second visual representation (1000) of FIG. 10, the second visual representation (1010) of FIG. 10, the second visual representation (1210) of FIG. 12, the second visual representation (1310) of FIG. 13, the second visual representation (1613) of FIG. 16, and the second visual representation (1803) of FIG. 18).

[0236] The User Query / Response Interface (2110) can receive a user's input (e.g., a prompt (700) of FIG. 7). The user's input may be in the form of natural language, an image, and / or a video. Furthermore, context information may also be transmitted when the user's input is transmitted. The context information may include various additional information at the time of the user's input. For example, it may include information on the application the user is currently using or information on the user's location. Furthermore, the user's input may be in a mixed form of the aforementioned natural language, image, sound, and context information. Furthermore, the user's input may also be in a non-natural language form, such as selecting a menu. The User Query / Response Interface (2110) can output the results of the generative artificial intelligence system (2100) to the user. The output may be in the form of natural language or specific content, and may also be provided in the form of an action requested by the user. The User Query Interface / Response Interface (2110) can output the results of the generative artificial intelligence system (2100) to the user. The output can be in the form of natural language or specific content, and can also be provided in the form of an action requested by the user.

[0237] The AI ​​framework (2120) can receive user input and coordinate and control each component necessary to perform the user's intention based on the user's query.

[0238] User input received from the User Query / Response Interface (2110) can be transmitted to the Prompt design component (2130). The Prompt design component (2130) can be used to generate prompts suitable for inputting the user input into large language models (LLMs) or large multimodal models (LMMs). The Prompt design component (2130) can be an AI component that uses a machine learning algorithm or a neural network to develop better prompts over time. The Prompt design component (2130) can access a knowledge component (i.e., knowledge repositories (2170)) including user preference data, a prompt library, and prompt examples based on the user input to generate a prompt, and transmit the generated prompt to the LLM or LMM.

[0239] The API / Plugins management component (2140) can communicate with external information when there is a request for additional information when passing user input as input to a generative model. The API / Plugins management component (2140) establishes a channel for communication with the outside of the AI ​​Interface through an API, and can enable access to various data sources through the established channel. In addition, the API / Plugins management component (2140) can request an action through an API (application programming interface; API) when an application or service needs to perform an action that performs the user input as a final result rather than an intermediate result. Information obtained from an external source can be used to generate a prompt in the Prompt design component (2130) along with the user input, or can be passed as input to the generative model.

[0240] The Refiner component (i.e., the output modification component (2150)) can fine-tune the output from the generative model. For example, the Refiner component can verify that the content generated by the LLM and / or LMM is not irrelevant, biased, or harmful. Furthermore, the Refiner component can determine the degree to which the output matches the user's desired result and, if necessary, perform additional processing. The Refiner component can also configure and provide hints to the user to avoid undesirable output.

[0241] Generative AI Model (2160) can generally refer to an artificial intelligence neural network that creates new types of data based on user input information. Generative AI Model (2160) can include an image-generating model and / or a language-generating model. Representative models for generating images include a generative adversarial network (GAN) and a variational auto encoder (VAE), and examples include a diffusion-based generative model that uses a VAE and a transformer structure. A language-generating model is a model trained to statistically output the most appropriate output based on input values, and representative examples include models such as CHAT-GPT 3 and CHAT-GPT 4. In addition, there is also an LMM that can recognize various types of data input, such as text, images, and voice, and generate new data corresponding to them.

[0242] An electronic device according to an embodiment of the present disclosure (e.g., the electronic device 101 of FIG. 1 and the electronic device 601 of FIG. 6) may include a memory (e.g., the memory 130 of FIG. 1) that includes instructions. The electronic device may include a processor (e.g., the processor 120 of FIG. 1) that executes the instructions. The instructions, when executed by the processor, may cause the electronic device to receive an input for generating a visual representation of a document. The instructions, when executed by the processor, may cause the electronic device to generate a first visual representation (e.g., the first visual representation 210 of FIG. 2 and the first visual representation 520 of FIG. 5) corresponding to the visual representation if a condition for generating the visual representation satisfies a first condition. The instructions, when executed by the processor, may cause the electronic device to display the first visual representation on a display module (e.g., the display module 160 of FIG. 1) in association with a document. The instructions, when executed by the processor, may cause the electronic device to transmit a request to a server for generation of a second visual representation corresponding to the visual representation (e.g., the second visual representation (220) of FIG. 2, the second visual representation (820) of FIG. 8, the second visual representation (900) of FIG. 9, the second visual representation (1000) of FIG. 10, the second visual representation (1010) of FIG. 10, the second visual representation (1210) of FIG. 12, the second visual representation (1310) of FIG. 13, the second visual representation (1613) of FIG. 16, and the second visual representation (1803) of FIG. 18) if a condition for generation of the visual representation satisfies a second condition. The instructions, when executed by the processor, may cause the electronic device to display, on a display module, the second visual representation received from the server (e.g., the server (108) of FIG. 1 and the server (608) of FIG. 6) in response to the request, in association with a document. The first visual representation may include at least some of the characters contained in the document.The second visual representation may include an indicator indicating a classification for the document (e.g., an icon (1403) of FIG. 14).

[0243] According to one embodiment, the instructions, when executed by the processor, may cause the electronic device to transmit a prompt (e.g., prompt (700) of FIG. 7) to an artificial intelligence computing device operatively connected to a server, the prompt requesting generation of a second visual representation that includes at least some of the characters included in the document.

[0244] In one embodiment, the prompt may include instructions for transmitting to the server a plurality of categories including the categories, so as to cause the server to determine a category for the document based on at least some of the characters contained in the document.

[0245] In one embodiment, the first condition may be whether the number of characters contained in the document is less than or equal to a threshold value. The second condition may be whether the number of characters contained in the document exceeds the threshold value.

[0246] According to one embodiment, the second visual representation may include a summary (e.g., summary (920) of FIG. 9) generated based on at least some characters highlighted by indicators, if the document includes indicators highlighting at least some characters included in the document.

[0247] According to one embodiment, the second visual representation may include a background (e.g., background (930) of FIG. 9) that includes at least one of a color specified by the classification or a style specified by the classification.

[0248] According to one embodiment, the second visual representation may include an image generated based on at least one of multimedia content included in the document or multimedia content associated with the document.

[0249] According to one embodiment, the second visual representation may include an image generated based on at least some of the characters contained in the document.

[0250] According to one embodiment, the instructions, when executed by the processor, may cause the electronic device to, in response to receiving input for modifying a second visual representation (e.g., user input (1401) of FIG. 14), display an icon for each of a plurality of classifications determined for the document, according to a priority order of the plurality of classifications.

[0251] According to one embodiment, the instructions, when executed by the processor, may cause the electronic device to, in response to receiving input for modifying the second visual representation, request a server to generate a second visual representation corresponding to a classification different from the classification.

[0252] According to one embodiment, the instructions, when executed by the processor, may cause the electronic device to identify another document related to the document and to display a second visual representation in association with the document and the other document.

[0253] An electronic device according to one embodiment of the present disclosure may include a memory including instructions. The electronic device may include a processor that executes the instructions. The instructions, when executed by the processor, may cause the electronic device to receive an input for generating a visual representation of a document. The instructions, when executed by the processor, may cause the electronic device to initiate generation of a first visual representation if the number of characters included in the document is less than or equal to a threshold value. The instructions, when executed by the processor, may cause the electronic device to initiate generation of a second visual representation if the number of characters included in the document exceeds the threshold value. The instructions, when executed by the processor, may cause the electronic device to display the first visual representation or the second visual representation on a display module in association with the document. The first visual representation may include at least some of the characters. The second visual representation may include at least one of a summary, a classification, and a background for the document generated based on at least some of the characters.

[0254] According to one embodiment, the second visual representation may vary the arrangement of at least one of the summary, classification, and background for the document depending on the classification.

[0255] In one embodiment, the instructions, when executed by the processor, may cause the electronic device to transmit a prompt to the server requesting generation of a second visual representation including at least some of the characters if the number of characters exceeds a threshold.

[0256] A method of operating an electronic device according to one embodiment of the present disclosure may include receiving an input for generating a visual representation for a document. The method of operating the electronic device may include generating a first visual representation corresponding to the visual representation if a condition for generating the visual representation satisfies a first condition. The method of operating the electronic device may include displaying the first visual representation on a display module in association with the document. The method of operating the electronic device may include transmitting a request for generating a second visual representation corresponding to the visual representation to a server if the condition for generating the visual representation satisfies a second condition. The method of operating the electronic device may include receiving the second visual representation from the server in response to transmitting the request for generating the second visual representation to the server. The method of operating the electronic device may include displaying the second visual representation on the display module in association with the document. The first visual representation may include at least a portion of sentences included in the document. The second visual representation may include an icon representing a classification of the document determined based on at least a portion of characters included in the document.

[0257] According to one embodiment, the act of sending a request for generation of a second visual representation corresponding to a visual representation may generate a prompt requesting generation of the second visual representation including at least some of the characters included in the document.

[0258] In one embodiment, the prompt may include instructions for transmitting to the server items used in classifying the document to cause the server to determine a classification for the document based on at least some of the characters contained in the document.

[0259] In one embodiment, the first condition may be whether the number of characters contained in the document is less than or equal to a threshold value. The second condition may be whether the number of characters contained in the document exceeds the threshold value.

[0260] According to one embodiment, the second visual representation may include a summary generated based on at least some characters highlighted by the indicators, if the document includes indicators highlighting at least some characters included in the document.

[0261] A method of operating an electronic device according to one embodiment of the present disclosure may include an operation of requesting the generation of a visual representation of a file based on information related to the size of the content contained in the file. The method may include an operation of displaying a visual representation including a summary and classification of the content in association with the file. The method may include an operation of providing content based on a selection input received from a user regarding the visual representation.

[0262] In one embodiment, information related to the size of the content may include the number of characters contained in the content. The requesting action may include transmitting a request for generation to an artificial intelligence computing device if the number of characters exceeds a threshold.

[0263] In one embodiment, the requesting action may include transmitting to the artificial intelligence computing device, as at least part of the request, a prompt that includes conditions associated with the generation of at least some of the characters and visual representations of the characters.

[0264] In one embodiment, the prompt may include, as at least part of the condition, information about what kind of language to generate the summary or classification in.

[0265] In one embodiment, the requested action may be based on a determination of whether the file contains, as at least part of the content, content that is restricted in use with respect to generating a visual representation.

[0266] A computer-readable recording medium according to one embodiment of the present disclosure may include one or more computer programs including commands for executing any one of the above-described methods.

[0267]

[0268] The embodiments of the present invention disclosed in this specification and drawings are merely specific examples presented to easily explain the technical contents according to the embodiments of the present invention and to help understand the embodiments of the present invention, and are not intended to limit the scope of the embodiments of the present invention. Therefore, the scope of the various embodiments of the present invention should be interpreted as including all changes or modified forms derived based on the technical ideas of the various embodiments of the present invention in addition to the embodiments disclosed herein.

Claims

1. In an electronic device (101, 601), a memory (130) containing instructions; and Processor (120) executing the above commands Including, The above instructions, when executed by the processor (120), cause the electronic device (101, 601) to: Receive input for generating a visual representation for a document, and if a condition for generating the visual representation satisfies a first condition, generate a first visual representation (210, 520) corresponding to the visual representation, and display the first visual representation (210, 520) on a display module (160) in association with the document, and if the condition for generating the visual representation satisfies a second condition, transmit a request for generating a second visual representation (220, 820, 900, 1000, 1010, 1210, 1310, 1613, 1803) corresponding to the visual representation to a server (108, 608), and in response to the request, transmit the second visual representation (220, 820, 900, 1000, 1010, 1210, 1310, 1613, 1803) received from the server (108, 608). 1613, 1803) to be displayed on the display module (160) in connection with the above document, The above first visual representation (210, 520) includes at least some of the characters included in the document, The second visual representation (220, 820, 900, 1000, 1010, 1210, 1310, 1613, 1803) includes an indication (1403) indicating a classification for the document. Electronic devices (101, 601).

2. In paragraph 1, The above instructions, when executed by the processor (120), cause the electronic device (101, 601) to: Transmitting a prompt (700) to an artificial intelligence computing device operatively connected to the server, requesting generation of the second visual representation (220, 820, 900, 1000, 1010, 1210, 1310, 1613, 1803) including at least some of the characters included in the said document; Electronic devices (101, 601).

3. In either of paragraphs 1 and 2, The above prompt (700) is, Including a command for transmitting a plurality of classifications including the classification to the server (108, 608) so as to cause the server to determine the classification for the document based on at least some of the characters included in the document. Electronic devices (101, 601).

4. In any one of paragraphs 1 to 3, The first condition above is, Whether the number of characters included in the above document is less than or equal to the threshold, The second condition is, Whether the number of characters included in the above document exceeds the above threshold value; Electronic devices (101, 601).

5. In any one of paragraphs 1 to 4, The above second visual representation (220, 820, 900, 1000, 1010, 1210, 1310, 1613, 1803) is If the above document includes indicators that emphasize at least some characters included in the above document, a summary (920) generated based on the at least some characters emphasized by the indicators, Electronic devices (101, 601).

6. In any one of paragraphs 1 to 5, The above second visual representation (220, 820, 900, 1000, 1010, 1210, 1310, 1613, 1803) is A background (930) comprising at least one of the colors specified according to the above classification or the styles specified according to the above classification, Electronic devices (101, 601).

7. In any one of paragraphs 1 to 6, The above second visual representation (220, 820, 900, 1000, 1010, 1210, 1310, 1613, 1803) is An image generated based on at least one of the multimedia contents included in the above document or multimedia contents related to the above document, Electronic devices (101, 601).

8. In any one of paragraphs 1 to 7, The above second visual representation (220, 820, 900, 1000, 1010, 1210, 1310, 1613, 1803) is An image generated based on at least some of the characters contained in the above document, Electronic devices (101, 601).

9. In any one of paragraphs 1 to 8, The above instructions, when executed by the processor (120), cause the electronic device (101, 601) to: In response to receiving an input (1401) for modifying the above second visual representation (220, 820, 900, 1000, 1010, 1210, 1310, 1613, 1803), displaying an icon (1403) for each of the plurality of classifications according to the priority of the plurality of classifications determined for the document. Electronic devices (101, 601).

10. In any one of paragraphs 1 to 9, The above instructions, when executed by the processor (120), cause the electronic device (101, 601) to: In response to receiving an input for modifying the above second visual representation (220, 820, 900, 1000, 1010, 1210, 1310, 1613, 1803), requesting the server (108, 608) to generate a second visual representation (220, 820, 900, 1000, 1010, 1210, 1310, 1613, 1803) corresponding to a classification different from the above classification. Electronic devices (101, 601).

11. In any one of paragraphs 1 to 10, The above instructions, when executed by the processor (120), cause the electronic device (101, 601) to: Identifying other documents related to said document, and displaying said second visual representation (220, 820, 900, 1000, 1010, 1210, 1310, 1613, 1803) in association with said document and said other documents. Electronic devices (101, 601).

12. In the electronic device (101, 601), a memory (130) containing instructions; and Processor (120) executing the above commands Including, The above instructions, when executed by the processor (120), cause the electronic device (101, 601) to: Receiving input for generating a visual representation for a document, initiating generation of a first visual representation (210, 520) if the number of characters included in the document is less than or equal to a threshold value, initiating generation of a second visual representation (220, 820, 900, 1000, 1010, 1210, 1310, 1613, 1803) if the number of characters included in the document exceeds the threshold value, and displaying the first visual representation (210, 520) or the second visual representation (220, 820, 900, 1000, 1010, 1210, 1310, 1613, 1803) on a display module (160) in association with the document, The first visual representation (210, 520) includes at least some of the characters, The second visual representation (220, 820, 900, 1000, 1010, 1210, 1310, 1613, 1803) comprises at least one of a summary (920), a classification and a background (930) for the document generated based on at least some of the characters. Electronic devices (101, 601).

13. In paragraph 12, The above second visual representation (220, 820, 900, 1000, 1010, 1210, 1310, 1613, 1803) is Depending on the above classification, at least one of the summary (920), classification and background (930) of the above document is arranged differently. Electronic devices (101, 601).

14. In any one of paragraphs 12 and 13, When the above commands are executed by the processor (120), the electronic device (101, 601) If the number of said characters exceeds said threshold, a prompt (700) is transmitted to the server (108, 608) requesting generation of said second visual representation, which includes at least some of said characters. Electronic devices (101, 601).

15. In the operating method of an electronic device (101, 601), An action requesting the generation of a visual representation of a file based on information regarding the size of the content contained in the file; An action to display said visual representation, including a summary and classification of said content, in association with said file; and An action to provide said content based on selection input for said visual representation received from the user. A method of operation, comprising:

Images