Electronic device for providing extended content by using generative ai model, and operating method thereof

Abstract

According to one embodiment, an electronic device may comprise a display, at least one processor, and a memory including instructions. According to one embodiment, when executed by the at least one processor, the instructions instruct the electronic device to: identify a command for displaying content for a book, including text and images; on the basis of identifying the command, display, on the display, a menu for extending a story of the content, focusing on a specific person from among persons appearing in the content; and, on the basis of a user input for selecting a first person from the menu, display, on the display, extended content for the first person, in which additional text and / or an additional image related to the first person is added to the content. According to one embodiment, the extended content can be generated on the basis of providing, to a generative AI model, the plurality of pieces of text included in the content, the plurality of images included in the content, and a prompt related to the first person.

Description

Electronic device providing expanded content using a generative AI model and method of operation thereof

[0001] The present disclosure relates to an electronic device that provides extended content using a generative AI model and a method of operating the same.

[0002] Thanks to remarkable advancements in information and communication technology and semiconductor technology, the proliferation and use of various electronic devices is rapidly increasing. Electronic devices are being developed to enable users to carry and communicate with one another. An electronic device can refer to any device that performs a specific function based on its embedded software, such as a mobile communication terminal, tablet PC, wearable electronic device, audio / video device, desktop / laptop computer, or in-vehicle navigation system.

[0003] Electronic devices can utilize artificial intelligence (AI) models to provide services for specific purposes. For example, AI models are being utilized in diverse fields such as content streaming, translation, photo editing, finance, new drug development, law, and the military. At least some of the various AI models for specific services can be implemented as generative AI models. Depending on the implementation, the AI ​​models can operate in a form where multiple AI models are linked together.

[0004] According to one embodiment, an electronic device may include a display, at least one processor, and a memory including instructions. According to one embodiment, the instructions, when executed by the at least one processor, may cause the electronic device to identify a command to display content for a book including texts and images. According to one embodiment, the instructions, when executed by the at least one processor, may cause the electronic device, based on identifying the command, to display a menu on the display for expanding a story of the content centered on a specific character among characters appearing in the content. According to one embodiment, the instructions, when executed by the at least one processor, may cause the electronic device, based on a user input selecting a first character from the menu, to display expanded content for a first character, in which at least one of additional text or additional image related to the first character is added to the content. According to one embodiment, the extended content may be generated based on providing a generative AI model with the plurality of texts included in the content, the plurality of images included in the content, and a prompt related to the first person.

[0005] According to one embodiment, a method of operating an electronic device may include an action of confirming a command to display content for a book including texts and images. According to one embodiment, the method of operating the electronic device may include an action of displaying a menu for expanding a story of the content centered on a specific character among characters appearing in the content on a display included in the electronic device based on the confirmation of the command. According to one embodiment, the method of operating the electronic device may include an action of displaying, on the display, expanded content for a first character in which at least one of additional text or additional image related to the first character is added to the content based on a user input of selecting a first character from the menu. According to one embodiment, the expanded content may be generated based on providing a generative AI model with the plurality of texts included in the content, the plurality of images included in the content, and a prompt related to the first character.

[0006] According to one embodiment, a computer-readable, non-transitory recording medium may store a program that, when executed by at least one processor included in an electronic device, causes the electronic device to perform the following actions: confirming a command to display content for a book including texts and images; displaying, on a display included in the electronic device, a menu for expanding a story of the content centered on a specific character among characters appearing in the content based on the confirmation of the command; and displaying, on the display, expanded content for a first character in which at least one of additional text or additional image related to the first character is added to the content based on a user input selecting the first character from the menu. According to one embodiment, the expanded content may be generated based on providing a generative AI model with the plurality of texts included in the content, the plurality of images included in the content, and a prompt related to the first character.

[0007] FIG. 1 is a block diagram of an electronic device within a network environment, according to one embodiment.

[0008] FIG. 2 is a block diagram illustrating a schematic configuration of an electronic device according to one embodiment.

[0009] Figure 3 is a schematic block diagram of an image generation model according to one embodiment.

[0010] FIG. 4 is a flowchart illustrating a method for an electronic device to display extended content based on original content for a book, according to one embodiment.

[0011] FIG. 5 is a flowchart illustrating a method for an electronic device to display extended content according to one embodiment.

[0012] FIG. 6 is a flowchart illustrating a method for an electronic device to obtain extended content according to one embodiment.

[0013] FIGS. 7A, 7B, and 7C are drawings of a user interface for an electronic device to display extended content, according to one embodiment.

[0014] FIG. 8 is a drawing for explaining extended content provided by an electronic device according to one embodiment.

[0015] FIGS. 9A and 9B are diagrams of prompts applied to the entire story of original content, according to one embodiment.

[0016] FIGS. 10A and 10B are diagrams of prompts applied to each page of original content, according to one embodiment.

[0017] FIGS. 11A, 11B, 11C and 11D are diagrams illustrating a method for generating a first page included in extended content depending on whether content about a first person appears on the first page included in the original content, according to one embodiment.

[0018] FIGS. 12A and 12B are drawings for explaining a method of generating an image included in extended content according to one embodiment.

[0019] FIG. 13 is a diagram illustrating a method for generating an image included in extended content according to one embodiment.

[0020] FIG. 14 is a drawing for explaining extended content provided by an electronic device according to one embodiment.

[0021] FIGS. 15A, 15B, and 15C are drawings illustrating a method for an electronic device to display extended content according to one embodiment.

[0022] FIG. 16 is a drawing for explaining additional images included in extended content according to one embodiment.

[0023] Fig. 17 is a diagram for explaining a generative artificial intelligence system according to one embodiment.

[0024] FIG. 1 is a block diagram of an electronic device (101) within a network environment (100) according to various embodiments. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with the electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of the electronic device (104) or the server (108) via a second network (199) (e.g., a long-range wireless communication network). In one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0025] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (121). For example, when the electronic device (101) includes the main processor (121) and the auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a given function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0026] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, in the electronic device (101) itself where artificial intelligence is performed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0027] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0028] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0029] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0030] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0031] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0032] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0033] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0034] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0035] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0036] The haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0037] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0038] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0039] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0040] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0041] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0042] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas, for example, by the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device via the selected at least one antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0043] In one embodiment, the antenna module (197) may generate a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high frequency band.

[0044] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0045] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0046] FIG. 2 is a block diagram illustrating a schematic configuration of an electronic device according to one embodiment.

[0047] Referring to FIG. 2, according to one embodiment, an electronic device (201) may include a camera (210), a processor (220), a memory (230), a display (260), and a communication circuit (290). For example, the electronic device (201) may be implemented in the same or similar manner as the electronic device (101) of FIG. 1.

[0048] According to one embodiment, the processor (220) (e.g., the processor (120) of FIG. 1) may control the overall operation of the electronic device (201). The processor (220) according to one embodiment may execute software (e.g., the program (140) of FIG. 1) to control at least one other component (e.g., a hardware or software component) of the electronic device (201) connected to the processor (220), and may perform data processing or calculation based on the instruction. The instruction according to one embodiment may include a command configured in a machine language that can be processed by the electronic device (201) or the processor (220). For example, the instruction may include a command corresponding to an operation instruction used in a program.

[0049] Meanwhile, although FIG. 2 illustrates that the electronic device (201) includes one processor (220), this is merely exemplary and the technical concept of the present invention may not be limited thereto. For example, the electronic device (201) may include at least one processor. For example, the processor (220) may be implemented as at least one processor.

[0050] According to one embodiment, the memory (230) (e.g., the memory (130) of FIG. 1) may store at least one command (or instruction) that causes at least one operation of the electronic device (201). The at least one instruction, when executed by the processor (220), may cause the electronic device (201) to perform a corresponding operation.

[0051] According to one embodiment, the processor (220) may display content (e.g., original content or extended content) for a book through the display (260). For example, the content for the book may include content for an electronic book (e-book). The content for the book may include a plurality of texts and a plurality of images. For example, the content for the book may include a plurality of pages, and each of the plurality of pages may include text and / or an image.

[0052] According to one embodiment, content for a book may be stored in an external device (e.g., a server) or in memory (230). For example, if content for a book is stored in an external device, the processor (220) may obtain information about the content for the book through the communication circuit (290).

[0053] According to one embodiment, the processor (220) may generate content (e.g., extended content) in which the story of the content is further expanded or added based on a specific character among the characters appearing in the content (e.g., original content) using a generative artificial intelligence (AI) model (e.g., the generative AI model (310) of FIG. 3). For example, the extended content may represent content in which additional text and / or additional images are further reflected or added to the content (e.g., original content). For example, the extended content may further add additional text that further emphasizes or adds the emotion of a specific character and / or additional images based on the gaze of a specific character to the content (e.g., original content). Alternatively, the extended content may edit and process some of the texts and / or images included in the content (e.g., original content) so that a specific character is emphasized (e.g., the emotion of a specific character or the viewpoint of a specific character). Alternatively, the extended content may include new text and / or images not included in the original content (e.g., the original content) to emphasize a specific character (e.g., a specific character's emotions or perspective). For example, the edited or added text and images may be acquired, modified, or generated using a generative AI model. For example, the generative AI model may be stored in memory (230) or on an external device (e.g., a server).

[0054] For convenience of explanation, below, content regarding a book stored on an external device (e.g., a server) or in memory (230) will be described as original content. Furthermore, content in which the description (e.g., a story and / or image) of content (e.g., original content) is further expanded or added based on a specific person using a generative AI model will be described as extended content.

[0055] According to one embodiment, the processor (220) may display the extended content through the display (260) based on a command requesting display of extended content for a specific person appearing in the original content. For example, the processor (220) may display the extended content based on the page order included in the original content. In this case, the original page where the specific person appears (e.g., a specific page included in the original content) may be replaced with a new page in which some text or some images are edited. In addition, the extended content may replace the original page where the specific person does not appear (e.g., a specific page included in the original content) with a new page that newly includes text and / or images for the specific person.

[0056] In one embodiment, the processor (220) may generate or obtain a prompt (e.g., a text prompt) for generating extended content. For example, the prompt may include instructions (e.g., text) for generating additional text and / or additional images based on the emotions or gazes of a specific person appearing in the original content. For example, the prompt may include instructions (e.g., text) for generating additional text that emphasizes or highlights the emotions of a specific person, or additional images that focus on the gazes or gazes of a specific person.

[0057] According to one embodiment, the processor (220) may generate a prompt for generating extended content for a specific person based on settings indicating a relationship between the extended content and the original content, the proportion of other people in the extended content besides the specific person, the mood of the extended content, and the background of the extended content. For example, the settings may be set to preset values. Additionally, the settings may be adjusted by the user. The processor (220) may provide an interface for adjusting the settings.

[0058] In one embodiment, the processor (220) may obtain extended content output from the generative AI model by providing original content (e.g., multiple texts and multiple images included in the original content) and a prompt to the generative AI model. The processor (220) may analyze information about a specific person appearing in the original content, information about the entire story of the original content, and / or information about each page included in the original content, and further provide information corresponding to the analyzed result to the generative AI model.

[0059] According to one embodiment, the processor (220) may provide original content (e.g., a plurality of texts and a plurality of images included in the original content) and a prompt to a generative AI model stored in a memory (230), and obtain extended content output from the generative AI model.

[0060] In another embodiment, the processor (220) may transmit information about the prompt and original content to an external device (e.g., a server) so that the information about the prompt and original content is input into a generative AI model stored in the external device. The processor (220) may receive or obtain information about the extended content from the external device (e.g., a server).

[0061] According to one embodiment, an operation of displaying content (e.g., original content or extended content) for a book may include a series of operations of displaying at least one of text or image included in a corresponding page in a specified order (e.g., page order). For example, the processor (220) may display a plurality of texts and a plurality of images included in the content for the book in a specified order (e.g., page order) through the display (260) based on a user input (e.g., a horizontal swipe input or a touch input).

[0062] According to one embodiment, the electronic device (201) may identify a user input for performing a specific function on content (e.g., original content or extended content) of a book. For example, the user input may include an input for selecting a specific object on a display (260) (e.g., a touch screen) (e.g., a touch input) or an input for changing a page of a book (e.g., a swipe input).

[0063] According to one embodiment, when the electronic device (201) is implemented as an HMD device or a VST device, the processor (220) can identify a user input based on an image acquired from the camera (210). For example, the processor (220) can analyze the image acquired from the camera (201) to identify a user input for performing a specific function on content (e.g., original content or extended content) of a book. For example, the user input can include an input for selecting a specific object (e.g., a touch gesture input) or an input for changing a page of a book (e.g., a swipe input).

[0064] Figure 3 is a schematic block diagram of a generative AI model according to one embodiment.

[0065] According to one embodiment, the generative AI model (310) may generate new extended content based on original content using at least one artificial intelligence (AI) model. For example, the generative AI model (310) may be stored in memory (230) or an external device (e.g., a server).

[0066] According to one embodiment, the generative AI model (310) may use original content such as text, audio, and / or images to generate new content similar to the original content. For example, the generative AI model may learn patterns of content and generate new content as an inference result. For example, the generative AI model (310) may perform at least one of an in-painting operation or an out-painting operation on an image included in the original content for a book to generate (or obtain, output) a new image. For example, the generative AI model (310) may generate (or obtain, output) new text based on text included in the original content for a book. Depending on the implementation, the generative AI model (310) may generate (or obtain, output) new audio or video based on audio or video included in the original content for a book.

[0067] In one embodiment, the generative AI model (310) may generate extended content for a specific person based on being provided (or input) with original content for a book (e.g., texts and images included in the original content) and a prompt. Furthermore, the generative AI model (310) may generate extended content for a specific person based on being provided (or input) with further information about the extent of each page included in the original content and / or analysis information about the original content. For example, the extended content may include extended content (e.g., text and images) for each page. For example, the number of pages included in the extended content may be the same as the number of pages included in the original content. Depending on the implementation, the number of pages included in the extended content may be more or less than the number of pages included in the original content.

[0068] In one embodiment, texts and images contained in the original content of a book may be distinguished by the pages of the original content. For example, the original content may include multiple pages. Each of the multiple pages may include at least one of the texts and images specified for that page.

[0069] According to one embodiment, the generative AI model (310) may analyze (or infer) information about each page of the original content based on the original content. The generative AI model may utilize information about each analyzed (or inferred) page when generating extended content. Depending on the implementation, the generative AI model (310) may further receive information about each page analyzed (or inferred) by a separate analysis model. For example, information about each page of the original content may include information about the characters appearing on the page, descriptions of the characters appearing on the page (e.g., text and / or images), and / or the order of the pages.

[0070] According to one embodiment, the generative AI model (310) may analyze (or infer) the entire story of the original content and utilize analysis information about the analyzed original content when generating extended content. Depending on the implementation, the generative AI model (310) may receive additional information about the original content (or the entire story of the original content) analyzed (or inferred) by a separate analysis model. For example, the analysis information about the original content may include information about the gender, age, personality, place of residence, relationships with other characters of a specific character appearing in the original content, and information about the behavior and / or emotions of a specific character on each page.

[0071] In one embodiment, the prompt may include instructions (e.g., text) to generate additional text and / or additional images based on the emotion or gaze of a specific character appearing in the original content. For example, the prompt may include instructions (e.g., text) to generate additional text that emphasizes or highlights the emotion of a specific character, or additional images that focus on the gaze or gaze of a specific character.

[0072] In one embodiment, the prompt may include a first prompt that applies to all pages of the expanded content and a second prompt that applies to some pages of the expanded content. For example, the first prompt may be determined based on the relationship between the expanded content and the original content, the proportion of characters other than the specific character in the expanded content, the mood of the expanded content, and settings that represent the background of the expanded content. For example, the settings may be set to preset values. Additionally, the settings may be adjusted based on user input to the user interface. For example, the second prompt may be determined based on whether a description (e.g., text and / or image) of a specific character exists on each page. For example, if a description of a specific character exists on the page, the second prompt may include a command to emphasize the emotion of the specific character or add an image of the specific character. Alternatively, if a description of the specific character does not exist on the page, the second prompt may include a command to create and add a new description and / or image of the specific character.

[0073] According to one embodiment, the generative AI model (310) may generate extended content in which content is expanded or additionally generated based on a specific character among characters appearing in the original content (e.g., E-book content) based on the original content, prompts, information about each page, and / or analysis information about the original content.

[0074] Through the above-described method, the electronic device (201) can provide content that includes an expanded story based on a specific character among multiple characters appearing in the content of a book (e.g., a story that emphasizes the emotions of a specific character or a story expanded from the perspective or viewpoint of a specific character). Through this, the electronic device (201) can provide the user with an expanded story based on various perspectives of a specific character within the scope of the story of the content of the book. The user can view stories from various perspectives based on a single content.

[0075] At least some of the operations of the electronic device (201) described below may be performed by the processor (220) or the generative AI model (310). However, for convenience of explanation, the operations below will be described as being performed by the electronic device (201).

[0076] FIG. 4 is a flowchart illustrating a method for an electronic device to display extended content based on original content for a book, according to one embodiment.

[0077] Referring to FIG. 4, according to one embodiment, at operation 401, an electronic device (e.g., electronic device (201) of FIG. 2) may identify a command to display original content for a book including texts and images. For example, the command may include a command to execute an application (e.g., an e-book reader application) for displaying (or reading) content for the book (e.g., an e-book) in response to a user input.

[0078] In one embodiment, in operation 403, the electronic device (201) may display a menu (or user interface) for expanding the story of the original content centered on a specific character among the characters appearing in the original content. For example, the menu may provide a user interface for selecting one of the characters appearing in the original content.

[0079] In one embodiment, in operation 405, the electronic device (201) may display extended content for the first person in which at least one of additional text or additional image related to the first person is added to the original content, based on a user input selecting a first person from among the people appearing in the original content.

[0080] According to one embodiment, the extended content may be generated based on providing a plurality of texts included in the original content, a plurality of images included in the original content, and a prompt related to the first person to a generative AI model (e.g., the generative AI model (310) of FIG. 3). For example, the generative AI model may be stored in memory (230) or an external device (e.g., a server). For example, the prompt may include a command to expand (or add to) the description (or story) of the original story based on the emotion or gaze of the first person. The prompt may be generated by the electronic device (201).

[0081] In the following Figures 5 and 6, a method for an electronic device (201) to obtain extended content using a generative AI model (310) will be specifically described.

[0082] FIG. 5 is a flowchart illustrating a method for an electronic device to display extended content according to one embodiment.

[0083] Referring to FIG. 5, according to one embodiment, in operation 501, an electronic device (e.g., the electronic device (201) of FIG. 2) may, in response to a user input selecting a first person among the characters appearing in the original content of a book, determine whether extended content for the first person has been generated. For example, the user input selecting the first person may represent an input requesting that the original content of the book be displayed as extended content that is expanded based on the emotion or gaze of the first person.

[0084] According to one embodiment, in operation 503, the electronic device (201) may determine whether the extended content for the first person has been parasitized. For example, the electronic device (201) may determine whether the extended content has been parasitized based on whether the extended content, which is based on the first person among the characters appearing in the original content for the book, has been stored in the memory (230).

[0085] According to one embodiment, if it is determined that extended content for the first person has been generated (example of operation 503), in operation 505, the electronic device (201) may display the extended content for the first person through the display.

[0086] According to one embodiment, if it is determined that extended content for the first person has been generated (example of operation 503), in operation 507, the electronic device (201) may generate or obtain extended content using a generative AI model (e.g., the generative AI model (310) of FIG. 3). For example, if the generative AI model (310) is stored in a memory (e.g., the memory (230) of FIG. 2), the electronic device (201) may provide the generative AI model (310) with information about the original content and a prompt (e.g., a prompt for expanding the original content based on the emotion or gaze of the first person), thereby generating and obtaining the extended content. Alternatively, if the generative AI model (310) is stored in an external device (e.g., a server), the electronic device (201) may provide the external device with information about the original content and a prompt, thereby obtaining the extended content from the external device.

[0087] According to one embodiment, after acquiring the extended content, the electronic device (201) may display the extended content for the first person on the display (260).

[0088] FIG. 6 is a flowchart illustrating a method for an electronic device to obtain extended content according to one embodiment.

[0089] Referring to FIG. 6, according to one embodiment, in operation 601, an electronic device (e.g., electronic device (201) of FIG. 2) may generate a prompt related to a first person based on determining that extended content for the first person is not parasitic.

[0090] According to one embodiment, in operation 603, the electronic device (201) may provide text included in the original content, images included in the content, and prompts to a generative AI model (e.g., the generative AI model (310) of FIG. 3). For example, if the generative AI model (310) is stored in an external device (e.g., a server), the electronic device (201) may transmit the text included in the original content, images included in the content, and prompts to the external device. Depending on the implementation, the electronic device (201) may provide information about each page included in the original content and analysis information about the entire content or the entire story of the original content to the generative AI model (310).

[0091] According to one embodiment, in operation 605, the electronic device (201) may obtain extended content using the generative AI model (310). For example, the extended content may include content (e.g., text and / or images) that emphasizes or expands the emotions or perspectives of a first character among the characters appearing in the original content while maintaining the overall story of the original content.

[0092] Through the above-described method, the electronic device (201) can provide content that includes an expanded story based on a specific character among multiple characters appearing in the content of a book (e.g., a story that emphasizes the emotions of a specific character or a story expanded from the perspective or viewpoint of a specific character). Through this, the electronic device (201) can provide the user with an expanded story based on various perspectives of a specific character within the scope of the story of the content of the book.

[0093] FIGS. 7A through 7C are drawings of a user interface for an electronic device to display extended content, according to one embodiment.

[0094] Referring to FIG. 7A, according to one embodiment, an electronic device (e.g., the electronic device (201) of FIG. 2) may display a first screen (710) through a display (e.g., the display (260) of FIG. 2) based on confirming a command to display original content for a book. For example, the first screen (710) may include an image representing a cover of the original content. Thereafter, the electronic device (201) may display a second screen (720) through the display (260) including a first UI (722) for selecting a reading type. For example, the first UI (722) may include a first object (724) for selecting original content and a second object (726) for selecting extended content.

[0095] According to one embodiment, the electronic device (201) may display a third screen (730) through the display (260) based on a user input for the second object (726). For example, the third screen (730) may include a second UI (730) for selecting a specific person among the people appearing in the original content. For example, the second UI (722) may include objects (734, 735, 736) representing the people appearing in the original content. For example, the electronic device (201) may check whether extended content for the people appearing in the original content has been generated. If it is determined that extended content for a specific person has been generated, the electronic device (201) may display an object (e.g., 736) representing the person so as to be visually distinct from other objects (734, 735). For example, the electronic device (201) may display objects (734, 735) representing people for whom extended content has not been created by deactivating or blurring them.

[0096] According to one embodiment, the electronic device (201) may display a fourth screen (740) through the display (260) based on a user input for selecting an object (735) representing a first person among the people appearing in the original content. For example, the fourth screen (740) may display an object (e.g., a fairy object) selected by the user input so as to be visually distinct from other objects. For example, extended content for a person (e.g., a fairy) corresponding to the selected object (e.g., a fairy object) may be generated based on the user input. For example, if the extended content for the person (e.g., a fairy) corresponding to the selected object (e.g., a fairy object) has not been generated in advance before the user input is confirmed, the electronic device (201) may perform an operation of generating the extended content before displaying the extended content. Alternatively, if the extended content for a person (e.g., a fairy) corresponding to the selected object (e.g., a fairy object) has been previously generated before the user input is confirmed, the electronic device (201) may perform an operation for displaying the extended content without performing an operation for generating the extended content. The electronic device (201) may activate an object (744) for displaying (or playing) the extended content based on the selection of the object. For example, the activated object (744) may be displayed to be visually distinct compared to before being activated. The electronic device (201) may display a fifth screen (750) based on the user input for the object (744). For example, the fifth screen (750) may include an image indicating the display (or playing) of the extended content. For example, the electronic device (201) may display pages of extended content in a specified order based on user input (e.g., swipe input) to the fifth screen (750).

[0097] Referring to FIG. 7B, according to one embodiment, the electronic device (201) may confirm a user input for selecting a second object (735) corresponding to a second person on a third screen (730). For example, extended content for a person (e.g., a fairy) corresponding to the selected second object (735) may not be generated in advance before the user input is confirmed.

[0098] According to one embodiment, the electronic device (201) may display a fourth screen (740) through the display (260) based on a user input for selecting an object (735) representing a second person among the people appearing in the original content. For example, the fourth screen (740) may display an object (e.g., a fairy object) selected by the user input so as to be visually distinct from other objects. Thereafter, the electronic device (201) may display a sixth screen (760) based on a user input for an object (744) for displaying (or playing) extended content. For example, the sixth screen (760) may include an image indicating that extended content for the second person (e.g., a fairy) corresponding to the selected object is being generated. While displaying the sixth screen (760), the electronic device (201) may generate or obtain extended content for the second person using the generative AI model (310). When extended content for the second person is generated or acquired, the electronic device (201) may display a seventh screen (770). For example, the seventh screen (770) may include a message notifying that extended content for the second person is generated. In addition, the seventh screen (770) may include an object (775) for displaying (or playing) the extended content for the second person. The electronic device (201) may perform an operation of displaying (or playing) the extended content for the second person based on a user input for the object (775). For example, the electronic device (201) may display a fifth screen (750) as part of an operation of displaying (or playing) the extended content for the second person.

[0099] Referring to FIG. 7C, according to one embodiment, the electronic device (201) may display a fourth screen (740) through the display (260) based on a user input for selecting an object (735) representing a second person among the people appearing in the original content. For example, the fourth screen (740) may include a setting object (746) for adjusting setting values ​​for the second extended content. For example, when a specific person (e.g., the second person) is selected by the user input, the electronic device (201) may activate the setting object (746). The electronic device may display a setting screen (780) through the display based on the user input for the setting object (744). For example, the setting screen (780) may provide an interface (e.g., an adjustment bar) for adjusting setting values ​​for the extended content. For example, the settings screen (780) may provide an interface (781) for adjusting the correlation between the extended content and the original content, an interface (783) for adjusting the proportion of characters appearing in the extended content, an interface (785) for adjusting the overall mood (or atmosphere) of the extended content, and an interface (787) for adjusting the background of the extended content. In addition, the settings screen (780) may include an object (789) for applying the adjusted settings to the extended content. The electronic device (201) may generate or obtain extended content based on the corresponding settings based on a user input for the object (789). For example, the corresponding settings may be applied to the entire story or entire pages of the extended content. For example, if there is a change in the settings, the electronic device (201) may newly generate or obtain extended content based on the corresponding settings. Alternatively, the electronic device (201) may not newly create or acquire extended content based on the settings if the extended content based on the settings is pre-stored and the settings have not been changed.

[0100] FIG. 8 is a drawing for explaining extended content provided by an electronic device according to one embodiment.

[0101] Referring to FIG. 8, according to an embodiment, the story of the extended content may be determined based on the story of the original content. For example, the original content may include pages corresponding to a designated timeline (810). The electronic device (201) may sequentially display (or play) the pages of the original content according to the designated timeline (810). The timeline of the extended content may be determined based on the timeline (810) designated in the original content. Additionally, the pages included in the extended content may include pages based on the determined timeline. For example, the number of pages may be the same as the number of pages included in the original content.

[0102] According to one embodiment, the extended content for a first person (e.g., a fairy) may include a story corresponding to the first person's perspective. The story corresponding to the first person's perspective may be composed of pages related to the first person based on a determined timeline. In this case, pages related to the first person among the pages included in the original content may be included in the extended content. In this case, the pages related to the first person included in the extended content may further include additional text or additional images to emphasize the emotions or perspectives of the first person. In addition, pages included in the original content that are not related to the first person may be excluded from the extended content. In addition, a newly created page centered on the first person or a page in which a portion of an existing page is edited centered on the first person may be added to the extended content instead of the excluded page.

[0103] Similarly, in one embodiment, extended content for a second character (e.g., a lumberjack) and extended content for a third character (e.g., a deer) may also include stories corresponding to the perspectives of those characters. These stories corresponding to the perspectives of those characters may be structured in a manner identical to or similar to the method used to structure the story for the first character described above.

[0104] FIGS. 9A and 9B are diagrams of prompts applied to the entire story of original content, according to one embodiment.

[0105] Referring to FIG. 9a, according to one embodiment, an electronic device (e.g., electronic device (201) of FIG. 2) may provide an interface (e.g., setting screen (780) of FIG. 7c) for adjusting settings for extended content.

[0106] According to one embodiment, the electronic device (201) may generate a prompt (910) for generating extended content for a specific person (hereinafter, referred to as a first person) based on setting values ​​indicating a relationship between the extended content and the original content, the proportion of other people in the extended content other than the specific person, the mood of the extended content, and the background of the extended content. For example, the setting values ​​may be set to preset default values. The electronic device (201) may check the adjusted setting values ​​based on a user input to the interface. The electronic device (201) may generate the prompt (910) based on the setting values. For example, the prompt (910) may include at least one command (e.g., a phrase including texts) to be applied to the entire story or entire page of the extended content.

[0107] Referring to FIG. 9B, according to one embodiment, when settings are adjusted, the prompt (910) (or the text included in the prompt) may change. For example, as settings are adjusted, some of the content of an existing prompt may change. Additionally, as settings are adjusted, new prompts may be added or some of the content of an existing prompt may be deleted.

[0108] According to one embodiment, when a setting value related to a "mood" is changed (e.g., a dynamic value increases), the electronic device (201) may generate a prompt (920) based on the setting value related to the changed "mood." Additionally, when a setting value related to a "background" is changed (e.g., a background country is changed to the United States), the electronic device (201) may generate a prompt (930) based on the setting value related to the changed "background." Prompts (920 and 930) may be applied to the entire story or entire pages of the extended content. For example, pages included in the extended content may include texts expressing dynamic emotions. For example, pages included in the extended content may include images representing the United States, and the races and clothing of characters appearing in the extended content may also be adjusted to represent the United States.

[0109] Meanwhile, the prompts illustrated in FIGS. 9a and 9b are exemplary, and the technical idea of ​​the present invention may not be limited thereto.

[0110] FIGS. 10A and 10B are diagrams of prompts applied to each page of original content, according to one embodiment.

[0111] Referring to FIGS. 10A and 10B , according to one embodiment, an electronic device (e.g., electronic device (201) of FIG. 2 ) may generate a prompt that applies to each page of the original content depending on whether the page includes a description (e.g., text and / or image) of a first person.

[0112] Referring to FIG. 10A, according to one embodiment, the electronic device (201) may identify the page as the first case if the page includes both text and an image of the first person (or if the first person appears in both the text and the image). The electronic device (201) may identify the page as the second case if the page includes only text of the first person (or if the first person appears only in the text). The electronic device (201) may identify the page as the third case if the page includes only an image of the first person (or if the first person appears only in the image). The electronic device (201) may identify the page as the fourth case if the page does not include both text and an image of the first person (or if the first person does not appear in the text and the image).

[0113] According to the above-described method, the electronic device (201) can generate a prompt that applies to the entire story (or entire page) of the extended content and provide the generated prompt to the generative AI model (310).

[0114] Referring to FIG. 10b, according to one embodiment, the electronic device (201) may generate a prompt to enhance the emotional fingerprint of the first person based on identifying the page as the first case. The generated prompt may be applied only to the page. For example, the page of extended content may further include a fingerprint (or text) that emphasizes the emotional fingerprint of the first person.

[0115] In one embodiment, the electronic device (201) may generate a prompt to generate a picture (or image) of the first person based on identifying the page as a second case. The generated prompt may be applied only to the page. For example, the corresponding page of the extended content may further include a new image including the first person. For example, the new image may be implemented as an image that further includes an object representing the first person in the existing image. Alternatively, the new image may replace the existing image on the page.

[0116] In one embodiment, the electronic device (201) may generate a prompt to add (or generate) a fingerprint of the first person based on identifying the page as a third case. The generated prompt may be applied only to the page. For example, the page of extended content may further include a fingerprint (or text) of the first person.

[0117] In one embodiment, the electronic device (201) may generate a prompt to create a new story and picture (or image) for the first person based on identifying the page as the fourth case. The generated prompt may then replace the page with the newly created page. For example, the expanded content may include a newly created page instead of an existing page of the original content. Additionally, the newly created page may include the story and picture of the first person.

[0118] According to the above-described method, the electronic device (201) can generate a prompt applicable to each page of the extended content and provide the generated prompt to the generative AI model (310).

[0119] Through the above-described method, the electronic device (201) can provide content that includes an expanded story based on a specific character among multiple characters appearing in the content of a book (e.g., a story that emphasizes the emotions of a specific character or a story expanded from the perspective or viewpoint of a specific character). Through this, the electronic device (201) can provide the user with an expanded story based on various perspectives of a specific character within the scope of the story of the content of the book.

[0120] FIGS. 11A to 11D are drawings for explaining a method of generating a first page included in extended content depending on whether content about a first person appears on the first page included in the original content, according to one embodiment.

[0121] Referring to FIGS. 11A to 11D , according to an embodiment, an electronic device (e.g., the electronic device (201) of FIG. 2 ) can obtain extended content, in which the original content of a book is extended for a first person, by using a generative AI model (e.g., the generative AI model (310) of FIG. 3 ).

[0122] Referring to FIG. 11a, according to one embodiment, if the first page (1110) of the original content includes a first text (1115) related to a first person (e.g., a fairy) and a first image related to the first person, the first page (1120) of the extended content may include the first image, the first text (1115), and additional text (1125). For example, the additional text may include a fingerprint (or description) of the first person's emotions. For example, the extended content may correspond to the first case of FIGS. 10a and 10b.

[0123] Referring to FIG. 11B, in one embodiment, when a second page (1130) of the original content includes a second text (1135) related to a first person (e.g., a fairy) and a second image not related to the first person, a second page (1140) of the extended content may include an image including an additional image (1142) of the first person (e.g., a fairy) in addition to the existing second image, a second text (1135), and additional text (1145). For example, the additional text may include a fingerprint (or description) of the emotion of the first person. For example, the additional image may include an image representing the first person. Depending on the implementation, the second page (1140) of the extended content may also include a new additional image (e.g., an image featuring the first person) that is completely different from the existing second image. For example, the extended content may correspond to the second case of FIGS. 10A and 10B.

[0124] Referring to FIG. 11c, according to one embodiment, if the third page (1150) of the original content includes third text (1155) unrelated to the first person (e.g., a fairy) and a third image related to the first person, the third page (1160) of the extended content may include the third image, the third text (1155), and additional text (1165). For example, the additional text may include a fingerprint of the first person's emotions. Alternatively, the additional text may include an additional description of the first person. For example, the extended content may correspond to the third case of FIGS. 10a and 10b.

[0125] Referring to FIG. 11d, in one embodiment, when the fourth page (1170) of the original content includes text unrelated to the first person (e.g., a fairy) and a fourth image unrelated to the first person, the fourth page (1180) of the extended content may include new text (1185) and a new image about the first person (e.g., a fairy) that are completely different from the existing text and the fourth image. The new text (1185) may include a description or a fingerprint of the emotion of the first person. For example, the new image may include an image about the first person that matches the new text (1185). For example, the extended content may correspond to the fourth case of FIGS. 10a and 10b.

[0126] Although the texts (1115, 1125, 1135, 1145, 1155, 1165, and 1185) in FIGS. 11A through 11D are shown separately from the images (1120, 1140, 1160, and 1180) of the extended content, the texts (1115, 1125, 1135, 1145, 1155, 1165, and 1185) may be displayed together with the images (1120, 1140, 1160, and 1180). For example, the texts (1115, 1125, 1135, 1145, 1155, 1165, and 1185) may be displayed over the images (1120, 1140, 1160, and 1180).

[0127] Meanwhile, the text or images illustrated in FIGS. 11a to 11d are exemplary, and the technical ideas of the present invention may not be limited thereto.

[0128] FIGS. 12A and 12B are drawings for explaining a method of generating an image included in extended content according to one embodiment.

[0129] Referring to FIG. 12A, according to an embodiment, the electronic device (201) may analyze the original content. For example, the electronic device (201) may analyze or confirm information about a character and background appearing on each page included in the original content. For example, the electronic device (201) may confirm information about a character (e.g., a fairy) appearing on a first page (1210) and the character's clothing (e.g., human clothing). In addition, the electronic device (201) may confirm an object (1215) representing the character on the first page (1210). For example, the electronic device (201) may confirm a character (e.g., a fairy) appearing on a second page (1220) and the character's clothing (e.g., heavenly clothing). In addition, the electronic device (201) may confirm an object (1225) representing the character on the second page (1220). For example, the electronic device (201) can identify a background (e.g., heaven) appearing on the third page (1230) and an object (1235) representing the background (e.g., heaven palace).

[0130] Referring to FIG. 12B, according to one embodiment, the electronic device (201) may provide analyzed information (e.g., information about a person (e.g., a fairy), the person's clothing (e.g., a heavenly clothing), and a background (e.g., a heavenly country)) to the generative AI model (310) when generating an image representing a new story. In addition, the electronic device (201) may provide information about an object (1225) representing the person and an object (1235) representing the background to the generative AI model (310). In addition, the electronic device (201) may generate a prompt related to the analyzed information and provide the generated prompt to the generative AI model (310).

[0131] In one embodiment, an image representing a new story may be generated based on an object (1225) representing a character and an object (1235) representing a background. For example, if the new story is about a fairy in heaven, the image included in the extended content may include images identical or similar to the object (1225) representing a character and the object (1235) representing a background.

[0132] In one embodiment, text representing a new story may be generated based on information describing the character (e.g., clothing information) and information describing the background. For example, if the new story is about a fairy in heaven, the text included in the extended content may include a description describing the fairy and a description describing the background of heaven.

[0133] Through the above-described method, the electronic device (201) can obtain and provide extended content including a story that matches the story of the original content.

[0134] FIG. 13 is a diagram illustrating a method for generating an image included in an extended image according to one embodiment.

[0135] Referring to FIG. 13, according to one embodiment, a first additional image (1315) newly created on one page of extended content may be used on other subsequent pages. For example, the first additional image (1315) included in the newly created first page (1310) may be reflected on the second page (1320). For example, the first additional image (1315) representing the "hair decoration" of the fairy included in the first page (1310) may be reflected on the "head" of the fairy included in the second page (1320). That is, the second additional image (1325) corresponding to the first additional image (1315) may be added to the "head" of the fairy included in the second page (1320).

[0136] Through the above-described method, the electronic device (201) can obtain and provide a new story that matches the entire story when generating extended content.

[0137] FIG. 14 is a drawing for explaining extended content provided by an electronic device according to one embodiment.

[0138] Referring to FIG. 14, according to one embodiment, the story of the extended content may be determined based on the story of the original content. For example, the extended content may include a page in which a portion of the original content page has been edited and / or a newly created page based on the emotions or gaze of the person in question. For example, the extended content may include a page in which a portion of an image included in a page in the original content has been edited and / or an image on that page has been replaced with a new image.

[0139] According to one embodiment, the original content may include a first page (1410), a second page (1420), a third page (1430), and a fourth page (1440). For example, the electronic device (201) may sequentially display the first page (1410), the second page (1420), the third page (1430), and the fourth page (1440) based on a user input (e.g., a swipe input).

[0140] According to one embodiment, an expanded page for a first character (e.g., a lumberjack) may include a first page (1410), a second page (1425), a third page (1435), and a fourth page (1440). For example, the second page (1425) may include an image in which a portion of an image included in the second page (1420) of the original content is edited. For example, the second page (1425) of the expanded content may further include an additional image (e.g., an image representing the lumberjack) in addition to the image included in the second page (1420) of the original content. Similarly, the third page (1435) may include an image in which a portion of an image included in the third page (1430) of the original content is edited. For example, the second page (1425) and the third page (1435) may correspond to the second case of FIGS. 10A and 10B . The electronic device (201) can sequentially display a first page (1410), a second page (1425), a third page (1435), and a fourth page (1440) based on a user input (e.g., a swipe input).

[0141] According to one embodiment, an expanded page for a second character (e.g., a fairy) may include a first page (1415), a second page (1420), a third page (1430), and a fourth page (1445). For example, the first page (1415) may include an image that is completely different from the image included in the first page (1410) of the original content (e.g., an image generated based on the second character's perspective). For example, the first page (1415) of the expanded content may include a new image that includes the second character. Similarly, the fourth page (1445) may include an image that is completely different from the image included in the fourth page (1440) of the original content (e.g., an image generated based on the second character's perspective). For example, the first page (1415) and the fourth page (1445) may correspond to the fourth case of FIGS. 10A and 10B . The electronic device (201) can sequentially display a first page (1415), a second page (1425), a third page (1435), and a fourth page (1445) based on a user input (e.g., a swipe input).

[0142] According to one embodiment, an expanded page for a third person (e.g., a deer) may include a first page (1410), a second page (1427), a third page (1437), and a fourth page (1447). For example, the second page (1427) may include an image in which a portion of an image included in the second page (1420) of the original content is edited. For example, the second page (1427) of the expanded content may further include an additional image (e.g., an image representing a deer) in addition to the image included in the second page (1420) of the original content. For example, the third page (1437) may include an image that is completely different from the image included in the third page (1430) of the original content (e.g., an image generated based on the third person's perspective). For example, the third page (1437) of the expanded content may include a new image including the third person. Similarly, the fourth page (1447) may include an image that is completely different from the image included in the fourth page (1440) of the original content (e.g., an image generated based on the perspective of a third person). For example, the second page (1427) may correspond to the second case of FIGS. 10A and 10B . In addition, the third page (1437) and the fourth page (1447) may correspond to the fourth case of FIGS. 10A and 10B . The electronic device (201) may sequentially display the first page (1410), the second page (1427), the third page (1437), and the fourth page (1447) based on a user input (e.g., a swipe input).

[0143] According to the above-described method, the electronic device (201) can provide the user with a story based on the perspective of the corresponding person as extended content.

[0144] FIGS. 15A to 15C are drawings illustrating a method for an electronic device to display extended content according to one embodiment.

[0145] Referring to FIGS. 15A to 15C , according to one embodiment, when a page (1510) included in extended content is newly created, the page (1510) may include an object (1515) to indicate that the page (1510) is a page that did not exist in the original content. For example, when an input for the object (1515) is confirmed, the electronic device (201) may display a message indicating that the page (1510) is a page that did not exist in the original content. Alternatively, when an input for the object (1515) is confirmed, the electronic device (201) may display a message indicating that the page (1510) is newly created by a generative AI model (e.g., the generative AI model (310) of FIG. 3).

[0146] Referring to FIG. 15A, according to one embodiment, an electronic device (e.g., the electronic device (201) of FIG. 2) may display a next page or a previous page included in the expanded content based on a horizontal swipe input. For example, the electronic device (201) may display a next page (1520) based on a currently displayed page (1510) based on a rightward swipe input. Alternatively, the electronic device (201) may display a previous page (1510) based on a currently displayed page (1520) based on a leftward swipe input.

[0147] Referring to FIG. 15B, according to one embodiment, the electronic device (201) may display a page included in the original content based on a vertical swipe input. For example, the electronic device (201) may display a page (1530) of the original content in an order corresponding to the currently displayed page (1510) of the expanded content based on an upward swipe input. Thereafter, the electronic device (201) may display the page (1510) of the expanded content again based on a downward swipe input.

[0148] Referring to FIG. 15C, according to one embodiment, the electronic device (201) may visually distinguishably display a specific object (e.g., a star-shaped object) included in a page (1510) of extended content based on an input (e.g., a touch input or a touch gesture) for the object. For example, the electronic device (201) may display a page (1540) in which the specific object is displayed as blinking. Alternatively, the electronic device (201) may display a page (1540) in which the specific object moves to a designated location or an animation effect is reflected for the specific object.

[0149] FIG. 16 is a drawing for explaining additional images included in extended content according to one embodiment.

[0150] Referring to FIG. 16, according to one embodiment, the extended content may include additional images based on the perspective of a specific person. For example, the additional images may include images representing objects viewed by the specific person.

[0151] According to one embodiment, the extended content for a first character (e.g., a lumberjack) may include a first page and a second page. For example, the first page may include a first image (1610). The second page may include a second image (1620). The electronic device (201) may sequentially display the first page and the second page based on a user input (e.g., a swipe input).

[0152] According to one embodiment, the second image (1620) may be a newly generated image by a generative AI model (e.g., the generative AI model (310) of FIG. 3). For example, the second image (1620) may include an image corresponding to a scene viewed by the first person. That is, the electronic device (201) may display not only an image in which the first person appears, but also an image corresponding to a scene viewed by the first person as a newly generated image (or additional image).

[0153] According to one embodiment, the electronic device (201) may generate a prompt to generate an image corresponding to a scene viewed by the first person. The electronic device (201) may provide the prompt to a generative AI model (310) when generating or obtaining extended content. The electronic device (201) may use the generative AI model (310) to obtain extended content including an image corresponding to the scene viewed by the first person.

[0154] According to one embodiment, the electronic device (201) may provide a theatrical function. For example, the electronic device (201) may receive a speech corresponding to a line (or text) to be delivered from the user to an object viewed by the first person while displaying a second image. Based on the speech received from the user to the object viewed by the first person, the electronic device (201) may output the line of the corresponding object as a voice. Through the above-described method, the electronic device (201) may provide a theatrical function related to a book.

[0155] The generative artificial intelligence system described in Figure 17 below can be applied identically or similarly to the generative AI model (310) described above.

[0156] Fig. 17 is a diagram for explaining a generative artificial intelligence system according to one embodiment.

[0157] According to one embodiment, a user query / response interface (1710) may receive a user's input. The user's input may be in the form of natural language, images, and / or videos, but is not limited thereto. Furthermore, context information may also be transmitted when the user's input is transmitted. The context information may include various additional information at the time of the user's input. For example, the additional information may include information about the application currently being used by the user or information about the user's location. Furthermore, the user's input may be in a mixed form of the aforementioned natural language, images, sounds, and context information. Furthermore, the user's input may also be in a non-natural language form, such as selecting a menu. The user query / response interface (1710) may output the results of the generative artificial intelligence system to the user. The output may be in the form of natural language or specific content, and may also be provided in the form of an action requested by the user. The user query / response interface (1710) may output the results of the generative artificial intelligence system to the user. The output can be in natural language form, in the form of specific content, or in the form of actions requested by the user.

[0158] The AI ​​framework (1740) can receive user input and coordinate and control each component necessary to perform the user's intention based on the user's query.

[0159] User input received from the user query / response interface (1710) can be transmitted to a prompt design component (1741). The prompt design component (1741) can be used to generate prompts suitable for inputting user input into a large language model (LLM) or a large multimodal model (LMM). The prompt design component (1741) can be an AI component that uses a machine learning algorithm or a neural network to develop better prompts over time. The prompt design component (1741) can access a knowledge component including user preference data, a prompt library, and prompt examples based on the user input to generate prompts, and transmit the generated prompts to the LLM or LMM.

[0160] The API / Plug-in management component (1742) can communicate with external information when there is a request for additional information when passing user input as input to the generative model. The API / Plug-in management component (1742) can establish a channel for communicating with the outside of the AI ​​Interface through the API, and can enable access to various data sources (e.g., knowledge repositories (1720)) through the established channel. In addition, the API / Plug-in management component (1742) can request the application / service component (1730) through the API for an action that ultimately performs the user input, rather than an intermediate result, when the action needs to be performed in the application or service. Information obtained from the outside can be used to generate a prompt in the prompt design component (1741) together with the user input, or can be passed as an input to the generative model.

[0161] The output modification component (also called a refiner component) (1743) can fine-tune the output from the generative model. For example, the output modification component (1743) can verify that the content generated through the LLM and / or LMM is not irrelevant, does not contain biased content, or does not contain harmful content. In addition, the output modification component (1743) can determine to what extent it matches the result desired by the user and, if necessary, can perform additional processing. The output modification component (1743) can additionally configure and provide the user with hints to avoid unwanted output.

[0162] A generative AI model (1760) can generally refer to an artificial intelligence neural network that creates new types of data based on user input information. A generative AI model (1760) can include an image-generating model and / or a language-generating model. Representative models for generating images include a generative adversarial network (GAN) and a variational autoencoder (VAE), and examples include a VAE and a Diffusion-based generative model that uses a Transformer structure. A language-generating model is a model trained to statistically output the most appropriate output based on input values, and representative examples include models such as CHAT-GPT 3 and CHAT-GPT 4. In addition, there are also large multimodal models (LMMs) that can recognize various types of data input, such as text, images, and voice, and generate new data corresponding to them.

[0163] According to one embodiment, the electronic device (201) may include a display (260), at least one processor (220), and a memory (130, 230) including instructions. According to one embodiment, the instructions, when executed by the at least one processor, may cause the electronic device to identify a command to display content for a book including texts and images. According to one embodiment, the instructions, when executed by the at least one processor, may cause the electronic device to display a menu on the display for expanding the story of the content centered on a specific character among characters appearing in the content, based on identifying the command. In one embodiment, the instructions, when executed by the at least one processor, may cause the electronic device to display, on the display, extended content for the first person, wherein at least one of additional text or additional image related to the first person is added to the content, based on a user input of selecting the first person from the menu. In one embodiment, the extended content may be generated based on providing the plurality of texts included in the content, the plurality of images included in the content, and a prompt related to the first person to a generative AI model (310).

[0164] In one embodiment, the instructions, when executed by the at least one processor, may cause the electronic device to determine, in response to the user input, whether the extended content for the first person has been generated parasitic. In one embodiment, the instructions, when executed by the at least one processor, may cause the electronic device to generate a prompt related to the first person based on determining that the extended content has not been generated parasitic. In one embodiment, the instructions, when executed by the at least one processor, may cause the electronic device to obtain the extended content based on providing the prompt to the generative AI model.

[0165] According to one embodiment, the instructions, when executed by the at least one processor, may cause the electronic device to generate a prompt to emphasize the emotion of the first person or generate the extended content based on the viewpoint of the first person, based on the settings of the extended content indicating the relevance of the content, the proportion of characters other than the first person, the mood, and the background.

[0166] In one embodiment, the instructions, when executed by the at least one processor, may cause the electronic device to generate the prompt to generate the additional text that emphasizes the emotion of the first person or to generate the additional image based on a viewpoint of the first person.

[0167] In one embodiment, if the first page of the content includes a first text related to the first person and a first image related to the first person, the first page of the extended content may include the first image, the first text, and the additional text.

[0168] In one embodiment, if the first page of the content includes a first text related to the first person and does not include a first image related to the first person, the first page of the extended content may include a second image related to the first person generated by the generative AI model, the first text, and the additional text.

[0169] In one embodiment, the second image may be generated by adding an additional image related to the first person to an image included in the first page of content.

[0170] In one embodiment, if the first page of the content does not include first text related to the first person but includes a first image related to the first person, the first page of the extended content may include additional text related to the first person and the first image generated by the generative AI model.

[0171] In one embodiment, if the first page of the content does not include any text and images related to the first person, the first page of the extended content may include additional text and additional images related to the first person generated by the generative AI model.

[0172] According to one embodiment, the content may include content for an electronic-book.

[0173] According to one embodiment, a method of operating an electronic device (201) may include an operation of confirming a command to display content for a book including texts and images. According to one embodiment, the method of operating the electronic device may include an operation of displaying a menu for expanding a story of the content centered on a specific character among characters appearing in the content on a display (260) included in the electronic device based on confirming the command. According to one embodiment, the method of operating the electronic device may include an operation of displaying, on the display, expanded content for the first character in which at least one of additional text or additional image related to the first character is added to the content based on a user input of selecting the first character from the menu. According to one embodiment, the expanded content may be generated based on providing the plurality of texts included in the content, the plurality of images included in the content, and a prompt related to the first character to a generative AI model (310).

[0174] According to one embodiment, the method of operating the electronic device may further include an operation of determining, in response to the user input, whether the extended content for the first person has been generated parasitically. According to one embodiment, the method of operating the electronic device may further include an operation of generating a prompt related to the first person based on determining that the extended content has not been generated parasitically. According to one embodiment, the method of operating the electronic device may further include an operation of obtaining the extended content based on providing the prompt to the generative AI model.

[0175] According to one embodiment, the action of generating the prompt may include an action of generating a prompt that emphasizes the emotion of the first person or generates the extended content based on the viewpoint of the first person, based on the setting values ​​of the extended content indicating the relevance to the content, the proportion of characters other than the first person, the mood, and the background.

[0176] In one embodiment, the act of generating the prompt may include generating the prompt to generate additional text that emphasizes the emotion of the first person or to generate the additional image based on the viewpoint of the first person.

[0177] In one embodiment, the instructions, when executed by the at least one processor, may cause the electronic device to generate the prompt to generate the additional text that emphasizes the emotion of the first person or to generate the additional image based on a viewpoint of the first person.

[0178] In one embodiment, if the first page of the content includes a first text related to the first person and a first image related to the first person, the first page of the extended content may include the first image, the first text, and the additional text.

[0179] In one embodiment, if the first page of the content includes a first text related to the first person and does not include a first image related to the first person, the first page of the extended content may include a second image related to the first person generated by the generative AI model, the first text, and the additional text.

[0180] In one embodiment, the second image may be generated by adding an additional image related to the first person to an image included in the first page of content.

[0181] In one embodiment, if the first page of the content does not include first text related to the first person but includes a first image related to the first person, the first page of the extended content may include additional text related to the first person and the first image generated by the generative AI model.

[0182] In one embodiment, if the first page of the content does not include any text and images related to the first person, the first page of the extended content may include additional text and additional images related to the first person generated by the generative AI model.

[0183] According to one embodiment, a computer-readable, non-transitory recording medium (130, 230) may store a program that, when executed by at least one processor (220) included in an electronic device (201), causes the electronic device to perform the following actions: confirming a command to display content for a book including texts and images; displaying, on a display (260) included in the electronic device, a menu for expanding a story of the content centered on a specific person among characters appearing in the content based on the confirmation of the command; and displaying, on the display, expanded content for a first person in which at least one of additional text or additional image related to the first person is added to the content based on a user input selecting the first person from the menu. According to one embodiment, the expanded content may be generated based on providing a generative AI model (310) with the plurality of texts included in the content, the plurality of images included in the content, and a prompt related to the first person.

[0184] The embodiments of this document and the terminology used herein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another component (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0185] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0186] Various embodiments of the present document may be implemented as software (e.g., a program) including one or more instructions stored in a storage medium (e.g., built-in memory or external memory) readable by a machine (e.g., an electronic device). For example, a processor (e.g., a processor) of the machine (e.g., an electronic device) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one instruction called. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' only means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and this term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily in the storage medium.

[0187] According to one embodiment, the method according to various embodiments of the present disclosure may be provided as a computer program product. The computer program product may be traded between sellers and buyers as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or may be provided through an application store (e.g., Play Store). TM ) or directly between two user devices (e.g., smartphones), online distribution (e.g., downloading or uploading). In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily created in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0188] According to embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to embodiments, operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In the electronic device (201), display (260); At least one processor (220); and A memory (230) including instructions, wherein the instructions, when executed by the at least one processor, cause the electronic device to: Check the command to display content for a book containing text and images, Based on the confirmation of the above command, a menu for expanding the story of the above content centered on a specific character among the characters appearing in the above content is displayed on the above display, Based on a user input of selecting a first person from the above menu, display extended content for the first person, wherein at least one of additional text or additional image related to the first person is added to the content, on the display; An electronic device characterized in that the extended content is generated based on providing the plurality of texts included in the content, the plurality of images included in the content, and a prompt related to the first person to a generative AI model (310).

2. In the first paragraph, the instructions, when executed by the at least one processor, cause the electronic device to: In response to the user input, determine whether the extended content for the first person has been generated, Based on the determination that the above extended content is not parasitic, generate a prompt related to the first person, An electronic device that obtains the extended content based on providing the above prompt to the generative AI model.

3. In any one of paragraphs 1 to 2, the instructions, when executed by the at least one processor, cause the electronic device to: An electronic device that generates a prompt to emphasize the emotions of the first person or generate the extended content based on the viewpoint of the first person, based on the settings of the extended content indicating the relevance to the content, the weight, atmosphere, and background of characters other than the first person.

4. In any one of paragraphs 1 to 3, the instructions, when executed by the at least one processor, cause the electronic device to: An electronic device that generates a prompt to generate said additional text that emphasizes the emotions of said first person or to generate said additional image based on the viewpoint of said first person.

5. In any one of paragraphs 1 to 4, An electronic device in which a first page of the content includes a first text related to the first person and a first image related to the first person, and a first page of the extended content includes the first image, the first text, and the additional text.

6. In any one of paragraphs 1 to 5, An electronic device in which the first page of the content includes a first text related to the first person and does not include a first image related to the first person, and the first page of the extended content includes a second image related to the first person generated by the generative AI model, the first text, and the additional text.

7. In any one of paragraphs 1 to 6, An electronic device characterized in that the second image is generated by adding an additional image related to the first person to the image included in the first page of content.

8. In any one of paragraphs 1 to 7, An electronic device in which the first page of the content does not include first text related to the first person but includes a first image related to the first person, and the first page of the extended content includes additional text related to the first person generated by the generative AI model and the first image.

9. In any one of paragraphs 1 to 8, An electronic device wherein, if the first page of the content does not include any text and images related to the first person, the first page of the extended content includes additional text and additional images related to the first person generated by the generative AI model.

10. In any one of paragraphs 1 to 9, The above content is an electronic device including content for an electronic-book.

11. In the operating method of an electronic device (201), An action that verifies the command to display content for a book containing text and images; Based on confirming the above command, an action of displaying a menu for expanding the story of the content centered on a specific character among the characters appearing in the content on the display (260) included in the electronic device; and Based on a user input of selecting a first person from the above menu, an action is included to display on the display extended content for the first person, wherein at least one of additional text or additional image related to the first person is added to the content. An operating method of an electronic device, characterized in that the extended content is generated based on providing the plurality of texts included in the content, the plurality of images included in the content, and a prompt related to the first person to a generative AI model (310).

12. In paragraph 11, In response to the user input, an action of determining whether the extended content for the first person has been parasitized; An action of generating a prompt related to the first person based on determining that the above extended content is not parasitic; and A method of operating an electronic device further comprising an action of obtaining said extended content based on providing said prompt to said generative AI model.

13. In any one of paragraphs 11 to 12, the action of generating the prompt comprises: An operating method of an electronic device, comprising an action of generating a prompt to emphasize the emotion of the first person or to generate the extended content based on the viewpoint of the first person, based on the setting values ​​of the extended content indicating the relevance of the content, the proportion of characters other than the first person, the atmosphere, and the background.

14. In any one of paragraphs 11 to 13, the action of generating the prompt comprises: A method of operating an electronic device comprising generating a prompt to generate additional text that emphasizes the emotion of the first person or to generate an additional image based on the viewpoint of the first person.

15. In any one of paragraphs 11 to 14, A method of operating an electronic device, wherein a first page of the content includes a first text related to the first person and a first image related to the first person, and a first page of the extended content includes the first image, the first text, and the additional text.

Images