Electronic device for generating video, operating method thereof, and storage medium

Abstract

According to an embodiment, an electronic device may comprise a display, at least one processor, and a memory configured to store instructions. According to an embodiment, the instructions, when executed individually or collectively by the at least one processor, may cause the electronic device to identify at least one first image for generating a video among a plurality of images stored in the electronic device. According to an embodiment, the instructions, when executed individually or collectively by the at least one processor, may cause the electronic device to identify at least one object included in the at least one first image and metadata of the at least one first image. According to an embodiment, the instructions, when executed individually or collectively by the at least one processor, may cause the electronic device to determine a character and a main character of the video on the basis of at least one of an area corresponding to the at least one object or the metadata. According to an embodiment, the instructions, when executed individually or collectively by the at least one processor, may cause the electronic device to acquire a first text describing the at least one first image by image captioning the at least one first image. According to an embodiment, the instructions, when executed individually or collectively by the at least one processor, may cause the electronic device to acquire a plurality of responses to the at least one first image on the basis of a plurality of predesignated queries for generating a story of the video. According to an embodiment, the instructions, when executed individually or collectively by the at least one processor, may cause the electronic device to acquire a text prompt for generating a story of the video by using the information about the character and the main character, the metadata, the first text, and the plurality of responses. According to an embodiment, the instructions, when executed individually or collectively by the at least one processor, may cause the electronic device to acquire the story of the video on the basis of providing the text prompt to a first generative artificial intelligence model. According to an embodiment, the instructions, when executed individually or collectively by the at least one processor, may cause the electronic device to acquire the video on the basis of providing the story and the at least one first image to a second generative artificial intelligence model.

Description

Electronic device for generating video, method of operation thereof, and storage medium

[0001] Embodiments of the present disclosure relate to an electronic device for generating video, a method of operating the same, and a storage medium.

[0002] Driven by the remarkable advancements in information and communication technology and semiconductor technology, the distribution and use of various electronic devices are increasing rapidly. Electronic devices are being developed to allow users to carry them around and communicate. The term "electronic device" may refer to a device that performs specific functions according to an installed program, such as mobile communication terminals, tablet PCs (personal computers), wearable electronic devices, video / audio devices, desktop / laptop computers, or vehicle navigation systems.

[0003] The information described above may be provided as related art for the purpose of aiding understanding of this document. None of the foregoing is to be claimed as prior art related to this document, nor is it to be used to determine prior art.

[0004] According to one embodiment, the electronic device may include a display, at least one processor, and a memory for storing instructions. According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may cause at least one first image to be identified among a plurality of images stored in the electronic device for creating a video. According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may cause at least one object included in the at least one first image and metadata of the at least one first image. According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may cause a character and a protagonist of the video to be determined based on at least one of the area corresponding to the at least one object or the metadata. According to one embodiment, the instructions may cause the electronic device to obtain a first text describing the at least one first image by image captioning the at least one first image when executed individually or collectively by the at least one processor. According to one embodiment, the instructions may cause the electronic device to obtain a plurality of responses to the at least one first image based on a plurality of predetermined queries for generating a story of the video when executed individually or collectively by the at least one processor.According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be caused to obtain a text prompt for generating a story of the video using information about the characters and the protagonist, the metadata, the first text, and the plurality of responses. According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be caused to obtain the story of the video based on providing the text prompt to a first generative artificial intelligence model. According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be caused to obtain the video based on providing the story and the at least one first image to a second generative artificial intelligence model.

[0005] According to one embodiment, a method of operating an electronic device may include an operation of identifying at least one first image for generating a video among a plurality of images stored in the electronic device based on user input. According to one embodiment, a method of operating an electronic device may include an operation of identifying at least one object included in the at least one first image and metadata of the at least one first image. According to one embodiment, a method of operating an electronic device may include an operation of determining a character and a protagonist of the video based on an area corresponding to the at least one object among the at least one first image or at least one of the metadata. According to one embodiment, a method of operating an electronic device may include an operation of obtaining a first text describing the at least one first image by image captioning the at least one first image. According to one embodiment, a method of operating an electronic device may include an operation of obtaining a plurality of responses to the at least one first image based on a plurality of predetermined queries for generating a story of the video. According to one embodiment, a method of operation of an electronic device may include an operation of obtaining a text prompt for generating a story of the video using information about the characters and the protagonist, the metadata, the first text, and the plurality of responses. According to one embodiment, a method of operation of an electronic device may include an operation of obtaining the story of the video based on providing the text prompt to a first generative artificial intelligence model. According to one embodiment, a method of operation of an electronic device may include an operation of obtaining the video based on providing the story and at least one first image to a second generative artificial intelligence model.

[0006] According to one embodiment, in a non-transient storage medium storing computer-readable instructions, the instructions cause the electronic device to perform at least one operation when executed by at least one processor of the electronic device, and the at least one operation is, based on user input, an operation of identifying at least one first image for generating a video among a plurality of images stored in the electronic device; an operation of identifying at least one object included in the at least one first image and metadata of the at least one first image; an operation of determining a character and a protagonist of the video based on at least one of the region corresponding to the at least one object or the metadata; an operation of obtaining a first text describing the at least one first image by image captioning the at least one first image; an operation of obtaining a plurality of responses to the at least one first image based on a plurality of predetermined queries for generating a story of the video; an operation of obtaining a text prompt for generating a story of the video using information about the character and the protagonist, the metadata, the first text, and the plurality of responses; and a first generative artificial intelligence The method may include an operation to acquire the story of the video based on providing it to a model, and / or an operation to acquire the video based on providing the story and at least one first image to a second generative artificial intelligence model.

[0007] In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components.

[0008] FIG. 1 is a block diagram of an electronic device in a network environment according to various embodiments.

[0009] FIG. 2 is a schematic block diagram of an electronic device according to one embodiment.

[0010] FIG. 3 is a block diagram showing a plurality of modules for generating a video according to one embodiment.

[0011] FIG. 4 is a flowchart illustrating the operation of an electronic device acquiring a video from at least one first image according to one embodiment.

[0012] FIGS. 5a and 5b are drawings for illustrating an operation in which an electronic device, according to one embodiment, identifies at least one object included in at least one image to be reflected as at least part of a video.

[0013] FIGS. 6a and FIGS. 6b are drawings for explaining the operation of an electronic device acquiring metadata of at least one image to be reflected as at least part of a video, according to one embodiment.

[0014] FIGS. 7a and FIGS. 7b are drawings for explaining the operation of an electronic device determining the characters and protagonist of a video according to one embodiment.

[0015] FIGS. 8a, FIGS. 8b, and FIGS. 8c are drawings for explaining the operation of an electronic device acquiring text information for at least one image to be reflected as at least part of a video, according to one embodiment.

[0016] FIGS. 9a and 9b are drawings for explaining a method of obtaining multiple responses for at least one image based on multiple queries for determining the story of a video according to one embodiment.

[0017] FIG. 10 is a diagram illustrating a method for obtaining multiple responses based on multiple queries to determine the story of a video according to one embodiment.

[0018] FIG. 11 is a diagram illustrating a method for obtaining text information based on a first text and a plurality of responses according to one embodiment.

[0019] FIG. 12a is a drawing for illustrating a pre-specified template for generating a text prompt according to one embodiment.

[0020] FIG. 12b is a diagram illustrating a method for obtaining a text prompt using a template according to one embodiment.

[0021] FIG. 13 is a diagram illustrating a method for obtaining a story based on a text prompt according to one embodiment.

[0022] FIGS. 14 and FIGS. 15 are drawings for explaining a method of obtaining user requests that a user wants to add to a video, according to one embodiment.

[0023] FIG. 16 is a diagram illustrating a method for obtaining a text prompt in consideration of user requests according to one embodiment.

[0024] FIGS. 17a and FIGS. 17b are drawings for explaining the operation of an electronic device identifying facial features of a protagonist included in at least one first image, according to one embodiment.

[0025] FIG. 18 is a diagram illustrating a method for obtaining a text prompt based on metadata (e.g., time, place, relationship) of at least one first image representing an image, according to one embodiment.

[0026] Hereinafter, embodiments of the present disclosure are described in detail with reference to the drawings so that those skilled in the art can easily practice them. However, the present disclosure may be embodied in various different forms and is not limited to the embodiments described herein. In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Furthermore, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and brevity.

[0027] FIG. 1 is a block diagram of an electronic device (101) in a network environment (100) according to various embodiments. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network) or may communicate with at least one of an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through a server (108). According to one embodiment, the electronic device (101) may include a processor (120), memory (130), input module (150), sound output module (155), display module (160), audio module (170), sensor module (176), interface (177), connection terminal (178), haptic module (179), camera module (180), power management module (188), battery (189), communication module (190), subscriber identification module (196), or antenna module (197). In some embodiments, at least one of these components (e.g., connection terminal (178)) may be omitted from the electronic device (101), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (176), camera module (180), or antenna module (197)) may be integrated into a single component (e.g., display module (160)).

[0028] The processor (120) can control at least one other component (e.g., hardware or software component) of the electronic device (101) connected to the processor (120) by executing software (e.g., program (140)), for example, and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (120) can store commands or data received from other components (e.g., sensor module (176) or communication module (190)) in volatile memory (132), process the commands or data stored in volatile memory (132), and store the resulting data in non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., central processing unit or application processor) or an auxiliary processor (123) that can operate independently or together with it (e.g., graphics processing unit, neural processing unit (NPU), image signal processor, sensor hub processor, or communication processor). For example, if the electronic device (101) includes a main processor (121) and an auxiliary processor (123), the auxiliary processor (123) may be configured to use lower power than the main processor (121) or to be specialized for a designated function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part thereof.

[0029] The auxiliary processor (123) may control at least some of the functions or states associated with at least one component of the electronic device (101) (e.g., display module (160), sensor module (176), or communication module (190)) on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (123) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (180) or communication module (190)). According to one embodiment, the auxiliary processor (123) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (101) itself where the artificial intelligence is performed, or through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0030] The memory (130) can store various data used by at least one component of the electronic device (101) (e.g., processor (120) or sensor module (176)). The data may include, for example, input data or output data for software (e.g., program (140)) and related commands. The memory (130) may include volatile memory (132) or non-volatile memory (134).

[0031] The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0032] The input module (150) can receive commands or data to be used for a component of the electronic device (101) (e.g., processor (120)) from outside the electronic device (101) (e.g., user). The input module (150) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0033] The sound output module (155) can output a sound signal to the outside of the electronic device (101). The sound output module (155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0034] The display module (160) can visually provide information to an external (e.g., user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0035] The audio module (170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150) or output sound through the sound output module (155) or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (101).

[0036] The sensor module (176) can detect the operating state of the electronic device (101) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0037] The interface (177) may support one or more specified protocols that can be used for the electronic device (101) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (102)). According to one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0038] The connection terminal (178) may include a connector through which the electronic device (101) can be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0039] The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module (179) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0040] The camera module (180) can capture still images and video. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0041] The power management module (188) can manage the power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0042] The battery (189) can supply power to at least one component of the electronic device (101). According to one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0043] The communication module (190) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may include one or more communication processors that operate independently of the processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (104) through a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (199) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (196).

[0044] The wireless communication module (192) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (192) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (192) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), external electronic device (e.g., electronic device (104)), or network system (e.g., second network (199)). According to one embodiment, the wireless communication module (192) can support a Peak data rate (e.g., 20 Gbps or more) for realizing eMBB, loss coverage (e.g., 164 dB or less) for realizing mMTC, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for realizing URLLC.

[0045] An antenna module (197) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (198) or a second network (199), may be selected from the plurality of antennas, for example, by a communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally created as part of the antenna module (197).

[0046] According to one embodiment, the antenna module (197) can create a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0047] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0048] According to one embodiment, commands or data may be transmitted or received between an electronic device (101) and an external electronic device (104) through a server (108) connected to a second network (199). Each of the external electronic devices (102, or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations performed on the electronic device (101) may be performed on one or more of the external electronic devices (102, 104, or 108). For example, if the electronic device (101) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (101) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (101) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or neural networks. According to one embodiment, the external electronic device (104) or the server (108) may be included within a second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0049] Functions related to artificial intelligence according to the present disclosure are operated through a processor and memory. The processor may be composed of one or more processors. In this case, the one or more processors may be general-purpose processors such as a CPU (central processing unit), AP (application processor), or DSP (digital signal processor), graphics-dedicated processors such as a GPU (graphic processing unit) or VPU (vision processing unit), or artificial intelligence-dedicated processors such as an NPU. The one or more processors control the processing of input data according to predefined operation rules or artificial intelligence models stored in memory. Alternatively, if the one or more processors are artificial intelligence-dedicated processors, the artificial intelligence-dedicated processors may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

[0050] The predefined rules of operation or artificial intelligence models are characterized by being created through learning. Here, being created through learning means that a predefined rules of operation or artificial intelligence models configured to perform desired characteristics (or objectives) are created by a basic artificial intelligence model being trained using multiple learning data by a learning algorithm. Such learning may be performed on the device itself where the artificial intelligence according to the present disclosure is executed, or it may be performed through a separate server and / or system. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but are not limited to the examples described above.

[0051] An artificial intelligence model may be composed of multiple neural network layers. Each of the multiple neural network layers has multiple weight values ​​and performs neural network operations through operations between the results of previous layers and the multiple weights. The multiple weights possessed by the multiple neural network layers can be optimized based on the learning results of the artificial intelligence model. For example, the multiple weights may be updated so that the loss value or cost value obtained by the artificial intelligence model during the learning process is reduced or minimized. The artificial neural network may include a deep neural network (DNN), and examples include, but are not limited to, convolutional neural networks (CNN), deep neural networks (DNN), recurrent neural networks (RNN), restricted Boltzmann machines (RBM), deep belief networks (DBN), bidirectional recurrent deep neural networks (BRDNN), or deep Q-networks.

[0052] FIG. 2 is a schematic block diagram of an electronic device according to one embodiment. According to one embodiment, the electronic device (201) can automatically generate a video using an artificial intelligence (AI) model. For example, the electronic device (201) can generate a video using a specific image selected by a user using an AI model. In this case, to generate a video using an AI model, a prompt (e.g., a text prompt) that commands the AI ​​model to generate a video that reflects not only the specific image but also the story intended by the user must be provided to the AI ​​model. For example, the prompt may include a command (e.g., text) for creating new content (e.g., an image and / or video) or for modifying, changing, or editing at least one element or at least one object included in the content (e.g., an image and / or video). For example, the prompt may include a command in the form of text that can be recognized by a generative AI model.

[0053] Previously, users had to manually input or generate text prompts for video acquisition and provide them to the AI ​​model. However, there could be difficulties in obtaining results that match the user's intent using the AI ​​model. Generally, prompts (e.g., text prompts) should be clear and specific, and may include examples if necessary. Additionally, prompts should include as much information as possible so that the AI ​​model can better understand and respond. Consequently, users may have faced difficulties in directly inputting or generating prompts that accurately reflect their intentions.

[0054] According to one embodiment, an electronic device (201) (e.g., the electronic device (101) of FIG. 1) can use a generative AI (artificial intelligence) model (e.g., a generative AI model for story generation) to analyze at least one image (hereinafter, at least one first image) selected by a user and automatically generate a text prompt containing a story of a video that reflects the user's intention (e.g., content of the video, genre of the video, background of the video, and characters and protagonist of the video). Additionally, the electronic device (201) can use a generative AI model (e.g., a generative AI model for video generation) to automatically generate a video based on at least one first image selected by a user and the text prompt. For example, the generative AI model (e.g., a generative AI model for story generation and / or a generative AI model for video generation) may be stored in memory (210). Alternatively, the generative AI model (e.g., a generative AI model for story generation and / or a generative AI model for video generation) may be stored on an external electronic device (e.g., the server (108) of FIG. 1).

[0055] Referring to FIG. 2, according to one embodiment, an electronic device (201) (e.g., electronic device (101) of FIG. 1) may include a memory (210) (e.g., memory (130) of FIG. 1), a processor (220) (e.g., processor (120) of FIG. 1), a display (260) (e.g., display module (160) of FIG. 1), and a communication circuit (290) (e.g., communication module (190) of FIG. 1).

[0056] According to one embodiment, the electronic device (201) may be implemented in the same or similar manner as the electronic device (101) of FIG. 1.

[0057] According to one embodiment, the processor (220) can control the overall operation of the electronic device (201). For example, the processor (220) may be implemented identically or similarly to the processor (120) of FIG. 1. According to one embodiment, the processor (220) can control at least one other component (e.g., hardware or software component) of the electronic device (201) connected to the processor (220) by executing software (e.g., program (140) of FIG. 1), and can perform data processing or operations based on instructions. According to one embodiment, the instructions may include instructions composed of machine language that can be processed by the electronic device (201) or the processor (220). For example, the instructions may include instructions corresponding to operation instructions used in the program.

[0058] Meanwhile, although FIG. 2 illustrates that the electronic device (201) includes one processor (220), this is exemplary and the technical concept of the present invention may not be limited thereto. For example, the electronic device (201) may include at least one processor. For example, the processor (220) may be implemented as at least one processor.

[0059] According to one embodiment, the processor (220) can control the operations of the electronic device (201) by executing instructions stored in the memory (210). For example, the processor (220) may correspond to a plurality of processors that divide and collectively perform a plurality of operations among the processors.

[0060] According to one embodiment, the memory (210) (e.g., the memory (130) of FIG. 1) may store at least one instruction (or instruction) that causes at least one operation of the electronic device (201). When executed by the processor (220), the at least one instruction may cause the electronic device (201) to perform the corresponding operation.

[0061] According to one embodiment, the processor (220) can identify at least one content for generating a video based on user input. For example, the at least one content may include images, sound, video, and / or various other data that serve as materials for generating a video. For example, the at least one content may be an image and / or video stored in memory (210). Alternatively, the at least one content may be an image and / or video obtained through a communication circuit (290).

[0062] According to one embodiment, the processor (220) can identify at least one first image for video generation based on user input. For example, the processor (220) can identify user input to select at least one first image for video generation from a plurality of images stored in the electronic device (201) through the display (260). For example, user input may include touch input, tap input, or long press input on the display (260).

[0063] According to one embodiment, the processor (220) can analyze at least one first image to generate a text prompt for generating a story of a video. For example, the processor (220) can obtain, based on the analysis of at least one first image, information about at least one object included in at least one first image, metadata of at least one first image (e.g., information about time, place, relationship), and text information describing at least one first image. The processor (220) can obtain a text prompt for generating a story of a video using the information about at least one object included in at least one first image, metadata of at least one first image (e.g., information about time, place, relationship), and text information describing at least one first image. The processor (220) can provide the text prompt to a generative AI model (e.g., a generative AI model for story generation) to generate or obtain a story of a video. The processor (220) can provide at least one first image and a story to a generative AI model (e.g., a generative AI model for video generation) to obtain a video containing at least a part of at least one first image (e.g., a video reflecting a story contained in a text prompt).

[0064] According to one embodiment, the processor (220) can identify at least one object included in at least one first image based on user input selecting at least one first image. For example, at least one object may include a person, object, animal, plant, and / or background. According to one embodiment, the processor (220) can obtain information about at least one object and a region corresponding to at least one object included in at least one first image by using a first artificial intelligence model (310) (e.g., the first artificial intelligence model (310) of FIG. 3). For example, the first artificial intelligence model (310) may include an artificial intelligence model for segmenting at least one object from an image. The processor (220) can identify a region corresponding to at least one object based on the result of segmenting at least one object. Additionally, the processor (220) may obtain information about the area of ​​the region (e.g., information about the width and / or size of the area) and / or information indicating what kind of object is included in the region (e.g., information indicating whether the object is a person or an animal and / or identification information of the object (e.g., the name of the person or animal)). For example, the first artificial intelligence model (310) may be stored on a separate server (or, external electronic device). According to another embodiment, the processor (220) may obtain information about the region corresponding to at least one object included in at least one first image and about at least one object using an algorithm stored in memory (210).

[0065] According to one embodiment, the processor (220) can check the metadata of at least one first image. For example, the processor (220) can check information about the time and / or place where at least one first image was taken based on the metadata. Additionally, the processor (220) can check information about the relationships between objects (e.g., people) included in at least one first image (e.g., identification information for objects, friendship, pet relationship, family relationship, or romantic relationship) based on the metadata. For example, the metadata of at least one first image may include at least one of the place where at least one first image was taken, the time it was taken, or information about the relationships between the objects included in at least one first image. The relationships between the objects may represent pet relationships, friendship relationships, romantic relationships, or family relationships. For example, the relationships between the objects may be pre-specified based on user settings. Alternatively, the processor (220) may check the relationships between the objects using a specified algorithm. For example, the processor (220) can perform image clustering on images stored in the electronic device (201) to identify relationships between objects. According to one embodiment, the processor (220) can determine the characters in a video generated using at least one first image based on information about an area corresponding to at least one object (e.g., information about the area size). According to one embodiment, the processor (220) can determine at least one of the objects included in at least one first image as a character in the video. For example, the processor (220) can determine at least one identifiable object among the objects as a character in the video. For example, the processor (220) can determine the remaining objects among the objects, excluding the object whose face is not identified, as characters in the video.For example, the processor (220) may determine that the face of a specific object is not identified if only the back view of the object is identified. In this case, the processor (220) may exclude the object whose face is not identified from the characters in the video. According to one embodiment, the processor (220) may exclude a specific object from the characters in the video if the size of the area corresponding to the specific object is smaller than a specified size. For example, the specified size may represent a reference size for determining the character in the video to be acquired. For example, the specified size may be set by the user or automatically set by the processor (220).

[0066] According to one embodiment, the processor (220) may determine a character among the characters that satisfies a specified rule (e.g., an identifiable object among the objects) as the main character of the video. The processor (220) may determine the remaining characters among the characters other than the main character as supporting characters of the video. For example, the processor (220) may determine the main character based on the area size of regions corresponding to objects included in at least one first image. For example, the processor (220) may determine the object among the objects that has the largest area size corresponding to the object as the main character of the video.

[0067] According to one embodiment, the processor (220) may determine the object that appears most frequently in multiple images among the objects as the protagonist. Alternatively, the processor (220) may determine multiple characters among the characters as the protagonist. For example, the processor (220) may determine multiple characters as the protagonist by using information about relationships obtained based on the metadata of at least one first image (e.g., identification information about the object, friendship, romantic relationship, family relationship). Alternatively, the processor (220) may determine a specific character as the protagonist and another character as a supporting character by using information about relationships obtained based on the metadata of at least one first image (e.g., identification information about the object, friendship, romantic relationship, family relationship).

[0068] According to one embodiment, the processor (220) can determine that at least one object is singular and / or the main character of the video.

[0069] According to one embodiment, the processor (220) may perform image captioning for at least one first image to obtain a first text representing at least one first image (or a first text describing at least one first image). For example, the first text may include text describing or depicting at least one first image. For example, the first text may include text regarding the background or overall composition included in at least one first image, or the behavior or facial expression of at least one object included in at least one first image. For example, the processor (220) may obtain a first text representing at least one first image by using a second artificial intelligence model (320) (e.g., the second artificial intelligence model (320) of FIG. 3). For example, the second artificial intelligence model (320) may include an artificial intelligence model for image captioning. For example, the second artificial intelligence model (320) may be stored in memory (210) or a separate server (or external electronic device). According to one embodiment, the processor (220) may obtain a first text representing at least one first image based on a separate algorithm stored in memory (210).

[0070] According to one embodiment, the processor (220) may obtain various text information related to at least one first image in addition to the first text. For example, the processor (220) may obtain a plurality of responses related to at least one first image based on a plurality of pre-specified queries to generate or determine the story of the video. For example, the plurality of pre-specified queries may include queries related to the facial expressions, emotions, and actions of a main object necessary to generate the story of the video, and queries related to the background, time, and place not related to the main object. For example, the plurality of pre-specified queries may include queries related to the genre of the video (or story). For example, the genre may include romance, fantasy, science fiction, mystery, horror, thriller, chick lit, drama, war, historical drama, or adventure. For example, a plurality of pre-specified queries may be set by a user or automatically set by an electronic device (201). For example, a plurality of pre-specified queries may be obtained from a separate server (or an external electronic device). For example, a plurality of pre-specified queries may consist of questions that help determine the genre exemplified above. For example, a plurality of responses may include answers related to at least one first image for a plurality of pre-specified queries. For example, when a pre-specified query inquires about the facial expression of a specific object, the response may include information (e.g., text describing the facial expression) indicating (or describing) the facial expression of the specific object included in the first image.Alternatively, when a pre-specified query inquires about the background of the first image, the response may include information indicating (or describing) the included background of the first image (e.g., text describing the background). The processor (220) may use a plurality of responses as text information for at least one first image.

[0071] According to one embodiment, the processor (220) can obtain multiple responses corresponding to a plurality of predetermined queries by using a third artificial intelligence model (330) (e.g., the third artificial intelligence model (330) of FIG. 3). For example, the processor (220) can obtain multiple responses based on providing the third artificial intelligence model (330) (e.g., the third artificial intelligence model (330) of FIG. 3) with a plurality of predetermined queries and the at least one first image. For example, the third artificial intelligence model (330) may include an artificial intelligence model for visual question answering (VQA). For example, the third artificial intelligence model (330) may be stored in memory (210) or a separate server (or external electronic device).

[0072] According to one embodiment, the processor (220) may obtain additional information from the user in addition to the information above. For example, the processor (220) may display a user interface for obtaining additional information from the user through a display (260). The processor (220) may obtain information regarding user considerations based on information entered into the user interface (e.g., characters in the video, protagonist, separate requests). The processor (220) may use the information regarding user considerations as text information for at least one first image.

[0073] According to one embodiment, the processor (220) can obtain a text prompt for generating a story of a video by using various information (e.g., information about at least one object included in at least one first image, metadata of at least one first image, and text information describing at least one first image) as described above. According to one embodiment, the processor (220) can obtain a text prompt for generating a story of a video based on a separate algorithm stored in memory (210). For example, the processor (220) can obtain a text prompt using a pre-specified template and the various information above. For example, the text prompt may represent a prompt in the form of text that can be recognized by the first generative artificial intelligence model (340).

[0074] According to one embodiment, the processor (220) can generate or acquire a story corresponding to a text prompt using a first generative artificial intelligence model (340) (e.g., the first generative artificial intelligence model (340) of FIG. 3). For example, the processor (220) can acquire a story of a video based on providing a text prompt to the first generative artificial intelligence model (340). For example, the first generative artificial intelligence model (340) may include an artificial intelligence model for a large language model (LLM). For example, the first generative artificial intelligence model (340) may be stored in memory (210) or a separate server (or external electronic device). According to one embodiment, the processor (220) may acquire (or generate) a story based on a separate algorithm stored in memory (210). According to one embodiment, the processor (220) may generate or acquire a video reflecting the story using a second generative artificial intelligence model (350) (e.g., the second generative artificial intelligence model (350) of FIG. 3). The processor (220) may acquire (or generate) the video based on providing at least one first image and story to the second generative artificial intelligence model (350). According to one embodiment, the processor (220) may acquire (or generate) the video based on a separate algorithm stored in memory (210). For example, the second generative artificial intelligence model (350) may include an artificial intelligence model (e.g., an LLM model) that enables the acquisition of a video from text and / or images. For example, the second generative artificial intelligence model (350) may be stored in memory (210) or a separate server (or external electronic device).

[0075] According to one embodiment, the processor (220) may acquire a video (e.g., a video reflecting the facial features of the protagonist) that takes into account the features of the image, based on providing the second generative artificial intelligence model (350) with an image corresponding to at least one object (e.g., an image representing the face of the protagonist).

[0076] According to one embodiment, the processor (220) may stop the operation of generating a video if it is confirmed that at least one first image is an obscene image or a violent image. At this time, the processor (220) may display notification information indicating that the image is obscene or violent through the display (260).

[0077] According to one embodiment, the processor (220) may stop the operation of generating a video if it is determined that the text prompt (or story of the video) contains obscene or violent text. At this time, the processor (220) may display notification information through the display (260) indicating that the text prompt (or story of the video) contains obscene or violent text.

[0078] According to one embodiment, the processor (220) may stop the operation of generating the video if it is confirmed that at least part of the acquired video contains an obscene or violent image (frame). At this time, the processor (220) may display notification information through the display (260) indicating that it contains an obscene or violent image. Depending on the implementation, if the processor (220) confirms that the acquired video contains an obscene or violent image (frame), it may delete the video or set it so that it cannot be played.

[0079] The operations of the electronic device (201) described in the drawings below may be performed by a processor (220). However, for the sake of convenience of explanation, the operations performed by the processor (220) will be described as being performed by the electronic device (201).

[0080] FIG. 3 is a block diagram showing a plurality of modules for generating a video according to one embodiment.

[0081] Referring to FIG. 3, according to one embodiment, a plurality of modules (300) may include an object recognition module (311), a metadata extraction module (312), a character (protagonist) determination module (313), a text extraction module (314), a story generation module (315), and a video generation module (317). For example, the object recognition module (311), the metadata extraction module (312), the character (protagonist) determination module (313), and the text extraction module (314) may be included in a single module (305) for generating a text prompt. For example, the module (305) may generate or obtain a text prompt using data obtained by the object recognition module (311), the metadata extraction module (312), the character (protagonist) determination module (313), and the text extraction module (314) and a pre-specified template.

[0082] According to one embodiment, memory (210) (e.g., memory (210) of FIG. 2) may store a plurality of modules (300). Depending on the implementation, memory (210) may store only some of the plurality of modules (300). According to another embodiment, at least some of the plurality of modules (300) may be stored in an external electronic device (e.g., server (108) of FIG. 1). For example, the plurality of modules (300) may be implemented in at least one of hardware or software.

[0083] According to one embodiment, the object recognition module (311) can recognize an object included in at least one first image required to generate a video. For example, the object recognition module (311) can recognize the object using a first artificial intelligence model (310). For example, the first artificial intelligence model (310) may include an artificial intelligence model for segmenting at least one object included in the image. For example, the object may include a person, object, animal, plant, and / or background included in the image. This is referred to as identifying the class of the object. For example, the first artificial intelligence model (310) can identify an area corresponding to an object included in the image based on performing segmentation. For example, the object recognition module (311) can also identify information about the area of ​​the corresponding region (e.g., information about the size of the area). Additionally, the object recognition module (311) may identify information about an object included in the corresponding region (class, type, color, and / or shape). For example, the object recognition module (311) can determine whether the object is a person or an animal. At this time, the object recognition module (311) may also check information about the object by performing clustering on related data.

[0084] According to one embodiment, the metadata extraction module (312) can verify the metadata of at least one first image. For example, the metadata of at least one first image may include at least one of the location where at least one first image was taken, the time of taking, identification information of an object included in at least one first image, and information regarding the relationship between objects. For example, the identification information of an object can be verified based on image clustering information for objects stored in the electronic device (201). The electronic device (201) can obtain image clustering information for each object used to identify each object by analyzing a previously taken image or a stored image. For example, the electronic device (201) can verify the identification information (e.g., name or nickname) of a recognized object by comparing the object recognized by the object recognition module (311) with the image information of each object stored in the server. Alternatively, the electronic device (201) may verify the identification information of an object using image clustering information for each object stored in a server. For example, the relationship between objects may represent a pet relationship, a friend relationship, a romantic relationship, or a family relationship. For example, the metadata extraction module (312) can verify information about the time and / or place where at least one first image was taken based on the metadata. Additionally, the metadata extraction module (312) can verify information about the relationship between objects (e.g., people) included in at least one first image (e.g., identification information about the object, a friend relationship, a pet relationship, a family relationship, or a romantic relationship) based on the metadata. The metadata extraction module (312) can verify information about the person's identification information (e.g., a name or nickname) when the object is a person. At this time, the relationship between objects can be verified by performing image clustering on the images stored in the electronic device (201).For example, the metadata extraction module (312) can determine the relationship between objects by considering the “folder name” where the image is stored. Additionally, the metadata extraction module (312) can determine the relationship between objects based on user information (e.g., user’s friend information, pet information, and family information).

[0085] According to one embodiment, the character (protagonist) determination module (313) can determine the characters and protagonist of the video among the objects included in at least one first image. The character (protagonist) determination module (313) can determine the characters of the video generated using at least one first image based on information regarding regions corresponding to the objects identified by the object recognition module (311) (e.g., information regarding area size). For example, the character (protagonist) determination module (313) can determine at least one identifiable object among the objects as a character of the video. For example, the character (protagonist) determination module (313) can exclude an object from the characters of the video when the face of a specific object is not confirmed. The character (protagonist) determination module (313) can exclude an object from the characters of the video when the size of the region corresponding to a specific object is smaller than a specified size.

[0086] According to one embodiment, the character (protagonist) determination module (313) may determine at least one object among the objects determined as characters as the protagonist of the video. Additionally, the character (protagonist) determination module (313) may determine the remaining characters among the characters other than the protagonist as supporting characters of the video. For example, the character (protagonist) determination module (313) may determine the protagonist based on the area size of regions corresponding to the objects. For example, the processor (220) may determine the object among the objects that has the largest area size corresponding to the object as the protagonist of the video. Alternatively, the character (protagonist) determination module (313) may determine the object among the objects that appears most frequently in multiple images as the protagonist. Alternatively, the character (protagonist) determination module (313) may determine an object specified by the user as the protagonist. For example, multiple characters may be determined as protagonists. The character (protagonist) determination module (313) may determine related objects as protagonists by using information about relationships obtained from the metadata extraction module (312) (e.g., identification information about objects, friendship, romantic relationship, family relationship). For example, the character (protagonist) determination module (313) may determine objects corresponding to friendship (or family relationship) as protagonists.

[0087] According to one embodiment, the text extraction module (314) can extract text information related to at least one first image. The text extraction module (314) may include a second artificial intelligence model (320) and a third artificial intelligence model (330). For example, the text information may include texts corresponding to the results of analysis by the second artificial intelligence model (320) and the third artificial intelligence model (330) on at least one first image. Additionally, the text information may further include text separately entered by a user request. For example, the second artificial intelligence model (320) may include an artificial intelligence model for image captioning. For example, the second artificial intelligence model (320) may be stored on a separate server (or external electronic device). For example, the third artificial intelligence model (330) may include an artificial intelligence model for visual question answering (VQA). For example, the second artificial intelligence model (320) and / or the third artificial intelligence model (330) may be stored on a separate server (or external electronic device).

[0088] According to one embodiment, the text extraction module (314) can obtain a first text describing (or representing) at least one first image by using a second artificial intelligence model (320). For example, the second artificial intelligence model (320) can obtain a first text describing (or representing) at least one first image by performing image captioning for at least one first image.

[0089] According to one embodiment, the text extraction module (314) can obtain a plurality of responses related to at least one first image based on a plurality of pre-specified queries to generate or determine the story of a video using a third artificial intelligence model (330). For example, the plurality of responses may include answers corresponding to the results analyzed by the third artificial intelligence model for the plurality of queries. For example, the third artificial intelligence model (330) can obtain a plurality of responses based on receiving a plurality of pre-specified queries and at least one first image.

[0090] According to one embodiment, the electronic device (201) can obtain a text prompt based on information obtained from modules (311, 312, 313, 314) (e.g., object information, metadata, character and protagonist information, text information). For example, the electronic device (201) can obtain a text prompt using information obtained from modules (311, 312, 313, 314) and a pre-specified template. Depending on the implementation, the operation of obtaining a text prompt may be performed by a separate module. Alternatively, the operation of obtaining a text prompt may be performed by a story generation module (315).

[0091] According to one embodiment, the story generation module (315) can generate or obtain a story of a video based on a text prompt by using a first generative artificial intelligence model (340). The story generation module (315) may include a first generative artificial intelligence model (340) trained to generate a story. For example, the first generative artificial intelligence model (340) may include an artificial intelligence model based on a large language model (LLM). For example, the first generative artificial intelligence model (340) may obtain a text prompt containing a story of a video based on receiving a text prompt. Depending on the implementation, the first generative artificial intelligence model (340) may generate a text prompt based on receiving text information and generate a story of a video based on the generated text prompt.

[0092] According to one embodiment, the video generation module (317) can generate or obtain a video based on a story generated by the story generation module (315) and at least one first image using a second generative artificial intelligence model (350). The video generation module (317) may include a second generative artificial intelligence model (350) trained to generate a video. For example, the second generative artificial intelligence model (340) may include an artificial intelligence model based on a Text-to-Video model. For example, the second generative artificial intelligence model (350) may generate a video by receiving text (e.g., a story) and an image (e.g., at least one first image). When video generation is requested, the electronic device (201) may provide the video output from the video generation module (317) as a result.

[0093] At least some of the operations of the electronic device (201) described below may be performed or controlled by the processor (220). However, for convenience of explanation, the subject of the operation will be described as the electronic device (201). Additionally, in the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0094] FIG. 4 is a flowchart illustrating the operation of an electronic device acquiring a video from at least one first image according to one embodiment.

[0095] Referring to FIG. 4, according to one embodiment, in operation 411, an electronic device (201) (e.g., the electronic device (201) of FIG. 2) can identify at least one first image for generating a video among a plurality of images stored in the electronic device (201) based on user input. For example, user input may include various forms of input for selecting at least one first image with respect to a display (260) (e.g., the display (260) of FIG. 2).

[0096] According to one embodiment, in operation 413, the electronic device (201) can identify at least one object included in at least one first image and metadata of at least one first image. For example, at least one object may include a person, object, animal, plant, or background. According to one embodiment, the electronic device (201) can use an object recognition module (e.g., object recognition module (311) of FIG. 3) to identify each region corresponding to at least one object included in at least one first image and obtain information about at least one object included in the region (e.g., type, shape, size of the object). For example, the object recognition module (311) can recognize or identify the object using an artificial intelligence model for segmenting at least one object included in at least one first image (e.g., first artificial intelligence model (310) of FIG. 3). According to one embodiment, the electronic device (201) can identify metadata of at least one first image. For example, the electronic device (201) can verify or extract metadata of at least one first image using a metadata extraction module (312). The electronic device (201) can use the metadata to obtain information about the place and / or time at least one first image was taken, and the relationships between objects included in at least one first image.

[0097] According to one embodiment, in operation 415, the electronic device (201) can determine the character and protagonist of the video among at least one identified object. For example, the electronic device (201) can identify or determine the character and protagonist among the identified objects using a character (protagonist) determination module (313). For example, the electronic device (201) can determine the object whose face is identified among the identified objects as the character of the video. Alternatively, the electronic device (201) can determine the object having an area larger than a specified size among the identified objects as the character of the video. The electronic device (201) can determine the protagonist of the video among the objects determined as the character. For example, the electronic device (201) can compare the size of the area occupied by each object (e.g., area size) and determine the object having a larger area as the protagonist of the video. Alternatively, the electronic device (201) can determine the protagonist of the video by considering the relationship between the identified objects based on metadata. Alternatively, the electronic device (201) may determine a designated object as the main character based on a user's request. According to one embodiment, in operation 417, the electronic device (201) may image caption at least one first image to obtain a first text describing (or representing) at least one first image. For example, the electronic device (201) may obtain or generate the first text using a text extraction module (314) (or a second artificial intelligence model (320)). For example, the first text may represent text describing or depicting at least one first image. For example, the first text may include a background, atmosphere, behavior of at least one object, facial expression of at least one object, and overall composition included in at least one first image. The electronic device (201) may obtain the first text as text information for at least one first image.

[0098] According to one embodiment, in operation 419, the electronic device (201) may obtain a plurality of responses to at least one first image based on a plurality of queries for determining or generating a story of a video. For example, the predetermined plurality of queries may include queries related to the facial expression, emotion, or behavior of at least one object included in at least one first image, and the background (and / or atmosphere) excluding at least one object included in at least one first image. For example, the predetermined plurality of queries may include queries related to the genre of the video. For example, the genre may include romance, fantasy, science fiction, mystery, horror, thriller, chick lit, drama, war, historical drama, or adventure. For example, the plurality of responses may include responses (or answers) based on the results of the third artificial intelligence model analyzing at least one first image in response to the above plurality of queries. For example, the plurality of responses may include information regarding the facial expression, emotion, behavior, and background (and / or atmosphere) of at least one object included in at least one first image. Alternatively, the plurality of responses may include information related to the genre of the video. For example, the electronic device (201) may acquire or generate the plurality of responses using a text extraction module (314) (or the third artificial intelligence model (340)). For example, the electronic device (201) may acquire the plurality of responses as text information regarding at least one first image.

[0099] According to one embodiment, in operation 421, the electronic device (201) can obtain a text prompt for generating a story of a video using information about a character and a protagonist, metadata, a first text, and a plurality of responses. For example, the electronic device (201) can obtain a text prompt using data obtained in operations 413 through 419 and a pre-specified template.

[0100] According to one embodiment, in operation 423, the electronic device (201) can obtain a story of a video based on providing a text prompt to a first generative artificial intelligence model (340) (e.g., the first generative artificial intelligence model (340) of FIG. 3). For example, the first generative artificial intelligence model (340) may include an artificial intelligence model for a large language model (LLM). For example, the text prompt may include text that the first generative artificial intelligence model (340) can recognize and / or command. For example, the text prompt may include text that causes the video to be generated.

[0101] According to one embodiment, in operation 425, the electronic device (201) may acquire a video based on providing a story and at least one first image to a second generative artificial intelligence model (350) (e.g., the second generative artificial intelligence model (350) of FIG. 3). For example, the second generative artificial intelligence model (350) may include an artificial intelligence model that acquires a video from text and / or images. For example, according to one embodiment, the electronic device (201) may acquire a video reflecting the facial features of the protagonist based on providing the second generative artificial intelligence model (350) with an image of the protagonist (or information representing the facial features of the protagonist).

[0102] Based on the method described above, the electronic device (201) can use a generative AI (artificial intelligence) model (e.g., a generative AI model for story generation and a generative AI model for video generation) to analyze at least one image (hereinafter, at least one first image) selected by the user and automatically generate a video that reflects the user's intention (e.g., content of the video, genre of the video, background of the video, mood of the video, and characters and protagonist of the video).

[0103] FIGS. 5a and 5b are drawings for illustrating an operation in which an electronic device, according to one embodiment, identifies at least one object included in at least one image to be reflected as at least part of a video.

[0104] Referring to FIGS. 5a and 5b, according to one embodiment, in operation 501, an electronic device (e.g., the electronic device (201) of FIG. 2) can identify at least one image (506, 508) to be reflected as at least part of a video based on user input (e.g., user input selecting an image that serves as material for a video). For example, the electronic device (201) can identify a first image (506) corresponding to (a) of FIG. 5b and a second image (508) corresponding to (b) based on user input.

[0105] According to one embodiment, in operation 503, the electronic device (201) can obtain an area corresponding to an object included in at least one image and information about said object. For example, the electronic device (201) can identify a first area (510) corresponding to a first object (e.g., a dog) included in the first image (506) and a second area (520) corresponding to a second object (e.g., a person). Additionally, the electronic device (201) can identify a third area (530) corresponding to a third object (e.g., a person kayaking) included in the second image (508). The electronic device (201) can obtain information about an object included in each identified area. For example, the electronic device (201) can obtain information about the area size of the first area (510) and information about the type and shape of an object included in the first area (510) (e.g., a dog sitting with the second object). For example, the electronic device (201) can obtain information about the area size of the second area (520) and information about the type and shape of an object included in the second area (520) (e.g., a person sitting with the first object). At this time, the electronic device (201) can verify information about the faces of the first object and the second object. For example, the electronic device (201) can obtain information about the area size of the third area (530) and information about the type and shape of an object included in the third area (530) (e.g., a person kayaking). At this time, the electronic device (201) can verify that the face of the third object is not identifiable.

[0106] According to one embodiment, the electronic device (201) can manage information obtained according to the method described above as object recognition information of at least one image (506, 508).

[0107] FIGS. 6a and FIGS. 6b are drawings for explaining the operation of an electronic device acquiring metadata of at least one image to be reflected as at least part of a video, according to one embodiment.

[0108] Referring to FIGS. 6a and 6b, according to one embodiment, in operation 601, an electronic device (e.g., the electronic device (201) of FIG. 2) can obtain metadata of at least one image (606, 608) to be reflected as at least part of a video. For example, the electronic device (201) can verify the metadata of the first image (606) corresponding to (a) of FIG. 6b and the metadata of the second image (608) corresponding to (b) of FIG. 6b. For example, the metadata may include information about the time, place, and relationship of the corresponding image.

[0109] According to one embodiment, in operation 603, the electronic device (201) can determine information regarding the time, place, and relationship of at least one image (606, 608) based on metadata. For example, the electronic device (201) can determine information regarding the time of each image based on the time at which the first image (606) and the second image (608) were taken. The electronic device (201) can determine information regarding the place of each image based on information regarding the location at which the first image (606) and the second image (608) were taken (e.g., GPS (global positioning system) information). For example, the electronic device (201) can determine that the background (640) of the first image (606) is a beach at a specific time based on information regarding the time and place of the first image (606). For example, the electronic device (201) can determine, based on information regarding the time and place of the second image (608), that the background (650) of the second image (608) is on the sea at a specific time. Additionally, the electronic device (201) can determine, based on information regarding the time and place of the first image (606) and the second image (608), whether the two images are related to each other (e.g., whether they were taken at a similar time and place).

[0110] According to one embodiment, the electronic device (201) can obtain information about objects (610, 620, 630) included in the first image (606) and the second image (608). For example, the electronic device (201) can verify information about the relationship between the first object (610) and the second object (620) included in the first image (606). For example, the electronic device (201) can obtain information about the relationship between the first object (610) and the second object (620) (e.g., pet and owner) based on clustering the images stored in the electronic device (201). Alternatively, the electronic device (201) can verify information about the identification information (e.g., name or nickname) or gender of each object. For example, the electronic device (201) can verify information about the name of each object based on the folder name where the image is stored. Additionally, the electronic device (201) can also guess whether the third object (630) and the second object (620) are the same person based on whether the two images are stored in the same folder.

[0111] According to one embodiment, the electronic device (201) can manage information obtained according to the method described above as metadata information of at least one image (606, 608).

[0112] FIGS. 7a and FIGS. 7b are drawings for explaining the operation of an electronic device determining the characters and protagonist of a video according to one embodiment.

[0113] Referring to FIGS. 7a and 7b, according to one embodiment, in operation 701, an electronic device (e.g., the electronic device (201) of FIG. 2) can identify objects contained in at least one image (706, 708) to be reflected as at least part of a video. For example, the electronic device (201) can identify objects contained in at least one image based on object recognition information performed in operation 503 of FIG. 5a. For example, the electronic device (201) can identify a first object (710) and a second object (720) contained in the first image (706). Additionally, the electronic device (201) can identify a third object (730) contained in the second image (708).

[0114] According to one embodiment, in operation 703, the electronic device (201) can determine the characters appearing in the video among the identified objects. For example, the electronic device (201) can determine the objects among the identified objects whose faces are identified as the characters. For example, the electronic device (201) can determine the first object (710) and the second object (720) included in the first image (706), whose faces are identified, as the characters. Additionally, the electronic device (201) can exclude the third object included in the second image (708), whose faces are not identified, from the characters. Alternatively, the electronic device (201) can determine the object among the areas corresponding to the identified objects that has an area size larger than a specified size as the characters in the video.

[0115] According to one embodiment, in operation 705, the electronic device (201) can determine the main character based on at least one of the sizes of regions corresponding to the objects among the objects determined as characters or metadata (e.g., relationships). For example, the electronic device (201) can determine the two objects (710, 720) as the main characters of the video because the first object (710) and the second object (720) have substantially the same area size. If the area size of the region corresponding to the second object (720) is larger than the area size of the region corresponding to the first object (710), the electronic device (201) can determine the second object (720) as the main character. At this time, the electronic device (201) can determine the first object (710) as the supporting character of the video. Alternatively, the electronic device (201) can determine, based on metadata, that the relationship between the first object (710) and the second object (720) is that of a pet and its owner. At this time, the electronic device (201) can determine the two objects (710, 720) as the main characters of the video.

[0116] According to one embodiment, the electronic device (201) can manage information obtained according to the method described above as information about characters and protagonists.

[0117] FIGS. 8a, FIGS. 8b, and FIGS. 8c are drawings for explaining the operation of an electronic device acquiring text information for at least one image to be reflected as at least part of a video, according to one embodiment.

[0118] Referring to FIG. 8a and FIG. 8b, according to one embodiment, in operation 801, an electronic device (e.g., the electronic device (201) of FIG. 2) may perform captioning (e.g., image captioning) for at least one image (806, 808) to be reflected as at least part of a video. For example, the electronic device (201) may perform image captioning for the first image (806) to obtain a first text (840) describing (or representing) the first image (806). Additionally, the electronic device (201) may obtain a second text (850) describing (or representing) the second image (808). Referring to FIG. 8c, the electronic device (201) may use a video in addition to a still image as material for creating a new video. For example, the electronic device (201) can identify a first image (806) and a first video (809) for a new video by user input and perform image captioning for the first image (806) and the first video (809). At this time, the electronic device (201) can obtain a third text (860) that describes (or indicates) the video by performing image captioning for the first video (809). For example, the third text (860) may include content summarizing the video considering the characteristics of the video. Alternatively, the third text (860) may include content describing information about the motion of an object included in the video, movement considering the results of scene analysis, and events related to motion or movement. Additionally, the third text (860) may include content related to the viewpoint, movement, and motion of a global camera (e.g., a camera that controls the entire scene of the video). For example, the third text (860) may include content describing various movements such as zoom in, zoom out, and pan, and may include content that allows focusing on a specific scene or securing an overall view.

[0119] According to one embodiment, in operation 803, the electronic device (201) can acquire data to determine the story of the video and the genre of the story. For example, the electronic device (201) can acquire a plurality of responses to at least one image (806, 808) as text information regarding the video. The electronic device (201) can determine the story of the video and the genre of the story based on the plurality of responses. This will be explained in detail in FIGS. 9a and 9b below.

[0120] According to one embodiment, in operation 805, the electronic device (201) may acquire additional data for the story of the video. An explanation of this will be described in detail in FIG. 10 below. For example, the electronic device (201) may acquire additional responses to at least one image (806, 808) as additional data for the video. The electronic device (201) may determine or generate the story of the video by further considering the additional data.

[0121] According to one embodiment, the electronic device (201) can manage information obtained according to the method described above as text information of at least one image (806, 808).

[0122] FIGS. 9a and 9b are drawings for explaining a method of obtaining multiple responses for at least one image based on multiple queries for determining the story of a video according to one embodiment.

[0123] Referring to FIG. 9a, according to one embodiment, an electronic device (201) (e.g., the electronic device (201) of FIG. 2) may obtain a plurality of responses (920) corresponding to a plurality of queries (910) for determining or generating a story of a video obtained from a first image (806) (e.g., the first image (806) of FIG. 8a) and a second image (808) (e.g., the second image (808) of FIG. 8a). For example, the plurality of queries (910) may include queries related to the facial expressions, emotions, and behaviors of a first object (810) (e.g., the first object (810) of FIG. 8a) and a second object (820) (e.g., the second object (820) of FIG. 8a) included in the first image (806), and the background (and / or atmosphere) excluding at least one object included in at least one image (806, 808). A plurality The queries (910) may include queries related to the genre of the video. For example, the genre may include romance, fantasy, science fiction, mystery, horror, thriller, chick lit, drama, war, historical drama, or adventure. For example, multiple queries (910) may be specified by the user or automatically specified by the electronic device (201).

[0124] According to one embodiment, an electronic device (201) can obtain a plurality of responses (920) corresponding to a plurality of queries (910) based on providing a first image (806) and a plurality of queries (910) to a third artificial intelligence model (330) (e.g., the third artificial intelligence model (330) of FIG. 3). The plurality of responses may include answers related to the first image (806) for the plurality of queries.

[0125] According to one embodiment, multiple responses may be determined as “yes” or “no” as answers to multiple queries. For example, multiple responses may be obtained as text corresponding to answers to multiple queries other than “yes or no”.

[0126] According to one embodiment, the electronic device (201) can determine or generate a story of a video based on multiple responses to multiple queries (e.g., queries 1 through 11). For example, the electronic device (201) can include the acquired multiple responses (920) in a pre-specified template (e.g., 1210, 1221, 1231, 1232, 1233, 1281 of FIG. 12a). For example, the pre-specified template may represent a template for generating a text prompt.

[0127] Meanwhile, in FIG. 9, for convenience of explanation, the text (930) to be included in the template is illustrated as an example where the response is “Yes” and “A”, but the technical features of the present invention may not be limited thereto.

[0128] According to one embodiment, the electronic device (201) can determine the genre of the story of the video based on multiple responses to multiple queries (e.g., queries 1 through 11).

[0129] For example, query 1 (e.g., What emotions are the main characters feeling towards each other in this picture?) may represent a query related to the romance genre. For example, A may include love, friendship, comfort, or intimacy.

[0130] For example, query 2 (e.g., are the main characters in this photo holding hands or hugging each other?) may represent a query related to the romance genre.

[0131] For example, question 3 (e.g., Are there magical, mythical, or unrealistic elements in this photo?) may represent a question related to the fantasy genre.

[0132] For example, question 4 (e.g., Does the background (and / or atmosphere) in the photo look futuristic, cosmic, techno-oriented, or like an alien planet?) may represent a question related to the science fiction genre.

[0133] For example, question 5 (e.g., Is there anything suspicious in the photo or any elements related to crime?) may represent a question related to the mystery genre.

[0134] For example, question 6 (e.g., Does the photo evoke a scary atmosphere or a sense of fear?) may represent a question related to the horror genre.

[0135] For example, question 7 (e.g., Is the background of the photo set in a specific era? Does it depict the lifestyle or historical events of that era?) may represent a question related to the chick lit genre.

[0136] For example, question 8 (e.g., Does the photo make you feel tension or anxiety?) may represent a question related to the thriller genre.

[0137] For example, query 9 (e.g., Does it feature female characters and show elements such as fashion, friendships, daily life, and work life?) may represent a query related to the drama genre.

[0138] For example, query 10 (e.g., are there traces of war, weapons, destroyed buildings, military uniforms, etc. in the background of the photo, or is it a combat situation?) may represent a query related to the war genre.

[0139] For example, query 11 (e.g., Does the background of the photo have a unique or eye-catching landscape, or represent a travel destination or tourist attraction?) may represent a query related to the adventure genre.

[0140] A plurality of predetermined queries according to one embodiment may not be limited to the example described in FIG. 9a and may include various queries. Referring to FIG. 9b, according to one embodiment, an electronic device (201) may sequentially inquire about each of the plurality of queries (960, 980) to a third artificial intelligence model (e.g., the third artificial intelligence model (330) of FIG. 3) and obtain answers (970, 990) thereto as a plurality of responses. For example, the operation of inquiring about the plurality of queries and the operation of obtaining the plurality of responses may not be displayed through the display (260), and FIG. 9b may be displayed in a UI (user interface) manner for convenience of explanation.

[0141] According to one embodiment, the electronic device (201) may inquire of a third artificial intelligence model (330) with a first query (960) among a plurality of queries (e.g., Are the main characters in this photo holding hands or hugging each other?). For example, the electronic device (201) may obtain a first response (970) (e.g., Yes) corresponding to the first query (960) through the third artificial intelligence model (330).

[0142] According to one embodiment, the electronic device (201) may inquire of a third artificial intelligence model (330) with a second query (980) (e.g., Does the photo evoke a scary atmosphere or a sense of fear?) among a plurality of queries. For example, the electronic device (201) may obtain a second response (990) (e.g., No) corresponding to the second query (980) through the third artificial intelligence model (330).

[0143] Based on the method described above, the electronic device (201) can obtain multiple responses corresponding to multiple queries through the third artificial intelligence model (330).

[0144] Meanwhile, in FIG. 9a, the process of obtaining multiple responses to the second image (808) is omitted, and this can be implemented in the same or similar way as the method of obtaining multiple responses to the first image (806).

[0145] FIG. 10 is a diagram illustrating a method for obtaining multiple responses based on multiple queries to determine the story of a video according to one embodiment.

[0146] Referring to FIG. 10, according to one embodiment, an electronic device (201) (e.g., the electronic device (201) of FIG. 2) may inquire of a third artificial intelligence model (330) with a plurality of queries (1010) to generate or determine a story of a video obtained from a first image (806) (e.g., the first image (806) of FIG. 8a) and a second image (808) (e.g., the second image (808) of FIG. 8a), and obtain a plurality of responses (1020) through the third artificial intelligence model (330). For example, the plurality of queries (1010) may be queries additionally generated in consideration of user requests. Or, the plurality of queries (1010) may be queries additionally generated through a text extraction module (314). For example, the electronic device (201) can obtain texts (1030) to be used in a template by using multiple responses (1020) to multiple queries (1010). For example, when a specific query is “What season is this photo?” and the response to it is “B,” the electronic device (201) can obtain text such as “The season of this story is B” (e.g., text to be used in a template) based on the specific query and response. In addition, the electronic device (201) can obtain texts regarding characters, places, and facial expressions of characters by using multiple responses (1020) to multiple queries (1010).

[0147] According to one embodiment, the electronic device (201) may include or reflect a plurality of responses (1020) in a specified template (e.g., template (1281) of FIG. 12a) to generate a text prompt.

[0148] Meanwhile, in FIG. 9b, the process of obtaining multiple responses to the second image (808) is omitted, and this can be implemented in the same or similar way as the method of obtaining multiple responses to the first image (806).

[0149] FIG. 11 is a diagram illustrating a method for obtaining text information based on a first text and a plurality of responses according to one embodiment.

[0150] Referring to FIG. 11, according to one embodiment, an electronic device (201) (e.g., the electronic device (201) of FIG. 2) can use a second artificial intelligence model (320) (e.g., the second artificial intelligence model (320) of FIG. 3) to caption a first image and obtain a first text (1110) that describes (or indicates) the first image (806) as a result of captioning.

[0151] According to one embodiment, the electronic device (201) can obtain a plurality of responses (1121, 1122, 1123, 1124) corresponding to a plurality of queries using a third artificial intelligence model (330) (e.g., the third artificial intelligence model (330) of FIG. 3).

[0152] According to one embodiment, the electronic device (201) may obtain a plurality of responses (1130) corresponding to a plurality of additional queries.

[0153] Based on the method described above, the electronic device (201) can obtain text information for the first image (806).

[0154] Meanwhile, the content of obtaining text information for the second image (808) has been omitted, and this can be implemented in the same or similar way as the method of obtaining multiple responses for the first image (806).

[0155] FIG. 12a is a drawing for illustrating a pre-specified template for generating a text prompt according to one embodiment.

[0156] Referring to FIG. 12a, according to one embodiment, data for a pre-specified template may be stored in memory (210) (e.g., memory (210) of FIG. 2). The pre-specified template may include a template for generating a text prompt to be provided to a first generative artificial intelligence model (340) (e.g., the first generative artificial intelligence model (340) of FIG. 3).

[0157] According to one embodiment, an electronic device (201) (e.g., the electronic device (201) of FIG. 2) may reflect information obtained by analyzing at least one first image that is reflected as at least part of a video into a pre-specified template. For example, the electronic device (201) may add to the pre-specified template the time of the video (1210), the story genre of the video (1221), the main character (1231), the characters (1232), text about the story of the video (1233), and additional content about the story to be considered (1281).

[0158] FIG. 12b is a diagram illustrating a method for obtaining a text prompt using a template according to one embodiment.

[0159] Referring to FIG. 12b, according to one embodiment, an electronic device (201) (e.g., the electronic device (201) of FIG. 2) can obtain a text prompt (1200) based on adding information analyzing at least one first image to a template specified in FIG. 12a.

[0160] According to one embodiment, the text prompt (1200) may include text (1210) indicating the story time of the video. For example, the story time may be determined by the user or automatically determined by the electronic device (201). For example, the story time in the text prompt (1200) may be determined to be 3 minutes.

[0161] According to one embodiment, the text prompt (1200) may include candidate groups for the genre of the video story (e.g., romance, fantasy, science fiction, mystery, horror, thriller, period drama, chick lit, drama, war, adventure). For example, the candidate groups for the genre of the video story may each correspond to a plurality of queries.

[0162] According to one embodiment, the text prompt (1200) may include text (1230) representing a first object (810) (e.g., the first object (810) of FIG. 8a) and a second object (820) (e.g., the second object (820) of FIG. 8a) included in a first image (806) (e.g., the first image (806) of FIG. 8a) determined to be the protagonist of the story, and text (1240) representing the first image (806).

[0163] According to one embodiment, the text prompt (1200) may include text information (1250, 1260, 1270) for a plurality of responses based on a plurality of queries.

[0164] According to one embodiment, the text prompt (1200) may include text information (1280) for multiple responses based on multiple additional queries.

[0165] Based on the method described above, the electronic device (201) can automatically generate a text prompt for generating a story of a video.

[0166] FIG. 13 is a diagram illustrating a method for obtaining a story based on a text prompt according to one embodiment.

[0167] Referring to FIG. 13, according to one embodiment, an electronic device (201) (e.g., the electronic device (201) of FIG. 2) can generate a story (1301) of a video using a story generation module (315) (e.g., the story generation module (315) of FIG. 3). For example, the electronic device (201) can acquire or generate a story of a video based on providing a text prompt (1200) (e.g., the text prompt (1200) of FIG. 12b) to a first generative artificial intelligence model (340) (e.g., the first generative artificial intelligence model (340) of FIG. 3).

[0168] According to one embodiment, the story (1301) may include the genre of the story (e.g., drama), the title of the story (e.g., Haeundae's Day: Summer, Friends, and Happy), and the content of the story.

[0169] According to one embodiment, the electronic device (201) can acquire a video based on a story (1301) by using a video generation module (e.g., the video generation module (317) of FIG. 3). For example, the electronic device (201) can acquire or generate a video based on providing the story (1301) and at least one first image to a second generative artificial intelligence model (350) (e.g., the second generative artificial intelligence model (350) of FIG. 3).

[0170] FIGS. 14 and FIGS. 15 are drawings for explaining a method of obtaining user requests that a user wants to add to a video, according to one embodiment.

[0171] Referring to FIGS. 14 and 15, according to one embodiment, in operation 1401, an electronic device (201) (e.g., the electronic device (201) of FIG. 2) can check user requests (e.g., main character, separate request content) based on user input. For example, the electronic device (201) can display a UI screen for inputting user requests on a display (e.g., the display (260) of FIG. 2). User input may be input to a UI screen set to input user requests.

[0172] According to one embodiment, the electronic device (201) can select an image (1510) used to generate a video among a plurality of images stored in the electronic device (201) based on a first user input. The electronic device (201) can display the selected image (1510) on a UI screen based on the user input.

[0173] According to one embodiment, the electronic device (201) may display indicators (1521, 1522, 1523) that can select the main character of the video on a UI screen. For example, the indicators (1521, 1522, 1523) may include thumbnail images corresponding to objects included in the selected image (1510). The electronic device (201) may determine the selected object among the objects included in the selected image (1510) as the main character of the video based on a second user input regarding the indicators (1521, 1522, 1523).

[0174] According to one embodiment, the electronic device (201) can display indicators (1531, 1532, 1533) that allow selecting a character in a video on a UI screen through a display (260). For example, the indicators (1531, 1532, 1533) may include thumbnail images corresponding to objects included in the selected image (1510). The electronic device (201) can determine a selected object among the objects included in the selected image (1510) as a character in the video based on a third user input regarding the indicators (1531, 1532, 1533).

[0175] According to one embodiment, the electronic device (201) can display a list (1540) on a UI screen through a display (260) for selecting the genre of the story of the video. For example, the list (1540) may include romance, fantasy, science fiction, mystery, horror, thriller, period drama, chick lit, drama, war, and adventure. The electronic device (201) can determine the genre of the video based on a fourth user input to the list (1540).

[0176] According to one embodiment, the electronic device (201) may display an interface (1550) on a UI screen to receive content to be included in the story of a video through a display (260). The electronic device (201) may add content to be included in the story based on a fifth user input to the interface (1550). For example, the fifth user input may include various inputs for entering text (e.g., touch input, voice input). According to one embodiment, when content to be added to the video is entered by the fifth user input, the electronic device (201) may display the corresponding content on the interface (1550).

[0177] According to one embodiment, in operation 1403, the electronic device (201) may generate or obtain a text prompt with further consideration of user requests. A text prompt generated with further consideration of user requests will be illustrated exemplarily in FIG. 16 below.

[0178] FIG. 16 is a diagram illustrating a method for obtaining a text prompt in consideration of user requests according to one embodiment.

[0179] Referring to FIG. 16, according to one embodiment, the text prompt (1600) may include text (1610) indicating the story time of the video.

[0180] According to one embodiment, the text prompt (1600) may include text (1620) representing the selected protagonist (or character) based on a second user input or a third user input (e.g., user input selecting the protagonist or character in FIG. 15).

[0181] According to one embodiment, the text prompt (1600) may include text (1630) indicating the genre of the story of the selected video based on a fourth user input (e.g., user input selecting a genre in FIG. 15).

[0182] According to one embodiment, the text prompt (1600) may include text (1640) representing the story of a video based on the results of analyzing a selected image. For example, the text (1640) may include a story about the main characters of the video.

[0183] According to one embodiment, the text prompt (1600) may include text (1650) for a user request based on a fifth user input (e.g., user input entering a user request in FIG. 15).

[0184] According to one embodiment, an electronic device (201) (e.g., the electronic device (201) of FIG. 2) can generate or acquire a story of a video based on providing a text prompt (1600) to a story generation module (e.g., the story generation module (315) of FIG. 3) (or the first generative artificial intelligence model (340)).

[0185] Meanwhile, FIG. 16 describes generating a story for a video by taking into account information regarding the main character (or characters), genre, and user requests provided by the user, but this is merely one example, and various other information may also be provided by the user.

[0186] As described above, the electronic device (201) can generate or acquire a video story that takes into account user requests. The electronic device (201) can automatically generate a video that reflects user requests using the story.

[0187] FIGS. 17a and FIGS. 17b are drawings for explaining the operation of an electronic device identifying facial features of a protagonist included in at least one first image, according to one embodiment.

[0188] Referring to FIG. 17a, according to one embodiment, in operation 1701, an electronic device (e.g., the electronic device (201) of FIG. 2) can verify an image containing a protagonist. For example, when the protagonist is determined, the electronic device (201) can verify at least one image containing an object corresponding to the protagonist.

[0189] According to one embodiment, in operation 1703, the electronic device (201) can identify the facial features of the protagonist by analyzing an image containing the protagonist (or an object corresponding to the protagonist included in the image). Additionally, the electronic device (201) can identify the motion or physical features of the protagonist.

[0190] According to one embodiment, in operation 1705, the electronic device (201) can generate a video by taking into account information regarding the facial features (and motion or physical features) of the identified protagonist. For example, the electronic device (201) can provide information regarding the facial features (and motion or physical features) of the protagonist to a second generative artificial intelligence model.

[0191] According to one embodiment, the electronic device (201) can perform image clustering to analyze images containing objects corresponding to the protagonist stored in the electronic device (201). Based on the results of analyzing the corresponding images, the electronic device (201) can obtain information regarding the protagonist's facial features (and motion or physical features). Additionally, the electronic device (201) can generate a video reflecting the protagonist's features by taking into account the obtained information.

[0192] According to one embodiment, the electronic device (201) may additionally receive an image containing an object corresponding to the protagonist or an image reflecting the facial features of said object. The electronic device (201) may analyze the additionally received image to obtain information regarding the facial features (and motion or physical features) of the protagonist. The electronic device (201) may generate a video reflecting the features of the protagonist by taking into account the obtained information.

[0193] Referring to (a) and (b) of FIG. 17b, according to one embodiment, an electronic device (201) (e.g., the electronic device (201) of FIG. 2) can identify a first image (1710) and a second image (1720) containing a main character. For example, the first image (1710) and the second image (1720) may be images selected by user input for video generation. Alternatively, the first image (1710) and the second image (1720) may be images identified by clustering images stored in the electronic device (201). Alternatively, the first image (1710) and the second image (1720) may be images additionally provided by the user.

[0194] According to one embodiment, the electronic device (201) can identify a first image portion (1711) representing the face of the first protagonist included in the first image (1710). The electronic device (201) can analyze the first image portion (1711) and identify the facial features of the first protagonist based on the analysis result. The electronic device (201) can identify a second image portion (1721) representing the face of the first protagonist included in the second image (1720) and a third image portion (1722) representing the face of the second protagonist. The electronic device (201) can analyze the second image portion (1721) and identify and update the facial features of the first protagonist based on the analysis result. The electronic device (201) can analyze the third image portion (1722) and identify the facial features of the second protagonist based on the analysis result.

[0195] Fig. 17b describes a method for analyzing and verifying the facial features of the protagonist, but the electronic device (201) can analyze and verify the motion or physical features of the protagonist in the same or similar way.

[0196] Based on the method described above, the electronic device (201) can automatically generate a video that reflects the facial features of the protagonist. Through this, the electronic device (201) can realistically generate a video intended by the user.

[0197] FIG. 18 is a diagram illustrating a method for obtaining a text prompt based on metadata (e.g., time, place, relationship) of at least one first image representing an image, according to one embodiment.

[0198] Referring to FIG. 18, according to one embodiment, the text prompt (1800) may include text (1840) indicating the story time of the video.

[0199] According to one embodiment, the text prompt (1800) may include text (1850) representing candidate genres for the story of the video (e.g., romance, fantasy, science fiction, mystery, horror, thriller, period drama, chick lit, drama, war, adventure). For example, the candidate genres for the story of the video may each correspond to a plurality of queries.

[0200] According to one embodiment, the text prompt (1800) may include text (1860) representing the protagonist and characters of the story. For example, the text (1860) may include text representing a first object (1711) included in the first protagonist (e.g., the first image (1710) of FIG. 17a) and a first object (1721) included in the second image (1720)) and a second protagonist (e.g., the second object (1722) included in the second image (1720) of FIG. 17a)) determined to be the protagonist and characters.

[0201] According to one embodiment, the text prompt (1800) may include text (1870) obtained by captioning the first image (1710) and the second image (1720). The text prompt (1800) may include text (1880) obtained by considering a plurality of responses related to the first image (1710) and the second image (1720). For example, the plurality of responses may include answers to a plurality of queries obtained through a third artificial intelligence model (e.g., the third artificial intelligence model (330) of FIG. 3).

[0202] According to one embodiment, the text prompt (1800) may include text (1890) based on the results of analyzing the metadata of the first image (1710) and the second image (1720). For example, the metadata of the first image (1710) may include the time and place where the first image (1710) was taken, and a designation (or, name) assigned to the first object (1711) (e.g., b). For example, the metadata of the second image (1720) may include the time and place where the second image (1720) was taken, a designation (or, name) assigned to the first object (1721) (e.g., b), and a designation (or, name) assigned to the second object (1722) (e.g., Ted).

[0203] According to one embodiment, the electronic device (201) can analyze the metadata of the first image (1710) and the second image (1720) to identify information regarding the names of the first object (1721) and the second object (1722), as well as the background, location, and time of the images (1710, 1720). Additionally, the electronic device (201) can obtain information regarding the relationship (e.g., lovers) between the first object (1721) and the second object (1722). The electronic device (201) can obtain text (1890) based on the information obtained by analyzing the metadata and add the obtained text (1890) to a text prompt (1800). For example, the text (1890) may include content based on information about the time (e.g., summer), place (e.g., Haeundae, Busan) where the images (1710 and 1720) were taken, and the relationship between the objects included in the images (e.g., romantic relationship).

[0204] According to one embodiment, the electronic device (201) may additionally check other images stored in the electronic device (201) that were taken at the same or similar time and place (e.g., Haeundae, Busan) as the images (1710 and 1720) were taken. Additionally, the electronic device (201) may additionally check images related to the images (1710 and 1720) (e.g., images included in the same folder, e.g., the “Ted and Busan Trip” folder). The electronic device (201) may also reflect information on the analysis of the additionally checked images in the text prompt (1800).

[0205] According to one embodiment, the electronic device (201) may generate a text prompt (1800) to reflect text (1890) obtained by analyzing metadata as an option (e.g., optional) of the story of the video. Depending on the implementation, the electronic device (201) may also generate a text prompt (1800) to reflect text (1890) obtained by analyzing metadata as a required item of the story of the video.

[0206] According to one embodiment, the electronic device may include a display, at least one processor, and a memory for storing instructions. According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may cause at least one first image to be identified among a plurality of images stored in the electronic device for creating a video. According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may cause at least one object included in the at least one first image and metadata of the at least one first image. According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may cause a character and a protagonist of the video to be determined based on at least one of the area corresponding to the at least one object or the metadata. According to one embodiment, the instructions may cause the electronic device to obtain a first text describing the at least one first image by image captioning the at least one first image when executed individually or collectively by the at least one processor. According to one embodiment, the instructions may cause the electronic device to obtain a plurality of responses to the at least one first image based on a plurality of predetermined queries for generating a story of the video when executed individually or collectively by the at least one processor.According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be caused to obtain a text prompt for generating a story of the video using information about the characters and the protagonist, the metadata, the first text, and the plurality of responses. According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be caused to obtain the story of the video based on providing the text prompt to a first generative artificial intelligence model. According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be caused to obtain the video based on providing the story and the at least one first image to a second generative artificial intelligence model.

[0207] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device (201) may be caused to obtain the plurality of responses based on providing the predetermined plurality of queries and the at least one first image to an artificial intelligence model for visual question answering (VQA), wherein the predetermined plurality of queries include queries related to the facial expression, emotion, behavior of the at least one object included in the at least one first image and the background excluding the at least one object included in the at least one first image.

[0208] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device (201) may be caused to determine the remaining objects as the characters, excluding the objects among the at least one objects included in the at least one first image whose faces are not identified.

[0209] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device (201) may cause the first object to be determined as the main character based on confirming that the size of the first area corresponding to the first object among the objects determined as the main character included in the first image is larger than the size of the second area corresponding to the second object.

[0210] According to one embodiment, the metadata may include at least one of the place where the at least one first image was taken, the time when the at least one first image was taken, or information about the relationship between the at least one object included in the at least one first image, and the relationship between the at least one object may represent a friendship, a romantic relationship, or a family relationship.

[0211] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device (201) may cause the electronic device (201) to display a first screen for selecting the character and the protagonist among the at least one object through the display, and to determine the character and the protagonist based on a first user input for the first screen.

[0212] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device (201) may be caused to identify an image including the main character, analyze the image to identify the facial features of the main character, and further consider information regarding the facial features to acquire the video.

[0213] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device (201) may stop generating the video based on confirming that the text prompt contains designated text related to at least one of violence or obscenity, and may cause the display to show notification information indicating that the text prompt contains designated text related to at least one of violence or obscenity.

[0214] According to one embodiment, the instructions, when executed individually or collectively by the at least one processor, may cause the electronic device (201) to obtain information about the at least one object and a region corresponding to the at least one object included in the at least one first image, based on providing the at least one first image to an artificial intelligence model for image segmentation.

[0215] According to one embodiment, the instructions may cause the electronic device (201), when executed individually or collectively by the at least one processor, to obtain the first text describing the at least one first image based on providing information about a region corresponding to the at least one object and the at least one first image to an artificial intelligence model for image captioning.

[0216] According to one embodiment, a method of operating an electronic device may include an operation of identifying at least one first image for generating a video among a plurality of images stored in the electronic device based on user input. According to one embodiment, a method of operating an electronic device may include an operation of identifying at least one object included in the at least one first image and metadata of the at least one first image. According to one embodiment, a method of operating an electronic device may include an operation of determining a character and a protagonist of the video based on an area corresponding to the at least one object among the at least one first image or at least one of the metadata. According to one embodiment, a method of operating an electronic device may include an operation of obtaining a first text describing the at least one first image by image captioning the at least one first image. According to one embodiment, a method of operating an electronic device may include an operation of obtaining a plurality of responses to the at least one first image based on a plurality of predetermined queries for generating a story of the video. According to one embodiment, a method of operation of an electronic device may include an operation of obtaining a text prompt containing the story of the video based on providing information about the character and the protagonist, the metadata, the first text, and the plurality of responses to a first generative artificial intelligence model. According to one embodiment, a method of operation of an electronic device may include an operation of obtaining the video based on providing the text prompt and the at least one first image to a second generative artificial intelligence model.

[0217] According to one embodiment, a method of operation of an electronic device (201) may include the operation of obtaining the plurality of responses based on providing the predetermined plurality of queries and the at least one first image to an artificial intelligence model for visual question answering (VQA), wherein the predetermined plurality of queries include queries related to the facial expression, emotion, and behavior of the at least one object included in the at least one first image, and the background excluding the at least one object included in the at least one first image.

[0218] According to one embodiment, the method of operation of the electronic device (201) may include the operation of determining the remaining objects as the characters, excluding the object whose face is not identified among the at least one object included in the at least one first image.

[0219] According to one embodiment, the method of operation of the electronic device (201) may include the operation of determining the first object as the main character based on confirming that the size of the first area corresponding to the first object among the objects determined to be the characters included in the first image is larger than the size of the second area corresponding to the second object.

[0220] According to one embodiment, the metadata includes at least one of information about the place where the at least one first image was taken, the time when the at least one first image was taken, or the relationship between the at least one object included in the at least one first image, and the relationship between the at least one object may represent a friendship, a romantic relationship, or a family relationship.

[0221] According to one embodiment, the method of operation of the electronic device (201) may include displaying a first screen for selecting the character and the protagonist among the at least one object through the display, and confirming the character and the protagonist based on a first user input for the first screen.

[0222] According to one embodiment, the operation of acquiring the video may include the operation of verifying an image including the main character, the operation of verifying the facial features of the main character by analyzing the image, and the operation of acquiring the video by further considering information regarding the facial features.

[0223] According to one embodiment, a method of operation of an electronic device (201) may include an operation of stopping the generation of the video based on confirming that the text prompt contains a designated text related to at least one of violence or obscenity, and an operation of displaying, through the display, notification information indicating that the text prompt contains the designated text related to at least one of the violence or obscenity.

[0224] According to one embodiment, a method of operation of an electronic device (201) may include an operation of obtaining information about an at least one object and a region corresponding to the at least one object included in the at least one first image, based on providing the at least one first image to an artificial intelligence model for image segmentation.

[0225] According to one embodiment, in a non-transient storage medium storing computer-readable instructions, the instructions cause the electronic device to perform at least one operation when executed by at least one processor of the electronic device, and the at least one operation is, based on user input, an operation of identifying at least one first image among a plurality of images stored in the electronic device for generating a video; an operation of identifying at least one object included in the at least one first image and metadata of the at least one first image; an operation of determining a character and a protagonist of the video based on at least one of the region corresponding to the at least one object or the metadata; an operation of obtaining a first text describing the at least one first image by image captioning the at least one first image; an operation of obtaining a plurality of responses to the at least one first image based on a plurality of predetermined queries for generating a story of the video; and a text prompt including the story of the video based on providing information about the character and the protagonist, the metadata, the first text, and the plurality of responses to a first generative artificial intelligence model. The method may include an operation to acquire the video based on acquiring the text prompt and / or providing the at least one first image to a second generative artificial intelligence model.

[0226] The electronic device according to the various embodiments disclosed in this document may be of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a consumer electronics device. The electronic device according to the embodiments of this document is not limited to the devices described above.

[0227] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, each of phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as “first,” “second,” or “first” or “second” may be used simply to distinguish a component from another component and do not limit the components in any other aspect (e.g., importance or order). Where any (e.g., first) component is referred to as “coupled” or “connected” to another (e.g., second) component, with or without the terms “functionally” or “communicationally,” it means that said component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0228] The term “module” as used in the various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0229] Various embodiments of this document may be implemented as software (e.g., program (140)) comprising one or more instructions stored in a storage medium (e.g., internal memory (136) or external memory (138)) readable by a machine (e.g., electronic device (101, 201)). For example, a processor (e.g., processor (120, 220)) of the machine (e.g., electronic device (101, 201)) may call at least one of the one or more instructions stored from the storage medium and execute it. This enables the machine to be operated to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' is a device in which the storage medium is tangible, and It merely means that it does not contain a signal (e.g., electromagnetic waves), and this term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily on a storage medium.

[0230] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0231] According to various embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to various embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In an electronic device, display; At least one processor; and It includes memory for storing instructions, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, Based on user input, at least one first image for generating a video among a plurality of images stored in the electronic device is identified, and Identify at least one object included in the at least one first image and the metadata of the at least one first image, and Based on a region corresponding to at least one of the above-mentioned object or at least one of the above-mentioned metadata, the characters and protagonists of the above-mentioned video are determined, and The above at least one first image is image captioned to obtain a first text describing the above at least one first image, and Obtaining multiple responses for at least one first image based on multiple predetermined queries for generating the story of the above video, and A text prompt for generating the story of the video using information about the above characters and the above protagonist, the above metadata, the above first text, and the above plurality of responses, and Based on providing the above text prompt to a first generative artificial intelligence model, the above story of the above video is obtained, and An electronic device that causes the video to be acquired based on providing the above story and at least one first image to a second generative AI model.

2. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, Based on providing the aforementioned predetermined plurality of queries and the above at least one first image to an artificial intelligence model for visual question answering (VQA), the plurality of responses are obtained, and An electronic device comprising the above-mentioned multiple pre-specified queries, which include queries related to the facial expression, emotion, and behavior of the at least one object included in the at least one first image, and the background excluding the at least one object included in the at least one first image.

3. In any one of paragraphs 1 to 2, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, An electronic device that causes the remaining objects, excluding the object whose face is not identified among the at least one object included in the at least one first image, to be determined as the characters.

4. In any one of paragraphs 1 through 3, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, An electronic device that causes the first object to be determined as the protagonist based on confirming that the size of the first area corresponding to the first object among the objects determined as the characters included in the first image is larger than the size of the second area corresponding to the second object.

5. In any one of paragraphs 1 through 4, The above metadata includes at least one of information about the location where the at least one first image was taken, the time at least one first image was taken, or the relationship between the at least one object included in the at least one first image, and An electronic device representing a relationship between at least one of the above objects, such as a friendship, a romantic relationship, or a family relationship.

6. In any one of paragraphs 1 through 5, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, An electronic device that displays a first screen for selecting the character and the protagonist among at least one object through the above display, and causes the character and the protagonist to be determined based on a first user input to the first screen.

7. In any one of paragraphs 1 through 6, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, Check the image including the aforementioned protagonist, By analyzing the above image to identify the facial features of the above protagonist, and An electronic device that causes the acquisition of the above video by further considering information regarding the above facial features.

8. In any one of paragraphs 1 through 7, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, Based on confirming that the above text prompt contains specified text related to at least one of violence or obscenity, stop generating the above video, and An electronic device that causes the text prompt to display notification information indicating that the text prompt includes the specified text related to at least one of the violence or the obscenity, through the display.

9. In any one of paragraphs 1 through 8, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, An electronic device that causes to obtain information about at least one object and a region corresponding to at least one object included in the at least one first image, based on providing at least one first image to an artificial intelligence model for image segmentation.

10. In any one of paragraphs 1 through 9, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, An electronic device that causes to obtain the first text describing the at least one first image based on providing information about the region corresponding to the at least one object and the at least one first image to an artificial intelligence model for image captioning.

11. In a method of operating an electronic device, An operation of identifying at least one first image for generating a video among a plurality of images stored in the electronic device based on user input; An operation to verify at least one object included in at least one first image and the metadata of at least one first image; An operation to determine the characters and protagonists of the video based on a region corresponding to at least one of the above-mentioned object or at least one of the above-mentioned metadata; The operation of image captioning at least one first image to obtain a first text describing the at least one first image; The operation of obtaining a plurality of responses to the at least one first image based on a plurality of predetermined queries for generating a story of the above video; The operation of obtaining a text prompt for generating a story of the video using information about the above-mentioned characters and the above-mentioned protagonist, the above-mentioned metadata, the above-mentioned first text, and the above-mentioned plurality of responses; An operation to acquire the story of the video based on providing the above text prompt to a first generative artificial intelligence model; and A method of operation of an electronic device comprising the operation of acquiring the video based on providing the above story and at least one first image to a second generative artificial intelligence model.

12. In Paragraph 11, The method further includes the operation of obtaining the plurality of responses based on providing the aforementioned predetermined plurality of queries and the above at least one first image to an artificial intelligence model for visual question answering (VQA). The above-mentioned pre-specified multiple queries are, A method of operation of an electronic device comprising the facial expression, emotion, and behavior of the at least one object included in the at least one first image, and queries related to the background excluding the at least one object included in the at least one first image.

13. In any one of paragraphs 11 to 12, A method of operation of an electronic device further comprising the operation of determining the remaining objects as the characters, excluding the objects among the at least one object included in the at least one first image whose faces are not identified.

14. In any one of paragraphs 11 through 13, A method of operation of an electronic device further comprising the operation of determining the first object as the main character based on confirming that the size of the first area corresponding to the first object among the objects determined to be the characters included in the first image is larger than the size of the second area corresponding to the second object.

15. In a non-transient storage medium for storing computer-readable instructions, said instructions cause said electronic device (201) to perform at least one operation when executed by at least one processor (220) of said electronic device, said at least one operation is: An operation of identifying at least one first image for generating a video among a plurality of images stored in the electronic device based on user input; An operation to verify at least one object included in at least one first image and the metadata of at least one first image; An operation to determine the characters and protagonists of the video based on a region corresponding to at least one of the above-mentioned object or at least one of the above-mentioned metadata; The operation of image captioning at least one first image to obtain a first text describing the at least one first image; The operation of obtaining a plurality of responses to the at least one first image based on a plurality of predetermined queries for generating a story of the above video; The operation of obtaining a text prompt for generating a story of the video using information about the above-mentioned characters and the above-mentioned protagonist, the above-mentioned metadata, the above-mentioned first text, and the above-mentioned plurality of responses; An operation to acquire the story of the video based on providing the above text prompt to a first generative artificial intelligence model; and A storage medium comprising the operation of acquiring the video based on providing the above story and at least one first image to a second generative artificial intelligence model.

Images