Method for Generating Video Creation Prompts Based on Multimodal Reference Analysis and Video Creation System Using the Same

Abstract

A method for generating an image generation prompt based on multimodal reference analysis according to an embodiment of the present invention is performed in an image generation device and comprises: receiving a multimodal reference including an image or video from a user; performing feature analysis according to a plurality of abstraction levels on the multimodal reference to extract multilayer visual features on the multimodal reference; converting the extracted multilayer visual features into feature representation data; and generating prompt data for image generation from the feature representation data using a semantic feature mapping network that learns the mapping relationship between the feature representation data and control parameters of a generative artificial intelligence model.

Description

Technology Field

[0001] The present invention relates to the field of image generation technology using Generative AI, and more specifically, to a method for analyzing multimodal references provided in multiple forms, such as images and videos, and automatically generating a prompt for image generation based on the analysis results, and a system for automatically generating an image through a Generative AI model based on the prompt thus generated as a condition. Background Technology

[0002] With the recent advancement of Generative AI technology, various tools that generate images or videos based on the meaning of text prompts entered by users are being commercialized. For example, generative video tools such as Stable Diffusion, Midjourney, and Runway perform video synthesis based on given text descriptions.

[0003] However, this text-based image generation method has the following drawbacks. First, obtaining the desired result requires a high level of expertise in drafting prompts, and effective results can only be achieved by understanding the model's characteristics, parameters, and representation techniques.

[0004] In addition, there is a problem in that it is difficult to accurately express complex visual concepts, such as subtle visual elements like specific mood, style, composition, and relationships between objects, using text alone.

[0005] Furthermore, language barriers exist for non-English speakers, and expressions different from the language and culture learned by the model are often not reflected as intended. Additionally, the limited transmission of abstract concepts, emotions, and genre characteristics can lead to a gap between the user's desired outcome and the actual product.

[0006] To overcome these limitations, methods utilizing reference images or reference videos have been proposed in some technologies. However, conventional reference-based techniques have a limited scope of application, such as extracting only specific styles from a single image, and lack the capability to process multiple references simultaneously or to comprehensively analyze and fuse multilayered features of low, medium, and high levels.

[0007] Furthermore, conventional technology has limitations in that the process of mapping features extracted from reference media to input parameters of a generative model is not standardized, making it difficult to predict or control the quality of the desired output in real time. Prior art literature

[0008] Korean Patent Publication No. 10-2779588 (Date of publication: March 6, 2025) The problem to be solved

[0009] The present invention aims to solve the aforementioned problems. An embodiment according to the present disclosure receives various forms of multimodal references, such as images and videos, and multi-layeredly extracts visual features such as color, composition, object placement, movement patterns, atmosphere, style, and genre characteristics from the references. By automatically generating a prompt for video generation based on these features, the invention provides a technology that enables even non-experts to intuitively reflect the desired style and composition in the video generation process.

[0010] In addition, one embodiment according to the present disclosure aims to provide a multimodal feature mapping technology capable of selectively fusing features of multiple references by simultaneously analyzing the features of multiple references and providing an attention-based feature importance calculation and weight adjustment function based on user selection.

[0011] An embodiment according to the present disclosure aims to provide an advanced image generation technology that enables precise control over the style and composition of the desired result by the user, by converting extracted features into conditional inputs in conjunction with ControlNet, IP-Adapter, etc., without changing the structure of a generative artificial intelligence model, and by supporting advanced control functions such as applying negative references, extracting features by partial region, applying references by time interval, and adjusting style intensity in real time.

[0012] An embodiment according to the present disclosure aims to provide an integrated image generation system capable of generating customized images with high reproducibility and quality in real time across various industrial fields by integrating the above-described multimodal feature extraction, semantic-based mapping, conditional image generation, and advanced control functions into a single pipeline, thereby overcoming the linguistic and cultural limitations, lack of expressiveness, and style control constraints of existing text prompt-dependent methods.

[0013] However, the problems to be solved in this disclosure are not limited to those mentioned above, and may be expanded in various ways without departing from the spirit and scope of this disclosure. means of solving the problem

[0014] One technical aspect of the present invention proposes a method for generating an image generation prompt based on multimodal reference analysis. The method for generating an image generation prompt based on multimodal reference analysis is performed in an image generation device and comprises: receiving a multimodal reference including an image or video from a user; performing feature analysis on the multimodal reference according to a plurality of abstraction levels to extract multilayer visual features on the multimodal reference; converting the extracted multilayer visual features into feature representation data; and generating prompt data for image generation from the feature representation data using a semantic feature mapping network that learns the mapping relationship between the feature representation data and the control parameters of a generative artificial intelligence model.

[0015] In one embodiment, the method may further include the step of modifying the prompt data by reflecting user-specified control parameters in the prompt data; and the step of providing the modified prompt data to a generative artificial intelligence model to control the generation of an image using the generative artificial intelligence model.

[0016] In one embodiment, the step of extracting multilayer visual features for the multimodal reference may include: a step of preprocessing the multimodal reference to generate analysis target data including visual information and time information; and a step of performing feature analysis according to a plurality of abstraction levels on the analysis target data to extract multilayer visual features for the multimodal reference including at least one of low-level visual attributes, medium-level visual attributes, and high-level visual attributes.

[0017] In one embodiment, the step of extracting multilayer visual features for the multimodal reference may further include, when there are multiple multimodal references, a step of individually calculating the multilayer visual features for each multimodal reference; a step of calculating importance weights for each multimodal reference; and a step of generating integrated multilayer visual features for multiple multimodal references by integrating the importance weights with the individually calculated multilayer visual features.

[0018] In one embodiment, the step of extracting multilayer visual features for the multimodal reference may include: applying a spatial mask to selectively apply features of different multimodal references to different spatial regions; distinguishing time intervals to selectively apply features of different multimodal references to different time intervals; and applying a transition function at the boundaries of the different regions or time intervals to mitigate seams between features.

[0019] In one embodiment, the multimodal reference includes a negative reference, and the step of generating prompt data for image generation from the feature representation data may include: identifying a visual attribute to be excluded corresponding to the negative reference; and including control information to suppress the visual attribute to be excluded in the prompt data.

[0020] In one embodiment, the multimodal reference includes a negative reference, and the step of generating prompt data for image generation from the feature representation data may include: identifying a visual attribute to be excluded from the negative reference; and adding a constraint to the semantic feature mapping network that minimizes the similarity to the visual attribute to be excluded.

[0021] In one embodiment, the prompt data for generating the image may include a text instruction generated based on the multimodal reference; and attribute specification information regarding the scene, composition, color, style, mood, or specific object of the image. The attribute specification information may include at least one of control information that suppresses the visual attribute to be excluded corresponding to the negative reference, or a constraint that minimizes the similarity to the visual attribute to be excluded.

[0022] Another technical aspect of the present invention proposes an image generation system based on multimodal reference analysis. The image generation system based on multimodal reference analysis comprises: a user terminal in which a user provides a multimodal reference including an image or video; an image generation device connected to communicate with the user terminal, receiving the multimodal reference, performing feature analysis on the multimodal reference according to a plurality of abstraction levels to extract multilayer visual features, converting the extracted multilayer visual features into feature representation data, and generating prompt data for image generation from the feature representation data using a semantic feature mapping network that learns the mapping relationship between the feature representation data and the control parameters of a generative artificial intelligence model; and an image generation artificial intelligence based on an image generation artificial intelligence model that receives the prompt data and automatically generates an image according to the prompt. Effects of the invention

[0023] According to various embodiments of the present disclosure, by analyzing multimodal references such as images and videos, a video generation prompt based on the reference can be automatically generated without changing the structure of a pre-trained generative artificial intelligence model, so that even a user without professional knowledge of creating prompts can intuitively generate a video that reflects the desired style and composition.

[0024] According to various embodiments of the present disclosure, by simultaneously analyzing the features of multiple references and providing attention-based importance calculation and user weight adjustment functions, low-level, medium-level, and high-level features of multiple references are selectively fused, thereby enabling more precise and diverse style expressions than existing single-reference-based generation methods.

[0025] According to various embodiments of the present disclosure, styles or features that the user wishes to exclude can be automatically detected and suppressed by applying a negative reference, thereby enabling high-quality results with minimized unnecessary or unintended visual elements.

[0026] According to various embodiments of the present disclosure, by integrating and supporting advanced control functions such as partial region-specific feature extraction, time interval-specific reference application, and real-time style intensity adjustment, detailed elements of the generated image can be precisely controlled, and work efficiency can be improved through iterative adjustment based on real-time preview.

[0027] According to various embodiments of the present disclosure, by integrating multimodal feature extraction, semantic-based mapping, conditional input transformation, and advanced control functions into a single pipeline, the limitations of expression and language barriers of existing text-centric prompt methods can be overcome, and a flexible and highly scalable image generation infrastructure capable of consistently generating customized high-quality images in various industrial fields can be provided.

[0028] The effects obtainable from the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art to which the present disclosure belongs from the description below. Brief explanation of the drawing

[0029] FIG. 1 is a block diagram showing the overall configuration of a multimodal reference analysis-based image generation system according to one embodiment of the present invention. FIG. 2 is a flowchart illustrating a method for generating an image generation prompt based on multimodal reference analysis according to an embodiment of the present invention. FIG. 3 is a block diagram illustrating an example configuration of an image generation device (100) based on multimodal reference analysis according to one embodiment of the present invention. FIG. 4 is a flowchart illustrating the process of generating exclusion attribute suppression control conditions using a negative reference according to one embodiment of the present invention. FIG. 5 is a block diagram illustrating an internal configuration module of a semantic feature mapping network (150) according to one embodiment of the present invention. FIG. 6 is a block diagram illustrating an example of the internal configuration of a prompt generation unit (160) according to one embodiment of the present invention. FIG. 7 is a flowchart illustrating a method for generating final prompt data based on user control parameter reflection according to an embodiment of the present invention. Specific details for implementing the invention

[0030] Hereinafter, embodiments of the present disclosure are described in detail with reference to the drawings so that those skilled in the art can easily practice them. However, the present disclosure may be embodied in various different forms and is not limited to the embodiments described herein. In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Furthermore, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and brevity.

[0031] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may each include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order). Where any (e.g., 1st) component is referred to as "coupled" or "connected" to another (e.g., 2nd) component, with or without the terms "functionally" or "communicationly," it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0032] The term “module” as used in the various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0033] Various embodiments of this document may be implemented as software (e.g., a program) comprising one or more instructions stored in a storage medium (e.g., memory) readable by a machine or device. For example, the processor of the machine or device may call at least one of the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by a machine may be provided in the form of a non-transitory storage medium. Here, "non-transitory" simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and this term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily in the storage medium.

[0034] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0035] According to various embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to various embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0036] In this disclosure, the term "processor" may refer to hardware capable of performing functions and operations according to each designation described herein, computer program code capable of performing specific functions and operations, or an electronic recording medium loaded with computer program code capable of performing specific functions and operations. According to an embodiment, the operation of the processor may be defined and / or interpreted as the operation of an image generating device, but is not limited thereto. The term "processor" may refer to a functional and / or structural combination of hardware for carrying out the technical concept of this disclosure and / or software for driving said hardware.

[0037] The image generation device in the present disclosure may be defined and / or referred to as an image generation server or a service server, which may mean a physically independent server but is not limited thereto, may be a virtual machine, and may be configured to encompass a module, program, or Docker running on a virtual or physical machine.

[0039] The meanings of the terms used in this disclosure are as follows.

[0040] "Multimodal reference" refers to reference data that includes two or more different modality data, such as images, video, and audio, which are referenced during the video generation process. Multimodal reference may include various sensory information regarding a single object or scene, and its sources are not limited to files uploaded by users, external databases, real-time captures, etc.

[0041] "Multilayered visual features" refers to a set of features extracted by analyzing visual information contained in a multimodal reference at different levels of abstraction, and may include, for example, low-level visual attributes (color distribution, saturation, brightness, texture patterns, etc.), mid-level visual attributes (object placement, composition, movement, gaze direction, depth relationships, etc.), and high-level visual attributes (scene mood, style, genre characteristics, semantic relationships, etc.).

[0042] "Feature representation data" refers to a data structure in which extracted multilayer visual features are transformed into a vector, matrix, or embedding form that can be processed by machine learning models. This data can be used for input to generative artificial intelligence models, control parameter mapping, prompt generation, and the like.

[0043] A "semantic feature mapping network" refers to a neural network-based module that learns the mapping relationship between feature representation data and control parameters of a generative artificial intelligence model to produce prompt data or control information corresponding to the input feature representation data.

[0044] "Prompt data" is data containing instructions and attribute specification information in a format recognizable by an image generation model, and may include text prompts, attribute tags, condition parameters, etc. Prompt data may specify or exclude the scene, composition, color, style, mood, or attributes of a specific object of the image.

[0045] "Negative reference" refers to a multimodal reference that specifies objects, colors, styles, scene elements, etc., to be excluded, serving as reference data intended to induce the exclusion of specific visual attributes during the image generation process.

[0047] FIG. 1 is a block diagram showing the overall configuration of a multimodal reference analysis-based image generation system according to one embodiment of the present invention.

[0048] As illustrated in FIG. 1, the image generation system of the present invention may be configured to include an image generation device (100), a user device (200), and an image generation artificial intelligence model (300).

[0049] The user device (200) provides a user interface that enables uploading of a multimodal reference, input of control parameters, and verification of the generated image, and can be connected to the image generation device (100) via a wired or wireless communication network.

[0050] The image generation artificial intelligence model (300) is a text-to-image or text-to-video generation model and can automatically generate an image (image or video clip) that satisfies specified attributes and constraints based on prompt data input from the image generation device (100).

[0051] The image generation device (100) includes at least one processor (101) and a memory (102), and performs the function of receiving a multimodal reference including an image or video from a user device (200), analyzing it, and generating prompt data for image generation.

[0052] The image generation device (100) can perform feature analysis according to multiple levels of abstraction on a multimodal reference to extract multilayered visual features including low-level, medium-level, and high-level visual attributes.

[0053] The image generation device (100) can convert the extracted multilayer visual features into vectorized feature representation data and generate prompt data through a semantic feature mapping network that learns the mapping relationship between the data and the control parameters of the image generation artificial intelligence model (300).

[0054] The image generation device (100) can reflect control parameters provided by the user in the prompt data, or reflect visual attributes to be excluded corresponding to negative references as suppression or similarity minimization constraints.

[0055] The image generation device (100) can provide the generated prompt data to the image generation artificial intelligence model (300) to control the model to automatically generate an image that reflects the scene, composition, color, style, atmosphere, and attributes of a specific object.

[0056] The processor (101) may include, as an example, at least one of a microprocessor, a central processing unit, a processor core, a multi-core processor, a multiprocessor, an Application-Specific Integrated Circuit (ASIC), or a Field Programmable Gate Array (FPGA), but is not limited thereto.

[0057] Memory (102) can store instructions (or programs) executable by the processor (110). Memory (120) may include volatile memory or non-volatile memory.

[0058] Volatile memory can be implemented as DRAM (dynamic random access memory), SRAM (static random access memory), T-RAM (thyristor RAM), Z-RAM (zero capacitor RAM), or TTRAM (Twin Transistor RAM).

[0059] Non-volatile memory can be implemented as EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory, MRAM (Magnetic RAM), Spin-Transfer Torque (STT)-MRAM, Conductive Bridging RAM (CBRAM), FeRAM (Ferroelectric RAM), PRAM (Phase change RAM), Resistive RAM (RRAM), Nanotube RRAM, Polymer RAM (PoRAM), Nano Floating Gate Memory (NFGM), holographic memory, Molecular Electronic Memory Device, or Insulator Resistance Change Memory.

[0060] In FIG. 1, the image generation artificial intelligence model (300) is shown separately from the image generation device (100), but is not limited thereto and may be implemented as a function of the image generation device (100) depending on the embodiment. For example, the processor (101) may include the image generation artificial intelligence model.

[0062] FIG. 2 is a flowchart illustrating a method for generating an image generation prompt based on multimodal reference analysis according to an embodiment of the present invention.

[0063] A method for generating an image generation prompt based on multimodal reference analysis (hereinafter referred to as the "image generation prompt generation method") described with reference to FIG. 2 can be performed by a video generation device (100) illustrated in FIG. 1.

[0064] In step S210, the image generating device (100) receives a multimodal reference containing an image or video from a user device (200).

[0065] This multimodal reference may be a single piece of data, or it may be in the form of a combination of multiple images, video, audio data, or text.

[0066] Next, in step S220, the image generating device (100) performs feature analysis according to multiple levels of abstraction on the received multimodal reference. Through this, multilayered visual features including at least one of low-level visual attributes, medium-level visual attributes, and high-level visual attributes can be extracted.

[0067] For example, low-level attributes include color histograms, saturation / brightness values, edge distribution, etc., mid-level attributes include object layout, character pose, background structure, etc., and high-level attributes may include the scene's theme, emotional mood, shooting style, etc.

[0068] In one embodiment, when there are multiple multimodal references, the image generation device (100) can individually calculate multilayer visual features for each reference, calculate importance weights for each reference, and integrate them to generate final multilayer visual features.

[0069] In one embodiment, the image generating device (100) may selectively reflect different reference features for each different spatial region by applying a spatial mask, or reflect different reference features for each interval by dividing time intervals, and in this case, a transition function may be applied to the boundary area to mitigate seams between features.

[0070] In one embodiment, the image generation device (100) can process by reflecting a negative reference. The negative reference is reference data intended to induce the exclusion of specific visual attributes during the image generation process, and may specify objects, colors, styles, scene elements, etc. that are not to be included, and may be set by the user.

[0071] In one embodiment, if the negative reference image provided by the user is a 'foggy background,' the image generation device (100) extracts spatial frequency characteristics related to the fog pattern, the ratio of low-saturation and high-brightness regions, and the degree of edge blur during the feature analysis process. These can then be tagged as 'visual attributes to be excluded.'

[0072] In one embodiment, if the negative reference has a specific hue (e.g., a strong green tone), the image generating device (100) can identify the proportion of the corresponding hue in the color histogram and generate a color control parameter to suppress it in a subsequent step.

[0073] In one embodiment, the image generating device (100) can compare and analyze a plurality of negative references to preferentially identify common visual attributes and aggregate the feature quantities of the attributes to assign exclusion priority.

[0074] In step S230, the image generation device (100) converts the extracted multilayer visual features into vectorized feature representation data. This feature representation data is structured so that it can subsequently be used as an input for controlling a generative artificial intelligence model.

[0075] In step S240, the image generation device (100) generates prompt data for image generation from the corresponding feature representation data using a semantic feature mapping network that learns the mapping relationship between the feature representation data and the control parameters of the generative artificial intelligence model.

[0076] In one embodiment, the prompt data may include two main components: a text instruction and an attribute specification. The text instruction is a phrase describing the main scenes, style, and mood of the image in a language format recognizable by a generative AI model, and the attribute specification may be a structured control parameter including scene composition, color palette, specific object placement, style constraints, etc.

[0077] In one embodiment, prompt data may include designation information regarding the scene, composition, color, style, mood, and specific object attributes of the image along with text instructions, and control information for suppressing visual attributes to be excluded or similarity minimization constraints may be reflected within this designation information.

[0078] As in the example above, a multimodal reference may include a negative reference, and in this case, control information to identify the visual attribute to be excluded corresponding to the negative reference and to suppress it may be included in the prompt data.

[0079] For example, a positive reference-based prompt could be "A cinematic view of a mountain valley at sunrise, vibrant colors, wide-angle perspective, with flowing river and birds flying", and a negative reference-based constraint could be applied as "--exclude foggy background, --minimize similarity to misty textures, avoid greenish color cast".

[0080] In one embodiment, the image generation device (100) can prevent the attribute from being reflected in the image by adding a constraint to the semantic feature mapping network that minimizes the similarity to the visual attribute to be excluded.

[0081] Finally, in step S250, the image generation device (100) provides the prompt data generated as above to the image generation artificial intelligence model (300) to control the model to automatically generate an image reflecting specified attributes and constraints. The generated image can then be transmitted to the user device (200) and provided to the user.

[0083] FIG. 3 is a block diagram illustrating an example configuration of an image generation device (100) based on multimodal reference analysis according to one embodiment of the present invention.

[0084] The image generation device (100) illustrated in FIG. 3 may include a multimodal reference input unit (110), a preprocessing unit (120), a multilayer feature extraction unit (130), a feature representation conversion unit (140), a semantic feature mapping network (150), and a prompt generation unit (160).

[0085] The image generation device (100) is connected to communicate with a user terminal and, when necessary, transmits prompt data to an external image generation artificial intelligence model (300) to control the automatic generation of an image according to specified conditions.

[0086] The multimodal reference input unit (110) can receive a multimodal reference including an image or video from a user device.

[0087] Multimodal references can be one or multiple, and users may provide additional negative references to specify attributes they wish to exclude during the creation process.

[0088] For example, when user A uploads ‘a still image A with a cinematic tone (warm color tone, film grain)’, ‘a video B containing a person’s walking motion’, and ‘a negative video C containing a scene of thick fog’, the input unit (110) can add metadata such as the format, resolution, length, and timestamp of each reference and transmit it to the preprocessing unit (120).

[0089] In addition, if the user specifies control parameters such as prompt intensity, tone symbol, number of scenes, and length, such information can be utilized in the subsequent prompt correction process.

[0090] The preprocessing unit (120) can normalize the input multimodal reference into analysis target data suitable for analysis. For example, it can perform processing such as resolution normalization, color space unification, noise reduction, frame sampling, scene division, and timeline alignment.

[0091] The video reference is converted into a keyframe sequence, and each frame is tagged with position / time information. For example, 120 frames can be extracted by sampling at 2 frames per second from video B, and the shooting time and scene boundaries can be marked on each frame.

[0092] The multilayer feature extraction unit (130) can extract multilayer visual features including low-level visual attributes (color distribution, saturation / brightness, contrast, texture / edge pattern, etc.), medium-level visual attributes (object placement, composition, person pose / gaze, depth / layer relationship, camera work and movement pattern, etc.), and high-level visual attributes (scene atmosphere, style / genre, narrative / symbolism, domain-specific style, etc.) by performing feature analysis according to multiple levels of abstraction on the preprocessed data.

[0093] In addition, the multilayer feature extraction unit (130) can selectively reflect different reference features for each different spatial region or time interval through a spatial mask and a time interval separation function, and can apply a transition function in the boundary region to mitigate seams.

[0094] For example, pose and movement characteristics of image B are applied to the person area, and color and grain of image A are applied to the background area, and feathering can be applied in the scene transition section to suppress flicker.

[0095] When multiple references are input, the multilayer feature extraction unit (130) can independently calculate multilayer features for each reference and then generate integrated features by assigning importance weights.

[0096] The feature representation conversion unit (140) can convert the extracted multilayer visual features into feature representation data of a machine learning-friendly structure.

[0097] For example, color information is mapped to palette parameters, composition to layout keypoints and heatmaps, motion to motion embeddings, and mood and style to semantic tokens, and the result of integrating multiple references is aligned into a single integrated representation in the same representation space.

[0098] The semantic feature mapping network (150) is a neural network that learns the mapping relationship between feature representation data and control parameters of an image generation artificial intelligence model, and can support two types of suppression as follows when a negative reference is provided.

[0099] First, there is a method of explicitly specifying suppression attributes at the prompt stage. For example, if the negative reference is a 'dense fog' scene, the semantic feature mapping network (150) directly specifies the attribute in the attribute specification information of the prompt data as "exclude: dense_fog" so that the attribute does not appear when the image is generated.

[0100] Second, there is a method of implicitly suppressing during the learning or inference process. In this case, the semantic feature mapping network (150) extracts feature vectors such as the low-saturation-high-brightness pattern and texture distribution of 'fog' from the negative reference, and includes a constraint term in the loss function that minimizes similarity with the feature vector of the target to be generated. This method has the effect of automatically suppressing the corresponding attribute in the generated result, even if the prohibited attribute is not directly written in the prompt.

[0101] The prompt generation unit (160) can configure prompt data for image generation using the output of the semantic feature mapping network (150).

[0102] Prompt data consists of text instructions and attribute specification information. The attribute specification information may include information regarding scenes, composition, color, style, mood, and specific objects, as well as control information or similarity minimization constraints that suppress visual attributes targeted for exclusion corresponding to negative references. Additionally, control parameters set by the user, such as length, frame rate, number of scenes, and style intensity, may be reflected.

[0103] The prompt generation unit (160) configures prompt data for image generation using the output of the semantic feature mapping network (150), and if a negative reference is provided, it can be processed as follows.

[0104] For example, in the explicit suppression method, the exclusion attribute corresponding to the negative reference is directly included in the attribute specification information. For instance, by writing it as negative_control: {exclude: ["dense_fog"]}, the corresponding attribute is forcibly excluded during the generation process.

[0105] As another example, in the implicit suppression method, the exclusion attribute is not specified in the prompt data, but the similarity minimization constraint is already applied in the semantic feature mapping network (150), so the attribute does not appear in the result. This allows the prompt text to be concise while maintaining quality and control effects.

[0106] The prompt data configured in this way is provided as a conditional input to the image generation artificial intelligence model (300), thereby controlling the automatic generation of an image that reflects specified attributes and constraints.

[0108] FIG. 4 is a flowchart illustrating the process of generating exclusion attribute suppression control conditions using a negative reference according to one embodiment of the present invention.

[0109] The embodiment of FIG. 4 relates to a procedure for receiving and analyzing negative references, identifying visual attributes to be excluded, generating control information for suppression and similarity minimization constraints, and reflecting them in prompt data.

[0110] In step S410, the image generation device (100) receives a negative reference from the user. The negative reference may be an image or image data containing specific visual attributes that the user wishes to exclude from the generated result.

[0111] For example, if a user does not want "dense fog" scenes to be included, they can upload actual footage of a foggy landscape. In this case, the input module collects metadata such as the format, resolution, length, and time of shooting of the reference and uses it for subsequent analysis.

[0112] In step S420, the image generation device (100) analyzes the negative reference to identify visual attributes to be excluded. The analysis can derive items to be excluded by extracting low-level (color, brightness, texture, etc.), medium-level (object placement, pose, movement, etc.), and high-level (genre, atmosphere, symbolism, etc.) attributes through a multi-layered feature extraction algorithm and comparing them with user-generated goals.

[0113] For example, if low-saturation, high-luminosity colors, blurred edges, or specific texture patterns are found in a negative image, they are identified as exclusion attributes such as "fog texture" and "low-saturation-high-luminosity combination."

[0114] In step S430, the image generating device (100) generates control information for suppression regarding the identified attribute. The control information for suppression may include direct instructions or parameters that prevent the attribute from appearing when the prompt is generated.

[0115] For example, items such as "exclude: dense_fog" or "exclude: fog" may be added to the control information, and when generating a style token, the method of deleting the corresponding attribute or setting the intensity to 0 is applied.

[0116] In step S440, the image generation device (100) generates a similarity minimization constraint for the identified attributes. This can function as an optimization item that causes the generation model to minimize visual and semantic similarity with the attributes during the learning or generation process. For example, a similarity measure (cosine similarity, L2 distance, etc.) can be calculated by referencing a feature vector called "fog_texture," and a constraint can be set so that the value is 0.1 or less. This induces the resulting image not to reproduce characteristics similar to negative attributes.

[0117] In step S450, the image generating device (100) generates final prompt data by reflecting the generated control information and constraints in the prompt data configuration process.

[0118] This prompt data may include not only text directives and attribute specification information, but also control information and constraints necessary for suppressing negative attributes. For example, structured items such as {negative_control: {exclude: ["dense_fog"], similarity_constraints: {fog_texture: 0.1}}} may be inserted into the attribute specification information of the final prompt.

[0119] To explain with an example of processing a single negative reference, when a user intends to generate an image of a "warm-toned evening landscape," an image of a "dawn mountain covered in thick fog" can be provided as a negative reference. The image generation device (100) extracts visual attributes such as fog texture, blurred edges, and low-saturation / high-brightness tones from the negative image and identifies them as attributes to be excluded. Subsequently, suppression control information (S430) such as "exclude: dense_fog" and a constraint (S440) such that the fog_texture similarity is 0.1 or less are generated. Reflecting this, the prompt generation unit constructs a final prompt that includes structured parameters for suppressing negative attributes, along with instructions such as "warm-toned evening mountain range, clear sky, sharp outline."

[0120] To illustrate with an example of multiple negative reference processing, a user may wish to exclude both "dense fog" and "low-light indoor shooting" attributes. In this case, the image generation device (100) receives negative references C1 (dense fog image) and C2 (low-light indoor image) together in S410. In S420, each reference is analyzed independently to identify attribute sets {dense_fog} and {low_light, indoor_shadow}. In S430, the two attribute sets are combined to generate control information exclude: [dense_fog, low_light, indoor_shadow], and in S440, similarity minimization constraints are set in parallel for each attribute. This ensures that multiple unnecessary attributes are suppressed simultaneously in the final product.

[0121] To explain with an example of attribute exclusion processing by time interval, in some generation scenarios, different attributes can be suppressed by time interval. For example, the 0-10 second interval can be set to suppress the negative C1 (fog) attribute, and the 10-20 second interval can be set to suppress the negative C2 (strong blue tone) attribute. To this end, the image generation device (100) identifies each attribute and the application interval together at S420, and generates time tag-based control information and constraints at S430 to S440. When generating the prompt, attribute suppression parameters per timeline are included, making scene-specific style control possible.

[0122] In this way, by applying the procedure and modified embodiment according to FIG. 4, single or multiple negative references, and furthermore, the exclusion of attributes by time interval, can be supported, thereby maximizing the quality of the generated result and the degree of reflection of the user's intention.

[0124] FIG. 5 is a block diagram illustrating an internal configuration module of a semantic feature mapping network (150) according to one embodiment of the present invention.

[0125] In the embodiment illustrated in FIG. 5, the semantic feature mapping network (150) includes a feature representation input module (151), an input feature normalization layer (152), a multimodal embedding module (153), an attribute-prompt mapping layer (154), a negative processing module (155), and an output prompt seed generation module (156).

[0126] The semantic feature mapping network (150) performs the function of converting visual and semantic features into an intermediate representation suitable for prompt generation and reflecting negative control conditions between the feature representation conversion unit (140) and the prompt generation unit (160).

[0127] The feature representation input module (151) receives feature representation data produced by the multilayer feature extraction unit (130) and the feature representation conversion unit (140). This feature representation data may include various forms of multimodal visual features, such as color palette vectors, composition layout keypoints, motion embeddings, and style tokens.

[0128] For example, motion embeddings extracted from a video reference (B) containing a person walking motion, color palette vectors extracted from a still image (A) with warm colors, and film grain style tokens are input as a single packed feature representation.

[0129] The input feature normalization layer (152) normalizes the input feature vectors into a form optimized for mapping network learning. This may include dimension scaling, mean-variance normalization, coordinate system transformation, etc. It corrects for differences in the distribution of features from different references, enabling comparison and combination in the same representation space.

[0130] For example, if the color palette value of A is in the range of 0 to 255 and the motion embedding value of B is in the range of -1 to 2, all values ​​can be scale-normalized to the range of -1 to 1.

[0131] The multimodal embedding module (153) maps normalized visual features to a language semantic space. Using a pre-trained multimodal model (CLIP, BLIP, etc.), visual attributes such as color, composition, and style are converted into corresponding language tokens or embedding vectors.

[0132] For example, "warm tone + film grain" can be converted into a language embedding "warm tone, film grain texture", and "person walking from left to right" can be converted into a motion instruction embedding "person walking left to right".

[0133] The attribute-prompt mapping layer (154) generates attribute specification information (scene, composition, color, style, mood, object, etc.) and text instruction elements of the prompt data based on multimodal embeddings. In this process, it is mapped into a structured data format (JSON, key-value, etc.) so that it can be directly used in the prompt generation unit (160) thereafter.

[0134] For example, attribute values ​​such as color=warm_palette, style=subtle_film_grain, motion=walk_smoothly, and mood=calm can be derived and mapped to text instructions such as "Calm cinematic sunset scene with a person walking smoothly, warm colors and subtle film grain".

[0135] The negative processing module (155), when a negative reference is provided, identifies the visual attributes to be excluded from the reference and creates a constraint that minimizes similarity with the attributes. This ensures that unnecessary visual attributes are automatically suppressed when generating a prompt.

[0136] For example, if the negative reference is "dense fog", extract attributes such as 'fog texture pattern', 'low saturation-high brightness contrast', and 'blurred edges', and include constraint information such as {exclude: ["dense_fog"], similarity_constraints: {fog_texture: 0.1}} in the prompt data.

[0137] The output prompt seed generation module (156) combines the data generated from the attribute-prompt mapping layer (154) and the negative processing module (155) to generate final seed data that can be transmitted to the prompt generation unit (160). This seed includes text instructions, attribute specification information, control parameters, negative suppression constraints, etc.

[0138] For example, in the seed, if text directive: "Warm-toned cinematic mountain valley at sunset, person walking smoothly, calm atmosphere", attribute information: {scene: "valley_sunset", style: "film_grain", color: "warm_palette", mood: "calm"}, then constraint: {exclude: ["dense_fog"], similarity_constraints: {fog_texture: 0.1}}

[0139] Control parameters: {duration: 60s, fps: 24} may be included.

[0140] In this way, the semantic feature mapping network (150) of FIG. 5 combines multilayer features and negative control information to provide a high-quality prompt seed that enables the prompt generation unit (160) to perform conditional image generation. Through this, the present invention can precisely control various visual attributes such as style, composition, motion, color, and atmosphere in reference-based customized image generation.

[0142] FIG. 6 is a block diagram illustrating an example of the internal configuration of a prompt generation unit (160) according to one embodiment of the present invention.

[0143] The prompt generation unit (160) performs the function of generating final prompt data by integrating feature information and constraints transmitted from higher stages (e.g., semantic feature mapping network (150)) and user control parameters at the final stage of the image generation device (100).

[0144] To this end, the prompt generation unit (160) may include an attribute data collection module (161), a negative constraint integration module (162), a user control parameter reflection module (163), a prompt text combination module (164), and a final prompt data generation module (165).

[0145] The attribute data collection module (161) collects structured attribute data derived from the attribute-prompt mapping layer (154) of FIG. 5. This attribute data is the result of converting low, medium, and high-level visual features derived from the multilayer feature extraction unit (130) and feature representation conversion unit (140) of FIG. 3 into semantic attributes, and includes various elements such as scene, layout, color, style, mood, and main object, and can be set as, for example, {scene: "valley_at_sunset", layout: "person_left_to_right", color: "warm_palette", style: "subtle_film_grain", mood: "calm"}.

[0146] The attribute data collection module (161) collects attribute-specific weight information (A: 0.6, B: 0.4, etc.) together so that the importance of each attribute can be reflected when combining them later.

[0147] The negative constraint integration module (162) combines the constraints generated in the procedures S410–S440 of FIG. 4 and the negative processing module (155) of FIG. 5 with the attribute data. The negative constraint integration module (162) can set negative constraints by including visual attributes to be excluded identified in the negative reference (e.g., dense fog) and conditions to minimize similarity for those attributes. This information is used to enable the generating model to visually suppress the attributes, even if the prohibition phrase is not specified in the final prompt.

[0148] The user control parameter reflection module (163) reflects control parameters such as prompt intensity, style intensity, video length, frame rate, and number of scenes input from the user terminal (200). The user control parameter reflection module (163) can adjust weights so that user-specified attributes are applied preferentially over automatically generated attributes, and selective reflection is also possible to overwrite only specific attributes with user values.

[0149] The user control parameter reflection module (163) reflects control parameters such as prompt intensity, style intensity, video length, frame rate, and number of scenes input from the user terminal (200). The user control parameter reflection module (163) can adjust weights so that user-specified attributes are applied preferentially over automatically generated attributes, and selective reflection is also possible to overwrite only specific attributes with user values.

[0150] The prompt text combination module (164) automatically generates natural language instructions that the generative model can understand based on attribute data, negative constraints, and user control parameters. At this time, the sentence can be corrected by reflecting the collected attributes and constraints based on the initial seed text provided by the output prompt seed generation module (156) of FIG. 5.

[0151] The final prompt data generation module (165) integrates the combined text instructions, structured attribute data, negative constraints, and user control parameters to pack them into final prompt data.

[0152] Below is an example of such final prompt data.

[0153] {

[0154] "text_prompt": "Cinematic mountain valley at sunset, warm tone, subtle film grain, person walking naturally from left to right, calm atmosphere, exclude dense fog",

[0155] "attributes": {

[0156] "scene": "valley_at_sunset",

[0157] "layout": "person_walks_left_to_right",

[0158] "color": "warm_palette",

[0159] "style": "subtle_film_grain",

[0160] "mood": "calm"

[0161] },

[0162] "negative_constraints": {

[0163] "exclude": ["dense_fog"],

[0164] "similarity_constraints": {"fog_texture": 0.1}

[0165] },

[0166] "user_parameters": {

[0167] "duration": "60s",

[0168] "fps": 24,

[0169] "shots": 3

[0170] }

[0171] }

[0173] FIG. 7 is a flowchart illustrating a method for generating final prompt data based on user control parameter reflection according to an embodiment of the present invention.

[0174] One embodiment of FIG. 7 is intended to provide a conditional input optimized for an image generation artificial intelligence model (300) by reflecting the user's direct control intention in the prompt data generated in FIG. 6.

[0175] First, in step S710, the prompt generation unit (160) may receive one or more user control parameters to be applied to the video generation process from the user terminal (200). The user control parameters may include values ​​regarding visual and compositional attributes of the video, such as video length, frame rate, number of scenes, style intensity, color saturation, camera viewpoint, scene transition frequency, and depth of focus. In one embodiment, the user control parameters may be input through various interfaces such as sliders, dropdowns, presets, or natural language commands, and in the case of natural language commands, they may be converted into structured parameter data by a Language Model (LM) based interpretation module.

[0176] Next, in step S720, the prompt generation unit (160) can load previously generated prompt data from memory (102). The prompt data is structured data including attribute data, negative constraint information, basic user settings, etc., and may be the result generated in a previous step (e.g., the final prompt data generation module (165) of FIG. 6). In one embodiment, a version control function is applied to selectively load prompt data at a specific point in time, or to recover damaged or missing elements through a data integrity verification procedure.

[0177] In step S730, the prompt generation unit (160) can adjust the attribute values ​​of the loaded prompt data according to the user control parameters. The adjustment can be performed using a simple substitution method or a non-linear correction method that considers the interdependence between attributes. For example, if the user sets the 'style intensity' to high, the associated 'texture detail' or 'color saturation' attributes can be automatically proportionally adjusted. In one embodiment, if conflicting attributes are entered simultaneously, a priority rule or an AI-based adjustment algorithm can be applied to calculate a balanced final attribute value. Additionally, the changed attribute values ​​can be provided as a preview screen through a real-time visual feedback module.

[0178] In one embodiment of step S730, the prompt generation unit (160) can adjust the attribute values ​​of the loaded prompt data according to user control parameters. This adjustment process is not limited to a simple substitution method, but can apply a multidimensional optimization algorithm that considers the interdependence between attributes, thereby enabling the setting of an attribute weight adjustment algorithm and non-linear style-content balance correction so that the entire combination of attributes aligns with both the user intent and the characteristics of the generation model.

[0179] For example, for each attribute ai, an initial value vi(0) and a user-defined change value △vi are input, and an adjusted value vi(adj) can be calculated as shown in the following [Equation 1] by considering the attribute importance weight wi and the correlation coefficient cij with other attributes.

[0180] [Mathematical Formula 1]

[0181]

[0182] Here, α is the user priority weighting coefficient, β is the attribute correlation correction coefficient, represents the model's recommended default value for the corresponding attribute. This ensures that even if the user makes extreme changes to a specific attribute, other attributes are automatically balanced, preventing degradation of the resulting image quality.

[0183] In addition, to compensate for the inverse relationship between style intensity S and content fidelity C, a non-linear function such as the following [Equation 2] may be applied.

[0184] [Mathematical Formula 2]

[0185]

[0186] Here, γ is the style-content correction factor, and Sopt is the optimal style intensity. Through this correction, important visual elements such as facial expressions or background details are not lost even when the style intensity is increased.

[0187] In step S740, the validity of the adjusted prompt data can be verified. Validation can be divided into formal verification, semantic verification, and constraint verification, and each may include checking the data type and range of attribute values, checking logical consistency between attributes, and checking for conflicts with negative constraints. In one embodiment, the result of the validation is calculated as a validity score in the range of 0 to 1, and if it falls below a certain standard, the system may automatically generate a correction suggestion or automatically correct minor errors.

[0188] Finally, in step S750, the final prompt data that has passed verification is provided to an image generation artificial intelligence model (300) to control the automatic generation of an image reflecting specified attributes and constraints. In one embodiment, a low-resolution preview image may be generated first before being provided to the generation model, and the main generation procedure may be performed after user approval. Additionally, a multi-shot mode may be applied to parallel process multiple prompt variations and adopt the result with the highest quality score.

[0189] Accordingly, one embodiment of the present invention according to FIG. 7 can improve the quality and suitability of the final prompt data by effectively reflecting user control parameters, and maximize the accuracy and controllability of the resulting image through advanced functions such as interdependent attribute correction, collision resolution, validity scoring and automatic correction, real-time feedback, and multi-shot parallel generation.

[0191] In the present disclosure, the image generation AI model is implemented as a generative deep learning model (Generative AI), and may be, for example, a diffusion model, a generative adversarial network (GAN), a Transformer-based visual-language model (Visual-Language Transformer), or a hybrid structure thereof.

[0192] In one embodiment, an image generation AI model receives user input information and prompt data corresponding to scene, object, and effect candidates, and can generate images that are independent of each layer while maintaining mutual physical and temporal consistency. For example, when generating a background layer, the color tone and atmosphere of the entire scene are determined by reflecting a light source map and an environment HDRI, and when generating an object layer, the generation result is output by considering the position, size, pose, motion vector, and occlusion relationship of the object.

[0193] According to various embodiments of the present disclosure, an image generating device may include a computer program stored on a computer-readable recording medium to perform operations described with reference to the drawings above, which are combined with a computer (or computing device) which is hardware.

[0194] Additionally, the image generation device may be implemented as a computing device comprising at least one processor that executes instructions of programs loaded into memory, and a program comprising instructions described to execute operations illustrated in the drawings above may be loaded into memory.

[0195] The device described above may be implemented as a hardware component, a software component, and / or a combination of a hardware component and a software component. For example, the device and components described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and one or more software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. In addition, other processing configurations, such as parallel processors, are also possible.

[0196] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or command the processing unit independently or collectively. Software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave so as to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computer systems and may be stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

[0197] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either alone or in combination. The program instructions recorded on the medium may be those specifically designed and configured for the embodiment, or they may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

[0198] Although the embodiments have been described above with reference to limited examples and drawings, those skilled in the art can make various modifications and variations from the description above. For example, suitable results can be achieved even if the described techniques are performed in a different order than described, and / or the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

[0199] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims set forth below.

[0200] Although specific embodiments have been described in the detailed description of this document, it will be obvious to those skilled in the art that various modifications are possible within the scope of this document. Explanation of the symbols

[0201] 100 : Image generating device

Claims

Claim 1 A method for generating an image generation prompt performed in an image generation device, comprising: receiving a multimodal reference including an image or video from a user; performing feature analysis according to a plurality of abstraction levels on the multimodal reference to extract multilayered visual features on the multimodal reference; converting the extracted multilayered visual features into feature representation data; and generating prompt data for image generation from the feature representation data using a semantic feature mapping network that learns the mapping relationship between the feature representation data and control parameters of a generative artificial intelligence model; wherein the step of extracting multilayered visual features on the multimodal reference comprises: preprocessing the multimodal reference to generate analysis target data including visual information and time information; and performing feature analysis according to a plurality of abstraction levels on the analysis target data to extract multilayered visual features including at least one of a low-level visual attribute, a medium-level visual attribute, and a high-level visual attribute on the multimodal reference. Claim 2 A method for generating an image generation prompt based on multimodal reference analysis, further comprising: a step of modifying the prompt data by reflecting user-specified control parameters in the prompt data in the first step; and a step of providing the modified prompt data to a generative artificial intelligence model to control the generative artificial intelligence model to automatically generate an image. Claim 3 delete Claim 4 A method for generating an image generation prompt based on multimodal reference analysis according to claim 1, wherein the step of extracting multilayer visual features for the multimodal reference further comprises: a step of individually calculating the multilayer visual features for each multimodal reference when there are multiple multimodal references; a step of calculating importance weights for each multimodal reference; and a step of generating integrated multilayer visual features for multiple multimodal references by integrating the importance weights with the individually calculated multilayer visual features. Claim 5 A method for generating an image generation prompt based on multimodal reference analysis according to claim 1, wherein the step of extracting multilayer visual features for the multimodal reference comprises: a step of selectively applying features of different multimodal references to different spatial regions by applying a spatial mask; a step of selectively applying features of different multimodal references to different time intervals by distinguishing time intervals; and a step of applying a transition function at the boundary of the different regions or time intervals to relax seams between features. Claim 6 A method for generating an image generation prompt based on multimodal reference analysis according to claim 1, wherein the multimodal reference includes a negative reference, and the step of generating prompt data for image generation from the feature representation data comprises: a step of identifying a visual attribute to be excluded corresponding to the negative reference; and a step of including control information to suppress the visual attribute to be excluded in the prompt data. Claim 7 A method for generating an image generation prompt based on multimodal reference analysis according to claim 1, wherein the multimodal reference includes a negative reference, and the step of generating prompt data for image generation from the feature representation data comprises: a step of identifying a visual attribute to be excluded from the negative reference; and a step of adding a constraint to the semantic feature mapping network that minimizes the similarity to the visual attribute to be excluded. Claim 8 A method for generating a multimodal reference-based image generation prompt according to claim 1, wherein the prompt data for generating the image comprises: a text instruction generated based on the multimodal reference; and attribute specification information regarding the scene, composition, color, style, mood, or specific object of the image; wherein the attribute specification information comprises at least one of control information for suppressing a visual attribute to be excluded corresponding to a negative reference or a constraint for minimizing similarity to a visual attribute to be excluded. Claim 9 A video generation system based on multimodal reference analysis, comprising: a user terminal that provides a multimodal reference including an image or video; a video generation device that is communicably connected to the user terminal, receives the multimodal reference, performs feature analysis on the multimodal reference according to a plurality of abstraction levels to extract multilayer visual features, converts the extracted multilayer visual features into feature representation data, and generates prompt data for video generation from the feature representation data using a semantic feature mapping network that learns the mapping relationship between the feature representation data and control parameters of a generative artificial intelligence model; and a video generation artificial intelligence based on a video generation artificial intelligence model that receives the prompt data and automatically generates a video according to the prompt. A multimodal reference analysis-based image generation system comprising: an image generation device, wherein, in order to extract multilayer visual features for the multimodal reference, the image generation device preprocesses the multimodal reference to generate analysis target data including visual information and time information, and performs feature analysis according to a plurality of abstraction levels on the analysis target data to extract multilayer visual features for the multimodal reference including at least one of low-level visual attributes, medium-level visual attributes, and high-level visual attributes. Claim 10 A computer-readable recording medium having a computer program stored therein for performing a method for generating an image generation prompt based on multimodal reference analysis according to any one of claims 1, 2, 4 through 8, combined with hardware.

Images