A method for acquiring video related to a scenario using a neural network model.

Abstract

This invention provides a method for acquiring video using a neural network model executed by one or more processors in a computing device. [Solution] The method includes the steps of: obtaining natural language input related to the video; obtaining at least one keyword based on the natural language input; obtaining a prompt for input to a neural network model based on the at least one keyword; determining weights for the prompt based on the at least one keyword; and obtaining the video based on the prompt whose weights have been determined using the neural network model.

Description

[Technical Field]

[0001] This disclosure relates to a method for acquiring video related to a scenario using a neural network model, and more specifically, to a method for acquiring prompts and their weights based on at least one keyword obtained from natural language input, and then using a neural network model to acquire video based on the prompts whose weights have been determined, thereby optimizing various scene conditions such as text, emotion, and location to acquire natural conditions, and through this, acquiring images in which emotional changes are reflected, background consistency is maintained, and continuity accompanying scene changes is visually expressed in a natural way. [Background technology]

[0002] Traditional image generation models generally rely on a single text input to generate images, which has the problem of making it difficult to maintain emotional shifts between scenes and consistency in the background. Furthermore, even when relying on a single text input, keywords are not weighted separately to focus on important objects depending on the scene. This leads to problems in fields where visual storytelling is important, such as movies, animation, and games, where emotional shifts in characters and emphasized facial expressions do not easily appear according to the flow of the scenario.

[0003] Additionally, there were challenges in reducing the resources and costs associated with manual work during the process of adjusting various conditions according to the unfolding scenario before proceeding with filming, depending on the situation.

[0004] Therefore, there is a growing need for a method that optimizes various scene conditions such as text, emotion, and location to obtain natural conditions, thereby acquiring images that reflect emotional changes, maintain background consistency, and visually represent continuity with scene transitions in a natural way.

[0005] On the other hand, while this disclosure is derived at least on the technical background described above, the technical challenges or objectives of this disclosure are not limited to solving the problems or shortcomings described above. In other words, this disclosure can cover a variety of technical issues related to the content described below, in addition to the technical issues described above. [Overview of the Initiative] [Problems that the invention aims to solve]

[0006] This disclosure relates to a method for acquiring video related to a scenario using a neural network model. More specifically, it aims to solve the problem of acquiring natural conditions by optimizing various scene conditions such as text, emotion, and location, by acquiring prompts and their corresponding weights based on at least one keyword obtained from natural language input, and acquiring video based on the prompts whose weights have been determined using a neural network model, thereby acquiring natural images in which emotional changes and background consistency are maintained, and video in which continuity accompanying scene transitions is visually naturally expressed.

[0007] On the other hand, the technical challenges that this disclosure aims to address are not limited to those mentioned above, and may include a variety of technical challenges that are obvious to those skilled in the art from the content described below. [Means for solving the problem]

[0008] To address the aforementioned challenges, one embodiment of the present disclosure discloses a method performed by a computing device. The method may include the steps of: acquiring video and associated natural language input; acquiring at least one keyword based on the natural language input; acquiring a prompt for input to a neural network model based on the at least one keyword; determining weights for the prompt based on the at least one keyword; and acquiring video based on the weighted prompt using the neural network model.

[0009] Alternatively, the step of obtaining at least one keyword based on the natural language input may include the steps of inputting the natural language input into a pre-trained language model; and obtaining at least one keyword based on one or more output data output based on the language model.

[0010] Alternatively, the step of obtaining at least one keyword based on the natural language input may include at least one of the following: obtaining a first keyword relating to distance based on the natural language input; obtaining a second keyword relating to an object based on the natural language input; obtaining a third keyword including details about the object based on the natural language input; or obtaining a fourth keyword relating to background based on the natural language input.

[0011] Alternatively, the step of obtaining a second keyword relating to an object based on the natural language input may include at least one of the following steps: obtaining a second-first keyword relating to a higher-level conceptual object based on the natural language input; or obtaining a second-second keyword relating to a lower-level conceptual object of the second-first keyword based on the natural language input.

[0012] Alternatively, the step of obtaining a third keyword containing details about an object based on the natural language input may include at least one of the following steps: obtaining a third-first keyword related to one or more actions based on the natural language input; or obtaining a third-second keyword related to one or more states based on the natural language input.

[0013] Alternatively, the neural network model may include a neural network model capable of performing at least one of the following operations on the input data: encoding or decoding.

[0014] Alternatively, the step of acquiring video based on the weights determined prompts using the neural network model may include at least one of the following: acquiring an image based on the weights determined prompts using the neural network model; or acquiring a video based on the weights determined prompts using the neural network model.

[0015] Alternatively, the step of determining the weight of the prompt based on the at least one keyword may include the step of determining the weight of the prompt based on the proportion of the at least one keyword included in the natural language input.

[0016] Alternatively, the step of determining the weight for the prompt based on at least one keyword may include a step of determining that the weight for the second keyword is greater than the weight for the second keyword if the natural language input includes a second keyword relating to a sub-conceptual object of the second keyword.

[0017] As an alternative, the step of determining the weight for the prompt based on the at least one keyword may include, when the natural language input includes a 3-1 keyword related to one or more actions, setting the weight of the keyword for an object related to the 3-1 keyword to be greater than the weight of the keyword for an object not related to the 3-1 keyword.

[0018] As an alternative, the step of determining the weight for the prompt based on the at least one keyword may include, when the natural language input includes a 3-2 keyword related to one or more states, setting the weight of the keyword for an object related to the 3-2 keyword to be greater than the weight of the keyword for an object not related to the 3-2 keyword.

[0019] As an alternative, the method may further include obtaining additional natural language input related to additional video; obtaining at least one additional keyword based on the additional natural language input; obtaining an additional prompt for input to a neural network model based on the at least one additional keyword; determining a weight for the additional prompt based on the at least one additional keyword; and obtaining additional video based on the video and the additional prompt for which the weight has been determined by utilizing the neural network model.

[0020] In order to achieve the problems described above, a computer program stored in a computer-readable recording medium according to an embodiment of the present disclosure is disclosed. When the computer program is executed by one or more processors, the one or more processors perform operations for obtaining an image by utilizing a neural network model, and the operations include: an operation of obtaining a natural language input related to the image; an operation of obtaining at least one keyword based on the natural language input; an operation of obtaining a prompt for inputting into the neural network model based on the at least one keyword; an operation of determining a weight for the prompt based on the at least one keyword; and an operation of obtaining an image based on the prompt for which the weight has been determined by utilizing the neural network model.

[0021] As an alternative, the operation of obtaining at least one keyword based on the natural language input may include: an operation of inputting the natural language input into a pre-trained language model; and an operation of obtaining the at least one keyword based on one or more output data output based on the language model.

[0022] As an alternative, the operation of obtaining at least one keyword based on the natural language input may include at least one of: an operation of obtaining a first keyword related to distance based on the natural language input; an operation of obtaining a second keyword related to an object based on the natural language input; an operation of obtaining a third keyword including details related to the object based on the natural language input; or an operation of obtaining a fourth keyword related to background based on the natural language input.

[0023] Alternatively, the operation to obtain a second keyword relating to an object based on the natural language input may include at least one of the following: an operation to obtain a second-first keyword relating to a higher-level conceptual object based on the natural language input; or an operation to obtain a second-second keyword relating to a lower-level conceptual object of the second-first keyword based on the natural language input.

[0024] Alternatively, the operation to retrieve a third keyword containing details about an object based on the natural language input may include at least one of the following: an operation to retrieve a third-first keyword related to one or more actions based on the natural language input; or an operation to retrieve a third-second keyword related to one or more states based on the natural language input.

[0025] Alternatively, the operation of acquiring video based on a prompt whose weights have been determined using the neural network model may include at least one of the following: the operation of acquiring an image based on a prompt whose weights have been determined using the neural network model; or the operation of acquiring a video based on a prompt whose weights have been determined using the neural network model.

[0026] Alternatively, the operation of determining the weight of the prompt based on the at least one keyword may include the operation of determining the weight of the prompt based on the proportion of the at least one keyword contained in the natural language input.

[0027] Alternatively, the operation of determining the weight of the prompt based on at least one keyword may include the operation of determining a greater weight for the second keyword than for the second keyword if the natural language input includes a second keyword relating to a sub-concept object of the second keyword.

[0028] Alternatively, the operation of determining the weight of the prompt based on at least one keyword may include, if the natural language input contains one or more action-related action-related action-related action, setting the weight of the keyword for objects associated with the action-related action-related action to be greater than the weight of the keyword for objects not associated with the action-related action-related action.

[0029] Alternatively, the operation of determining the weight of the prompt based on at least one keyword may include, if the natural language input contains one or more state-related third-second keywords, setting the weight of the keyword for objects associated with the third-second keyword to be greater than the weight of the keyword for objects not associated with the third-second keyword.

[0030] To address the aforementioned challenges, a computing device according to one embodiment of the present disclosure is disclosed. The device includes at least one processor and memory, the processor being configured to acquire video and associated natural language input; acquire at least one keyword based on the natural language input; acquire prompts for input to a neural network model based on the at least one keyword; determine weights for the prompts based on the at least one keyword; and utilize the neural network model to acquire video based on the weighted prompts. [Effects of the Invention]

[0031] This disclosure relates to a method for acquiring video related to a scenario using a neural network model, more specifically, to acquiring prompts and their weights based on at least one keyword obtained from natural language input, and acquiring video based on the prompts whose weights have been determined using a neural network model. This optimizes various scene conditions such as text, emotion, and location to acquire natural conditions, thereby enabling the acquisition of natural images in which emotional changes and background consistency are maintained, and video in which continuity accompanying scene transitions is visually naturally expressed.

[0032] On the other hand, the effects of this disclosure are not limited to those mentioned above, and a variety of effects may be included, as described below, within the scope of what is obvious to a person with ordinary skill in the art to which the present invention belongs. [Brief explanation of the drawing]

[0033] [Figure 1] This is a block diagram of a computing device for acquiring video using a neural network model according to one embodiment of the present disclosure. [Figure 2] This is a schematic diagram illustrating a network function according to one embodiment of the present disclosure. [Figure 3] This is a flowchart showing a method for acquiring video using a neural network model according to one embodiment of the present disclosure. [Figure 4] This is a schematic diagram illustrating a process of acquiring video based on at least one keyword using a neural network model according to one embodiment of the present disclosure. [Figure 5] This is a schematic diagram illustrating a process in which, according to one embodiment of the present disclosure, weights for prompts are determined based on at least one keyword, and a neural network model is used to acquire video based on the weights of the prompts. [Figure 6]This is a schematic diagram illustrating the process of acquiring additional video based on additional prompts whose weights have been determined by utilizing a neural network model according to one embodiment of the present disclosure. [Figure 7] This is a simplified and general schematic diagram of an exemplary computing environment in which embodiments of the embodiments of this disclosure may be realized. [Modes for carrying out the invention]

[0034] Various embodiments are described below with reference to the drawings, where similar drawing numbers are used to represent similar components. Various descriptions are provided herein to facilitate understanding of this disclosure. However, these embodiments can certainly be carried out without these specific descriptions.

[0035] In this specification, terms such as “component,” “module,” and “system” refer to computer-related entities, hardware, firmware, software, combinations of software and hardware, or software execution. For example, a component may be, but is not limited to, a processing procedure executed on a processor, a processor, an object, an execution thread, a program, and / or a computer. For example, both an application running on a computing device and the computing device itself can be components. One or more components may reside in a processor and / or an execution thread, and one component may be localized within one computer or distributed across two or more computers. Such components may also be executed from a variety of computer-readable media, each containing a variety of data structures. Components may communicate locally and / or remotely, for example, by signals containing one or more data packets (e.g., data transmitted over a network such as the Internet, through data and / or signals from one component interacting with other components in a local system or distributed system).

[0036] The term "or" is used with the intention of meaning an implicational "or," not an exclusive "or." That is, unless specifically specified and contextually clear, "X uses A or B" means one of the natural implicational substitutions. In other words, if X uses A; X uses B; or X uses both A and B, then "X uses A or B" can be any of these. Furthermore, the terms "and / or" in this specification should be understood to refer to and include all possible combinations of one or more of the related items discussed.

[0037] Furthermore, the term “includes” as a predicate and / or as a modifier should be understood to mean that the feature and / or component in question exists. However, the term “includes” as a predicate and / or as a modifier should be understood not to exclude the existence or addition of one or more other further features, components and / or groups thereof. Also, where the number is not specifically identified or where it is not clear from the context to indicate a singular form, “singular” in this specification and claims should generally be interpreted to mean “one or more.”

[0038] Furthermore, the phrase "at least one of A or B" should be interpreted as meaning "including only A," "including only B," or "a combination of A and B."

[0039] Those skilled in the art should further recognize that the various exemplary logical blocks, configurations, modules, circuits, means, logic, and algorithmic stages described herein as relating to the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of both. To clearly illustrate the interoperability between hardware and software, various exemplary components, blocks, configurations, means, logic, modules, circuits, and stages have been generally described above in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and design constraints of the overall system. Skilled technicians can implement the described functionality in various ways for individual specific applications; however, decisions regarding such implementation should not be construed as departing from the scope of this disclosure.

[0040] The descriptions relating to the embodiments shown herein are provided so that a person with ordinary skill in the art of this disclosure may utilize or practice the invention. Various modifications to such embodiments are obvious to a person with ordinary skill in the art of this disclosure, and the general principles defined herein can be applied to other embodiments without departing from the scope of this disclosure. Therefore, this disclosure is not limited by the embodiments shown herein, but should be interpreted in the broadest sense consistent with the principles and novel features shown herein.

[0041] This disclosure allows for the interchangeable use of network functions, artificial neural networks, and neural networks.

[0042] Figure 1 is a block diagram of a computing device for acquiring video using a neural network model according to one embodiment of the present disclosure.

[0043] The configuration of the computing device (100) shown in Figure 1 is merely a simplified example. In one embodiment of this disclosure, the computer device (100) may include other configurations for implementing the computing environment of the computer device (100), and it is also possible to configure the computer device (100) using only some of the disclosed configurations.

[0044] The computer device (100) may include a processor (110), memory (130), and a network unit (150).

[0045] In one embodiment of the present disclosure, the processor (100) may consist of one or more cores and may include processors for data analysis and deep learning, such as a central processing unit (CPU), a general-purpose graphics processing unit (GPGPU), and a tensor processing unit (TPU). The processor (110) can read computer programs stored in memory (130) and execute data processing for machine learning in one embodiment of the present disclosure. Based on one embodiment of the present disclosure, the processor (110) can perform calculations for training a neural network. In deep learning (DL), the processor (110) can perform calculations for training a neural network, such as processing input data for training, extracting features from input data, calculating errors, and updating the weights of the neural network using backpropagation.

[0046] At least one of the CPU, GPGPU, and TPU of the processor (110) can process network function training. For example, both the CPU and GPGPU can perform network function training and data classification using network functions. In one embodiment of this disclosure, the processors of multiple computing devices can be used together to perform network function training and data classification using network functions. Furthermore, in one embodiment of this disclosure, the computer program executed in the computing device can be a program that can be executed on the CPU, GPGPU, or TPU.

[0047] According to one embodiment of the present disclosure, the memory (130) can store any form of information generated or determined by the processor (110) and any form of information received by the network unit (150).

[0048] In one embodiment of the present disclosure, the memory (130) may include at least one type of storage medium from among flash memory type, hard disk type, multimedia card micro type, card type memory (e.g., SD or XD memory), RAM (Random Access Memory), SRAM (Static Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, magnetic disk, and optical disk. The computing device (100) may also operate in conjunction with web storage that performs the storage function of the memory (130) over the internet. The above descriptions of memory are illustrative and the present disclosure is not limited thereto.

[0049] In one embodiment of the present disclosure, the network unit (150) can use a variety of wired communication systems such as public switched telephone networks (PSTN), xDSL (x Digital Subscriber Line), RADSL (Rate Adaptive DSL), MDSL (Multi Rate DSL), VDSL (Very High Speed ​​DSL), UADSL (Universal Asymmetric DSL), HDSL (High Bit Rate DSL), and local area networks (LANs).

[0050] Furthermore, the network unit (150) in this specification can utilize a variety of wireless communication systems such as CDMA (Code Division Multi Access), TDMA (Time Division Multi Access), FDMA (Frequency Division Multi Access), OFDMA (Orthogonal Frequency Division Multi Access), SC-FDMA (Single Carrier-FDMA), and other systems.

[0051] In this disclosure, the network unit (150) can be configured regardless of the communication method, such as wired or wireless, and can be composed of various communication networks such as a Personal Area Network (PAN) or a Wide Area Network (WAN). Furthermore, the network may be the well-known World Wide Web (WWW), and wireless transmission technologies used for short-range communication, such as infrared (IrDA) or Bluetooth, can also be used. The technologies described in this disclosure can also be used in the other networks mentioned above. Figure 2 is a schematic diagram showing a network function according to one embodiment of the present disclosure.

[0052] Throughout this specification, the terms artificial intelligence model, artificial intelligence-based model, computational model, neural network, network function, and neural network may be used interchangeably.

[0053] A neural network can consist of a set of interconnected computational units, which can generally be called nodes. These nodes are sometimes also called neurons. A neural network consists of at least one node. The nodes (or neurons) that make up a neural network can be interconnected by one or more links.

[0054] Within a neural network, one or more nodes connected via links can form a relative input-output node relationship. The concepts of input and output nodes are relative; any node that is an output node to another node can be an input node to another node, and vice versa. As mentioned above, the input-output node relationship can be generated around links. One input node can be connected to one or more output nodes via links, and vice versa.

[0055] In a relationship between input and output nodes connected via a single link, the data of the output node can be determined based on the data input to the input node. Here, the link interconnecting the input and output nodes may have weights. The weights can be variable and can be changed by the user or algorithm to enable the neural network to perform desired functions. For example, if one or more input nodes are interconnected to a single output node by their respective links, the output node's value can be determined based on the values ​​input to the input nodes connected to the output node and the weights set for the links corresponding to each input node.

[0056] As mentioned earlier, a neural network consists of one or more nodes interconnected via one or more links, forming input-output node relationships within the neural network. The characteristics of a neural network can be determined by the number of nodes and links within the neural network, the relationships between nodes and links, and the weight values ​​assigned to each link. For example, if there are two neural networks with the same number of nodes and links but different link weight values, the two neural networks may be perceived as different from each other.

[0057] A neural network can consist of a set of one or more nodes. A subset of the nodes that make up a neural network can form a layer. Some of the nodes that make up a neural network can form a layer based on their distance from the first input node. For example, a set of nodes that are n in distance from the first input node can form an n-layer. The distance from the first input node can be defined by the minimum number of links that must be traversed to reach that node from the first input node. However, such a definition of a layer is arbitrary for illustrative purposes, and the difference in the number of layers within a neural network can be defined in a different way than described above. For example, a node layer can also be defined by its distance from the final output node.

[0058] In one embodiment of this disclosure, a collection of neurons or nodes can be defined as a layer.

[0059] The initial input node can refer to one or more nodes in a neural network that receive data directly without going through links in relation to other nodes. Alternatively, it can refer to a node in a neural network that does not have other input nodes connected by links in relation to other nodes based on links. Similarly, the final output node can refer to one or more nodes in a neural network that do not have other output nodes in relation to other nodes. Furthermore, hidden nodes can refer to nodes in a neural network that are neither the initial input node nor the final output node.

[0060] A neural network according to one embodiment of this disclosure may have the same number of nodes in the input layer as the number of nodes in the output layer, and the number of nodes may decrease as the network progresses from the input layer to the hidden layer, and then increase again. Another neural network according to another embodiment of this disclosure may have fewer nodes in the input layer than the number of nodes in the output layer, and the number of nodes may decrease as the network progresses from the input layer to the hidden layer. Yet another neural network according to yet another embodiment of this disclosure may have more nodes in the input layer than the number of nodes in the output layer, and the number of nodes may increase as the network progresses from the input layer to the hidden layer. Yet another neural network according to yet another embodiment of this disclosure may be a neural network that combines the aforementioned neural networks.

[0061] A deep neural network (DNN) can refer to a neural network that includes multiple hidden layers in addition to the input and output layers. Using deep neural networks, it is possible to grasp the latent structures of data. That is, the latent structures of photographs, text, videos, audio, protein sequence structures, gene sequence structures, peptide sequence structures, and music (e.g., what objects are in a photograph, what is the content and emotion of the text, what is the content and emotion of the audio, etc.), and / or the degree of binding affinity between peptides and MHCs. Deep neural networks can include convolutional neural networks (CNNs), recurrent neural networks (RNNs), autoencoders, restricted Boltzmann machines (RBMs), deep belief networks (DBNs), Q networks, U networks, Siam networks, generative adversarial networks (GANs), and transformers. The aforementioned description of deep neural networks is merely illustrative, and this disclosure is not limited thereto.

[0062] The artificial intelligence-based model described herein can be represented by a network structure of any of the aforementioned structures, including an input layer, a hidden layer, and an output layer.

[0063] The neural networks that can be used in the artificial intelligence-based models described herein can be trained using at least one of the following methods: supervised learning, unsupervised learning, semi-supervised learning, transfer learning, active learning, or reinforcement learning. Learning a neural network can be the process by which the neural network applies knowledge to perform a specific action.

[0064] Neural networks can be trained to minimize the error in their output. Training a neural network involves repeatedly inputting training data, calculating the network's output and target error for the training data, and updating the weights of each node in the neural network by backpropagating the error from the output layer to the input layer to reduce the error. In supervised learning, training data with the correct answer labeled is used (i.e., labeled training data), while in unsupervised learning, the training data may not have the correct answer labeled. For example, in supervised learning for data classification, the training data may be data with each training data point labeled with a category. The error can be calculated by inputting the labeled training data into the neural network and comparing the neural network's output (category) with the labels on the training data.

[0065] As another example, in unsupervised learning for data classification, the error can be calculated by comparing the input training data with the output of the neural network. The calculated error is backpropagated in the neural network (i.e., from the output layer to the input layer), and the connection weights of each node in each layer of the neural network may be updated according to the backpropagation. The amount of change in the connection weights of each node to be updated can be determined by the learning rate. The calculation of the neural network on the input data and the backpropagation of the error can constitute a learning cycle (epoch). The learning rate can be applied differently depending on the number of iterations of the neural network's learning cycle. For example, a high learning rate can be used in the early stages of learning to increase efficiency by allowing the neural network to quickly achieve a certain level of performance, while a low learning rate can be used in the later stages of learning to improve accuracy.

[0066] In neural network training, training data can generally be a subset of real-world data (i.e., data to be processed using the trained neural network). Therefore, there can be training cycles where errors on the training data decrease, but errors on the real-world data increase. Overfitting is a phenomenon where the training data is excessively trained, leading to increased errors on the real-world data. For example, a neural network trained to recognize a yellow cat as a cat may fail to recognize a cat of a different color as a cat; this is a type of overfitting. Overfitting can act as a cause of increased errors in machine learning algorithms. Various optimization methods can be used to prevent such overfitting. To prevent overfitting, methods such as increasing the amount of training data, regularization, dropout (deactivating some of the network nodes during the training process), and the use of a batch normalization layer can be applied.

[0067] A computer-readable medium storing a data structure according to one embodiment of the present disclosure is disclosed. The aforementioned data structure can be stored in the storage unit of the present disclosure, executed by a processor, and transmitted and received by a communication unit.

[0068] A data structure can refer to the organization, management, and storage of data that enables efficient access to and modification of that data. A data structure can also refer to the organization of data for solving a specific problem (e.g., data analysis, data retrieval, data storage, data modification). A data structure can also be defined as the physical or logical relationships between data elements designed to support specific data processing functions. Logical relationships between data elements can include linking relationships between user-defined data elements. Physical relationships between data elements can include actual relationships between data elements physically stored in a computer-readable medium (e.g., permanent storage). Specifically, a data structure can include a collection of data, relationships between data, and functions or instructions that can be applied to data. A well-designed data structure allows computing devices to perform operations while minimizing the use of their resources. Specifically, computing devices can improve the efficiency of operations such as arithmetic, reading, insertion, deletion, comparison, exchange, and retrieval through well-designed data structures.

[0069] Data structures can be classified into linear and nonlinear data structures depending on their form. A linear data structure may be one in which only one data item is linked after another. Linear data structures can include lists, stacks, queues, and deques. A list can represent a set of data items that have an internal order. A list can include linked lists. A linked list may be a data structure in which data is linked in a linear fashion, with each item having a pointer. Pointers in a linked list may contain linking information to preceding and succeeding data. A linked list can be represented as a single linked list, a double linked list, or a circular linked list, depending on its form. A stack may be a data list structure in which data can be accessed in a restricted manner. A stack may be a linear data structure in which data can only be processed (e.g., inserted or deleted) from one end of the data structure. Data stored in a stack may be a last-in, first-out (LIFO) data structure. A queue is a data structure that allows restricted access to data, and unlike a stack, it can be a first-in, first-out (FIFO) data structure. A deck can be a data structure that allows data to be processed from both ends of the data structure.

[0070] A nonlinear data structure can be a structure in which multiple data are concatenated after a single data. A nonlinear data structure can include a graph data structure. A graph data structure can be defined as a vertex and an edge, where an edge can contain a line connecting two distinct vertices, and can include a graph data structure tree. A tree data structure can be a data structure in which there is only one path connecting two distinct vertices among the multiple vertices contained in the tree. In other words, it can be a data structure that does not form a loop in a graph data structure.

[0071] Throughout this specification, the terms artificial intelligence-based model, computational model, neural network, network function, and neural network may be used interchangeably. Hereafter, the term neural network will be used consistently. A data structure may include a neural network, and such a data structure may be stored in a computer-readable medium. The data structure may also include pre-processed data for processing by the neural network, data input to the neural network, neural network weights, neural network hyperparameters, data obtained from the neural network, activation functions associated with each node or layer of the neural network, and loss functions for learning the neural network. The data structure may include any of the components in the disclosed configuration. That is, the data structure may include all or any combination of the following: pre-processed data for processing by the neural network, data input to the neural network, neural network weights, neural network hyperparameters, data obtained from the neural network, activation functions associated with each node or layer of the neural network, and loss functions for learning the neural network. Beyond the configurations described above, data structures including neural networks can contain any other information that determines the properties of the neural network. Furthermore, data structures can contain, and are not limited to, all forms of data used or generated during the computational processes of the neural network. Computer-readable media can include computer-readable recording media and / or computer-readable transmission media. A neural network can be structured as a collection of interconnected computational units, which may generally be referred to as nodes. These nodes may also be referred to as neurons.A neural network consists of at least one node.

[0072] A data structure may include data to be input to a neural network. A data structure containing data to be input to a neural network may be stored on a computer-readable medium. Data to be input to a neural network may include training data input during the training process of the neural network and / or input data to be input to a neural network after training is complete. Data to be input to a neural network may include pre-processed data and / or data to be pre-processed. Pre-processing may include data processing processes to prepare data for input to a neural network. Therefore, a data structure may include data to be pre-processed and data generated by pre-processing. The data structures described above are merely examples, and this disclosure is not limited thereto.

[0073] The data structure may include the weights of a neural network (in this specification, weights and parameters may be used interchangeably). The data structure containing the weights of a neural network may be stored in a computer-readable medium. A neural network may have multiple weights. The weights may be variable and can be varied by the user or algorithm to enable the neural network to perform desired functions. For example, if one or more input nodes are interconnected to an output node by their respective links, the output node can determine the data value output from the output node based on the values ​​input to the input nodes connected to the output node and the weights set on the links corresponding to each input node. The data structures described above are merely examples, and this disclosure is not limited thereto.

[0074] As an unrestricted example, weights may include weights that are variable during the learning process of a neural network and / or weights after the neural network has finished learning. Weights that are variable during the learning process of a neural network may include weights at the start of a learning cycle and / or weights that are variable during a learning cycle. Weights that have finished learning a neural network may include weights after a learning cycle has finished. Thus, a data structure containing the weights of a neural network may include a data structure containing weights that are variable during the learning process of a neural network and / or weights after the neural network has finished learning. Therefore, the aforementioned weights and / or each combination of weights shall be included in the data structure containing the weights of a neural network. The aforementioned data structures are illustrative only, and this disclosure is not limited thereto.

[0075] A data structure containing neural network weights can be stored in a computer-readable medium (e.g., memory, hard disk) after undergoing a serialization process. Serialization may be a process of storing the data structure in the same or other computing device and later reconstructing it into a usable form. The computing device can serialize the data structure and send and receive the data over a network. The serialized data structure containing neural network weights can be reconstructed in the same or other computing device by deserialization. The data structure containing neural network weights is not limited to serialization. Furthermore, the data structure containing neural network weights may include data structures that enhance computational efficiency while minimizing the use of computing device resources (e.g., B-Tree, R-Tree, Trie, m-way search tree, AVL tree, Red-Black Tree in nonlinear data structures). The foregoing are merely examples, and this disclosure is not limited thereto.

[0076] The data structure may include the hyperparameters of the neural network. Furthermore, the data structure containing the neural network's hyperparameters may be stored in a computer-readable medium. The hyperparameters may be variable variables that can be changed by the user. Examples of hyperparameters may include the learning rate, cost function, number of training cycles, weight initialization (e.g., setting the range of weight values ​​to be initialized), and the number of Hidden Units (e.g., the number of hidden layers, the number of nodes in the hidden layers). The data structures described above are merely examples, and this disclosure is not limited thereto.

[0077] A model according to one embodiment of the present disclosure can be trained to remove some or all of the predicted noise for Gaussian distributed noise to obtain data from which some or all of the predicted noise has been removed.

[0078] A computing device 100 according to one embodiment of the present disclosure can train a neural network to remove part or all of isotropic Gaussian distributed noise and obtain data from which part or all of the noise has been removed. In this case, the neural network may include a conditional noise prediction model. The conditional noise prediction model may include a U-Net structure in which the input and output have the same magnitude, and can take noisy data x(t) and a diffusion time step t as inputs, predict the diffusion noise contained in the noisy data x(t), and output it.

[0079] The computing device 100 can perform a forward process that repeatedly adds random Gaussian noise little by little to original data x(0) that does not contain noise over T time steps, thereby obtaining isotropic Gaussian distributed noise x(T). The type of original data x(0) can include a variety of examples other than image data, such as audio data. On the other hand, the forward process according to one embodiment of the present disclosure can be performed by the following mathematical formula.

[0080]

number

[0081] In the above equation 1, β t β can be used as a hyperparameter in the process of calculating the diffusion coefficient, t β can be set to any value, where 0 < β1 < β2 < ... < β T It can be set to a value of <1. For example, if β1 has a value of 0.0001, β T It can have a value of 0.02, and from β1 to β T up to β t The value of can increase linearly and may increase according to the cosine function. Furthermore, T, which represents the total number of diffusion steps, can be set to 1000. However, β t The fact that β increases linearly or according to a cosine function is merely illustrative, and according to embodiments of the present disclosure, β increases depending on the type of original data. t The amount of increase can be determined to be different, and a specific explanation will be given later by [Equation 2] below. Also, ε can mean random Gaussian distributed noise. The general formula that shows the noise-containing data x(t) at time step t in relation to the original data x(0) without noise and the included diffusion noise can be expressed as follows.

[0082]

Number

[0083] According to one embodiment of the present disclosure, the formulas (1) and (2) in the number 2 represent the specific meaning of the diffusion coefficient α ￣ t In the formula (1) of the number 2, the diffusion coefficient α at a specific time step t t can be calculated as a value obtained by subtracting 1 from the hyperparameter β t And in the formula (2) of the number 2, the diffusion coefficient α ￣ t may mean the diffusion coefficients sequentially accumulated from time step 1 to t. Therefore, the formula (3) of the number 2 represents the data x(t) including noise at time step t as a formula related to the original data x(0) without noise, the diffusion coefficient (α ￣ t ), and the random Gaussian distribution noise (ε). Also, the formula (4) in the number 2 may mean the noise ratio n(t) and the signal ratio s(t) determined based on the type of the original data according to one embodiment of the present disclosure. Therefore, the data x(t) including noise at time step t can be represented based on the noise ratio n(t) and the signal ratio s(t). Specifically, the data x(t) including noise at time step t becomes closer to the shape of the original data as the ratio of the original data x(0) without noise (i.e., the signal ratio s(t)) increases, and becomes closer to the shape of the random Gaussian distribution noise as the ratio of the random Gaussian distribution noise (ε) (i.e., the noise ratio n(t)) increases.

[0084] For example, the computing device 100 can acquire first data by adding first noise to original data x(0) that does not contain noise, in a forward process that adds random Gaussian noise little by little over T time steps, based on the determined noise ratio, and acquire second data by adding second noise to the first data based on the determined noise ratio. In one embodiment, the type of original data x(0) may include a variety of examples other than image data, such as text data and audio data.

[0085] This allows the computing device 100 to perform a forward process in which it repeatedly adds random Gaussian noise to the original data x(0) which is free of noise, based on "the ratio of noise determined by equation (4) in equation 2" over T time steps, thereby obtaining isotropic Gaussian distributed noise x(T). However, the forward process is not limited to the example in equation 2, and various processes for adding noise to data may be included in the forward process.

[0086] Furthermore, the computing device 100 can train the neural network to perform a reverse process in the opposite direction to the forward process, where random Gaussian noise is gradually removed from isotropic Gaussian distributed noise x(T) over T time steps, thereby obtaining original data x(0) that does not contain noise. In this regard, the mathematical formula representing the reverse execution process can be expressed as follows.

[0087]

number

[0088] In the above equation 3, equation (1) is the noise prediction result (ε) predicted by the neural network for "data x(t) containing noise". θ (x t This equation represents the reverse process of removing the noise (t) to obtain "the previous data x(t-1) from which some of the noise has been removed." For example, the computing device 100 can remove the noise prediction result predicted by the neural network from "the second data x(2) containing the second noise" to obtain "the previous first data x(1) from which the second noise has been removed." In equation 3, equation (2) is the diffusion coefficient α at the current time step t. t This means that in equation (3) above, σ t α represents the dispersion parameter, and the diffusion coefficient α t It can be calculated based on [Equation 3]. However, the reverse process is not limited to [Equation 3], and various processes for removing noise from data containing noise may be included in the reverse process.

[0089] Specifically, the computing device 100 inputs data x(t) containing noise and a time step t to a neural network, compares the noise prediction result predicted by the neural network with the actual diffusion noise, calculates a loss function, and performs gradient descent according to the loss function to train the neural network. For example, the loss function calculated by comparing the noise prediction result predicted by the neural network with the actual diffusion noise can be expressed by the following formula.

[0090]

number

[0091] In the above equation 4, the loss function is the actual diffusion noise (ε) and the noise prediction result (ε) predicted by the neural network. θ (x t The loss function can be calculated by comparing (t)). For example, the computing device 100 can predict "the first noise contained in the first data x(1)" and calculate the first loss function by comparing the predicted first noise with the first noise added to the original data x(0). The computing device 100 can predict "the second noise contained in the second data x(2)" and calculate the second loss function by comparing the predicted second noise with the second noise added to the first data x(1). However, the loss function is not limited to the example shown in Equation 4, and may include a variety of loss functions calculated by comparing the noise prediction result with the actually contained spreading noise.

[0092] Additionally, the neural network can be trained to predict the diffusion noise contained in the data x(2) containing the second noise, and to remove the predicted second noise, thereby obtaining "data x(1) from which the second noise has been removed." For example, the computing device 100 can also train the neural network based on at least one of the first loss function or the second loss function. Furthermore, the neural network can be trained to predict the diffusion noise contained in "data x(t) containing the noise," and to remove the predicted diffusion noise, repeating this process one or more times, thereby removing all the diffusion noise contained in "data x(t) containing the noise," and obtaining original data x(0) from which all noise has been removed. On the other hand, in the process of using the neural network to learn to remove diffusion noise from data containing noise, the neural network can be trained to generate higher quality data by determining the noise ratio based on the type of original data.

[0093] The computing device 100 can determine the noise ratio based on the size of the data that can be represented for the determined type of original data. Specifically, the computing device 10 can determine the noise ratio such that the size of the noise added to the original data is smaller than the minimum size of the data that can be represented for the determined type of original data. For example, if the type of original data is determined to be image data, the original data may have a depth of 8 bits per channel. Therefore, the computing device 10 can determine the first noise ratio such that the size of the noise added to the original data is smaller than the minimum size of the data that can be represented for the image data, which is 1 / (2^8) (= approximately 0.00039).

[0094] On the other hand, according to other embodiments of the present disclosure, when the type of original data is determined to be audio data, the original data may have a depth of 16 bits per sample. Therefore, the computing device 10 can determine the second noise ratio such that the amount of noise added to the original data is less than 1 / (2^16) (= approximately 0.000015), which is the minimum amount of data that can be represented for audio data. Thus, the second noise ratio when the original data is audio data can be determined to be less than the first noise ratio when the original data is image data. For example, the noise ratio can be determined by the following formula.

[0095]

number

[0096] Specifically, in one embodiment of the present disclosure, referring to equation (1) in [Equation 5], the noise ratio n(t) may be determined to increase exponentially, and referring to equation (2) in [Equation 5], the signal ratio s(t) may be determined to decrease exponentially. In this case, r may represent a hyperparameter that determines the shape of the curves for the noise ratio n(t) and the signal ratio s(t). However, [Equation 5] is merely an example, and the noise ratio and signal ratio can be determined in various other ways. This allows the neural network model to be trained so that the results generated through the neural network model are hardly affected by noise, even in data domains that are relatively sensitive to noise compared to image data (for example, audio data).

[0097] Figure 3 is a flowchart showing a method for acquiring video using a neural network model according to one embodiment of the present disclosure.

[0098] A computing device 100 according to one embodiment of this disclosure can directly acquire "information for acquiring video using a neural network model" or receive it from an external system. The external system may be a server, database, etc., that stores and manages information for acquiring video using a neural network model. The computing device 100 can use the directly acquired information or the information received from the external system as "input data for acquiring video using a neural network model".

[0099] According to one embodiment of the present disclosure, the computing device 100 can acquire natural language input related to video (S110). In this case, the natural language input related to video may include text and audio related to scenarios used in the creation of storyboards and storyboards for films, animations, etc., and the natural language input related to video may exemplify text related to a specific scenario or emotional changes of a specific character in a film, but is not limited to these examples, and various examples can be used. On the other hand, the computing device 100 can acquire at least one keyword based on the natural language input, which will be described below.

[0100] According to one embodiment of the present disclosure, the computing device 100 can acquire at least one keyword based on natural language input acquired through step S110 (S120). For example, the computing device 100 can input the natural language input into a pre-trained language model and acquire the at least one keyword based on one or more output data output based on the language model. In this case, the language model may be a pre-trained transformer (GPT series) model, and exemplary examples include Google's Bard, Chat-GPT, etc., but is not limited to these, and various language models may be used. The computing device 100 can also acquire at least one of a first keyword related to distance, a second keyword related to an object, a third keyword including details about the object, or a fourth keyword related to the background based on the natural language input. In this case, the first keyword related to distance may include a keyword related to the sense of distance of the background or object, such as "appears far away," and the second keyword related to an object may include a keyword related to an object that may be included in the output video, such as a character, actor, person, or animal. Furthermore, the third keyword, which includes details about the object, may include keywords that express the second keyword in more detail, and the fourth keyword, which includes the background, may include keywords that describe or explain the background area other than the object expressed in the foreground. More specifically, the computing device 100 can obtain a second-first keyword relating to the higher-level conceptual object, or a second-second keyword relating to a lower-level conceptual object of the second-first keyword, based on the natural language input. For example, the second-first keyword relating to the higher-level conceptual object may include concepts relating to comprehensive objects such as head and body, and the second-second keyword relating to a lower-level conceptual object of the second-first keyword may exemplify lower-level concepts of the head such as eyes, nose, mouth, and eyebrows, and lower-level concepts of the body such as legs, hands, fingers, and feet.As another example, the computing device 100 can acquire a third-first keyword related to one or more actions based on the natural language input, or a third-second keyword related to one or more states based on the natural language input. In this case, the third-first keyword related to one or more actions may include keywords related to the actions or activities of the object in the video, such as "moving eyes" or "unable to move," and the third-second keyword related to one or more states may include keywords related to the state of the object, such as "tense expression" or "overwhelmed," but a variety of other examples can be used. On the other hand, the computing device 100 can acquire a prompt for input to a neural network model based on at least one of the acquired keywords, and a specific explanation of this will be given later with reference to Figures 4 to 6.

[0101] According to one embodiment of the present disclosure, the computing device 100 can obtain a prompt for input to a neural network model based on at least one keyword obtained in step S120 (S130). In this case, the neural network model includes a neural network model capable of at least one of encoding or decoding of input data, and the computing device 100 can utilize a neural network model that has been trained to remove some or all of the noise from data containing noise and obtain restored data from which some or all of the noise has been removed as the neural network model. For example, a diffusion model, or a latent diffusion model such as stable diffusion or midjourney, may be used as the neural network model, but is not limited to these, and various examples may be used. Furthermore, the prompt may, but is not limited to, mean the result of transforming the keyword into a condition that can be used in the process of inputting the neural network model and acquiring video, based on a rule-based or language model such as GPT. It may also include a variety of commands corresponding to the input format in a computer program or system. On the other hand, the computing device 100 can determine the weight of the prompt based on at least one keyword, which will be explained below.

[0102] According to one embodiment of the present disclosure, the computing device 100 can determine the weights for the prompts acquired in step S130 based on at least one keyword acquired in step S120 (S140). In this regard, the computing device 100 can determine the weights for each prompt that is input to the neural network model as a generation condition. For example, if the computing device 100 inputs "steak" and "red velvet cake" as prompts to the neural network model in order to acquire images related to dinner, the computing device 100 can determine the weight for the keyword "steak" to be 0.8 and the weight for the keyword "red velvet cake" to be 0.2 and input these as generation conditions to the neural network model. As a result, in the video generated using prompts that reflect the aforementioned weights, features related to "steak" (for example, the characteristics of well-cooked meat, such as a light brown and reddish cross-section and dark brown top and bottom edges) can be expressed relatively more strongly than features related to "red velvet cake" (for example, the characteristics of a red cake, such as a dark red cross-section and the texture of layered sponge). On the other hand, if the computing device 100 determines a weight of 0.2 for the keyword "steak" and a weight of 0.8 for the keyword "red velvet cake" and inputs these as generation conditions to the neural network model 11, then in the resulting video, features related to "steak" can be expressed relatively less strongly than features related to "red velvet cake". Therefore, in the process of acquiring video using the neural network model, the computing device 100 can adjust (control) the visual features expressed in the generated video by adjusting the video information generation conditions (for example, the prompts and their weights).On the other hand, in addition to the examples in 0.8 and 0.2 above, a variety of values ​​can be determined as weights, and various examples such as text embedding by a multimodal encoder, not just prompts, can be used as generation conditions in the process of acquiring images or videos using the neural network model. On the other hand, various examples can be used as methods for assigning weights to the generation conditions of the neural network model 11, such as a method of generating a feature map that reflects the weights using a cross attention mechanism.

[0103] For example, the computing device 100 can determine the weight of the prompt based on the proportion of at least one keyword in the natural language input. Alternatively, if the natural language input includes a second keyword relating to a sub-concept object of the second keyword, the computing device 100 can determine that the weight of the second keyword is greater than the weight of the second keyword. Furthermore, if the natural language input includes a third keyword relating to one or more actions, the computing device 100 can set the weight of the keyword relating to the third keyword and its associated objects greater than the weight of the keyword relating to objects not relating to the third keyword. As another example, if the natural language input includes a third keyword relating to one or more states, the computing device 100 can set the weight of the keyword relating to the third keyword and its associated objects greater than the weight of the keyword relating to objects not relating to the third keyword. In this regard, the computing device 100 can optimize and reflect various scene conditions such as text, emotion, and location in the process of acquiring video, as described later, by determining a higher weight for prompts so that the more specific the keyword relating to the details of the object, the more detailed it is represented and the larger the proportion it occupies on the screen. On the other hand, the computing device 100 can acquire video based on the prompts whose weights have been determined by utilizing the neural network model, which will be explained later with reference to Figures 4 to 6.

[0104] According to one embodiment of the present disclosure, the computing device 100 can acquire video based on prompts whose weights are determined in step S140 by utilizing the neural network model (S150). For example, the computing device 100 can acquire an image based on the weighted prompts by utilizing the neural network model, or acquire a video based on the weighted prompts. In this regard, the computing device 100 can acquire video by determining a higher weight for the prompts so that the more specific the keywords relating to the details of the object, the more detailed they are expressed and the larger the proportion they occupy on the screen. This optimizes and reflects various scene conditions such as text, emotion, and location, and enables the acquisition of natural video in which visual features related to scenes requiring detailed depiction, such as emotional changes, are well expressed.

[0105] Additionally, the computing device 100 can acquire additional natural language input related to the additional video for the video, and acquire at least one additional keyword based on the additional natural language input. Furthermore, the computing device 100 can acquire additional prompts for input to a neural network model based on the at least one additional keyword, and can determine weights for the additional prompts based on the at least one additional keyword. Subsequently, the computing device 100 can use the neural network model to acquire additional video based on the video and the additional prompts whose weights have been determined. In this regard, in the process of acquiring the additional video, the computing device 100 can enhance the continuity between scenes and acquire video in which each scene is naturally connected by utilizing not only the additional keywords whose weights have been determined, but also visual features such as lighting and emotional changes of people included in the previously acquired video. Meanwhile, a specific explanation of the process by which the computing device 100 acquires additional video based on the additional prompts whose weights have been determined will be shown below with reference to Figure 6.

[0106] Figure 4 is a schematic diagram illustrating the process of acquiring video based on at least one keyword using a neural network model according to one embodiment of the present disclosure.

[0107] Referring to Figure 4, the computing device 100 can acquire natural language input related to the image. The natural language input 20 related to the image may include text relating to a scenario used in the creation of storyboards and storyboards for films, animations, etc., such as "Scene #1: Protagonist A was so overwhelmed by the majestic mountain scenery in the distance that he could not move." The natural language input 20 related to the image may, but is not limited to, text relating to a specific scenario or emotional change of a specific character in a film.

[0108] Furthermore, the computing device 100 can acquire at least one keyword 21-24 based on the acquired natural language input 20. The computing device 100 can input the natural language input 20 into a pre-trained language model and acquire at least one keyword 21-24 based on one or more output data output based on the language model. In this case, a pre-trained transformer (GPT series) model may be used as the language model, and exemplary examples include Google's Bard and Chat-GPT, but it is not limited to these, and a variety of language models can be used.

[0109] Specifically, the computing device 100 can acquire, based on the natural language input 20, a first keyword 21 relating to distance, "visible in the distance," a second keyword 22 relating to an object, "protagonist A," a third keyword 23 containing details about the object, "overwhelming" 23-1 and "immobile" 23-2, or a fourth keyword 24 relating to the background, "mountain scenery." Subsequently, the computing device 100 can acquire a prompt to input to the neural network model 11 based on at least one of the acquired keywords 21-24. In this case, the neural network model 11 includes a neural network model capable of at least one of encoding or decoding the input data, and the computing device 100 can utilize as the neural network model 11 a neural network model that has been trained to remove some or all of the noise from data containing noise and to acquire restored data from which some or all of the noise has been removed. For example, a diffusion model, or a latent diffusion model such as Stable Diffusion or Midjourney, may be used as the neural network model 11, but it is not limited to these, and a variety of other examples may be used. Furthermore, the prompt may mean the result of the keywords 21-24 being converted into conditions that can be used in the process of inputting the neural network model 11 and acquiring video, based on a rule-based or language model such as GPT, but it is not limited to this, and may include a variety of commands corresponding to input formats in computer programs and systems.

[0110] Subsequently, the computing device 100 can determine (30) the weights for the prompts based on at least one keyword 21-24. Specifically, the computing device 100 can determine (30) the weights for each prompt input to the neural network model 11 as generation conditions, and in this process, it can determine (30) the weights for the prompts based on the proportion of at least one keyword 21-24. For example, the computing device 100 can determine the weight for the fourth keyword, "mountain scenery" 24, as 1.5, which is higher than the weight of the second keyword, "protagonist A" 22, because the keyword "magnificent mountain scenery visible in the distance" has a larger proportion than "protagonist A". Alternatively, the computing device 100 can determine the weight for "protagonist A" 22 as 0.5, which is less than the reference value of 1, based on the first keyword related to distance, "visible in the distance" 21, and input this as a generation condition to the neural network model 11. As a result, in the video 40 generated using the prompt 31 that reflects the aforementioned weights, features related to "a majestic mountain landscape visible in the distance" (for example, the features of a large, tall mountain that occupies a larger proportion of the screen than protagonist A and towers above protagonist A's head) can be expressed relatively more strongly than features related to a specific person, "protagonist A" (for example, the features of the back view of a motionless person wearing a hooded hat).Therefore, in the process of acquiring video using the neural network model 11, the computing device 100 can adjust the visual features expressed in the resulting video 40 by adjusting the video information generation conditions (for example, the prompt and its weight 31). The more specifically described a keyword occupies a large proportion of the natural language input, the more detailed it is expressed and the higher the weight assigned to the prompt so that it occupies a larger proportion of the screen. By determining a higher weight for the prompt and acquiring the video, it is possible to acquire a more natural video that reflects visual importance and the context of the scenario compared to simply acquiring a video using only keywords. On the other hand, a variety of values ​​other than the examples in 1.5 and 0.5 can be determined as weights, and a variety of examples can be used as generation conditions in the process of acquiring images or videos using the neural network model 11. On the other hand, a variety of examples can be used as methods for assigning weights to the generation conditions of the neural network model 11, such as using a cross-attention mechanism to generate a feature map that reflects the weights. On the other hand, the computing device 100 can determine the weights for prompts based on the detailed content of the keywords and acquire video, which will be explained later in Figure 5.

[0111] Figure 5 is a schematic diagram illustrating the process by which, according to one embodiment of the present disclosure, weights for prompts are determined based on at least one keyword, and a neural network model is used to acquire video based on the weights of the prompts.

[0112] Referring to Figure 5, the computing device 100 can acquire natural language input related to the video, and the natural language input 20' related to the video may include text relating to a scenario used in the creation of storyboards and storyboards for movies, animations, etc., such as "Scene #11: Protagonist B kept moving his eyes incessantly, his face tense." The natural language input 20' related to the video may also include, but is not limited to, text relating to a specific scenario or emotional change of a specific character in a movie. Furthermore, the computing device 100 can acquire at least one keyword 22-1'~23-2' based on the acquired natural language input 20', and the computing device 100 can input the natural language input 20' into a pre-trained language model and acquire at least one keyword 22-1'~23-2' based on one or more output data output based on the language model. In this case, a pre-trained transformer (GPT series) model can be used as the language model. For example, Google's Bard and Chat-GPT can be used, but the model is not limited to these, and various language models can be used.

[0113] Specifically, the computing device 100 can obtain, based on the natural language input 20', the second-first keyword 22-1' relating to the higher-level conceptual object, "protagonist B", and the second-second keyword 22-2' relating to the lower-level conceptual object of "protagonist B" 22-1, "protagonist B's eyes". The computing device 100 can also obtain the third-first keyword "move eyes" 23-2' relating to the action of "protagonist B's eyes" 22-2', and the third-second keyword "tense expression" 23-1' relating to the state of "protagonist B" 22-1'. Subsequently, the computing device 100 can obtain a prompt to input to the neural network model 11 based on at least one of the obtained keywords 22-1' to 23-2'. In this case, the neural network model 11 includes a neural network model capable of at least one of encoding or decoding the input data. The computing device 100 can utilize as the neural network model 11 a neural network model that has been trained to remove some or all of the noise from data containing noise and to obtain restored data from which some or all of the noise has been removed. For example, a diffusion model, or a latent diffusion model such as Stable Diffusion or Midjourney, may be used as the neural network model 11, but is not limited to these, and various examples may be used. Furthermore, the prompt may mean the result of converting the keywords 22-1'~23-2' into conditions that can be used in the process of inputting the neural network model 11 and acquiring video, based on a rule-based or language model such as GPT, but is not limited to this, and may include a variety of commands corresponding to the input format in a computer program or system.

[0114] Subsequently, the computing device 100 can determine (30') the weights for the prompts based on at least one keyword 22-1' to 23-2'. Specifically, the computing device 100 can determine (30') the weights for each prompt input to the neural network model 11 as generation conditions. In this process, if the natural language input 20' includes the second keyword "protagonist B's eyes" 22-2', which relates to a sub-concept object of the second keyword "protagonist B" 22-1', the computing device 100 can determine that the weight for the second keyword "protagonist B's eyes" 22-2' is greater than the weight for "protagonist B" 22-1'. For example, the computing device 100 can determine the weights for "protagonist B's eyes" 22-2' and "moving the eyes" 23-2' as weights of 2, which are higher than the weights of 1 for "protagonist B" 22-1' and "tense expression" 23-1', and input these as generation conditions to the neural network model 11. As a result, in the video 40' generated using the prompt 31' that reflects these weights, features related to "protagonist B's eyes moving" (for example, the features of protagonist B's eyes that are trembling due to tension) can be expressed relatively more strongly than features related to "protagonist B with a tense expression" (for example, the features of protagonist B's entire face, such as dry lips and sweat).Therefore, in the process of acquiring video using the neural network model 11, the computing device 100 can adjust the visual features expressed in the resulting video 40' by adjusting the conditions for generating video information (for example, the prompt and its weight 31'). The more specific the keyword relating to the details of an object, the more detailed it will be expressed, and the higher the weight assigned to the prompt so that it occupies a larger proportion of the screen. By optimizing and reflecting the conditions of various scenes such as text, emotion, and location, the computing device 100 can acquire natural video in which the visual features of scenes requiring detailed depiction, such as emotional changes, are well expressed. On the other hand, in addition to the examples in 2 and 1 above, a variety of other examples can be determined as weights, and these various examples can be used as conditions for generating images or videos using the neural network model 11. On the other hand, the computing device 100 can acquire additional video based on additional prompts whose weights have been determined, and this will be explained later with reference to Figure 6.

[0115] Figure 6 is a schematic diagram illustrating the process of acquiring additional video based on weighted additional prompts, utilizing a neural network model according to one embodiment of the present disclosure.

[0116] Referring to Figure 6, the computing device 100 may include "Scene #2: Overwhelmed by the scenery, protagonist A was moved to tears." as additional natural language input 50 related to additional images for the video 40. However, the additional natural language input 50 related to the video is not limited to the above examples, and a variety of examples can be used. Based on the additional natural language input 50, the computing device 100 can obtain at least one additional keyword such as "protagonist A" 51, "overwhelmed" 52-1, "moved to tears" 52-2, etc. Furthermore, based on the at least one additional keyword 51 to 52-2, the computing device 100 can obtain an additional prompt for input to the neural network model 11, and determine the weights for the additional prompts (60) based on the at least one additional keyword 51 to 52-2. In this process, the examples described in this disclosure can be used when the computing device 100 determines the weights for the additional prompts. Subsequently, the computing device 100 can use the neural network model 11 to acquire additional video 70 based on the video 40 and the additional prompts 61 whose weights have been determined. Specifically, in the process of acquiring the additional video 70, the computing device 100 utilizes not only the additional prompts 61 whose weights have been determined, but also visual features from video 40 of scene #1 acquired by example in Figure 4, such as "the visual feature of illumination (sunlight) shining from the mountains toward protagonist A," "the visual feature of the faintly blue sky at dawn," and "the emotional changes of protagonist A." By doing so, it can enhance the continuity between scenes and acquire video in which each scene requiring detailed depiction, such as emotional changes, is naturally connected.

[0117] Figure 7 is a simplified and general schematic diagram of an exemplary computing environment in which embodiments of the present disclosure can be realized.

[0118] While it has been stated that this disclosure can generally be embodied by computing devices, those skilled in the art will understand that this disclosure can also be embodied in combination with computer executable instructions and / or other program modules that can be run on one or more computers, and / or as a combination of hardware and software.

[0119] Generally, modules as defined herein include routines, programs, components, data structures, and so on, that perform a specific task or implement a specific abstract data type. Furthermore, those skilled in the art will understand that the methods disclosed herein can be implemented in configurations of other computer systems, including single-processor or multi-processor computer systems, minicomputers, mainframe computers, as well as personal computers, handheld computing devices, microprocessor boards, or programmable consumer electronics, and so on (each of which can operate in conjunction with one or more associated devices). The embodiments described herein can further be implemented in a distributed computing environment in which a task is performed by remote processing units connected via a communication network. In a distributed computing environment, program modules can reside in both local and remote memory storage devices.

[0120] Computers include a variety of computer-readable media. Any media accessible by a computer can be computer-readable, but such computer-readable media include volatile and non-volatile media, transient and non-transitory media, and portable and non-portable media. By example, rather than by limitation, computer-readable media may include computer-readable storage media and computer-readable transmission media. Computer-readable storage media include volatile and non-volatile media, transient and non-transitory media, portable and non-portable media implemented by any method or technique for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer-readable storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, DVD (digital video disk) or other optical disk storage devices, magnetic cassettes, magnetic tapes, magnetic disk storage devices or other magnetic storage devices, or any other media that can be accessed by a computer and used to store information. Computer-readable transmission media typically include all information transmission media that implement computer-readable instructions, data structures, program modules, or other data on a modulated data signal, such as a carrier wave or other transport mechanism. The term modulated data signal means a signal in which one or more of its characteristics have been set or modified to encode information within the signal. By example, rather than by limitation, computer-readable transmission media include wired media such as wired networks or direct-wired connections, and wireless media such as acoustic, RF, infrared, and other wireless media. Any combination of any of the aforementioned media is also included in the scope of computer-readable transmission media.

[0121] An exemplary environment (1100) is shown that realizes various aspects of this disclosure, including a computer (1102), the computer (1102) including a processor (1104), system memory (1106), and a system bus (1108). The system bus (1108) connects system components, including (but not limited to) system memory (1106), to the processor (1104). The processor (1104) can be any processor from a variety of commercial processors. Dual processors and other multiprocessor architectures can also be used as the processor (1104). The system bus (1108) can be one of several types of bus structures that can be further interconnected to a local bus using any of the following: a memory bus, a peripheral bus, and various commercial bus architectures. System memory (1106) includes read-only memory (ROM) (1110) and random access memory (RAM) (1112). The basic input / output system (BIOS) is stored in non-volatile memory (1110) such as ROM, EPROM, or EEPROM, and this BIOS includes basic routines that support the exchange of information between multiple components within the computer (1102) during startup, etc. RAM (1112) may also include high-speed RAM such as static RAM for caching data.

[0122] The computer (1102) also includes an internal hard disk drive (HDD) (1114) (e.g., EIDE, SATA)—this internal hard disk drive (1114) can also be configured for external use in a suitable chassis (not shown), a magnetic floppy disk drive (FDD) (1116) (e.g., for reading from and writing to a portable diskette (1118)), and an optical disk drive (1120) (e.g., for reading CD-ROM disks (1122) or for reading from and writing to other high-capacity optical media such as DVDs). The hard disk drive (1114), magnetic disk drive (1116), and optical disk drive (1120) can be connected to the system bus (1108) by a hard disk drive interface (1124), a magnetic disk drive interface (1126), and an optical drive interface (1128), respectively. The interface (1124) for implementing external drives includes, for example, at least one or both of the following: USB (Universal Serial Bus) or IEEE 1394 interface technology. These drives and computer-readable media provide non-volatile storage of data, data structures, computer-executable instructions, and so on. In the case of a computer (1102), the drives and media correspond to storing any data in a suitable digital format. While the above description of computer-readable storage media refers to HDDs, portable magnetic disks, and portable optical media such as CDs or DVDs, those skilled in the art will understand that other types of computer-readable storage media, such as zip drives, magnetic cassettes, flash memory cards, cartridges, and so on, can also be used in exemplary operating environments, and that any of these media may contain computer-executable instructions for performing the methods of the present disclosure.

[0123] Numerous program modules, including an operating system (1130), one or more application programs (1132), other program modules (1134), and program data (1136), can be stored in the drive and RAM (1112). All or part of the operating system, applications, modules, and / or data can also be cached in RAM (1112). It will be understood that this disclosure can be implemented by various commercially available operating systems or combinations of operating systems. The user can input commands and information to the computer (1102) through one or more wired or wireless input devices, such as a keyboard (1138) and a pointing device such as a mouse (1140). Other input devices (not shown in the diagram) may include a microphone, IR remote control, joystick, gamepad, stylus pen, touchscreen, and so on. These and other input devices are often connected to the processing unit (1104) through an input device interface (1142) connected to the system bus (1108), but they can also be connected through other interfaces such as parallel ports, IEEE1394 serial ports, game ports, USB ports, IR interfaces, and so on.

[0124] A monitor (1144) or other type of display device is also connected to the system bus (1108) through an interface such as a video adapter (1146). In addition to the monitor (1144), the computer generally includes other peripheral output devices such as speakers, printers, and so on (not shown in the illustration). A computer (1102) can operate in a networked environment by utilizing logical connections to one or more remote computers (1148), such as multiple remote computers (1148), via wired and / or wireless communication. The multiple remote computers (1148) can be workstations, server computers, routers, personal computers, portable computers, microprocessor-based entertainment devices, peer devices, or other typical network nodes, and generally include many or all of the components described for a computer (1102), although for simplification only a memory storage device (1150) is illustrated. The illustrated logical connections include wired and wireless connections in a short-range network (LAN) (1152) and / or a larger network, such as a long-range network (WAN) (1154). Such LAN and WAN networking environments are common in offices and companies, facilitating enterprise-wide computer networks such as intranets, all of which can connect to global computer networks, such as the Internet.

[0125] When used in a LAN networking environment, the computer (1102) connects to the local network (1152) via a wired and / or wireless network interface, or via an adapter (1156). The adapter (1156) facilitates wired or wireless communication to the LAN (1152), which also includes a wireless access point installed therein to communicate with the wireless adapter (1156). When used in a WAN networking environment, the computer (1102) may include a modem (1158), connect to a communication server on the WAN (1154), or have other means of establishing communication through the WAN (1154), such as via the Internet. The modem (1158), which can be internal or external, and wired or wireless, connects to the system bus (1108) via a serial port interface (1142). In a networked environment, a program module or part thereof described for a computer (1102) can be stored in a remote memory / storage device (1150). While the illustrated network connection is illustrative, it is readily apparent that other means of establishing communication links between multiple computers may be used. The computer (1102) operates to communicate with any wireless device or unit that is arranged and operates wirelessly, such as a printer, scanner, desktop and / or portable computer, PDA (portable data assistant), communication satellite, any equipment or location relating to a wirelessly discoverable tag, and a telephone. This includes at least Wi-Fi and Bluetooth® wireless technologies. Thus, the communication may be a predefined structure like a conventional network, or simply ad hoc communication between at least two devices.

[0126] Wi-Fi (Wireless Fidelity) enables internet access and other connectivity without a wired connection. Wi-Fi is a wireless technology, similar to cell phones, that allows devices like computers to send and receive data indoors and outdoors, i.e., anywhere within the range of a base station. Wi-Fi networks use IEEE 802.11 (a, b, g, etc.) wireless technology to provide secure, reliable, and high-speed wireless connectivity. Wi-Fi can be used to connect computers to each other, to the internet, and to wired networks (using IEEE 802.3 or Ethernet). Wi-Fi networks can operate in unlicensed 2.4 and 5 GHz wireless bands at data rates such as 11 Mbps (802.11a) or 54 Mbps (802.11b), or in products that include both bands (dual band). A person with ordinary skill in the art of this disclosure will understand that information and signals can be represented using any variety of different techniques and methods. For example, the data, instructions, commands, information, signals, bits, symbols and chips referenced in the foregoing description can be represented by voltage, current, electromagnetic waves, magnetic fields, etc. or particles, optical fields, etc. or particles, or any combination thereof.

[0127] A person with ordinary skill in the art of this disclosure will understand that the various exemplary logic blocks, modules, processors, means, circuits, and algorithmic stages described herein can be implemented by electronic hardware, various forms of programs or design code (referred to herein for convenience as “software”), or a combination of all of these. To illustrate this interoperability of hardware and software, various exemplary components, blocks, modules, circuits, and stages have been generally described above with respect to their functions. Whether such functions are implemented in hardware or software depends on the design constraints imposed on a particular application and the overall system. A person with ordinary skill in the art of this disclosure can implement the functions described in various ways for individual specific applications, but such decisions should not be construed as departing from the scope of this disclosure. The various embodiments described herein can be realized by methods, apparatus, or manufactured articles using standard programming and / or engineering techniques. The term “manufactured article” includes computer programs, carriers, or media accessible from any computer-readable device. For example, computer-readable storage media include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, magnetic strips, etc.), optical disks (e.g., CDs, DVDs, etc.), smart cards, and flash memory devices (e.g., EEPROMs, cards, sticks, key drives, etc.). The various storage media described herein also include one or more devices and / or other machine-readable media for storing information.

[0128] It should be understood that the specific order or hierarchical structure of the multiple stages in the presented process is an example of an exemplary approach. It should be understood that, based on design priorities, the specific order or hierarchical structure of the stages in the process can be rearranged within the scope of this disclosure. The appended method claims provide a variety of stage elements in sample order, but are not limited to the specific order or hierarchical structure shown. The descriptions relating to the embodiments provided herein are provided so that any person with ordinary skill in the art of the present disclosure may utilize or implement the present disclosure. Various variations of such embodiments are readily apparent to a person with ordinary skill in the art of the present disclosure, and the general principles defined herein can be applied to other embodiments without departing from the scope of the present disclosure. Accordingly, the present disclosure is not limited by the embodiments provided herein and should be interpreted in the broadest sense consistent with the principles and novel features provided herein.

Claims

1. A method for acquiring images using a neural network model, which is executed by one or more processors in a computing device, The stage of acquiring natural language input related to the video, A step of obtaining at least one keyword based on the aforementioned natural language input, A step of obtaining a prompt to input to a neural network model based on at least one of the aforementioned keywords, A step of determining a weight for the prompt based on at least one of the keywords, Using the aforementioned neural network model, the process involves acquiring video based on the prompt whose weights have been determined, and Includes, The step of obtaining at least one keyword based on the aforementioned natural language input is: The steps include obtaining a second-first keyword relating to a higher-level conceptual object based on the aforementioned natural language input, The step includes obtaining a second keyword relating to a sub-concept object of the second keyword based on the natural language input, The step of determining the weight for the prompt based on at least one keyword is: If the natural language input includes a second keyword relating to a sub-concept object of the second keyword, the process includes determining that the weight of the second keyword is greater than the weight of the second keyword. method.

2. The step of obtaining at least one keyword based on the aforementioned natural language input is: The steps include: inputting the aforementioned natural language input into a pre-trained language model; The step of obtaining at least one keyword based on one or more output data generated based on the language model, The method according to claim 1.

3. The step of obtaining at least one keyword based on the aforementioned natural language input is: The steps include obtaining a first keyword related to distance based on the aforementioned natural language input, The steps include obtaining a second keyword relating to an object based on the aforementioned natural language input, A step of obtaining a third keyword containing details about the object based on the natural language input, The process includes at least one of the following steps: obtaining a fourth keyword related to the background based on the natural language input; The method according to claim 2.

4. The step of obtaining a third keyword containing details about the object based on the aforementioned natural language input is: The steps include obtaining a third-order keyword related to one or more activities (actions) based on the aforementioned natural language input, The step of obtaining a third-second keyword related to one or more states based on the aforementioned natural language input, and at least one of these steps, The method according to claim 3.

5. The aforementioned neural network model is A neural network model capable of performing at least one of the following operations on input data: encoding or decoding. The method according to claim 1.

6. The step of acquiring video based on prompts whose weights have been determined using the aforementioned neural network model is as follows: Using the aforementioned neural network model, the step of acquiring an image based on the prompt whose weights have been determined, The process includes at least one of the following steps: using the neural network model to acquire a video based on the weights of the prompts, The method according to claim 1.

7. The step of determining the weight for the prompt based on at least one keyword is: The step includes determining a weight for the prompt based on the weighting of at least one keyword included in the natural language input, The method according to claim 3.

8. The step of determining the weight for the prompt based on at least one keyword is: If the natural language input includes one or more third-first keywords related to an action, the step includes setting the weight of the keyword for objects related to the third-first keyword to be greater than the weight of the keyword for objects not related to the third-first keyword. The method according to claim 4.

9. The step of determining the weight for the prompt based on at least one keyword is: If the natural language input includes one or more state-related keywords, the step includes setting the keyword weight for objects associated with the keyword to be greater than the keyword weight for objects not associated with the keyword. The method according to claim 4.

10. The aforementioned method, The stage of obtaining additional natural language input related to the additional video, The steps include obtaining at least one additional keyword based on the aforementioned additional natural language input, A step of obtaining an additional prompt to input to the neural network model based on at least one additional keyword, A step of determining a weight for the additional prompt based on the at least one additional keyword, The process further includes the step of using the neural network model to acquire additional video based on the video and additional prompts whose weights have been determined, The method according to claim 1.

11. A computer program stored on a computer-readable recording medium, When the computer program is executed by one or more processors, the one or more processors are configured to perform operations to acquire images using a neural network model. The aforementioned operation is, The operation of obtaining natural language input related to the video, The operation of obtaining at least one keyword based on the aforementioned natural language input, An operation to obtain a prompt for input to a neural network model based on at least one of the aforementioned keywords, An operation to determine a weight for the prompt based on at least one of the keywords, Using the aforementioned neural network model, the operation of acquiring video based on the prompt whose weights have been determined, Includes, The operation of obtaining at least one keyword based on the aforementioned natural language input is: The operation of obtaining a second-first keyword relating to a higher-level conceptual object based on the aforementioned natural language input, This includes the operation of obtaining a second keyword relating to a subordinate conceptual object of the second keyword based on the natural language input, The operation of determining a weight for the prompt based on at least one keyword is: If the natural language input includes a second keyword relating to a sub-concept object of the second keyword, the operation includes determining that the weight of the second keyword is greater than the weight of the second keyword. A computer program stored on a computer-readable recording medium.

12. The operation of obtaining at least one keyword based on the aforementioned natural language input is: The operation of inputting the aforementioned natural language input into a pre-trained language model, The operation includes obtaining at least one keyword based on one or more output data generated based on the language model, A computer program stored on a computer-readable recording medium according to claim 11.

13. The operation of obtaining at least one keyword based on the aforementioned natural language input is: The operation of obtaining a first keyword related to distance based on the aforementioned natural language input, The operation of obtaining a second keyword related to the object based on the aforementioned natural language input, An operation to obtain a third keyword containing details about the object based on the aforementioned natural language input, The operation includes at least one of the following: obtaining a fourth keyword related to the background based on the aforementioned natural language input. A computer program stored on a computer-readable recording medium according to claim 12.

14. The operation of obtaining a third keyword containing details about an object based on the aforementioned natural language input is as follows: The operation of obtaining one or more third-order keywords related to an action based on the aforementioned natural language input, The operation includes at least one of the following: obtaining a third-second keyword related to one or more states based on the aforementioned natural language input. A computer program stored on a computer-readable recording medium as described in claim 13.

15. The operation of acquiring video based on prompts whose weights have been determined using the aforementioned neural network model is as follows: Using the aforementioned neural network model, the operation of acquiring an image based on the prompt whose weights have been determined, The operation includes at least one of the following: using the neural network model to acquire a video based on the prompt whose weights have been determined, A computer program stored on a computer-readable recording medium according to claim 11.

16. The operation of determining a weight for the prompt based on at least one keyword is: This includes determining a weight for the prompt based on the weighting of at least one keyword included in the natural language input, A computer program stored on a computer-readable recording medium as described in claim 13.

17. The operation of determining a weight for the prompt based on at least one keyword is: If the natural language input includes one or more third-order keywords related to an action, the operation includes setting the weight of the keyword for objects related to the third-order keyword to be greater than the weight of the keyword for objects not related to the third-order keyword. A computer program stored on a computer-readable recording medium as described in claim 14.

18. The operation of determining a weight for the prompt based on at least one keyword is: If the natural language input includes one or more state-related keywords, the operation includes setting the weight of the keyword for objects associated with the keyword to be greater than the weight of the keyword for objects not associated with the keyword. A computer program stored on a computer-readable recording medium as described in claim 14.

19. A computing device, It includes at least one processor and memory, The at least one processor is Acquire natural language input related to the video, Based on the aforementioned natural language input, obtain at least one keyword, Based on at least one of the aforementioned keywords, a prompt is obtained for input to the neural network model. A weight for the prompt is determined based on at least one of the keywords. The system is configured to acquire video based on prompts whose weights have been determined, utilizing the aforementioned neural network model. The process of obtaining at least one keyword based on the aforementioned natural language input is as follows: Based on the aforementioned natural language input, obtain the second-first keyword relating to the higher-level conceptual object. This process includes obtaining a second keyword relating to a subordinate conceptual object of the second keyword based on the natural language input, The process of determining the weight for the prompt based on at least one keyword is: If the natural language input includes a second keyword relating to a sub-concept object of the second keyword, the process includes determining that the weight of the second keyword is greater than the weight of the second keyword. Computing device.

Images