Generative video engine capable of outputting videos of various durations, resolutions, and aspect ratios.
By employing a diffusion-transformer architecture and a multimodal cue-based visual media generative response engine, this approach addresses the memory and interface limitations of AI tools when generating videos, enabling high-quality, flexible video generation and supporting video creation across various resolutions, aspect ratios, and lengths.
Patent Information
- Application Number
- CN202580027529.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2025-01-31
- Filing Date
- 2025-02-12
- Publication Date
- 2026-11-17
AI Technical Summary
Existing AI tools are limited by model attention and insufficient memory when generating videos and long texts. Furthermore, their interfaces mainly rely on text input and have limited multimodal prompt responses, resulting in insufficient generation quality and flexibility.
The visual media generative response engine, which employs a diffusion-transformer structure, decomposes video frames into spatiotemporal blocks and combines them with image and text prompts to provide a user interface to guide video creation, thereby achieving high-quality video generation.
It enables video generation at various resolutions, aspect ratios, and lengths, improving the quality and flexibility of generated videos, supporting multimodal input, and enhancing artists' control over the generation process.
Smart Images

Figure CN123125625A_ABST
Abstract
Description
[0001] Cross-references to related applications This application claims priority to U.S. Application No. 19 / 042463, filed January 31, 2025, and U.S. Provisional Application No. 63 / 553637, filed February 14, 2024, and U.S. Provisional Application No. 63 / 729367, filed December 7, 2024, under USC §119(e), all of which are expressly incorporated herein by reference in their entirety. Background Technology
[0002] The past few years have seen many exciting developments in the fields of machine learning and artificial intelligence (AI). Large language models built on transformer architectures and image generation models built on diffusion models have demonstrated significant high-level capabilities in producing software-generated content of high quality. Such tools have not only provided efficiency gains by leveraging AI tools to perform procedural tasks, but AI tools have also been found to be useful in the creative process. Some artists use AI tools to aid in brainstorming or to enhance ideas.
[0003] Recently, artists have begun using AI tools to bring their artistic visions to life through AI-guided creations and images, such as works and images guided by artists and other creative professionals. Therefore, we are entering an exciting new era of developing new forms of new media creation.
[0004] Meanwhile, AI tools are still in their early stages of development. Many technical challenges remain to be addressed. One such problem is the limitation on model attention, which poses a challenge to the amount of memory available to the processing unit. This limitation severely restricts the ability of AI tools to generate videos and long-form text.
[0005] The interfaces of AI tools are even more rudimentary. Until recently, most prompts from AI tools involved text input. Only recently have multimodal prompts (a combination of images, text, and / or audio) become possible. Even with multimodal prompts, the prompts are still delivered, and the AI tool responds to them. Prompts are typically provided in a chat interface or via an application programming interface (API).
[0006] Therefore, AI tools have shown great promise, and many anticipate they will have a significant impact in the future. However, many technical and user interface limitations remain to be overcome before AI tools become truly useful for creating high-quality video and longer-form outputs in any modality. Attached Figure Description
[0007] Details of one or more aspects of the subject matter described in this disclosure are set forth in the accompanying drawings and the following description. However, the drawings illustrate only some typical aspects of this disclosure and are therefore not to be considered as limiting its scope. Other features, aspects, and advantages will become apparent from the specification, drawings, and claims.
[0008] Figure 1 An example system architecture for a visual media generative response engine according to some embodiments of the present technology is shown.
[0009] Figure 2 A noise frame input to a diffusion model according to some embodiments of the present technology is shown.
[0010] Figure 3A and Figure 3B Examples of encoding noise frames and cue frames into a set of spatiotemporal patches according to some embodiments of the present technology are shown.
[0011] Figure 4 The vectors representing the potential representations according to some embodiments of the present technology are shown in more detail.
[0012] Figure 5 An example process is shown that uses a visual media generative response engine to generate visual media of different durations, resolutions, and aspect ratios, according to some embodiments of the present technology.
[0013] Figure 6 An example process is shown that illustrates additional functionality of the input layer according to some embodiments of the present technology.
[0014] Figure 7 An example process for training a visual media generative response engine according to some embodiments of the present technology is shown.
[0015] Figure 8 An example system supporting a generative response engine during inference operations is shown, according to some embodiments of the present technology.
[0016] Figure 9A , Figure 9B and Figure 9C An example converter architecture according to some embodiments of the present technology is shown.
[0017] Figure 10 An example process for creating prompts for generating visual media by presenting and using a graphical user interface according to some embodiments of the present technology is shown.
[0018] Figure 11 Example library interfaces according to some embodiments of the present technology are shown.
[0019] Figure 12 A resolution input control menu according to some embodiments of the present technology is shown.
[0020] Figure 13 An aspect ratio input control menu according to some embodiments of the present technology is shown.
[0021] Figure 14A and Figure 14B The style control menu and configuration interface according to some embodiments of the present technology are shown.
[0022] Figure 15 A quantity input control for generating quantities according to some embodiments of the present technology is shown.
[0023] Figure 16 An example process for creating new visual media from existing visual media cues, according to some embodiments of the present technology, is illustrated.
[0024] Figure 17A Example frames from previously created visual media are shown, representing some embodiments of the present technology.
[0025] Figure 17B Editing options according to some embodiments of this technology are shown.
[0026] Figure 18 An example process is shown, according to some embodiments of the present technology, for helping users generate prompts for a visual media generative response engine through a storyboard user interface.
[0027] Figure 19A , Figure 19B , Figure 19C and Figure 19D Some aspects of the storyboard user interface according to some embodiments of the present technology are shown.
[0028] Figure 20A , Figure 20B and Figure 20C Examples of generated videos and corresponding storyboard user interfaces according to some embodiments of the present technology are shown.
[0029] Figure 21 Examples of using a storyboard user interface to transition visual media into different scenes are shown, according to some embodiments of the present technology.
[0030] Figure 22A and Figure 22B Examples of enhancing prompts to be more descriptive according to some embodiments of the present technology are shown.
[0031] Figure 23An example process for inducing a visual media generative response engine to reconstruct existing visual media according to some embodiments of the present technology is shown.
[0032] Figure 24A and Figure 24B A remixing intensity control menu according to some embodiments of the present technology is shown.
[0033] Figure 25 An example process for influencing the fusion of a first visual media and a second visual media, according to some embodiments of the present technology, is shown.
[0034] Figure 26A and Figure 26B Examples of fusion interfaces according to some embodiments of the present technology are shown.
[0035] Figure 27A , Figure 27B and Figure 27C A frame of fused visual media using a transition fusion curve is shown according to some embodiments of the present technology.
[0036] Figure 28 A loop interface according to some embodiments of the present technology is shown.
[0037] Figure 29 The clip frame interface buttons according to some embodiments of the present technology are shown.
[0038] Figure 30 An example process for requesting visual media from a visual media generative response engine using an API, according to some embodiments of the present technology, is shown.
[0039] Figure 31 Examples of API calls according to some embodiments of this technology are shown.
[0040] Figure 32 An example process for modulating visual media generated by a visual media generative response engine, according to some embodiments of the present technology, is shown.
[0041] Figure 33 This is a block diagram illustrating an example machine learning platform for implementing various aspects of this disclosure, based on some aspects of the present technology.
[0042] Figure 34 Examples of systems for implementing some embodiments of this technology are shown. Detailed Implementation
[0043] Various embodiments of this disclosure are discussed in detail below. While specific implementations are discussed, it should be understood that this is done for illustrative purposes only. Those skilled in the art will recognize that other components and configurations can be used without departing from the spirit and scope of this disclosure.
[0044] Recently, artists have begun using AI tools to bring their artistic visions to life through guided creations and images, such as those guided by AI tools from artists and other creative professionals. Therefore, we are entering an exciting new era of developing new forms of new media creation.
[0045] Meanwhile, AI tools are still in their early stages of development. Many technical challenges remain to be addressed. One such problem is the limitation on model attention, which poses a challenge to the amount of memory available to the processing unit. This limitation severely restricts the ability of AI tools to generate videos and long-form text.
[0046] The interfaces of AI tools are even more rudimentary. Until recently, most prompts in AI tools involved text input. Only recently have multimodal prompts (a combination of images, text, and / or audio) become possible. Even with multimodal prompts, the prompts are still delivered, and the AI tool responds to them. Prompts are typically provided in a chat interface or through an application programming interface (API).
[0047] Therefore, AI tools have shown great promise, and many anticipate they will have a significant impact in the future. However, many technical and user interface limitations remain to be overcome before AI tools become truly useful for creating high-quality video and longer-form outputs in any modality.
[0048] This technology solves these and more problems, resulting in a class of first-of-its-kind visual media generative response engine capable of producing high-quality video. This engine can also generate visual media of various aspect ratios, resolutions, and lengths.
[0049] To build a visual media generative response engine capable of producing such videos, numerous technical challenges have been overcome. Furthermore, some solutions to these challenges are applicable to fields beyond visual media generation. Therefore, while this description is presented in the context of a visual media generative response engine, it should be understood that many of the embodiments presented herein are themselves innovative techniques that can be used in generative response engines to generate audio, text, or audiovisual content, or other multimodal content.
[0050] As described above, this technology is the first of its kind. It is not only the first technology capable of producing high-quality videos of various lengths, resolutions, and aspect ratios, but also the first to utilize a diffusion-transformer architecture. This architecture combines the advantages of both diffusion and transformer models, with the diffusion model demonstrating excellent performance in image generation, while the transformer model is a relatively scalable model.
[0051] Another improvement offered by this technology is that its visual media generative response engine represents the physical world and objects within it with remarkable accuracy. This is a phenomenon unique to the massive amounts of video on which this technology has been trained, and the fact that it is trained on native video. Native video refers to video that has not undergone preprocessing. Most machine learning models require uniform input data. Therefore, previous attempts to provide input for image and video models have yielded inputs with uniform resolution, aspect ratio, and video length. However, this leads to distortion in visual media. Whether scaling or cropping frames, the images within a frame can become distorted. Even truncating a video can warp the beginning and end of a concept depicted by a visual artist. Of course, the ability of the visual media generative response engine to learn on images and videos of any resolution, aspect ratio, and video length presents its own technical challenges. These challenges are addressed by breaking down the frames of the input photos and videos into spatiotemporal blocks, where each block is of uniform size, but any number of spatiotemporal blocks can be used to capture the entire frame.
[0052] Another innovation offered by this technology is its ability to receive and utilize both image and text cues to guide video creation. For example, this technology enables novel use cases such as generating videos that begin with an input image or video, generating videos that end with an input image or video, merging one image or video into another, and extracting features from an image or video by rendering feature-rich frames. These use cases are achieved by passing some visual frames and some text cues to the diffusion-transformer. Of course, a mixture of these input types is also needed to train the diffusion-transformer. Visual and text inputs can be passed to the diffusion-transformer at certain frame locations, instructing the model what visual input to retain across generated frames and how generated frames should relate to the visual input throughout the video process. This innovation provides the benefit of making visual media generative response engines more general and gives artists greater ability to communicate their visions to the visual media generative response engine.
[0053] Another innovative aspect of this technology is providing a user interface that allows users to provide input to the visual media generative response engine. This technology includes several innovative graphical user interfaces. One such graphical user interface is a timeline interface, where visual cue frames can be provided as input at certain points on the timeline, while text cueing can be provided at other points. In this way, artists can guide the visual media generative response engine to generate videos that more faithfully realize the artist's vision.
[0054] This article will elaborate on these and other innovations in more detail.
[0055] Figure 1 An example system architecture of a visual media generative response engine 100 according to some embodiments of the present technology is shown. Although the example system depicts specific system components and their arrangement, this depiction is for the purpose of facilitating the discussion of the present technology and should not be considered limiting unless specified in the appended claims. For example, some components shown as separate may be combined with other components, some components may be divided into separate components, some components may be absent or unnecessary, and additional components may be present.
[0056] Figure 1 The visual media generative response engine 100 is shown to start from a front-end 124 that receives user input, through a diffusion-transformer layer 114 that generates visual media, and to the output of a block decoder layer 120 that outputs visual media frames.
[0057] Frontend 124 can be a user interface or an application programming interface (API) (see...) Figure 8 API 806 in the Visual Media Generative Response Engine 100 receives a prompt regarding the desired output visual media. This prompt may include prompt text 108 and / or prompt frames 110. In some embodiments, text is required as part of the prompt, but one or more visual media frames may optionally be provided. As will be explained later, the prompt may also include various other inputs that may specify visual formats such as resolution and aspect ratio, as well as other properties.
[0058] The cues received by front-end 124 are passed to input layer 122. Input layer 122 is responsible for determining how to process certain aspects of the cues and determining the attributes of the visual format of the visual media (which may or may not be provided through the input to front-end 124). More specifically, some parts of the cues may bring several types of input to diffusion-transformer layer 114. Cue text 108 is provided to diffusion-transformer layer 114 as an embedding created by embedding layer 112. However, cue frame 110 may be provided to diffusion-transformer layer 114 as an embedding provided by embedding layer 112 or as an encoded frame provided by encoding layer 106. Input layer 122 sends frames to encoding layer 106 when it determines that selected frames should be part of the output visual media, and sends frames provided to express the concept to embedding layer 112. This distinction will be explained in more detail herein. Front-end 124 may also receive input defining one or more of duration, resolution, and aspect ratio 126.
[0059] like Figure 1 As shown, input layer 122 also generates noise frames 102 for input to encoding layer 106. Noise frames 102 convey the number of frames, aspect ratio, and resolution that should be present in the output visual media. Therefore, input layer 122 determines the length of the visual media to be output and provides the necessary number of frames to create the visual media. If the output is a single image, only one frame is provided. If the output is a 20-second video at 30 frames per second (fps), 60 frames are provided. In some embodiments, input layer 122 determines the aspect ratio, length, and resolution from explicit input provided to front end 124. In some embodiments, input layer 122 infers the aspect ratio, length, and resolution from other cues.
[0060] Noisy frame 102 is a typical input to a diffusion model, which is used to iteratively denoise random noise patterns based on cues to generate cue-based images. A diffusion model is a generative model used in machine learning that learns to generate data by progressively denoising a signal. It starts with the distribution of random noise and, through a series of steps mimicking the process of backdiffusion, gradually refines the noise into data with the desired characteristics. This method is particularly known for its ability to generate high-quality, richly detailed images. Figure 2 This process is illustrated by showing a noisy frame 202 input to the diffusion model (in this case, a diffusion-transformer), which progressively denoises the frame to produce an iteratively denoised frame 204 and a final output frame 206.
[0061] In some embodiments, such as Figure 1As shown, by encoding pixel-based images or embedding some cue frames and cue text, the inputs (noise frame 102, cue text 108, cue frame 110) are transformed into simplified representations. Encoding layer 106 reduces dimensionality and simplifies the representation of image frames in pixel space into a compressed vector-type representation. Images tend to contain a lot of redundant information because adjacent pixels in a frame often exhibit high similarity. For example, in many images, the background in one part of the image tends to share properties of the background in another part of the image (e.g., the sky is considered part of the background; although there are some differences in how the sky appears in the image, there are many similarities in the sky in most images, and even more similarities between pixels that are close to each other when viewed pixel by pixel). This redundant information is compounded in cross-frame video because each frame in the video is an image that changes only slightly from the previous and next frames.
[0062] The coding layer 106 can receive noise frames 102 and cue frames 110. The coding layer 106 processes noise frames 102 and cue frames 110 in a similar manner, except that cue frames 110 are not subject to added noise. Note that not all cue frames 110 are provided to the coding layer 106. Some cue frames 110 may be provided to the embedding layer 112.
[0063] Encoding layer 106 can be a variational autoencoder (VAE), a generative model that uses Bayesian inference principles to generate new data points. The variational autoencoder encodes data into a low-dimensional space and then decodes it back (block decoder layer 120). Encoding layer 106 processes the input visual media by compressing the frames of the visual media both spatially and temporally. When the visual media is video, frames are divided into groups of four adjacent frames and encoded together. This compression reduces the redundancy inherent in video data, where adjacent pixels and frames often exhibit high similarity. By reducing the number of pixels that the diffusion transformer needs to process, the visual encoder improves computational efficiency and facilitates the processing of high-resolution video. The image encoder simplifies the complexity of the data into a more manageable representation, facilitating a more efficient generation process in terms of computational resources and time.
[0064] Figure 3A and Figure 3B An example is shown where noisy frames 102 and cue frames 110 are encoded into a set of spatiotemporal blocks. Coding layer 106 encodes the frames into spatiotemporal blocks, which effectively divide the video into manageable cubic units containing both spatial (height and width) and temporal (time) dimensions. Although Figure 3A and Figure 3BThey are shown separately, but this is for illustrative purposes, and in most embodiments, the cue frame 110 will be combined with the noise frame 102 in sequence. Typically, the cue frame 110 will be arranged as the first or last frame, or at some other specific position indicated by the cue.
[0065] These spatiotemporal blocks from noisy frame 102 and any cue frames 110 are then flattened into a single vector by vector representation layer 104 to generate the basis for subsequent processing by diffusion-transformer layer 114. This lexicalization approach ensures that each block can reference and interact with all other blocks throughout the video, thereby promoting a high degree of consistency and continuity.
[0066] Figure 3A The diagram shows how noisy frames 102 (in groups of four) are converted into spatiotemporal boxes 302, and then vectors 304 representing the potential representation are generated. Figure 3B The diagram illustrates the potential representation 306 that converts cue frames 110 (up to four cue frames) into spatiotemporal facets 302 and generates input frames.
[0067] Figure 4 Vectors representing potential representations according to some embodiments of the present technology are shown in more detail. In particular, Figure 4 The latent representation vector 304 is shown, consisting of the latent representations of spatiotemporal block 402. The latent representation 402 of the spatiotemporal block represents the latent embedding of a frame (or a set of 4 consecutive frames) of the input video after it has been processed by the visual encoder. The latent representation has a spatial dimension (I×I) and C channels. This format comes from compressing the video into a compact latent space, reducing redundancy while preserving core visual and temporal information.
[0068] The latent representation 402 of the spatiotemporal block is divided into smaller non-overlapping blocks of size p×p. Each block corresponds to a portion of the spatial and temporal dimensions of the video. This block-based processing reduces the computational complexity of processing high-dimensional video data. The number of blocks generated from the latent representation is proportional to (I / p). 2 Proportional. This represents the number of blocks extracted from the spatial dimension of the latent representation.
[0069] After a frame or frame group has been processed by the block decoder layer 120, the latent representation 402 of the spatiotemporal block is flattened into a 1D sequence of T tokens by the vector representation layer 104, where T = (I / p). 2 This sequence can be processed by a transformer model. Each term in the sequence has a feature dimension d, which encodes the visual and temporal features of the corresponding block.
[0070] As described above, the prompt text 108 and some prompt frames 110 are transformed into embeddings by the embedding layer 112. The embedding layer 112 can be a multimodal model, such as contrastive language-image preprocessing (CLIP), or it can be multiple embedding models, such as an embedding model for images and an embedding model for text.
[0071] CLIP, developed by OpenAI, is a model that aligns text and visual embeddings using a contrastive learning-centric training framework. CLIP employs a dual-encoder architecture, consisting of a text encoder and an image encoder, trained simultaneously on a large dataset of image-text pairs. Each encoder generates embeddings in a shared feature space, allowing for the alignment of corresponding images and text based on semantic similarity. During training, the model is optimized to assign higher similarity scores to matching image-text pairs while minimizing the scores of mismatched pairs. In other words, CLIP creates similar embeddings for an image and the text describing that image. The embedding space created by CLIP is essentially a transformation space between text and image.
[0072] The embeddings created by embedding layer 112 and the spatiotemporal blocks created by coding layer 106 are fused by vector representation layer 104. The roles of coding layer 106 and embedding layer 112 are to create latent representations of the input. Vector representation layer 104 takes the latent representations and flattens them into one or more vectors, which are the latent representations of the visual media to be generated. Most of the vectors of the latent representations consist of representations from noisy frame 102, but the first or last representation (or some other specific frame) may be as follows: Figure 3B One of the latent representations of the input frame is depicted as 306. Furthermore, the representation of a certain interval in the vector of the latent representation will come from the cue embedding. The cue embedding can convey the intent of the cue to the diffusion-transformer layer. As mentioned above, some cue embeddings are derived from text cue, while others are derived from image cue.
[0073] The diffusion-transformer layer 114 receives vectors of the latent representation and processes frames across the entire visual medium simultaneously. Importantly, as will be described in more detail here, the diffusion-transformer layer 114 leverages knowledge of other frames in the visual medium to process the frames of the visual medium. This holistic approach allows for comprehensive spatial and temporal consistency across the generated video, thereby ensuring consistency and realism in motion and object interactions.
[0074] Linearization layer 118 receives the output from diffuse-transformer layer 114, which is the latent representation of the denoised frames, and rearranges them into a vector comprising a series of denoised frames. Block decoder layer 120 is essentially the reverse of coding layer 106, and it transforms the frames from their latent representations back into pixel representations. Recall that a group of four frames is compressed into a single representation. Therefore, block decoder layer 120 expands any representation into four frames represented in pixels.
[0075] Figure 5 Example processes according to some embodiments of the present technology are illustrated for generating visual media of varying durations, resolutions, and aspect ratios using a visual media generative response engine. Although the example processes depict operations in a specific order, this order can be changed without departing from the scope of this disclosure. For example, some of the described operations may be performed in parallel or in a different order that does not substantially affect the functionality of the process. In other examples, different components of the example device or system implementing the process may perform their functions substantially simultaneously or in a specific order.
[0076] While some operations are described as being performed by specific components or services, this is for illustrative purposes only, and it should be understood that references to specific components or services do not preclude the possibility that higher-level devices or services, or different devices or services, may perform the same functionality. The explicit intent is that if a function is performed by a service on a system, device, container, or virtual machine, it should be understood that the system, device, container, or virtual machine is performing that function as part of the execution of a service.
[0077] According to some examples, the method includes receiving input prompts in box 502 by a visual media generative response engine. For example, Figure 1 The input layer 122 shown can receive input prompts from a user interface or an application programming interface (API). In some embodiments, the input layer 122 may be part of the front end 124.
[0078] In some embodiments, the input cue is text. In some embodiments, the input cue includes visual media and text. In some embodiments, the input cue may also include duration, resolution, aspect ratio, style, and storyboard specifications.
[0079] As described herein, an advancement in the prior art provided by this technology is the ability to generate visual media with various durations, resolutions, and / or aspect ratios. In some embodiments, the duration, resolution, and / or aspect ratio of the visual media can be determined from natural language prompts provided in the front end 124. In some embodiments, the user interface or API can specify the desired duration, resolution, and aspect ratio 126.
[0080] According to some examples, the method includes providing at least one noise frame corresponding to the duration, resolution, and / or aspect ratio specification as input to the diffusion-transformer in box 504, in addition to the input cue. For example, in addition to the input cue, Figure 1 The input layer 122 shown may also provide at least one noise frame as input, corresponding to the duration, resolution, and / or aspect ratio specifications. As explained above, the diffusion model starts with a noise frame and gradually denoises the frames to the desired output. Therefore, providing at least one noise frame can be a method of controlling the resolution and / or aspect ratio of the generated video by arranging randomly initialized blocks in a grid of appropriate size to generate at least one noise frame. The duration of the visual media can be controlled by the number of noise frames provided. If a still image is the desired visual media, only one frame is needed, but if a 20-second video at 60 frames per second is desired, 1200 frames are needed.
[0081] As will be explained in more detail in this paper, the visual media generative response engine has been trained on native videos with various frame rates, aspect ratios, durations, and resolutions, and can therefore output visual media with various different attributes.
[0082] According to some examples, the method includes compressing at least one noisy frame into a corresponding representation in a lower-dimensional latent space in box 506. For example, Figure 1 The coding layer 106 shown can compress at least one noisy frame and one or more frames of optional cue video or image frames into a corresponding representation in a lower-dimensional latent space.
[0083] The cues also need to be converted into potential representations. Another innovation of this technology is that the visual media generative response engine 100 can generate visual media from cues that include both visual media and text. Another innovation of this technology is that the visual media generative response engine 100 can merge visual media input into output visual media. Figure 5 The remainder of the description will illustrate embodiments in which the hints may or may not include visual media, while Figure 6 The description will illustrate an embodiment in which visual media in the prompt is incorporated into the output visual media.
[0084] According to some examples, the method includes generating a cue embedding from the text and / or visual media of the input cue in box 508. For example, Figure 1 The embedding layer 112 shown can generate cue embeddings from the text and / or visual media of the input cue. In some embodiments, the embedding layer is a contrastive language-image preprocessing (CLIP) embedding layer. In some embodiments, the embedding layer may use a different embedding model for images and another embedding model for text.
[0085] Therefore, the output of embedding layer 112 is one or more embeddings derived from the cues. In a simple example, if the cues are text cues, embedding layer 112 might output a single embedding. Alternatively, if the cues are text cues that define an action or story to be represented by the video, the text cues might be broken down into several different text cues that describe the frames of the video that will be transformed by the action or story. In this example, embedding layer 112 will output several embeddings derived from several text cues. If the cues are to provide visual media as input, where the text cues visual media generative response engine uses the same style features depicted in the visual media to generate the video, the embedding layer might also output embeddings based on one or more frames of the visual media. (All of this implies that some logic occurs in input layer 122, which will refer to...) Figure 6 (To be described in more detail).
[0086] Now, the visual media generative response engine 100 has encoded the noisy frame 102 that will constitute the output of the visual media generative response engine 100, and has converted the cue into a cue embedding, which is a joint representation of the noisy frame 102 and the cue in a latent form, which needs to be combined into an input that can be efficiently processed by the diffusion-transformer layer 114. According to some examples, the method includes decomposing the corresponding representation in the lower-dimensional latent space into spatiotemporal blocks representing at least one frame in box 510. For example, Figure 1 The vector representation layer 104 shown can decompose the various representations in the lower-dimensional latent space into spatiotemporal blocks representing at least one frame. These spatiotemporal blocks are included in a spatiotemporal block vector representing frames in the output visual media. In some embodiments, such as when cues correspond to descriptions of frames appearing throughout the output video, these cues are scattered across noisy frames. In some embodiments, cue-based embeddings may be interleaved with noisy frames at predefined intervals (e.g., every 6 frames, every 10 frames, every 15 frames, every 20 frames, etc.). In the vector of spatiotemporal blocks, a first spatiotemporal block may consist of embeddings from cues, followed by several spatiotemporal blocks consisting of encoded noisy frames, followed by another embedding, followed by more noisy frames, and so on. This vector composed of spatiotemporal blocks... Figure 3A and Figure 4 The vector 304 is depicted as a potential representation.
[0087] According to some examples, the method includes embedding a processing spacetime block in box 512 based on prompts. For example, Figure 1The diffusion-transformer layer 114 shown can be processed into spatiotemporal blocks based on cues. In some embodiments, the diffusion-transformer layer includes multiple diffusion-transformer processing units, each processing a corresponding spatiotemporal block in parallel. This is also an innovative feature of this technology. Most transformer architectures process lemmas serially, but this technology processes all spatiotemporal blocks in parallel while predicting all lemmas. Since some of these lemmas are part of a spatiotemporal block derived from the cues, the diffusion-transformer layer 114 works to ensure consistency between spatiotemporal blocks, thereby affecting the output of the diffusion-transformer layer 114 to be consistent with the cues in visual media.
[0088] According to some examples, the method includes outputting visual media in box 514 in response to input prompts. For example, Figure 1 The visual media generative response engine 100 shown can output visual media in response to input prompts. (See reference...) Figure 1 As described herein, before the visual media can be output, the linearization layer 118 arranges the output from the diffusion-transformer processing unit back into a spatiotemporal block, which is now processed for denoising, representing the latent representation of the output visual media. Then, the block decoder layer 120 decodes the latent representation of the output visual media into pixel frames that constitute the conventional visual media.
[0089] Output visual media matching duration, resolution, and / or aspect ratio specifications. When the duration is zero seconds, and the visual media responding to the input prompt is a still image - a single frame.
[0090] Figure 6 Example processes illustrating additional functionality of the input layer according to some embodiments of the present technology are shown. Although the example processes depict operations in a specific order, this order can be changed without departing from the scope of this disclosure. For example, some of the described operations may be performed in parallel or in a different order that does not substantially affect the functionality of the process. In other examples, different components of the example device or system implementing the process may perform functions substantially simultaneously or in a specific order.
[0091] As described above, prompts can include visual media. Although Figure 5 An embodiment in which a prompt is input to the embedding layer 112 is described, but Figure 6 The addition of input layer 122 enhances the cue, and parts of the cue can be input into encoding layer 106 for encoding together with the noise frame 102.
[0092] According to some examples, the method includes determining the intent from the prompt in decision box 602. For example, Figure 1The input layer 122 shown can determine the intent from the cue. For example, the input layer 122 can determine whether the intent of the cue is to generate visual media that includes a frame or a portion of a frame provided in the cue. If the intent of the cue is to include a frame from the cue in the output visual media, the input layer 122 can provide that frame for encoding so that it can be provided to the diffusion-transformer layer 114, as described with reference to box 506. Alternatively, the input visual media cue can be provided to help express the vision of the visual artist providing the cue, and in this case, the frame from the input visual media cue does not need to be included in the output visual media. In this case, the visual media can be provided to the embedding layer 112, as described with reference to box 508.
[0093] Another aspect of determining the intent of the cues is determining whether the cues should be modified or enhanced. In some embodiments, the cues may not include much detail; however, when highly descriptive cues are provided, the diffusion-transformer layer 114 will provide better and more consistent results. Therefore, in some embodiments, at block 604, the input layer 122 may send the cues to the language model to generate more detailed cues. In some embodiments, the input cues may depict a blurred progression of the desired output visual media. In such embodiments, the input layer 122 may send the cues to the language model to generate a series of cues that act as a storyboard, describing specific frames that may appear in the progression suggested by the cues.
[0094] For example, an intent may be found in the prompt that includes at least one frame from the input visual media, which includes the visual media and text instructing the visual media generative response engine 100 to generate video by using a certain attribute of the input visual media or modifying a certain aspect of the visual media. Therefore, the input layer 122 will provide the image to the encoding layer 106 so that it is provided to the diffusion-transformer layer 114 along with noisy frames, as described with respect to box 506. The output from the diffusion-transformer layer 114 will be a video in response to the input prompt, which includes images, i.e., at least a portion of the images that the prompt wishes to retain, as part of the video in response to the input prompt. In a more specific example, the input visual media may include a picture of a toy truck, and the prompt may request to see the toy truck driving on a beach. The output will extract the toy truck from the input visual media and re-render the rest of the visual media while rendering additional frames.
[0095] In another example, the input cue may include a cue video as visual media, and text instructing the visual media generative response engine 100 to generate extended video from the cue video in a forward or backward temporal dimension. Therefore, the input layer 122 will provide all frames of the input video to the encoding layer 106, positioning the frames constituting the input video in the correct order relative to the noise frames. In other words, if the cue requests the output video to extend the input video, the input frames will be positioned first, and the noise frames that will be generated as new frames to extend the input video will be positioned after the input frames. The output from the spread-transformer layer 114 will be the video in response to the cue that has extended the input video with additional frames.
[0096] In another example, the input cue includes at least two cue visual media items, and the text instructs the visual media generative response engine 100 to create a merged video that merges from the first visual media item to the second visual media item to produce a merged video. Therefore, the input layer 122 will provide the encoding layer 106 with at least one frame of the first input visual media and at least one frame of the second visual media, and will position the frames constituting the first and second visual media in an appropriate order relative to the noise frames—in other words, the noise frames will be positioned between the frames from the first and second visual media and these frames will be separated, so that the diffusion-transformer layer 114 can generate a merged frame between the two input visual media.
[0097] Figure 7 Example processes for training a visual media generative response engine according to some embodiments of the present technology are illustrated. Although the example processes describe operations in a specific order, this order may be changed without departing from the scope of this disclosure. For example, some of the described operations may be performed in parallel or in a different order that does not substantially affect the functionality of the process. In other examples, different components of the example device or system implementing the process may perform functions substantially simultaneously or in a specific order.
[0098] Along with several innovations regarding the functionality and performance of the Visual Media Generative Response Engine 100, some of which are rooted in the training of the Visual Media Generative Response Engine 100, more specifically, the Visual Media Generative Response Engine 100 is trained on native visual media. Furthermore, this visual media includes visual media of various aspect ratios, resolutions, and lengths, which presents several technical challenges.
[0099] According to some examples, the method includes initially training a diffusion-transformer layer of the visual media generative response engine in box 702 by providing a training set of visual media. The visual media is native visual media, which includes visual media of varying durations, resolutions, and / or aspect ratios. Native visual media means that the visual media has not been edited for training the visual media generative response engine.
[0100] Most machine learning models require uniform input data. Therefore, previous attempts at image and video models provided input with uniform resolution, aspect ratio, and video length. However, this leads to distortion of the visual media. Whether scaling or cropping frames, the images within a frame can become distorted. Even truncated videos can warp the beginning and end of a concept depicted by a visual artist.
[0101] A key and desirable outcome of training on native visual media is that the visual media generative response engine learns accurate scaling and framing. Scaling and framing can become distorted when visual media is cropped and scaled to fit typical visual models. This technique avoids these consequences by training on native visual media.
[0102] Another important and desirable outcome of the design of visual media generative response engines (especially attention mechanisms) is their ability to learn how objects interact in the real world. In a sense, a visual media generative response engine is a visual learning model that can learn to simulate the physical interactions of objects in the physical world. Specifically, a visual media generative response engine can learn the causes and effects of object interactions and replicate them in the visual media it generates. For example, in a video of an artist painting, the model can display the interaction between the brush and the canvas during brush strokes, resulting in the applied paint and texture. The video can be created while preserving and building upon the results of the brush strokes. In this way, the model is able to permanently update the object state across the entire visual media.
[0103] The visual media generative response engine's ability to maintain 3D consistency and object persistence ensures that the generated videos present stable and realistic object interactions and movements. This consistency is essential for applications where the integrity of visual narratives and object states is crucial, such as in storytelling, education, and simulation training.
[0104] After this training on a large amount of native visual media, the visual media generative response engine can be fine-tuned.
[0105] According to some examples, the method includes providing at least one noisy frame and training cues describing the existing visual media in the fine-tuning training set in box 704.
[0106] According to some examples, the method includes selecting sample frames from visual media in the fine-tuning training set in box 706.
[0107] According to some examples, the method includes generating corresponding embedding frames from sample frames in box 708. The corresponding embedding frames are generated using the Contrastive Language-Image Pre-training (CLIP) technique. An important aspect of fine-tuning the visual media generative response engine is training on CLIP embeddings. Training on CLIP embeddings provides the diffusion-transformer layer with the ability to receive visual cues to guide the generation of visual media. For example, training on CLIP embeddings enables the visual media generative response engine to create visual media that can be transformed from one set of input frames to another, to re-represent the input frames, etc.
[0108] According to some examples, the method includes providing corresponding CLIP embedding frames interleaved between some noisy frames in box 710. CLIP embeddings from training cues can be interleaved in a certain percentage of training. For example, CLIP embeddings can be provided in 50% of the training samples. Furthermore, text embeddings can also be interleaved in a certain proportion of the training samples, thus also making the model sensitive to text input.
[0109] Finally, reinforcement learning can be used to train the model, thereby the method includes generating training prompts from the image-to-text captioning model in box 712, the image-to-text captioning model being trained to create detailed descriptions of existing visual media.
[0110] Training cues can be provided to the visual media generative response engine.
[0111] According to some examples, the method includes a diffusion-transformer layer in box 714 that rewards accurate prediction of frames constituting existing visual media. The diffusion-transformer layer of the visual media generative response engine receives multiple frames including noisy frames and corresponding embeddings interleaved with the noisy frames, whereby the diffusion-transformer layer learns to predict noisy frames consistent with the corresponding embeddings.
[0112] Figure 8 Example systems supporting generative response engines during inference operations according to some embodiments of the present technology are illustrated. While the example systems depict specific system components and their arrangement, this depiction is for the purpose of facilitating discussion of the present technology and should not be considered limiting unless specified in the appended claims. For example, some components illustrated as separate may be combined with other components, and some components may be divided into separate components.
[0113] A generative response engine 810 is artificial intelligence (AI) that can generate content in response to prompts. These prompts can come from a human or a software entity (AI or application). Prompts are typically in natural language, but can also be in code that includes binary code. Some examples of generative response engines can include language models for generating language, such as CHATGPT, or other models such as DALL-E for generating images and SORA for generating videos. CHATGPT, DALL-E, and SORA are all provided by OPENAI, but generative response engines are not limited to AI provided by OPENAI. A generative response engine can also be any type of generative AI and can include AI developed using various architectures, such as diffusion models and transformers (e.g., generative pre-trained transformers) and model combinations.
[0114] In some cases, a language model, such as CHATGPT, can receive hints of images, videos, code, applications, etc., that it can provide by interfacing with one or more other models, as will be further elaborated in this paper.
[0115] Users and applications can interact with the generative responsive engine 810 through the front end 802. The front end 802 acts as an interface and mediator between the user and the generative responsive engine. It includes a graphical user interface 804 and an application programming interface (API) 806 that facilitates communication, input processing, and output presentation. Typically, users interact through the graphical user interface 804, which usually includes a dialog interface, and applications interact through the API 806, but this is not required.
[0116] The graphical user interface 804 is the platform through which the user interacts with the generative response engine 810. It can be a web-based chat window, a mobile application, or any interface that supports data input and output. The graphical user interface 804 facilitates dialogue between the user and the generative response engine because the user provides prompts for the generative response engine to respond within the graphical user interface 804, and those responses are presented within the graphical user interface 804. In some embodiments, the graphical user interface 804 presents a dialog interface that has the attribute of a dialogue thread between the user account and the generative response engine 810.
[0117] The graphical user interface 804 is configured to perform input processing, context management, and output rendering. The types of input that can be received may be related to the details of the generative response engine 810. However, even when the model does not directly accept certain types of input, the front end 802 may be able to receive different types of input that can be transformed into inputs that are accepted by the generative response engine 810. For example, a language model is typically configured to accept text, but the front end 802 can accept speech and convert it into text, or accept an image and create a text representation.
[0118] The graphical user interface 804 is also configured to maintain the context of the dialogue, which allows for consistent and relevant responses. For example, the graphical user interface 804 is responsible for providing the generative response engine with the dialogue thread and other relevant context accessible to the front-end 802, as well as specific prompts to the generative response engine. In one example, the dialogue between the user account and the generative response engine 810 may have gone through several rounds (prompt, response, prompt, response, etc.). When the user account provides further prompts, the graphical user interface 804 can provide those prompts to the generative response engine within the context of the entire dialogue.
[0119] In another example, frontend 802 can access memory 826 that already stores facts about the user's account. In some embodiments, these facts may have been identified as worthy of storage by the generative response engine, and frontend 802 has already stored these facts under the guidance of the generative response engine. Therefore, these facts can be provided to the generative response engine 810 along with user-provided prompts, allowing the generative response engine to access these facts when generating a response.
[0120] In another example, the graphical user interface 804 can be configured to provide system prompts along with user-provided hints. System prompts are hidden from the user account and are used to set the behavior and guidelines of the generative responsive engine. It can be used to define the AI's roles, styles, and constraints.
[0121] The graphical user interface 804 is also configured to display responses from the generative response engine, which may include text, code snippets, images, or interactive elements.
[0122] In some embodiments, the generative responsive engine 810 may provide instructions to the front end 802, directing the graphical user interface 804 on how to display some of the output from the generative responsive engine. For example, the generative responsive engine may instruct the graphical user interface 804 to render code in a code-specific format, or to render interactive graphics or static images. In other examples, the generative responsive engine may instruct the graphical user interface 804 to render an interactive document editor, whereby the graphical user interface 804 may be presented alongside the document editor, allowing the user account and the generative responsive engine to collaborate on documents. In some embodiments, the generative responsive engine 810 may provide instructions to the front end 802 to record facts in a personalized notepad. Therefore, the graphical user interface 804 does not always display all the output of the generative responsive engine.
[0123] As mentioned above, the front-end 802 can also provide one or more application programming interfaces (APIs) 806. APIs allow developers to integrate the capabilities of the generative responsive engine into external applications and services. They provide programmatic access to the generative responsive engine, allowing for customized interactions and functionalities.
[0124] API 806 can accept structured requests that include prompts, context, and configuration parameters. For example, the API can be used to provide prompts, categorizing them as system prompts and user prompts. In some embodiments, API 806 can provide specific inputs, and the generative response engine 810 is configured to respond to these inputs with specific behaviors. For example, the API can be used to specify that it requires output in a specific format or structured output. For example, in a chat completion API, the API call can specify parameters for the output, such as the maximum expected length of the output, and specify the tone of the language used in the response. Some common APIs are used for engaging in conversations (chat completion API), for providing single responses (completion API), for converting text into embeddings (embedding API), and so on. APIs can also be used to indicate specific decision boundaries that the generative response engine 810 can be trained to interpret. For example, a moderation API can leverage the generative response engine to make content moderation decisions. In the case of moderation APIs and other APIs, the API may grant access to services other than the generative response engine. For example, a moderation API could be an interface to a moderation system 838 as described below.
[0125] Other common APIs include: the Fine-Tuning API, which allows developers to customize the Generative Response Engine model using their own datasets; the Audio and Speech API, which enables the Generative Response Engine to output speech or audio; and the Image Generation API, which enables the Generative Response Engine to output images (this may require leveraging other models).
[0126] There can also be APIs that guide the generative response engine to interface with other applications or other generative AI engines. In this case, a specific application or AI engine can be specified, or the generative response engine can be allowed to choose another application or AI engine to respond to prompts.
[0127] In short, the graphical user interface 804 and API 806 can be used to provide prompts to the generative response engine. Prompts are sometimes categorized into different types. For example, system prompts can be hidden prompts that set behaviors and guidelines for the generative response engine. User prompts are explicit inputs provided by the user, which may include questions, commands, or information.
[0128] The system architecture server 820 is located between the front end 802 and the generative response engine 810. The function of the system architecture server 820 is to manage and organize the data flow between key subsystems, enabling the generative response engine 810 to generate context-sensitive, accurate responses that are enriched with additional information when needed.
[0129] Action 822 facilitates auxiliary tasks beyond basic text generation. In some embodiments, action 822 may be an action corresponding to API 806. In some embodiments, action 822 may be a proxy action that the generative response engine 810 decides to take to perform the user's intent as described in the prompt.
[0130] Prompt 824 is a request or command provided by a user account through frontend 802. In some embodiments, prompt 824 may be further supplemented by system prompts and other information that may be included by graphical user interface 804 or API 806. In some embodiments, prompt 824 may even be modified or enhanced by generative response engine 810, as further described below. Additionally, a conversation thread is formed when a user account provides a prompt and generative response engine 810 provides a response. When a user account provides a new prompt, this is appended to the entire conversation and added to prompt 824. Thus, a user account may consider a first user-provided message as a first prompt and a second user-provided message as a second prompt, and so on, but prompt 824 perceived by generative response engine 810 may include threads of user-provided messages and responses from generative response engine 810 in multi-turn conversations. Generally, prompt 824 will include the entire conversation thread, but in some cases, prompt 824 may need to be shortened if it exceeds the maximum acceptable length (typically measured by the number of tokens).
[0131] The system architecture server 820 can also route prompts and responses through an auditing system 838, which may be part of or separate from the system architecture server 820. In some embodiments, prompts are provided to a prompt security system 834 before being provided to the generative response engine 810. The prompt security system 834 is configured to evaluate prompts using one or more techniques to ensure that no audited content is generated by the request generative response engine 810. In some embodiments, the prompt security system 834 may utilize text pattern matching, classifiers, and / or other AI techniques.
[0132] Because cues can evolve over time in a dialogue consisting of cues and responses, cues can be evaluated repeatedly in each round of the dialogue.
[0133] Memory 826 can facilitate continuity and personalization of the conversation. It allows the system to maintain user-specific context, preferences, or details that may inform future interactions. The memory file can be persistent data from previous interactions or sessions, providing contextual information to maintain continuity. In some embodiments, memory can be recorded at the instruction of the generative response engine 810 when it identifies facts or data that it determines should be saved in memory because it may be useful in later conversations or sessions.
[0134] Dialogue metadata 828 can aggregate dialogue-related data points, including user prompts 824, actions 822, and memory 826. This aggregated information package serves as input to the generative response engine 810. Dialogue metadata 828 can tag portions of the prompts as user-provided, provided by the generative response engine, system prompts, memory 826, data from actions 822, or data from tools 830 (described below).
[0135] The generative response engine is the core engine that processes the input (from the system architecture server 820) and generates the output. In some embodiments, the generative response engine is a generative pre-trained transformer (GPT), but it may utilize other architectures.
[0136] The core feature of the generative response engine 810 is the generation of content in response to prompts. When the generative response engine 810 is a GPT (Generative Perceptions Test), it is configured to receive input from front-end 802, which provides guidance on the desired output. The generative response engine can analyze the input and identify relevant patterns and associations in the data, and it has learned to generate the most likely continuation sequence of lexical terms predicted as the input. Guided by the patterns observed during its training, the generative response engine 810 generates responses by sampling from the probability distribution of possible lexical terms. In some embodiments, the generative response engine 810 can generate multiple possible responses before presenting the final response. The generative response engine 810 can generate multiple responses based on the input, and these responses are variants that the generative response engine 810 considers potentially relevant and consistent.
[0137] In some embodiments, the generative response engine 810 can evaluate the generated response based on certain criteria. These criteria may include relevance to the prompt, consistency, fluency, and sometimes adherence to specific guidelines or rules, depending on the application. Based on this evaluation, the generative response engine 810 can select the most suitable response. This selection is typically based on the highest-scoring response that balances relevance, informativeness, consistency, and content moderation guidelines / training on the set criteria.
[0138] In some embodiments, instructions provided by API 806, system prompts, or decisions made by the generative response engine 810 may cause the generative response engine 810 to interpret and rewrite the prompt or improve the prompt for a desired purpose. For example, the generative response engine 810 may determine that a prompt to create an image is acceptable and enhance the prompt to produce a better image. In these cases, the generative response engine 810 may generate its own prompt, which may be provided to the tool 830 or to the generative response engine 810 to produce a better output response than the original prompt might have.
[0139] The generative response engine 810 can do more than simply generate content in response to a prompt. In some embodiments, the generative response engine 810 can utilize decision boundaries to determine the appropriate course of action based on the prompt. In some examples, the decision boundary can be used to make the generative response engine realize that it is being asked to provide a response in a specific format, causing it to generate its response constrained by that specific format. In some examples, the decision boundary can cause the model to refuse to generate a response output if the decision is that the response output would violate an censorship policy. In some examples, the decision boundary might make the generative response engine realize that it needs to interact with another AI model or application to respond to the prompt. For example, when the generative response engine is a language model, it might realize that it is being asked to output an image, and therefore, it needs to interact with a model capable of outputting images to provide a response to the prompt. In another example, the prompt might require searching the Internet before responding. The generative response engine can use the decision boundary to realize that it should perform an Internet search and use the results of that search to respond to the prompt. In another example, the prompt might request the generative response engine to take proxy actions on behalf of the user by interacting with a third-party service (e.g., help me book...), and the generative response engine can use decision boundaries to recognize that it needs to plan steps to locate the third-party service, contact the third-party service, and interact with the third-party service to complete the task, and then report to the user that the action has been completed.
[0140] When the generative response engine 810 determines that it should take an agent action on behalf of the user or that it should invoke a tool to help provide a quality response to the user account, the generative response engine 810 may invoke tool 830 or cause action 822 to be executed. As mentioned above, tool 830 may include an internet browser, an editor such as a code editor, other AI tools, etc. Action 822 is an action that the generative response engine 810 may use tool 830 to cause it to be executed. As used herein, action 822 should be considered to cover a broad array of actions that the generative response engine 810 may perform with or without tool 830. Tool 830 is considered to cover a wide variety of services and software, including tools such as computer operating systems, enabling the generative response engine 810 to control computer operating systems, robot actuators, search browsers, and specific applications on behalf of the user.
[0141] Additionally, the generative response engine 810 can also generate portions of the response that are not displayed to the user. For example, the generative response engine 810 can instruct the front-end 802 to provide specific behaviors, such as instructions on how to present the response from the generative response engine 810 to the user's account. In another example, the generative response engine 810 can provide a portion of the response defined by an API, where the portion of the response to the API can be consumed by the application invoking it, but is not presented to the end user.
[0142] In some embodiments, the output of the generative response engine can be further analyzed by the output security system 836. While the generative response engine 810 can perform some of its own audits, there may be situations where additional service review output is expected to comply with audit policies. Figure 8 The use of the dashed line distinguishes between paths that use the output security system 836 and those that do not.
[0143] Although Figure 8 The example shows that the response is provided directly back to frontend 802, but in some embodiments, the response may be returned via system architecture server 820.
[0144] Figure 9A , Figure 9B and Figure 9C Example transformer architectures according to some embodiments of the present technology are shown. Examples of ML models using transformer neural networks (e.g., transformer architecture 900) may include, for example, generative pre-trained transformer (GPT) models and bidirectional encoder representation (BERT) models from transformers. Figure 9A , Figure 9B and Figure 9CThe converter architecture 900 shown includes an input 902, an input embedding block 904, a position encoder 906, an encoder 908 including an encoding block 910, a decoder 912 including a decoding block 914, a linear block 916, a softmax block 918, and an output probability 920.
[0145] Input embedding block 904 is used to provide a representation of a word. For example, embeddings can be used in text analysis. According to some non-limiting examples, this representation is a real-valued vector that encodes the meaning of a word in such a way that words that are closer in the vector space are expected to be similar in meaning. Word embeddings can be obtained using language modeling and feature learning techniques, where words or phrases from a vocabulary are mapped to real-valued vectors. According to some non-limiting examples, input embedding block 904 can be a learned embedding to transform input and output lexical units into dimensional vectors with the same dimensions as, for example, positional encodings.
[0146] Positional encoding 906 provides information about the relative or absolute position of words in a sequence. According to some non-limiting examples, positional encoding 906 can be provided by adding positional encoding to the input embedding at the inputs of encoder 908 and decoder 912. Positional encoding has the same dimension as the embedding, making it possible to sum the embedding with the positional encoding. Several methods exist for implementing positional encoding, including learned encoding and fixed encoding. For example, sine and cosine functions with different frequencies can be used. That is, each dimension of the positional encoding corresponds to a sine curve. As those skilled in the art will understand, other techniques that convey positional information can also be used. For example, a similar result can be obtained by using learned positional embeddings instead. One advantage of using sine curve positional encoding instead of learned positional encoding is that doing so allows the model to extrapolate to sequence lengths longer than those encountered during training.
[0147] Encoder 908 can use stacked self-focused and point-by-point fully connected layers. Encoder 908 can be a stack of N identical layers (e.g., N=6), and each layer can be a coding block, as described by... Figure 9B The encoder block 910 is shown in the diagram. Each encoder block 910 has two sub-layers: (i) a first sub-layer with a multi-head concern block 922, and (ii) a second sub-layer with a feedforward block 926, which can be a position-by-position fully connected feedforward network. The feedforward block 926 can use a modified linear unit (ReLU).
[0148] Encoder 908 uses residual connections around each of the two sublayers, followed by normalized addition and norm blocks 924. For example, the output of each sublayer could be LayerNorm(x+Sublayer(x)). To facilitate these residual connections, all sublayers in the model, as well as the embedding layers, produce output data with the same dimensions.
[0149] Similar to encoder 908, decoder 912 uses stacked self-focused and pointwise fully connected layers. Decoder 912 can also be a stack of M identical layers (e.g., M=6), and each layer can be a decoding block, as described by... Figure 9B The decoding block 912 is shown in the diagram. In addition to the two sub-layers found in the encoding block 910 (i.e., sub-layer 922 with a multi-head attention block and sub-layer 926 with a feedforward block), the decoding block 914 may also include a third sub-layer that performs multi-head attention on the output of the encoder stack. Similar to the encoder 908, the decoder 912 uses residual connections around each sub-layer, followed by layer normalization. Additionally, the sub-layer 922 with the multi-head attention block can be modified in the decoder stack to prevent positional attention to subsequent positions. This masking, combined with the fact that the output embedding is offset by one position, ensures that the prediction of position i can depend only on known output data at positions less than i. Linear block 916 can learn linear transformations. For example, when transformer architecture 900 is being used to translate from a first language to a second language, linear block 916 can project the output from the last decoded softmax block 918 onto the word scores of the second language (e.g., the score value of each unique word in the target vocabulary) at each position in the sentence. For example, if the output sentence has seven words and the vocabulary provided for the second language has 10,000 unique words, then 10,000 score values are generated for each of those seven words. The score value indicates the probability that each word in the vocabulary will appear at that position in the sentence.
[0150] Then, the Softmax block 918 transforms the scores from the linear block 916 into output probabilities 920 (which sum to 1.0). At each position, an index provides the word with the highest probability, and this index is then mapped to the corresponding word in the vocabulary. These words then form the output sequence of the transformer architecture 900. The softmax operation is applied to the output from the linear block 916 to transform the raw numbers into output probabilities 920 (e.g., lexical probabilities).
[0151] Therefore, so far, this description has focused on the innovations relevant to the visual media generative response engine itself. As a novel generative tool, the visual media generative response engine is further accompanied by several innovative interface features. The following description turns to the features of the interface used to provide input and edit the output from the visual media generative response engine. While the discussion below illustrates the use of these interfaces with the visual media generative response engine described herein, it is explicitly anticipated that these interfaces will have utility with other visual media creation tools, and therefore, unless explicitly stated otherwise, the statements made herein should not be construed as limiting the interfaces described herein to use only with the visual media generative response engine described herein.
[0152] Figure 10 Example processes for presenting and using a graphical user interface to create prompts for generating visual media, according to some embodiments of the present technology, are illustrated. Although the example processes depict operations in a specific order, this order can be changed without departing from the scope of this disclosure. For example, some of the described operations may be performed in parallel or in a different order that does not substantially affect the functionality of the process. In other examples, different components of the example device or system implementing the process may perform functions substantially simultaneously or in a specific order.
[0153] According to some examples, the method includes presenting a graphical user interface, including a prompt editor, in box 1002. For example, Figure 1 The front end 124 shown can present a graphical user interface including a prompt editor. Figure 11 An example prompt editor 1102 is shown. The prompt editor 1102 can at least receive text input, and in some embodiments, it can also receive one or more visual media as input. In some embodiments, the prompt editor 1102 can also receive input for specifying the resolution, aspect ratio, and / or length of the expected visual media. In some embodiments, the prompt editor 1102 can also receive time-based or frame-based input to provide detailed control over visual media comprising a temporally sequenced frame.
[0154] According to some examples, the method includes receiving at least one text input from a prompting editor in box 1004. For example, Figure 1 The front-end 124 shown can receive at least one text input from the prompt editor 1102. The text input is a prompt describing the visual media to be generated by the visual media generative response engine.
[0155] In some embodiments, the prompt editor 1102 includes a visual media upload button 1104, which allows input visual media to be included as part of the prompt. Visual media can also be added to the prompt by selecting visual media from the gallery, as shown in the reference. Figure 16 A more detailed explanation.
[0156] According to some examples, the method includes receiving a command in box 1006 to generate visual media using at least one text input to a prompt editor. For example, Figure 1 The front-end 124 shown can receive commands to generate visual media using at least one text input from a prompt editor. This can be achieved by selecting... Figure 11 The generate button 1106 shown is used to receive the command.
[0157] If reference is at least Figure 11 , Figures 19A-19D , Figures 24A-24B and Figures 26A-26B In more detail, in addition to the cue text and / or visual media, other details can be provided as part of the cue to the visual media generative response engine 100.
[0158] According to some examples, the method includes determining at least one of the aspect ratio or resolution used in the generated visual media in box 1008. For example, Figure 1 The front-end 124 shown can determine at least one of the aspect ratio or resolution of the generated visual media. As illustrated herein, the visual media generative response engine is capable of generating visual media at multiple resolutions and / or aspect ratios. The resolution, aspect ratio, and duration of the output visual media are controlled by attributes of noise frames provided to the diffusion-transformer layer of the visual media generative response engine. The diffusion-transformer layer denoises the noise frames to output frames with the same aspect ratio and resolution as the input noise frames. The length of the video is controlled by the number of noise frames and the frame rate of the displayed output frames. Therefore, the front-end 124 can determine the aspect ratio, resolution, and length based on prompts using deterministic or artificial intelligence algorithms, or the user can explicitly indicate the desired aspect ratio, resolution, and / or length.
[0159] Figure 11 Examples of controls are shown, through which users can provide explicit input for style, aspect ratio, resolution, and / or length using style control 1108, aspect ratio input control 1110, resolution input control 1112, and / or duration input control 1114.
[0160] Style control 1108 allows users to select a visual style and apply it to the visual media to be generated.
[0161] The aspect ratio input control 1110 allows users to specify the aspect ratio of an image or video frame. Some common aspect ratios include 4:3 (traditionally used in older television broadcasts and computer monitors), 16:9 (widely used in HDTV, modern video content, and computer monitors as the standard widescreen format), 1.85:1 (typically used for theatrical films, providing a slightly wider field of view than 16:9), 2.39:1 or 2.35:1 (typically used for widescreen movies, also known as Cinemascope or Anamorphic format), 1:1 (frequently used on social media platforms such as Instagram for square images and videos), and 21:9 (appearing on ultrawide monitors and in some movie presentations, providing an immersive viewing experience).
[0162] The resolution input control 1112 allows users to set their desired pixel dimensions, which define the sharpness and detail of an image or video. Common display resolutions include a range of pixel dimensions to meet the needs of various devices and users. Starting with lower resolutions, 640x480 pixels, known as VGA, was once the standard for early computer monitors. Moving up, 1024x768 pixels represents XGA resolution, commonly used in older laptops and projectors. In the high-definition realm, 1280x720 pixels defines HD or 720p, primarily used in early HDTVs. Full HD, or 1080p, is characterized by 1920x1080 pixels and is prevalent in most modern televisions, computer monitors, and streaming services. Even for even higher sharpness, 2560x1440 pixels, or QHD, offers a sharper experience and is typically found in gaming monitors. 4K resolution, offering 3840x2160 pixels, is becoming increasingly common in UHD televisions and high-end monitors. Finally, the 8K resolution of 7680x4320 pixels provides extremely high detail for cutting-edge displays, although it is still emerging in the consumer market. Higher resolutions are expected to be utilized in the future and will be compatible with current technologies.
[0163] Additionally, the duration input control 1114 allows users to define the duration or length of time for which visual content should be presented. This length can be as short as zero seconds, which can result in a single frame as visual media output, or the length can be adjusted as needed. However, longer lengths may require longer processing times or resource scheduling to ensure that enough diffusion-transformer processing units are available to process the number of frames required to generate the visual media. Currently, videos up to one minute in length can be easily adapted using the techniques described in this paper with universally available cloud resources (GPUs in data centers). Longer videos are also possible by scheduling and scaling the number of GPUs. Very long videos, such as feature-length movies, may require additional optimizations, such as creating videos in segments and compressing frames into batches of 8 or 16 frames instead of the 4 frames described above.
[0164] In some embodiments, the user interface may also include a model selector 1120, which can be used to select a visual media generative response engine to be used to generate visual media. Some models may be smaller and faster, while others may be larger and take longer to process, but may output higher quality visual media.
[0165] In some embodiments, the graphical user interface may include an algebraic input control 1116, which can be used to enable the visual media generative response engine to generate more than one version of the output visual media. This allows the user to select from several candidate visual media that all correspond to the prompt.
[0166] According to some examples, the method includes sending at least the text input, aspect ratio, and resolution used by the generated visual media to the visual media generative response engine in box 1010. For example, Figure 1 The front end 124 shown can send at least the text input, aspect ratio, and resolution used to generate the visual media to the visual media generative response engine.
[0167] Therefore, although so far Figure 10 The graphical user interface that can be configured to provide prompts and send them to a visual media generative responsive engine has already been described, but... Figure 10 An embodiment is also described in which a third-party application can provide a graphical user interface, and the sending of the prompt at box 1010 is provided by an application programming interface (API).
[0168] According to some examples, the method includes receiving visual media generated based on cues in box 1012. For example, Figure 1 The front end 124 shown can receive visual media generated based on prompts. The visual media is generated according to the aspect ratio or resolution provided with the prompts. In some embodiments, the front end 124 may include a viewer or player that can display the visual media.
[0169] Figure 11 An example library interface 1100 according to some embodiments of the present technology is shown. Although Figure 11 A specific user interface is shown, but this technology should not be considered limited to use with such an interface. Instead, Figure 11 The user interface shown is provided to illustrate example options and features offered by this technology.
[0170] The library interface 1100 displays a library of visual media saved by a user account. Visual media can be created by a user account, or created by another user account and saved by that user account. As will be explained herein, any visual media in a user account's library can be used to create new visual media.
[0171] The library interface 1100 also includes an explore tab 1118 user interface object that points to a public library of previously generated visual media. Although not shown, the public library can be a rotating selection of visual media created by other user accounts. These visual media can provide inspiration and a starting point for the creation of new visual media.
[0172] As described above, the library interface 1100 also includes a tooltip editor 1102 for creating and adjusting tooltips used when generating visual media using the visual media generative response engine 100.
[0173] Figure 12 A resolution input control menu according to some embodiments of the present technology is shown. For example... Figure 12 As shown, the resolution input control 1112 includes options to select from 360p, 480p, 720p, and 1080p resolutions. While these are the only resolutions listed in the resolution input control 1112, this technology is capable of providing other resolutions, although the processing time will be adjusted proportionally. The user interface also informs the user that higher resolutions will increase the processing time for visual media generation.
[0174] Figure 13 A menu for aspect ratio input control according to some embodiments of the present technology is shown. For example... Figure 13 As shown, the aspect ratio input control 1110 includes options to select from aspect ratios of 16:9, 1:1, and 9:16. While these are the only aspect ratios listed in the aspect ratio input control 1110, the present invention is capable of providing other aspect ratios when provided with noisy frames of the same aspect ratio.
[0175] Figure 14A and Figure 14B The style control menu and configuration interface according to some embodiments of the present technology are shown.
[0176] Figure 14A A style control menu 1108 is shown, allowing selection from preset style options. In some embodiments, the preset style options are pre-configured as part of the user interface. In some embodiments, the preset style options are those previously configured by the user account.
[0177] Figure 14B The style configuration interface is shown. For example, Figure 14BThe description of the preset style is shown. Following the preset style is text describing that style. In some embodiments, the style acts as a system prompt, as the user does not see the text describing the style added to their prompt in the prompt editor or storyboard user interface; however, the text describing the style is added by the front end before the prompt is sent to the next layer of the visual media generative responsive engine.
[0178] Figure 15 An algebraic input control according to some embodiments of the present technology is shown. For example... Figure 15 As shown, the algebraic input control 1116 includes an option to generate 1, 2, or 4 visual media variations in response to a prompt. This allows the user to select from several candidate visual media that all correspond to the prompt.
[0179] Figure 16 Example processes for creating new visual media from existing visual media cues, according to some embodiments of the present technology, are illustrated. Although the example processes depict operations in a specific order, this order can be changed without departing from the scope of this disclosure. For example, some of the described operations may be performed in parallel or in a different order that does not substantially affect the functionality of the process. In other examples, different components of the example device or system implementing the process may perform their functions substantially simultaneously or in a specific order.
[0180] According to some examples, the method includes receiving a selection of existing visual media previously generated using a visual media generative response engine in box 1612. For example, by Figure 1 The user interface provided by the front-end 124 shown can accept selections of existing visual media previously generated using a visual media generative response engine. Existing visual media can be displayed in public or private galleries.
[0181] According to some examples, the method includes, in box 1614, in response to the selection, presenting the existing visual media selected via a user interface. For example, by Figure 1 The user interface provided by the front end 124 shown can respond to the selection by presenting the existing visual media of the selection via the user interface.
[0182] For example, Figure 17A A frame of existing visual media 1702 selected from a public gallery is shown. A prompt for existing visual media 1704 is presented to the selected existing visual media 1702. The prompt 1704 for existing visual media is a prompt for creating visual media, which is displayed in response to the selection of the display prompt 1706 option for visual media. In some embodiments, the user can select to display the prompt for this frame for visual media 1708, which causes the user interface to display a machine-generated description of the specific frame of the visual media.
[0183] According to some examples, the method includes receiving edits to the example prompt in box 1616. For example, by Figure 1 The user interface provided by the front end 124 shown can receive edits to the example cues. Whether the cues are for visual media or for frames of visual media, they can be used to create new visual media by adapting or refining the cues.
[0184] According to some examples, the method includes receiving instructions in box 1618 to generate new visual media based on edits to example prompts. For example, by Figure 1 The user interface provided by the front end 124 shown can receive instructions to generate new visual media based on editing of example prompts.
[0185] According to some examples, the method includes in box 1620, copying the required input details from existing visual media that are not provided through user interaction with the user interface. For example, by Figure 1 The user interface provided by the front end 124 shown can replicate the required input details from existing visual media that are not provided through user interaction with the user interface. In some cases, the user may omit specifying the expected output resolution, duration, aspect ratio, etc. In such cases, this technique can import any input details not specified by the user from the visual media that inspired the new visual media to be generated.
[0186] According to some examples, the method includes presenting a storyboard user interface in box 1622, pre-filled with text prompts and / or one or more frames describing existing visual media. For example, by Figure 1 The user interface provided by the front end 124 shown can present a storyboard user interface pre-filled with text prompts and / or one or more frames describing existing visual media. Figure 17A A display storyboard is shown for the Visual Media 1710 option, which can lead to the display of a storyboard that facilitates greater control over video generation. The storyboard user interface will at least reference... Figures 19A-19D To elaborate further.
[0187] According to some examples, the method includes using a visual media generative responsive engine in box 1624 to generate new visual media based on edits to the example prompt. For example, Figure 1 The visual media generative response engine 100 shown can generate new visual media based on editing of the example prompts.
[0188] According to some examples, the method includes presenting new visual media generated using a visual media generative responsive engine in box 1626. For example, by Figure 1The user interface provided by the front end 124 shown can present new visual media generated using a visual media generative response engine.
[0189] Figure 17A Example frames from previously created visual media are shown, illustrating some embodiments of the present technology. Although Figure 17A A specific user interface is shown, but this technology should not be considered limited to use with such an interface. Instead, Figure 17A The user interface shown is provided to illustrate example options and features offered by this technology.
[0190] In particular, Figure 17A Existing visual media 1702 is shown, along with a cue to existing visual media 1704, which is used to create a video in response to the cue.
[0191] As explained above, Figure 17A It also displays options for displaying visual media (1706), displaying visual media (1708), and displaying visual media (1710). Figure 17A It also provides Figure 17B The editing options described in section 1712.
[0192] Figure 17B Editing options according to some embodiments of this technology are shown. Figure 17B yes Figure 17A A closer view of the editing option 1712 shown.
[0193] One of the editing options 1712 is the storyboard user interface button 1714, which is used to launch the storyboard user interface. (See also...) Figure 18 , Figure 19A - Figure 19D , Figure 20C and Figure 21 A more detailed explanation of the storyboard user interface.
[0194] Another option in editing option 1712 is the clip frame interface button 1716, which is used to launch the interface to trim frames from the generated visual media. This interface... Figure 29 As shown in the image.
[0195] Another option in editing option 1712 is the remix interface button 1718, which is used to change the editing visual media. This interface is... Figure 24A and Figure 24B As shown in the image.
[0196] Another option in editing option 1712 is the merge interface button 1720, which is used to merge the first provided visual media with another visual media or text prompt. This interface is... Figure 26A and Figure 26BAs shown in the image.
[0197] Another option in editing 1712 is the loop interface button 1722, which is used to loop visual media, so that the visual media can be played continuously without the viewer needing to notice the beginning or end of the visual media. This interface is... Figure 28 As shown in the image.
[0198] Another option in editing option 1712 is the upgrade resolution button 1724, which can be used to upgrade visual media to a higher resolution. This can be useful when visual artists want to generate visual media more quickly from the visual media generative response engine, allowing them to iteratively generate draft visual media and only generate high-resolution versions when the visual artist has already achieved a lower-resolution version of their artistic vision.
[0199] Figure 18 Example processes according to some embodiments of the present technology are illustrated for assisting users in generating prompts for a visual media generative response engine through a storyboard user interface. Although the example processes depict operations in a specific order, this order can be changed without departing from the scope of this disclosure. For example, some of the described operations may be performed in parallel or in a different order that does not substantially affect the functionality of the process. In other examples, different components of the example device or system implementing the process may perform functions substantially simultaneously or in a specific order.
[0200] Because videos consist of a series of frames displayed over time, their creation typically involves planning transitions between frames or scenes so that they depict a visual story as they are played in sequence. In some embodiments, a user can clearly express the story in a single long cue, but when the visual media generative response engine is given a single text cue, the timing of various story elements and transitions becomes somewhat unpredictable. Therefore, this technology provides a storyboard user interface that allows users to provide visual or text cues for frames that appear at specific points in time within the video. In this way, visual artists can exert additional control over the progress of the visual media to be generated by the visual media generative response engine 100.
[0201] According to some examples, the method includes presenting a storyboard user interface in box 1802. For example, Figure 1 The front end 124 shown can display a storyboard user interface. The storyboard user interface includes a visual media timeline and a representation of the first frame located on the visual media timeline. Figure 19A - Figure 19D An example of a storyboard user interface 1900 is shown. Figure 19AAs shown, the storyboard user interface 1900 includes a time scale 1908 representing the duration of the video to be generated. One or more frames can be added to the time scale 1908, for example... Figure 19A The first frame in the video is 1906.
[0202] According to some examples, the method includes presenting the option to add a frame in box 1804 within the storyboard user interface. For example, Figure 1 The front end 124 shown can present the Add Frame option in the storyboard user interface. The Add Frame button can be used to display a representation of the second frame on the visual media timeline. Figure 19A The option to add a frame 1918 is shown, which is used to add a second frame 1914, as shown. Figure 19B As shown in the image.
[0203] According to some examples, the method includes receiving a prompt to be associated with a frame in box 1806. For example, Figure 1 The front end 124 shown can receive prompts to be associated with a frame. The prompts may include text or visual media. Figure 19A An example of the prompt editor 1912 is shown in the first frame 1906. A complete description of the prompts for Visual Media 2006 is provided below. Figure 20A and Figure 20C As shown in the image.
[0204] In some embodiments, several frames can be added and associated with corresponding prompts. For example, Figure 20C The example shows a first frame 1906 with a cue describing visual media 2006 and a second frame 1914 with a cue describing existing visual media 2010. Although the example shows a text cue, the cue can also be a visual cue. Figure 21 An example of the third frame 1916, including visual cues, is shown.
[0205] According to some examples, the method includes receiving a move frame command in box 1808. For example, Figure 1 The front end 124 shown can receive a frame move command. In some embodiments, a user can operate a user interface device to drag a frame to a new position on time scale 1908. In response to the frame move command, the method includes moving a frame initially located at a first timestamp to a second timestamp in box 1810.
[0206] According to some examples, the method includes receiving a command in box 1812 to generate output visual media by a visual media generative response engine. For example, Figure 1The visual media generative response engine 100 shown can receive commands to generate output visual media. A cue associated with at least a first frame, along with a first timestamp, is provided to the visual media generative response engine, indicating that the cue describes the expected output first frame at that timestamp. Cues associated with second and third frames, etc., can also be provided to the visual media generative response engine 100 along with their respective timestamps.
[0207] According to some examples, the method includes receiving output visual media with frames in box 1814, the frames corresponding at least to a cue at a first timestamp. For example, Figure 1 The front end 124 shown can receive output visual media with frames corresponding at least to a cue at a first timestamp, and, if provided, to cue at a second timestamp and a third timestamp, and so on. As described above, this technique can utilize embeddings to represent text or images, and can provide embeddings interwoven with noisy frames, such that the diffusion-transformer layer can refine the noisy frames into final frames consistent with the embeddings. Thus, this technique can utilize cueing provided in a storyboard user interface to generate output visual media.
[0208] Figure 19A , Figure 19B , Figure 19C and Figure 19D Some aspects of the storyboard user interface according to some embodiments of the present technology are shown. Although Figures 19A-19D A specific user interface is shown, but this technology should not be considered limited to use with such an interface. Instead, Figure 19A - Figure 19D The user interface shown is provided to illustrate example options and features offered by this technology.
[0209] Figure 19A A blank storyboard user interface 1900 is shown. The storyboard user interface 1900 includes a first frame 1906, which is also represented as a first frame thumbnail 1902 along the time scale 1908. The first frame 1906 includes a timestamp 1910 indicating that the frame occurs at 0 seconds, and the first frame thumbnail 1902 is located at the 0-second position on the time scale 1908.
[0210] The first frame 1906 is displayed in the middle of the storyboard user interface 1900, which indicates that the first frame 1906 is the frame that is in focus of editing via the prompt editor 1912.
[0211] You can use the Add Frame option 1918 to add one or more additional frames.
[0212] Figure 19BA blank storyboard user interface 1900 with two frames is shown. The first frame 1906 is at timestamp 0 seconds, and the second frame 1914 is at timestamp 3.14 seconds. Since the second frame 1914 is in the center of the storyboard user interface 1900, it is considered to be in the focus of editing, and a prompt can be entered for the second frame 1914 using the prompt editor 1912. The prompt can be text or visual media.
[0213] Figure 19C A blank storyboard user interface 1900 with three cue frames is shown. The first frame 1906 is at timestamp 0 seconds, the second frame 1914 is at timestamp 3.14 seconds, and the third frame 1916 has been added at timestamp 2 seconds. Since the third frame 1916 is in the center of the storyboard user interface 1900, it is considered to be in the focus of editing, and cue editor 1912 can be used to input cue for the second frame 1914. The cue can be text or visual media.
[0214] Figure 19D A blank storyboard user interface with two frames is shown 1900. Figure 19D Similar to Figure 19B However, another version with the added frame option 1918 is shown. Figure 19D The diagram shows that the Add Frame option 1918 appears when the user interface hovers over a frame on time scale 1908. Selecting Add Frame option 1918 on time scale 1908 causes a new frame to be added at that timestamp. Additionally, a frame at a given timestamp can be dragged to a new timestamp as desired.
[0215] Figure 20A , Figure 20B and Figure 20C Examples of generated videos and corresponding storyboard user interfaces according to some embodiments of the present technology are shown. Although Figures 20A-20C A specific user interface is shown, but this technology should not be considered limited to use with such an interface. Rather, Figures 20A-20C The user interface shown is provided to illustrate example options and features offered by this technology.
[0216] Figure 20A A visual media viewing interface for playing existing visual media is shown. The existing visual media may be selected from a public gallery or content library created or shared using a user account of a user interacting with the visual media viewing interface. Specifically, frame 2002 of the existing visual media is shown together with biographical information 2004 of the existing visual media. Below frame 2002 of the existing visual media, the user clicks on a text fragment showing a prompt for creating the existing visual media, which results in the display of a prompt describing visual media 2006.
[0217] Figure 20B The visual media viewing interface is shown playing existing visual media and displaying another frame 2008 of the existing visual media. This is a later frame in the visual media than frame 2002 of the existing visual media because, when using the overhead ring light as a reference point, it is clear that the woman has walked further down the hall.
[0218] Figure 20B User interface controls and pointers above the display storyboard for the Visual Media 1710 option are also shown.
[0219] Figure 20C The storyboard user interface 1900 is shown, which is generated after the visual media viewing interface has received a selection to display the storyboard for visual media 1710. At the location of the first frame 1906 is a prompt describing visual media 2006. Note that this prompt describes the entire visual media, not just the first frame. At the location of the second frame 1914 is a prompt describing another frame of existing visual media 2010. Note that although this description refers to frames as first frame, second frame, third frame, etc., this is merely to distinguish these frames from each other, and referring to a frame as first, second, or third frame does not mean that the frame is the first, second, or third frame of the visual media.
[0220] The second frame, 1914, has a timestamp of 2.12 seconds and a cue that reads, "The lady walks through the aisle, and the ring lights above create a halo effect as she walks." This cue guides the visual media generative response engine to generate frames in the video around that timestamp that typically correspond to this description.
[0221] Storyboard user interfaces are not always required to advance the plot or visual story at exactly the right pace. In some embodiments, storyboard user interfaces are particularly effective when they cause significant changes to occur.
[0222] Figure 21 Examples of using a storyboard user interface to transition visual media to different scenes are shown, according to some embodiments of the present technology. Figure 21 It also shows that the prompts can be visual cues. Although Figure 21 A specific user interface is shown, but this technology should not be considered limited to use with such an interface. Instead, Figure 21 The user interface shown is provided to illustrate example options and features offered by this technology.
[0223] On the left, Figure 21 The storyboard user interface 1900 is shown, featuring Figures 20A-20CThe first frame 1906 and the second frame 1914 show the visual media depicting a woman walking along a future corridor. The third frame 1916 is a frame of visual media set in a completely different scene, and the fourth frame 2102 is a cue describing the visual media in the third frame.
[0224] If cues included in the storyboard user interface are submitted to the visual media generative responsive engine, then the text and visual cues will be embedded. The embeddings will be inserted between noisy frames and provided to the diffusion-transformer layer. The diffusion-transformer layer will attempt to denoise the noisy frames into frames consistent with the embeddings. The result might begin with a scene of a woman walking down a future corridor, and at the end of the corridor, the woman might enter a scene of people having dinner by the sea. Alternatively, the result might begin with the woman walking along a future corridor (where some people in the scene are gathered), and then evolve to everyone meeting at a table in the future environment. In fact, new results are generated each time visual media is generated.
[0225] Figure 22A and Figure 22B Examples illustrating how prompts can be enhanced to be more descriptive, based on some embodiments of the present technology, are shown. Although Figure 22A and Figure 22B A specific user interface is shown, but this technology should not be considered limited to use with such an interface. Rather, Figure 22A and Figure 22B The user interface shown is provided to illustrate example options and features offered by this technology.
[0226] Figure 22A An example one-line prompt is shown. The user interface pointer is positioned above prompt enhancement option 2202. Figure 22B A prompt generated by selecting prompt enhancement option 2202 is shown. After the user interface receives a selection of prompt enhancement option 2202, a prompt from... Figure 22A A single-sentence prompt is sent to the language model for more detailed enhancement, and this is more useful in guiding the output of the visual media generative response engine.
[0227] The graphical user interface includes a prompt enhancement option 2202, which, when selected, effectively inputs prompts into the language model to generate enhanced prompts, and then replaces those prompts with enhanced prompts in the prompt editor. Prompt enhancement option 2202 can be useful because users, especially novice users, often provide advanced prompts that leave many details to infer. Prompt enhancement option 2202 can receive more detailed prompts from the language model, where the more detailed prompts clarify many additional details. The user interacting with the prompt editor 1102 can then view the detailed prompts and either accept them or edit details that are not part of the user's vision for the output visual media.
[0228] Figure 23 Example processes for inducing a visual media generative response engine to reconstruct existing visual media according to some embodiments of the present technology are illustrated. Although the example processes depict a specific sequence of operations, this sequence can be changed without departing from the scope of this disclosure. For example, some of the described operations may be performed in parallel or in a different order that does not substantially affect the functionality of the process. In other examples, different components of the example device or system implementing the process may perform functions substantially simultaneously or in a specific order.
[0229] As described above, this technology provides access to a library of existing visual media to inspire other visual artists, enabling them to use existing visual media as inspiration or a starting point to create new visual media. Figure 24A An interface showing an example of existing visual media is illustrated, which includes a prompt editor 1102 for creating text prompts that describe the desired new visual media to be created using existing visual media as inspiration.
[0230] According to some examples, the method includes presenting a remix intensity control menu in box 2302 to remix existing visual media. For example, Figure 1 The front end 124 shown can present a remix intensity control menu to remix existing visual media. The remix intensity control menu includes options for remixing the degree of remixing of existing visual media. Figure 24A and Figure 24B The remix intensity control menu 2402 is shown.
[0231] According to some examples, the method includes receiving a selection of a first remixing degree from the remix intensity control menu in box 2304. For example, Figure 1 The front end 124 shown can receive a selection of the first remixing level from the remix intensity control menu. For example, as Figure 24B As shown, you can choose the remixing level from strong, mild, slight, and customized.
[0232] As depicted in boxes 2306 and 2308, the intensity of remixing can be controlled by modifying the cue, or by adjusting a characteristic such as the amount of noise provided in the input visual media, or a combination of both.
[0233] According to some examples, the method includes requesting the narrative from the language model in box 2306. For example, Figure 1The front-end 124 shown can request a narrative from the language model. Hints to the language model include existing visual media or text titles of existing visual media, as well as instructions to generate a narrative based on existing visual media but varying according to a first degree of remixing.
[0234] According to some examples, the method includes providing existing visual media with a narrative generated by a language model to a visual media generative response engine in box 2308. For example, Figure 1 The front end 124 shown can provide the visual media generative response engine with existing visual media having a narrative generated by a language model.
[0235] As described above, another way to control the remixing intensity is to adjust the amount of noise supplied to the input visual media. For example, when an input frame is to be used as input to a visual media generative response engine, the amount of noise in that frame can be adjusted. Generally, when an input frame is encoded and provided as input to a visual media generative response engine, it is provided noise-free along with a set of noise frames refined to generate the visual media. However, when a remixing level is selected from the remixing intensity control menu, the input frame may include a certain amount of introduced noise. A low remixing intensity (“Slight” in the figure) may be associated with a small amount of noise, while a high remixing intensity (“Strong” in the figure) may be associated with a large amount of noise. The more noise in the input frame, the more susceptible that frame is to changes introduced by the visual media generative response engine.
[0236] According to some examples, the method includes receiving output visual media in box 2310, which is a reconstructed version of existing visual media. For example, Figure 1 The front end 124 shown can receive output visual media, which is a reconstructed version of existing visual media. The amount of reconstructing the existing visual media is controlled by a first remixing degree.
[0237] Figure 25 Example processes for influencing the fusion of a first visual medium with a second visual medium, according to some embodiments of the present technology, are illustrated. Although the example processes describe a specific order of operations, this order may be changed without departing from the scope of this disclosure. For example, some of the described operations may be performed in parallel or in a different order that does not substantially affect the functionality of the process. In other examples, different components of the example device or system implementing the process may perform their functions substantially simultaneously or in a specific order.
[0238] According to some examples, the method includes receiving a first input visual media and a second input visual media in box 2502 by a prompt editor. For example, Figure 1The front end 124 shown can receive the first input visual media and the second input visual media through the prompt editor.
[0239] According to some examples, the method includes presenting a merged interface in box 2504. For example, Figure 1 The front end 124 shown can present a fusion interface. The fusion interface displays at least one frame of the first input visual media and at least one frame of the second input visual media. Figure 26A An example of a fusion interface 2600 with a first input visual media 2602 and a second input visual media 2604 is shown.
[0240] According to some examples, the method includes adjusting the blending curve in box 2506 in response to user input. For example, Figure 1 The front end 124 shown can be used to adjust the fusion curve in response to user input. The fusion curve represents the effect of the first input visual media relative to the second input visual media over time.
[0241] In some embodiments, user input is the selection of a pre-configured fusion curve. For example, Figure 26B Pre-configured blending curves are shown, including transition, blending, and sampling. The transition blending curve is used to transition from the full influence of the first input visual media 2602 to the full influence of the second input visual media 2604, and in the middle of the visual media, the frame reflects the transition from the first input visual media 2602 to the second input visual media 2604. The blending blending curve is used to blend some aspects of the first input visual media 2602 into the second input visual media 2604. When combined with the second input visual media 2604, the sampling blending curve is used to adjust the amount of influence of the first input visual media 2602.
[0242] In some embodiments, users can also interact with the user interface to select and create custom blending curves.
[0243] By combining aspects of frames from the first and second visual media into a fused frame, a fusion curve selected in the user interface can be transformed into input to the visual media generative response engine. For example, by applying weighting factors to the corresponding frames corresponding to the selected fusion curve through mathematical operations, frames from the first visual media in the coded representation can be fused with frames from the second visual media in the coded representation. The coded representation of the combined frames can be introduced into the visual media generative response engine. This process can be repeated for several or even all frames of the visual media to guide the transition of the visual media from beginning with the first visual media to ending with the second visual media.
[0244] Another mechanism for combining these two frames could be to create CLIP embeddings from the corresponding frames and mathematically combine these embeddings in a mathematical operation similar to the mathematical operation for the encoded representation described above. This process can be repeated for several or even all frames of the visual medium to guide the transition of the visual medium from beginning with the first visual medium to ending with the second visual medium.
[0245] Another mechanism for combining these two frames could be by adding text prompts that instruct the visual media generative response engine to generate a combined frame using the degree of influence of the frame from the first visual media relative to the degree of influence of the frame from the second visual media. These text prompts can be provided across the entire length of the visual media and can vary to match the amount of fusion of the visual media frames.
[0246] Figure 27A , Figure 27B and Figure 27C Frames of fused visual media using transition fusion curves according to some embodiments of the present technology are shown. Figure 27A , Figure 27B and Figure 27C The visual media cues shown utilize a first input visual media and a second input visual media, along with text cues describing the desired effect as depicted by the transition fusion curve.
[0247] Figure 27A The first frame of the visual media generated by the visual media generative response engine is shown. The first frame is identical to the first input visual media. Figure 27B A transition frame illustrating some aspects of fusing the first and second input visual media is shown. The visual media ends at... Figure 27C The last frame shown ends here, which is the same as the second input visual media.
[0248] Figure 28 A looping interface according to some embodiments of the present technology is shown. The looping interface 2800 can be used to edit frames at the beginning and end of a video to allow for seamless looping of the video. For example... Figure 28 As shown, the user has cropped frames from the beginning and end of the video so that it will loop seamlessly when the video loops. Although the interface looks like a simple cropping interface, the cropping in this interface is input to the visual media generative response engine, which takes input frames and outputs additional frames to produce output visual media that ends at the exact frame that should precede the first frame of the visual media.
[0249] Although this technology provides a looping interface 2800, it is possible to generate video ready to be looped. Users can use a storyboard user interface and insert the same frames as the first and last frames of the visual media, or users can use the looping interface 2800 to provide a cue specifying that the generated visual media should end at a frame that can seamlessly loop back to the first frame. Both techniques can result in video ready to be looped without careful frame selection within the looping interface 2800.
[0250] Figure 29 A frame clipping interface button according to some embodiments of the present technology is shown. The frame clipping interface 2900 can be used to clip frames before and / or after a generated video by manipulating a slider in the interface, to include frames that should be retained as part of the video. Alternatively, the interface can be used to identify frames to be removed.
[0251] Figure 30 Example procedures for requesting visual media from a visual media generative response engine using an API, according to some embodiments of the present technology, are illustrated. Although the example procedures depict operations in a specific order, this order can be changed without departing from the scope of this disclosure. For example, some of the described operations may be performed in parallel or in a different order that does not substantially affect the functionality of the process. In other examples, different components of the example device or system implementing the process may perform functions substantially simultaneously or in a specific order.
[0252] While much of this specification has already mentioned that Front-End 124 provides a graphical user interface to guide users in creating prompts, this technology also supports other applications (whether first-party, second-party, or third-party) that can handle user input and output display by providing application programming interfaces (APIs) that can be called by other applications to request output from the visual media generative responsive engine. For example... Figure 8 As shown, the front-end 802 can also provide the function of receiving API calls.
[0253] According to some examples, the method includes sending a cue from the application to the visual media generative response engine in box 3002. The cue is provided to the visual media generative response engine using an application programming interface (API) provided by the engine. The API may define a text cue field for providing a text value describing the visual media to be generated by the engine. In some embodiments, the text cue field defines a subfield of the text value and an associated subfield of a timestamp associated with the text value. The text cue field may have multiple text cues and associated timestamps. The API may also define a visual media cue field, which further defines subfields of the visual media cue and an associated timestamp. The visual media cue field may also contain multiple visual media cues. Visual media can be provided in the API by providing a link to visual media that can be downloaded by the engine, or by transferring the visual media in an archive that matches the visual media cue and timestamp to the correct file.
[0254] The API defines fields for specifying the duration, resolution, and aspect ratio of the output visual media.
[0255] Tips can be adapted via API by combining the tips with their corresponding timestamps, such as those created by the storyboard user interface described above.
[0256] According to some examples, the method includes receiving output visual media in response to prompts by a third-party application in box 3004.
[0257] Figure 31 Examples of API calls according to some embodiments of this technology are shown. For example, Figure 31 The API call includes a cue at 0 seconds describing the overall visual media – “A tranquil forest with sunlight streaming through the trees.” The API call also includes a text cue at 5 seconds – “A deer walks gracefully near a stream,” and a corresponding visual media cue at 5.5 seconds, which includes an image of the deer. The API call also includes an additional visual media cue at 15 seconds showing an image of a mountain.
[0258] Finally, the API call also includes settings for the output media. The API request is for a 20-second duration video with a resolution of 1920x1080 and an aspect ratio of 16:9.
[0259] Figure 32Example processes for modulating visual media generated by a visual media generative response engine, according to some embodiments of the present technology, are illustrated. Although the example processes depict operations in a specific order, this order can be changed without departing from the scope of this disclosure. For example, some of the described operations may be performed in parallel or in a different order that does not substantially affect the functionality of the process. In other examples, different components of the example device or system implementing the process may perform their functions substantially simultaneously or in a specific order.
[0260] While visual media generative responsive engines are powerful tools that can help visual artists create high-quality visual art more easily than existing technologies, some security mechanisms are needed. Previous image generators have been used to create depth artifacts and graphics to spread misinformation, among other undesirable use cases. Therefore, this technology utilizes several moderation mechanisms to prevent the generation of harmful or undesirable content by visual media generative responsive engines.
[0261] While this document provides a description of the review strategy, it should be apparent to those skilled in the art that some content reviews reflect judgments that may evolve over time. No examples should be considered as statements of policy by the inventor or applicant regarding certain reviews. All discussion is intended to illustrate the technology and provide illustrative examples, and should not be construed as limiting the technology unless expressly stated in the claims tree.
[0262] The method begins when the prompt is passed to the review system 838.
[0263] In some embodiments, certain words or combinations of words are clearly intended to generate content that a moderation policy is configured to prohibit. In this case, text matching can be used as a first line of defense against such prompts. According to some examples, the method includes evaluating the text portion of the prompt against a block list at box 3202 to identify prohibited words. For example, Figure 8 The auditing system 838 shown can evaluate the text portion of a prompt against a block list to identify prohibited words or combinations of words. When a prompt includes words from the block list, it may be possible to rewrite the prompt without using words from the block list, and in this case, the prompt can be passed to decision box 3206, as will be further elaborated below.
[0264] Even when a prompt doesn't include a word from the block list, it may still lead to content that should be reviewed. In some cases, users may not intend to create a prompt that would result in reviewed content. To combat the possibility that a prompt might lead to content that should be reviewed under a review policy, this technology can utilize more sophisticated methods to evaluate the entire prompt. According to some examples, the method includes evaluating the prompt in decision box 3204 to classify it as a violation of a review policy. For example, Figure 8 The auditing system 838 shown can evaluate prompts to classify them as violations of auditing policies.
[0265] One consequence of the fact that users of visual media generative response engines expect a certain amount of latency when generating visual media is that these engines can utilize more sophisticated tools to evaluate cues and generated content. The time spent evaluating cues and generated content is relatively short compared to the time spent generating the visual media. Therefore, this technique is not limited to potentially overly simplistic and robust methods, and can leverage more sophisticated evaluation techniques and models.
[0266] For example, an moderation system could use embedding space to attempt to categorize prompts as content that might be subject to moderation policies. In another example, the moderation system includes a language model configured to recognize prompts for visual media that would violate moderation policies if the request for a visual media generative response engine outputs such prompts.
[0267] In some embodiments, the moderation system may utilize multiple different models, instances of the same model, or, in the case of a language model, a language modeling procedure that guides the language model to evaluate multiple prompts and responses in a more cautious manner. For example, the language model could be multiple instances of a language model, each configured to identify when a text prompt will violate a specific aspect of the moderation policy. Indeed, for a given aspect of the moderation policy, it may be desirable to have a specific instance of the model (or a specific prompt for the same model). This provides greater control over having smaller models or individual prompts for different aspects of the moderation policy compared to training a machine learning model to detect all prompts that might violate the moderation policy. Thus, if a change is needed to a specific part of the moderation policy, it is not necessary to train an entirely new model. Instead, only a smaller model or prompt relevant to that specific part of the moderation policy needs to be adjusted.
[0268] In some embodiments, the review system can be multimodal, meaning it can evaluate the parts of the prompt given in the form of text, visual media, speech, etc. For example, the language model used to evaluate the prompt can be a multimodal language model, thereby enabling it to evaluate both the image and text portions of the prompt.
[0269] According to some examples, the method includes determining in decision box 3206 whether the prompt should be rewritten to comply with the review policy. For example, Figure 8 The review system 838 shown determines whether a prompt can be rewritten to comply with the review policy. In some embodiments, when it is determined that the visual media portion of the prompt violates the review policy, the prompt is rejected and not modified. In some embodiments, it may be determined that the text portion of the prompt violates a specific aspect of the review policy and cannot be rewritten. When the prompt cannot be modified, the method proceeds to box 3208, as described below.
[0270] Based on some examples, the method includes rewriting the warning in box 3210 to conform to the review policy. For example, when the violation is a portion of the warning's text, Figure 8 The auditing system 838 shown can rewrite the prompts for violating the auditing policy to comply with the auditing policy.
[0271] An example of a prompt that can be rewritten might be a prompt asking for a picture of "a cartoon mouse like Mickey Mouse riding a train." In this case, the prompt could be rewritten as "a cartoon mouse with big ears and a cheerful expression riding an old-fashioned steam train." This prompt can be rewritten because the overall intent is indisputable as long as the output does not include Mickey Mouse. An example of a prompt that cannot be rewritten is one that might be requesting visual media describing self-harm. This prompt cannot be rewritten because its intent is to produce visual media that does not comply with review policies.
[0272] According to some examples, the method includes sending a prompt to the visual media generative response engine in box 3212. For example, when the prompt conforms to an approval policy, Figure 8 The review system 838 shown can send prompts to the visual media generative response engine.
[0273] According to some examples, the method includes receiving an instance of visual media generated by a visual media generative response engine in box 3214.
[0274] It's not just the prompts that should be evaluated in accordance with content moderation policies. Sometimes, even well-intentioned compliance prompts can lead to the generation of offensive visual media. According to some examples, the approach includes, in box 3216, using an image classifier to evaluate instances of visual media generated by a visual media generative response engine to determine when the output violates moderation policies. For example, Figure 8 The moderation system 838 shown can use an image classifier to evaluate instances of visual media generated by the visual media generative response engine to determine when the output violates the moderation policy.
[0275] When the generated visual media triggers the image classifier in box 3216, the method includes determining in decision box 3218 whether the image classifier is associated with a moderation policy for strictly prohibited or conditionally prohibited content. For example, Figure 8 The review system 838 shown can determine whether an image classifier is associated with a review policy that strictly prohibits or conditionally prohibits content.
[0276] An example of content that might be conditionally prohibited is content describing a person in a potentially offensive environment or manner. If this person is fictionalized by a visual media generative response engine, the content might be acceptable. However, if this person appears as part of a cue in the input visual media, the content might lean towards deepfake-type content and would be offensive. Therefore, according to some examples, the method includes determining in decision box 3220 whether the cue used to generate instances of conditionally prohibited visual media is a text cue. For example, Figure 8 The review system 838 shown can determine whether the prompt used to generate the instance of visual media is a text prompt. If the prompt is a text prompt, conditionally offensive visual media can be permitted, and the method can proceed to box 3222.
[0277] According to some examples, the method includes providing the user with instances of visual media generated by a visual media generative responsive engine in box 3222. For example, when the output is not blocked by a moderation policy, Figure 8 The review system 838 shown can provide users with instances of visual media generated by the visual media generative response engine.
[0278] If the cue used to generate conditionally offensive content is determined to be a visual cue, then in box 3208, the moderation system will not output instances of visual media generated by the visual media generative responsive engine. Furthermore, if output is strictly prohibited, visual media cannot be provided to the user.
[0279] According to some examples, the method includes, in box 3208, refusing to serve visual media from the visual media generative response engine when a prompt or output violates an audit policy. For example, when a prompt or output violates an audit policy... Figure 8 The review system 838 shown can refuse to provide visual media from the visual media generative response engine.
[0280] Figure 33This is a block diagram illustrating an example machine learning platform for implementing various aspects of this disclosure, according to some aspects of the present technology. Although the example system depicts specific system components and their arrangement, this depiction is intended to facilitate the discussion of the present technology and should not be considered limiting unless specified in the appended claims. For example, some components shown as separate may be combined with other components, and some components may be divided into separate components.
[0281] System 3300 may include a data input engine 3310, which may further include a data retrieval engine 3312 and a data transformation engine 3314. The data retrieval engine 3312 may be configured to access, interpret, request, or receive data, which may be adjusted, reformatted, or altered (e.g., interpreted by another engine such as the data input engine 3310). For example, the data retrieval engine 3312 may use an API to request data from a remote source. The data input engine 3310 may be configured to access, interpret, request, format, reformatted, or receive input data from one or more data sources 3301. For example, the data input engine 3310 may be configured to use the data transformation engine 3314 to perform reconfiguration or other changes to the data, such as data dimensionality reduction. In some embodiments, the one or more data sources 3301 may be associated with a single entity (e.g., an organization) or multiple entities. One or more data sources 3301 may include one or more of the following: training data 3302a (e.g., input data for feeding a machine learning model as part of one or more training processes), validation data 3302b (e.g., data that at least one processor can compare with to model outputs, for example, to determine the quality of the model outputs), and / or reference data 3302c. In some embodiments, the data input engine 3310 may be implemented using at least one computing device. For example, data from one or more data sources 3301 may be obtained through one or more I / O devices and / or network interfaces. Furthermore, data may be stored (e.g., during the execution of one or more operations) in a suitable storage device or system memory. The data input engine 3310 may also be configured to interact with a data storage device, which may be implemented on a computing device that stores data in storage devices or system memory.
[0282] System 3300 may include a feature extraction engine 3320. Feature extraction engine 3320 may include a feature annotation and labeling engine 3322 (e.g., configured to annotate or label features from a model or data that can be extracted by feature extraction engine 3324), a feature extraction engine 3324 (e.g., configured to extract one or more features from a model or data), and / or a feature scaling and selection engine 3326. Feature scaling and selection engine 3326 may be configured to determine, select, restrict, constrain, connect, or define features (e.g., AI features) for use with the AI model.
[0283] System 3300 may also include a machine learning (ML) modeling engine 3330, which can be configured to perform one or more operations on a machine learning model (e.g., model training, model reconfiguration, model validation, model testing), such as those described in the process described herein. For example, the ML modeling engine 3330 can perform operations for training a machine learning model, such as adding, removing, or modifying model parameters. Training of the machine learning model can be supervised, semi-supervised, or unsupervised. In some embodiments, training of the machine learning model may include multiple rounds of a machine learning model process (e.g., a training process) or data transfer (e.g., training data 3302a). In some embodiments, different rounds may have different degrees of supervision (e.g., supervised, semi-supervised, or unsupervised). Data entering the model to train the model may include input data (e.g., as described above) and / or data previously output from the model (e.g., forming recursive learning feedback). Model parameters may include one or more of the following: seed values, model nodes, model layers, algorithms, functions, model connections (e.g., between other model parameters or between models), model constraints, or any other digital components that affect the model output. Model connections can include or represent relationships between model parameters and / or models, which can be dependent or interdependent, hierarchical, and / or static or dynamic. The combinations and configurations of model parameters discussed herein, as well as the relationships between them, are cognitively infeasible for human thought to maintain or use. Without limiting the disclosed embodiments in any way, machine learning models can include millions, billions, or even trillions of model parameters. The ML modeling engine 3330 can include a model selector engine 3332 (e.g., configured to select a model from multiple models, for example, based on input data), a parameter engine 3334 (e.g., configured to add, remove, and / or change one or more parameters of a model), and / or a model generation engine 3336 (e.g., configured to generate one or more machine learning models, for example, based on model input data, model output data, comparison data, and / or validation data).
[0284] In some embodiments, the model selector engine 3332 may be configured to receive input and / or transmit output to the ML algorithm database 3370. Similarly, the feature extraction engine 3320 may utilize storage devices or system memory to store data and may utilize one or more I / O devices or network interfaces to transmit or receive data. The ML algorithm database 3370 may store one or more machine learning models, any of which may be fully trained, partially trained, or untrained. Machine learning models can be, or include, one or more of the following (e.g., in the case of meta-models): statistical models, algorithms, neural networks (NNs), convolutional neural networks (CNNs), generative neural networks (GNNs), Word2Vec models, bag-of-words models, term frequency-inverse document frequency (TF-IDF) models, GPT (Generative Pre-trained Transformer) models (or other autoregressive models), diffusion models, diffusion-transformer models, encoders such as BERT (Bidirectional Encoder Representation from Transformer) or LXMERT (Learned Cross-modal Encoder Representation from Transformer), Proximity Policy Optimization (PPO) models, nearest neighbor models (e.g., k-nearest neighbor models), linear regression models, k-means clustering models, Q-learning models, temporal difference (TD) models, deep adversarial network models, or any other type of model further described herein. Some ML algorithms in the ML algorithm database 3370 can be considered generative response engines. Generative response engines are those models that are often referred to as generative AI and can receive input prompts and generate additional content based on the prompts. GPT, diffusion models, and diffusion-transformer models are some non-limiting examples of generative response engines. Some concrete examples of generative response engines that can be stored in the ML algorithm database 3370 include versions DALL E, CHAT GPT, and SORA, all provided by OPEN AI.
[0285] System 3300 may also include a predictive output generation engine 3345 and an output validation engine 3350 (e.g., configured to apply validation data to the output of a machine learning model). The predictive output generation engine 3345 can analyze the input and identify relevant patterns and associations in the data it has learned to generate the most likely continuation sequence of words predicted by one or more models from an ML algorithm database 3370 for the input, aiming to provide a consistent and context-relevant answer. The predictive output generation engine 3345 generates a response by sampling from a probability distribution of possible words and sequences guided by patterns observed during its training. In some embodiments, the predictive output generation engine 3345 may generate multiple possible responses before presenting the final response. The predictive output generation engine 3345 may generate multiple responses based on the input, and these responses are variants that the predictive output generation engine 3345 considers potentially relevant and consistent. The output validation engine 3350 may evaluate these generated responses based on certain criteria. These criteria may include relevance to the prompt, consistency, fluency, and sometimes adherence to specific guidelines or rules, depending on the application. Based on this assessment, the output validation engine 3350 selects the most suitable response. This selection typically balances factors such as relevance, informativeness, and consistency, balancing the highest-scoring responses within the established criteria.
[0286] System 3300 may further include a feedback engine 3360 (e.g., configured to apply feedback from a user and / or machine to a model) and a model refinement engine 3355 (e.g., configured to update or reconfigure a model). In some embodiments, the feedback engine 3360 may receive input and / or transmit output (e.g., output from a trained, partially trained, or untrained model) to a result metric database 3365. The result metric database 3365 may be configured to store outputs from one or more models and may also be configured to associate outputs with one or more models. In some embodiments, the result metric database 3365 or other devices (e.g., the model refinement engine 3355 or the feedback engine 3360) may be configured to associate outputs, detect trends in output data, and / or infer changes in inputs or model parameters to elicit specific model outputs or specific types of model outputs. In some embodiments, the model refinement engine 3355 may receive outputs from a prediction output generation engine 3345 or an output validation engine 3350. In some embodiments, the model refinement engine 3355 may transmit the received output to the featureization engine 3320 or the ML modeling engine 3330 in one or more iteration cycles.
[0287] The engine of System 3300 may be a packaged functional hardware unit designed to be used as part of other components or programs performing specific functions (e.g., related functions). Any or each of these modules can be implemented using computing devices. In some embodiments, the functionality of System 3300 may be partitioned across multiple computing devices to allow distributed data processing, which can increase output speed and reduce computational load on individual devices. In some embodiments, System 3300 may use load balancing to maintain a stable resource load (e.g., processing load, memory load, or bandwidth load) across multiple computing devices and reduce the risk of computing devices or connections becoming overloaded. In these or other embodiments, different components may communicate through one or more I / O devices and / or network interfaces.
[0288] System 3300 may relate to different use cases or domains. The description of embodiments relating to specific domains (e.g., natural language processing or language modeling) is not intended to limit the disclosed embodiments to those specific domains, and embodiments consistent with this disclosure can be applied to any domain utilizing predictive modeling based on available data.
[0289] Figure 34 An example of a computing system 3400 is shown, which may be configured as follows: Figure 1 , Figure 8 and / or Figure 33 Any computing device or any component of any engine shown.
[0290] In some embodiments, the computing system 3400 is a single device or a distributed system, wherein the functions described herein may be distributed across a data center, multiple data centers, a peer-to-peer network, etc. In some embodiments, one or more described system components represent a plurality of such components, each performing some or all of the functions described therein. In some embodiments, a component may be a physical or virtual device.
[0291] In some embodiments, the computing system 3400 may include one or more computing resources supplied from a “cloud computing” provider, such as AMAZON ELASTIC COMPUTE CLOUD (“AMAZON EC2”) provided by Amazon Inc. of Seattle, Washington; SUN CLOUD COMPPUTER UTILITY provided by SUN MICROSYSTEMS of Santa Clara, California; AZURE provided by Microsoft Corporation of Redmond, Washington; and GOOGLE CLOUD PLATFORM provided by ALPHABET of Mountain View, California.
[0292] Example computing system 3400 includes at least one processing unit (CPU or processor) 3404 and a connection 3402 that couples various system components, including system memory 3408, such as read-only memory (ROM) 3410 and random access memory (RAM) 3412, to processor 3404. Memory 3408 may be a volatile or non-volatile memory device and may be a hard disk or other type of non-transitory computer-readable medium capable of storing computer-accessible data, such as magnetic tape, flash memory cards, solid-state memory devices, digital multifunction disks, random access memory (RAM), read-only memory (ROM), and / or some combination of these devices.
[0293] The memory 3408 may include software services, servers, logic, etc., which enable the system to perform functions when the processor 3404 executes the code defining such software. In some embodiments, hardware services that perform a particular function may include software components stored in a computer-readable medium along with necessary hardware components, such as the processor 3404, connection 3402, output device 3422, etc., to perform that function.
[0294] The computing system 3400 may include a cache of a high-speed memory 3406, which is directly connected to, adjacent to, or integrated into the processor 3404.
[0295] Connection 3402 can be a physical connection via a bus, or a direct connection to processor 3404, for example, in a chipset architecture. Connection 3402 can also be a virtual connection, a networking connection, or a logical connection.
[0296] Processor 3404 may include any general-purpose processor and hardware or software services stored in memory 3408 configured to control processor 3404 and dedicated processors, wherein software instructions are incorporated into the actual processor design. Processor 3404 may essentially be a fully self-contained computing system, containing multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric. Processor 3404 may be physical or virtual.
[0297] To enable user interaction, the computing system 3400 includes an input device 3426, which can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, voice, etc. The computing system 3400 may also include an output device 3422, which can be one or more of a variety of output mechanisms known to those skilled in the art. In some cases, a multimodal system allows the user to provide multiple types of input / output to communicate with the computing system 3400. The computing system 3400 may include a communication interface 3424, which typically governs and manages user input and system output. There are no limitations on operation on any particular hardware configuration; therefore, the basic features described herein can be readily replaced by improved hardware or firmware configurations on which they are developed.
[0298] In some embodiments, computing system 3400 may refer to a combination of an interactive personal computing device and components hosted in a data center, wherein both the computing device and the components are located in the data center. In such an example, both the components in the data center and the personal computing device may have processors, caches, memory, storage devices, etc.
[0299] For clarity, in some cases, this technology may be presented as comprising individual functional blocks, including devices, device components, steps, or processes in a method embodied in software or a combination of hardware and software.
[0300] Any steps, operations, functions, or processes described herein may be performed or implemented by hardware and software services or combinations of services, individually or in combination with other devices. In some embodiments, a service may be software residing in the memory of one or more servers of a client device and / or a content management system, and performs one or more functions when a processor executes the software associated with the service. In some embodiments, a service is a program or collection of programs that performs a specific function. In some embodiments, a service may be considered a server. Memory may be a non-transitory computer-readable medium.
[0301] In some embodiments, computer-readable storage devices, media, and memories may include cables or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer-readable storage media explicitly exclude media such as energy, carrier signals, electromagnetic waves, and the signals themselves.
[0302] The methods described in the examples above can be implemented using computer-executable instructions stored or otherwise available from a computer-readable medium. Such instructions may include, for example, instructions and data that cause or otherwise configure a general-purpose computer, special-purpose computer, or special-purpose processing device to perform a specific function or group of functions. Part of the computer resources used may be accessible via a network. Executable computer instructions may be, for example, binary, intermediate format instructions such as assembly language, firmware, or source code. Examples of computer-readable media that can be used to store instructions, information, and / or information created during the methods according to the examples include disks or optical discs, solid-state storage devices, flash memory, USB devices equipped with non-volatile memory, networked storage devices, etc.
[0303] Devices implementing the methods according to these disclosures may include hardware, firmware, and / or software, and may take any of a variety of form factors. Typical examples of such form factors include servers, laptops, smartphones, small form factor personal computers, personal digital assistants, etc. The functionality described herein may also be implemented in peripherals or add-in cards. By further example, such functionality may also be implemented on a circuit board in different chips or different processes that execute within a single device.
[0304] Instructions, the medium for conveying such instructions, the computing resources for executing them, and other structures for supporting such computing resources are components for providing the functionality described in these disclosures.
[0305] Some aspects: This technology includes, as set forth in some of the aspects of this technology presented below, computer-readable storage media for storing instructions and systems for executing any of the methods embodied in the instructions: Aspect 1: A visual media generative response engine, comprising: an input layer configured to receive cues, wherein the cues may include cue text and / or cue frames, the input layer being further configured to output at least one noise frame; a visual encoder layer configured to compress at least one noise frame, and when the cues include cue frames, the visual encoder layer is configured to also compress the cue frames into corresponding representations in a lower-dimensional latent space, wherein the visual encoder layer is a learned compression model for reducing the dimensionality of visual data, the visual encoder layer being configured to take raw visual media as input and output latent representations that are compressed both temporally and spatially; a vector representation layer configured to decompose the latent representations into spatiotemporal blocks representing at least one frame, wherein the spatiotemporal blocks act as transformer lexics; an embedding layer configured to create cue embeddings; and diffusion. A diffuse-transformer layer is configured to embed processing spatiotemporal blocks according to prompts to iteratively remove noise from the spatiotemporal blocks, thereby producing processed spatiotemporal blocks. The diffuse-transformer layer includes diffuse-transformer processing units that process the respective spatiotemporal blocks in parallel. Each diffuse-transformer processing unit has an associated attention head configured to communicate with the attention heads of other diffuse-transformer processing units to perform a self-attention operation, which informs the respective diffuse-transformer processing unit of information related to other spatiotemporal blocks being processed by the other diffuse-transformer processing units. A linearization layer is configured to linearize the processed spatiotemporal blocks from the respective diffuse-transformer processing units into an output sequence of encoded spatiotemporal blocks. A block decoder layer is configured to decode the output sequence of encoded spatiotemporal blocks into pixel space to produce an output video.
[0306] Aspect 2: The visual media generative response engine of Aspect 1, in which the diffusion-transformer processing unit is a graphics processing unit.
[0307] Aspect 3: A visual media generative response engine for any of Aspects 1-2, wherein the visual media generative response engine is trained on videos and images of different durations, resolutions, and aspect ratios.
[0308] Aspect 4: A visual media generative response engine for any of Aspects 1-3, wherein the visual media generative response engine is trained on videos and images at their native resolution and aspect ratio.
[0309] Aspect 5: A visual media generative response engine for any of Aspects 1-4, wherein the visual media generative response engine can output videos of different durations, aspect ratios and resolutions.
[0310] Aspect 6: A visual media generative response engine for any of Aspects 1-5, wherein spatiotemporal blocks constitute a single frame when a prompt requests an image as output.
[0311] Aspect 7: A visual media generative response engine of any of Aspects 1-6, wherein the visual media generative response engine is trained by providing at least one frame of input noise block and a training cue describing the image and by rewarding accurate prediction of the original block, wherein the training cue is visual media, and when the training cue is a text cue, the text cue is a detailed description of the original block, wherein the text cue is automatically generated from an image-to-text captioning model trained to create a detailed description of the original block from the original block and / or a short cue describing the original block; and a block decoder layer maps the blocks generated in the latent representation to pixel space.
[0312] Aspect 8: A visual media generative response engine according to any of Aspects 1-7, wherein the visual media generative response engine is trained by providing contrastive language-image pre-training (CLIP) embeddings interwoven within the spatiotemporal slices constituting the video frame to the diffusion-transformer layer.
[0313] Aspect 9: A method for generating visual media of different durations, resolutions, and aspect ratios using a visual media generative response engine, the method comprising: receiving an input cue from the visual media generative response engine at an input layer of the visual media generative response engine, wherein the input cue is text and may further include duration, resolution, and / or aspect ratio specifications; and outputting a video from the visual media generative response engine in response to the input cue, wherein the video matches the duration, resolution, and / or aspect ratio specifications.
[0314] Aspect 10: The method of aspect 9, wherein input prompts include visual media and text.
[0315] Aspect 11: The method of any one of Aspects 9-10 further includes: providing at least one noisy frame corresponding to a duration, resolution, and / or aspect ratio specification by an input layer as input supplementing the input cue; compressing at least one noisy frame and at least one frame of any cue video or image frame into a corresponding representation in a lower-dimensional latent space by a visual encoder layer; and decomposing the corresponding representation in the lower-dimensional latent space into spatiotemporal blocks representing at least one frame by a vector representation layer.
[0316] Aspect 12: A method of any one of Aspects 9-11, wherein providing at least one noise frame comprises controlling the resolution and / or aspect ratio of the generated video by arranging randomly initialized blocks in a grid of appropriate size to generate at least one noise frame, wherein the visual media generative response engine is configured to convert at least one noise frame into a full image frame having the same aspect ratio and resolution as at least one noise frame.
[0317] Aspect 13: The method of any one of Aspects 9-12 further includes: generating a prompt embedding from the text of the input prompt by an embedding layer.
[0318] Aspect 14: The method of any one of aspects 9-13 further includes: generating contrastive language-image pre-trained (CLIP) embeddings from one or more frames of the prompt by an embedding layer.
[0319] Aspect 15: The method of any one of Aspects 9-14 further includes: processing the spatiotemporal block by a diffusion-transformer layer based on cue embedding.
[0320] Aspect 16: A method of any one of Aspects 9-15, wherein the diffusion-transformer layer includes diffusion-transformer processing units that process respective blocks in the spatiotemporal block in parallel.
[0321] Aspect 17: The method of any one of Aspects 9-16 further includes: communicating with the attention head of other diffusion-converter processing units to perform a self-attention operation that notifies the respective diffusion-converter processing unit of information relating to other spatiotemporal blocks being processed by the other diffusion-converter processing units.
[0322] Aspect 18: A method of any of Aspects 9-17, wherein the duration is zero seconds and the video in response to the input prompt is a still image.
[0323] Aspect 19: A method of any one of Aspects 9-18, wherein the input prompt includes an image as visual media, and the text instruction generates a video from the image, and the video in response to the input prompt includes the image as part of the video in response to the input prompt.
[0324] Aspect 20: A method of any one of Aspects 9-19, wherein the input cue includes a cue video as visual media, and a textual instruction is given to generate an extended video in a forward or backward dimension from the cue video, and the extended video in response to the input cue includes a cue video with additional frames.
[0325] Aspect 21: A method of any one of Aspects 9-20, wherein the input prompt includes at least two prompt videos as visual media, and a text instruction is given to create a merged video that merges from the first of the at least two prompt videos to the second of the at least two prompt videos, and the merged video in response to the input prompt includes some aspects of the at least two prompt videos.
[0326] Aspect 22: A method of any one of Aspects 9-21, wherein the input prompt includes a prompt video as a visual medium, and a textual instruction to modify the prompt video is provided, and the video in response to the input prompt includes a prompt video as indicated by the input prompt for modification.
[0327] Aspect 23: A method for training a visual media generative response engine, the method comprising: initially training a diffusion-transformer layer of the visual media generative response engine by providing a training set of visual media, wherein the visual media have different durations, resolutions and / or aspect ratios, wherein the visual media are maintained in their native durations, resolutions and / or aspect ratios.
[0328] Aspect 24: The method of aspect 23 further includes: performing reinforcement learning by providing at least one noisy frame and training cues describing existing visual media in a fine-tuning training set, wherein the training cues are textual cues describing a detailed description of the existing visual media; and rewarding the diffusion-transformer layer for accurate predictions of frames constituting the existing visual media.
[0329] Aspect 25: A method of any one of Aspects 23-24, wherein fine-tuning of the visual media generative response engine further comprises: selecting sample frames of visual media from a fine-tuning training set; generating corresponding embedded frames from the sample frames; providing the corresponding embedded frames in place of at least one of the noise frames, wherein the diffusion-transformer layer of the visual media generative response engine receives a plurality of frames including the noise frames and the corresponding embedded frames interleaved with the noise frames, wherein the diffusion-transformer layer learns to predict the noise frames consistent with the corresponding embedded frames.
[0330] Aspect 26: A method of any one of Aspects 23-25, wherein the corresponding embedded frames are generated using the contrastive language-image pre-training (CLIP) technique.
[0331] Aspect 27: A method of any of Aspects 23-26, wherein training cues are generated from an image-to-text captioning model trained to create detailed descriptions of existing visual media.
[0332] Aspect 28: A method for generating a video including at least one frame of input visual media, the method comprising: receiving an input cue including text and the input visual media by an input layer; providing at least one frame of the input visual media to a diffusion-transformer layer of a visual media generative response engine in the form of an embedded or encoded frame, and providing an embedding of text to the diffusion-transformer layer; generating a video by the visual media generative response engine based on at least one frame of the input visual media and additional frames generated by the visual media generative response engine based on the text in the input cue, wherein the additional frames generated by the visual media generative response engine demonstrate spatial and temporal consistency with at least one frame of the input visual media.
[0333] Aspect 29: The method of aspect 28 further includes: determining an intent from text by an input layer, wherein the intent is to generate a video, wherein the video begins or ends at a portion of the input visual media, or wherein the intent is to generate a video based on features represented in the input visual media.
[0334] Aspect 30: A method of any one of Aspects 28-29, wherein the intent is to generate a video, wherein the video begins or ends at a portion of an input visual medium, the method further comprising: encoding at least one frame of the input visual medium to be included in the video into a latent representation that is compressed both temporally and spatially; creating an embedding from text of an input cue; providing a spatiotemporal block to a diffusion-transformer layer, wherein a portion of the spatiotemporal block is created from the latent representation and embedding of the at least one frame of the input visual medium to be included in the video; and receiving an output sequence of encoded spatiotemporal blocks, the output sequence comprising the spatiotemporal block created from the latent representation of the at least one frame of the input visual medium to be included in the video and additional encoded spatiotemporal blocks corresponding to the text from the input cue, thereby generating a video comprising the at least one frame from the input visual medium and the generated frame.
[0335] Aspect 31: A method of any one of Aspects 28-30, wherein the generated video begins with at least one frame from the input visual medium and continues with the generated frames.
[0336] Aspect 32: A method of any one of aspects 28-31, wherein the generated video ends at at least one frame from the input visual medium and begins at the generated frame.
[0337] Aspect 33: A method of any one of Aspects 28-32, wherein the intent is to generate a video based on features represented in an input visual medium, the method further comprising: creating a visual embedding of at least one frame describing the input visual medium; creating a text embedding from text of an input cue; and providing a spatiotemporal block to a diffusion-transformer layer, wherein at least one of the visual embedding and the text embedding of an intra-frame interleaved in the spatiotemporal block, wherein the embedding describes features represented in the input visual medium and a desired video based on the features represented in the input visual medium, and thereby guides the generation of a video in response to the cue.
[0338] Aspect 34: The method of any one of Aspects 28-33, wherein the generated video includes frames having features represented in a newly presented scene or style in the input visual media.
[0339] Aspect 35: A method of any one of Aspects 28-34, wherein the input visual medium includes at least one frame of the input visual medium and at least a second frame of the second input visual medium, and wherein the output sequence of encoded spatiotemporal blocks includes: a spatiotemporal block created from a potential representation of at least one frame of the input visual medium to be included in the video, a spatiotemporal block created from a potential representation of at least a second frame of the second input visual medium to be included in the video, and an additional encoded spatiotemporal block corresponding to text from an input prompt, wherein the video thereby generated begins with a video including at least one frame of the input visual medium and ends with a video including at least one frame of the second input visual medium.
[0340] Aspect 36: A method for requesting visual media from a visual media generative response engine, the method comprising: sending a prompt from the visual media generative response engine by a third-party application, wherein the prompt is provided to the visual media generative response engine using an application programming interface (API) provided by the visual media generative response engine, wherein the API defines a text prompt field for providing a text value describing visual media to be generated by the visual media generative response engine; and receiving output visual media in response to the prompt by the third-party application.
[0341] Aspect 37: The method of Aspect 36, wherein the text hint field defines a subfield of the text value and an associated subfield of the timestamp associated with the text value.
[0342] Aspect 38: A method of any of Aspects 36-37, wherein the text hint field allows multiple text hints and associated timestamps.
[0343] Aspect 39: A method of any one of Aspects 36-38, wherein the API defines a visual media cue field, which further defines subfields from the visual media cue and a subfield of an associated timestamp, the method comprising: sending a visual media cue by a third-party application to be used by the generative response engine when generating output visual media.
[0344] Aspect 40: A method of any one of Aspects 36-39, wherein the API is defined as a field specifying duration, resolution, and aspect ratio of the output visual media, the method further comprising: a third-party application sending values of duration, resolution, and aspect ratio of the requested output visual media; and a third-party application receiving visual media generated in response to the values of duration, resolution, and aspect ratio.
[0345] Aspect 41: A method for creating a new visual media from an existing visual media cue, the method comprising: receiving, via a user interface of a visual media generative response engine, a selection of existing visual media previously generated using the visual media generative response engine, wherein the existing visual media may be presented in a public gallery or a private gallery; in response to the selection, presenting the selected existing visual media via the user interface, wherein the selected existing visual media is presented with an example cue, wherein the example cue is a cue for creating the visual media, wherein the example cue is a machine-generated description of a specific frame of the visual media; receiving, via the user interface, an edit of the example cue; and receiving, via the user interface, instructions to generate the new visual media based on the edit of the example cue.
[0346] Aspect 42: The method of aspect 41, wherein the required input details, which are not provided by the user interacting with the user interface, are copied from existing visual media.
[0347] Aspect 43: The method of any one of aspects 41-42 further includes: presenting a storyboard user interface pre-filled with text prompts and / or one or more frames describing existing visual media.
[0348] Aspect 44: The method of any one of aspects 41-43 further includes: using a visual media generative response engine to generate new visual media based on editing of example prompts; and presenting the new visual media generated using the visual media generative response engine.
[0349] Aspect 45: A method comprising: presenting a graphical user interface including a prompt editor; receiving at least one text input to the prompt editor, wherein the text input is a prompt describing visual media to be generated by a visual media generative responsive engine; receiving a command to generate visual media using at least one text input in the prompt editor; determining at least one aspect ratio or resolution for generating the visual media before generating the visual media, wherein the visual media generative responsive engine is capable of generating visual media at multiple resolutions and / or aspect ratios; and receiving visual media generated based on the prompt, the visual media being generated at the determined aspect ratio or resolution.
[0350] Aspect 46: The method of aspect 45, wherein the graphical user interface further includes at least one of aspect ratio input control or resolution input control, wherein the determination of at least one of aspect ratio or resolution used in generating the visual media is based on explicit input provided using the aspect ratio input control or resolution input control.
[0351] Aspect 47: The method of any one of aspects 45-46, wherein at least one of the aspect ratio or resolution used to generate the visual media is determined based on inference derived from the text input.
[0352] Aspect 48: A method of any one of aspects 45-47, wherein the graphical user interface includes a visual media upload button that enables input visual media to be included as part of the prompt.
[0353] Aspect 49: The method of any one of aspects 45-48, wherein the graphical user interface further includes a duration input control, wherein the duration input control effectively controls the duration of the output visual media.
[0354] Aspect 50: A method of any one of Aspects 45-49, wherein the graphical user interface further includes an algebraic input control, wherein the algebraic input control effectively causes the visual media generative response engine to generate more than one output visual media.
[0355] Aspect 51: A method of any one of Aspects 45-51, wherein the graphical user interface includes a prompt enhancement option, wherein, when selected, the prompt enhancement option effectively inputs a prompt into a language model to generate an enhanced prompt and replaces the prompt with the enhanced prompt in a prompt editor.
[0356] Aspect 52: A method for assisting a user in generating prompts for a visual media generative response engine via a storyboard user interface, the method comprising: presenting a storyboard user interface, wherein the storyboard user interface includes a visual media timeline and a representation of a first frame located on the visual media timeline; and receiving, from the storyboard user interface, a prompt associated with the first frame, wherein the prompt may include text or visual media.
[0357] Aspect 53: The method of aspect 52 further includes: receiving a command to generate output visual media by a visual media generative response engine, wherein a cue associated with a first frame is provided to the visual media generative response engine together with a first timestamp indicating that the cue describes the desired output first frame at the first timestamp; and receiving output visual media having a frame corresponding to the cue at the first timestamp.
[0358] Aspect 54: The method of any one of aspects 52-53 further includes: presenting an add frame button in a storyboard user interface, wherein the add frame button effectively causes a representation of the second frame to be presented on the visual media timeline; and receiving a second prompt to be associated with the second frame from the storyboard user interface.
[0359] Aspect 55: The method of any one of aspects 52-54 further includes: receiving a move command via a storyboard user interface; and, in response to the move command, moving a first frame at a first timestamp to a second timestamp.
[0360] Aspect 56: A method for inducing a visual media generative response engine to reconstruct existing visual media, the method comprising: presenting a remix intensity control menu, wherein the remix intensity control menu includes options for the degree of remixing of the existing visual media; receiving a selection of a first remix degree from the remix intensity control menu; and receiving an output visual media as a reconstructed version of the existing visual media, wherein the amount of remixing of the existing visual media is controlled by the first remix degree.
[0361] Aspect 57: The method of aspect 56 further includes: in response to receiving a first remixing degree, requesting a narrative from a language model, wherein the prompts to the language model include existing visual media or text titles of existing visual media, having instructions to generate a narrative based on existing visual media but varying according to the first remixing degree.
[0362] Aspect 58: A method for influencing the fusion of a first visual media and a second visual media, the method comprising: receiving a first input visual media and a second input visual media by a prompt editor; presenting a fusion interface, wherein the fusion interface displays at least one frame of the first input visual media and at least one frame of the second input visual media; and adjusting a fusion curve in response to user input, wherein the fusion curve represents the influence of the first input visual media relative to the second input visual media over time.
[0363] Aspect 59: The method of aspect 58, wherein the user input is the selection of a pre-configured fusion curve, wherein the pre-configured fusion curve includes a transition, a blend, or a sample fusion curve, wherein the transition results in a transition from a first input visual media to a second visual media, the blend merges some aspects of the first input media with some aspects of the second visual media, and the sample affects the first visual media with elements of the second visual media.
[0364] Aspect 60: A method of any of Aspects 58-59, wherein the user input is to manipulate the blend curve to create a custom blend curve.
[0365] Aspect 61: A method for reviewing visual media generated by a visual media generative response engine, the method comprising: evaluating a cue by a review system to classify the cue as a violation of a review policy, wherein the review system includes a language model configured to recognize cue requesting visual media output by the visual media generative response engine that would violate the review policy; and using an image classifier to evaluate instances of visual media generated by the visual media generative response engine to determine when the output violates the review policy; and refusing to provide visual media from the visual media generative response engine when the cue or output violates the review policy.
[0366] Aspect 62: The method of aspect 61, wherein the language model configured to identify visual media prompts that will violate the moderation policy when the request visual media generative response engine outputs are multiple instances of a language model configured to identify specific aspects of the text prompts that will violate the moderation policy.
[0367] Aspect 63: The method of any one of Aspects 61-62, wherein the evaluation of the output further includes: determining that the image classifier has identified a strictly prohibited content category represented in a visual media instance generated by the visual media generative response engine, wherein the moderation system does not provide visual media instances generated by the visual media generative response engine.
[0368] Aspect 64: A method of any one of Aspects 61-63, wherein the evaluation of the output further includes: determining that the image classifier has identified the category of conditionally prohibited content represented in the output; determining that the cue used to generate an instance of visual media generated by the visual media generative response engine is a text cue; and providing the user with an instance of visual media generated by the visual media generative response engine, since the content was not prohibited when the conditionally prohibited content was generated from the text cue.
[0369] Aspect 65: A method of any one of Aspects 61-64, wherein the evaluation of the output further comprises: determining that the image classifier has identified the conditionally prohibited content category represented in the output; determining that the cue used to generate the content is a visual cue, wherein the moderation system does not provide an instance of visual media generated by the visual media generative response engine because the condition based on the instance of visual media generated by the visual media generative response engine is based on the conditions generated by the visual cue, and the conditionally prohibited content category is prohibited.
[0370] Aspect 66: The method of any one of aspects 61-65 further includes: evaluating the text portion of the prompt against a block list to identify prohibited words before the prompt is evaluated by a language model.
[0371] Aspect 67: A method of any one of aspects 61-66, wherein the language model used to evaluate the prompt is a multimodal language model, which can evaluate both the pictorial and textual components of the prompt.
[0372] Aspect 68: The method of any one of Aspects 61-67 further includes: after evaluating the prompt to classify the prompt as a violation of the review policy, when the violation of the review policy is a text portion of the prompt, rewriting the prompt that violates the review policy to conform to the review policy.
[0373] Aspect 69: The method of any one of aspects 61-68 further includes: determining that the image portion of the prompt violates the review policy, wherein the prompt is rejected.
[0374] Aspect 70: The method of any one of aspects 61-69 further includes: identifying a specific aspect of the text portion of the prompt that violates the review policy that it cannot be rewritten.
Claims
1. A visual media generative response engine, comprising: An input layer configured to receive cues, the input layer being further configured to output at least one noise frame; A vector representation layer, wherein the vector representation layer is configured to decompose the at least one noise frame into spatiotemporal blocks; An embedding layer configured to create a prompt embedding from at least a portion of the prompt; A diffusion-transformer layer is configured to embed the spatiotemporal block according to the cues to iteratively remove noise from the spatiotemporal block, thereby producing a processed spatiotemporal block. The diffusion-transformer layer includes diffusion-transformer processing units that process respective spatiotemporal blocks in parallel. Each diffusion-transformer processing unit has an associated attention head configured to communicate with the attention heads of other diffusion-transformer processing units to perform a self-attention operation, which informs the respective diffusion-transformer processing unit of information relating to other spatiotemporal blocks being processed by the other diffusion-transformer processing units. and A block decoder layer is configured to decode the processed spatiotemporal blocks into pixel space to produce output visual media.
2. The visual media generative response engine according to claim 1, further comprising: A visual encoder layer is configured to compress the at least one noisy frame, and when the cue includes a cue frame, the visual encoder layer is also configured to compress the cue frame into a corresponding representation in a lower-dimensional latent space.
3. The visual media generative response engine according to claim 1, further comprising: A linearization layer is configured to linearize the processed spatiotemporal block from the corresponding diffusion-transformer processing unit into an output sequence of encoded spatiotemporal blocks.
4. The visual media generative response engine according to claim 2, wherein, The visual encoder layer is a learning compression model that reduces the dimensionality of visual data. The visual encoder layer is configured to take the original visual media as input and output a latent representation that is compressed both temporally and spatially.
5. The visual media generative response engine according to claim 1, wherein, The cue includes at least one cue frame and cue text, wherein the visual media generative response engine is trained by providing the diffusion-transformer layer with spatiotemporally interwoven embeddings of the cue text.
6. The visual media generative response engine according to claim 1, wherein, The visual media generative response engine is trained by providing at least one frame of input noise block and training cues describing the image, and by rewarding the diffusion-transformer layer with accurate predictions of the original block.
7. The visual media generative response engine according to claim 1, further comprising: A graphical user interface including a prompt editor, wherein the prompt editor is configured to receive the prompts, wherein the prompt description is output visual media generated by the visual media generative response engine; The input layer is configured to determine at least one of the aspect ratio or resolution used to generate the output visual media, wherein the visual media generative response engine is capable of generating the output visual media at multiple resolutions and aspect ratios. The generated visual media is generated with a predetermined aspect ratio or resolution.
8. The visual media generative response engine according to claim 7, wherein, The graphical user interface further includes at least one of an aspect ratio input control or a resolution input control, wherein the determination of at least one of the aspect ratio or resolution used to generate the output visual media is based on explicit input provided using the aspect ratio input control or the resolution input control.
9. The visual media generative response engine according to claim 1, further comprising: An auditing system, wherein the auditing system is configured to evaluate the prompt to classify the prompt or generated visual media as violating or not violating the auditing policy using a language model and an image classifier, and to refuse to provide visual media from the visual media generative response engine when the prompt or the output visual media violates the auditing policy.
10. The visual media generative response engine according to claim 9, wherein, The evaluation of the output visual media further includes determining that the image classifier has identified a strictly prohibited content category represented in the output visual media, and that the review system blocks the transmission of the output visual media.
11. The visual media generative response engine according to claim 9, wherein, The evaluation of the output visual media further includes determining that the image classifier has identified the conditionally prohibited content category represented in the output, and verifying the output visual media when it is generated based on the text portion of the prompt.
12. A method for generating visual media using a visual media generative response engine, comprising: The input layer receives the prompts and outputs at least one noise frame; The vector representation layer decomposes the at least one noisy frame into spatiotemporal blocks; A prompt embedding is created by the embedding layer from at least a portion of the prompt; The spatiotemporal block is embedded and processed by the diffusion-transformer layer according to the cues to iteratively remove noise from the spatiotemporal block, thereby producing a processed spatiotemporal block. The diffusion-transformer layer includes diffusion-transformer processing units that process corresponding spatiotemporal blocks in parallel. Each diffusion-transformer processing unit has an associated attention head configured to communicate with the attention heads of other diffusion-transformer processing units to perform a self-attention operation, which informs the corresponding diffusion-transformer processing unit of information related to other spatiotemporal blocks being processed by the other diffusion-transformer processing units. as well as The processed spatiotemporal blocks are decoded into pixel space by the block decoder layer to produce the output visual media.
13. The method of claim 12, further comprising: The visual encoder layer compresses the at least one noisy frame, and when the cue includes a cue frame, the visual encoder layer is configured to further compress the cue frame into a corresponding representation in a lower-dimensional latent space.
14. The method according to claim 12, wherein, The cue includes at least one cue frame and cue text, wherein the visual media generative response engine is trained by providing the embedding of the cue text interwoven with the spatiotemporal block to the diffusion-transformer layer.
15. The method according to claim 12, wherein, The visual media generative response engine is trained by providing an input noise block of at least one frame and training cues describing the image, and the diffusion-transformer layer is rewarded for accurate prediction of the original block.
16. The method according to claim 12, in, The prompt editor is configured to receive the prompt, wherein the prompt description will be visual media generated by the visual media generative response engine; The input layer is configured to determine at least one of the aspect ratio or resolution used to generate the visual media, wherein the visual media generative response engine is capable of generating the visual media at multiple resolutions and aspect ratios. The output visual media is generated with a determined aspect ratio or resolution.
17. The method according to claim 16, wherein, Determine at least one of the aspect ratio or resolution used to generate the visual media based on explicit input provided using the aspect ratio input control or resolution input control in the prompt editor.
18. The method of claim 12, further comprising: The prompt is evaluated by the review system to classify the prompt or generated visual media as violating or not violating the review policy using a language model and an image classifier, and when the prompt or the output visual media violates the review policy, visual media is refused to be provided from the visual media generative response engine.
19. The method of claim 18, further comprising: Determine that the image classifier has identified strictly prohibited content categories represented in the output visual media; as well as Prevent the transmission of generated visual media.
20. The method of claim 18, further comprising: Determine that the image classifier has identified the category of conditionally prohibited content represented in the output visual media; as well as The output visual media is provided when the output visual media is generated based on the text in the prompt.
