Adaptive video compression using generative machine learning

Abstract

Various embodiments of the technology described herein relate to compression of video data, including selecting a pivot image from a video including a plurality of images, and causing a first machine learning model to generate a descriptor of the pivot image, where the descriptor includes a language description associated with the pivot image. In one example, pivot images and descriptors are provided to a decoder for reconstruction of a video. In an embodiment, the decoder includes a generative machine learning model that takes the pivot image and the descriptor as inputs. The decoder uses the pivot image to generate an image based at least in part on the descriptor. The image is combined with other images generated by the generative machine learning model to reconstruct the video.

Description

Background Technology

[0001] A significant portion of internet traffic includes streaming media, such as video and other graphical content. Furthermore, video is a common and universal media format, used in a variety of scenarios such as education, entertainment, art, and news. However, video streaming and transmission (e.g., over the internet) are costly and challenging, consuming substantial computing resources, network bandwidth, and time. Moreover, consumers of streaming video have high expectations for latency and video fidelity, which can be difficult to achieve with high network traffic competing for computing resources. Thus, video compression technology can benefit both producers and consumers of video content. Summary of the Invention

[0002] This summary is provided to introduce some concepts in a simplified form, which will be further elaborated in the detailed description below. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used alone to help determine the scope of the claimed subject matter.

[0003] Embodiments of the techniques described herein relate to video compression techniques that utilize generative machine learning and pivot images extracted from the video to preserve key concepts while reducing file size and maintaining video fidelity. Embodiments of the techniques described herein use generative machine learning models to reconstruct the video (e.g., generate a reconstructed video) based at least in part on a set of pivot images extracted from the original video and corresponding pivot image descriptors. Embodiments of the generative machine learning models take one or more pivot images (including frames and descriptors extracted from the video) as input and generate a set of images that can be combined to reconstruct the video.

[0004] In illustrative examples, the encoder generates compressed video data by extracting at least a pivot image from the video and generating a pivot image descriptor. In embodiments, the pivot image is selected based on an algorithm. For example, frames may be sampled from the video at time intervals (e.g., 10 milliseconds) or based on multiple frames (e.g., 20 frames). In other examples, the pivot image is selected at least in part based on the content of the video frame. Embodiments of the encoder include an object detection model (e.g., a neural network) that detects objects and / or background within video frames and selects the pivot image at least in part based on the detected objects and / or background. In one illustrative example, a change in the number of objects detected by the object detection model between consecutive frames of the video causes the encoder to select that frame as the pivot image. In this embodiment, the encoder (e.g., using an object detection model or other machine learning model) analyzes the video frames and extracts key concepts within the video as pivot images.

[0005] Furthermore, in various embodiments, the encoder includes a large language model (LLM) or other natural language model to generate descriptors for the pivot images. For example, the descriptors may include natural language descriptions of objects, background, interactions, and concepts included in the pivot images. In various embodiments, the descriptors for the pivot images, together with the pivot images, represent compressed video and may be provided to the decoder to reconstruct the video. Additionally, in various embodiments, the compression of the video can be adjusted (e.g., via user input) by modifying the number of pivot images and / or the length and number of descriptors.

[0006] Implementations of the decoder include one or more generative machine learning models (e.g., generative pre-trained transformers [GPT], Gaussian mixture models, and diffusion models) that generate video based on data provided by the encoder, such as a set of pivot images and descriptors. For example, a generative machine learning model included in the decoder takes a set of pivot images and their corresponding descriptors as input and generates a set of transition images and / or frames between consecutive pivot images to reconstruct the video. Furthermore, by implementing the decoder at the user device, in various embodiments, the amount of data for streaming video transmission can be reduced while maintaining video fidelity. Moreover, in such embodiments, coding metrics can be extended beyond pixel-level fidelity of the prior art. For example, since the generative model is capable of reconstructing video at the same or higher levels of fidelity, coding metrics can be used to evaluate or otherwise determine the extent to which the reconstructed video conveys the key concepts and / or ideas of the original content.

[0007] While some existing technologies allow for reduction in the size of video files and / or video data, the resulting videos have lower fidelity due to the limitations of compression, and still require a relatively large amount of computing resources to facilitate video streaming and / or transmission.

[0008] This disclosure provides one or more technical solutions that are effective in addressing various technical problems. For example, certain embodiments are effective in improving the compression rate of video data while maintaining the fidelity and conceptual information of the video. For example, instead of reducing the size of video data by eliminating redundant data, certain embodiments generate compressed video data including pivot images and descriptors, which can be used to reconstruct the video without losing any fidelity, while greatly reducing the amount and / or size of video data. Accordingly, one technical solution is to use a multimodal generative machine learning model to determine pivot images including key concepts and / or events in the video, generate natural language descriptors for the pivot images, and reconstruct the video. Consequently, the amount of network traffic required for streaming video services and / or video transmission is reduced, allowing these computational resources to be used for other tasks. For example, instead of transmitting a 1 megabyte (MB) compressed video stream to a viewer over the network, only a relatively small amount of pivot images and descriptors (e.g., text) of 45 kilobytes (KB) are transmitted. Attached Figure Description

[0009] This disclosure will now be described in detail with reference to the accompanying drawings, in which:

[0010] Figure 1 This is a block diagram of an example operating environment suitable for the implementation of this disclosure;

[0011] Figure 2 This is a block diagram of an example system according to embodiments of the present disclosure, including an encoder for generating compressed video data used by a decoder to reconstruct video.

[0012] Figure 3 This is a flowchart illustrating the generation and display of compressed video data according to embodiments of this disclosure;

[0013] Figure 4 This is a flowchart for generating compressed video data according to embodiments of the present disclosure;

[0014] Figure 5 This is a flowchart illustrating the generation of video based on compressed video data according to embodiments of the present disclosure;

[0015] Figure 6 This is a block diagram of a language model that makes specific predictions using specific inputs according to embodiments of the present disclosure;

[0016] Figure 7 This is a block diagram of an example computing environment suitable for use in implementing embodiments of the present disclosure; and

[0017] Figure 8 This is a block diagram of an example computing environment suitable for use in implementing embodiments of the present disclosure. Detailed Implementation

[0018] To meet legal requirements, the subject matter of various aspects of this disclosure has been specifically described herein. However, this description itself is not intended to limit the scope of this patent. Rather, the subject matter claimed in this disclosure may also be embodied in other ways, including different steps or combinations of steps similar to those described herein, in combination with other existing or future technologies. Furthermore, although the terms “step” and / or “box” may be used herein to imply different elements of the methods employed, these terms should not be construed as implying any particular order between the steps disclosed herein unless the order of the steps is explicitly described. Each method described herein may include a computational process that can be performed using any combination of hardware, firmware, and / or software. For example, various functions may be performed by a processor executing instructions stored in memory. Methods may also be embodied as computer-usable instructions stored on a computer storage medium. Methods may be provided by a standalone application, service, or managed service (independent of or in combination with another managed service), or a plug-in to another product, etc.

[0019] The various embodiments discussed herein relate to generating compressed video data including pivot images and descriptors that can be used as input to a generative machine learning model to reconstruct the video data. For example, an encoder can extract pivot images from a video and (e.g., using a multimodal generative model, such as a generative pre-trained transformer [GPT]) generate natural language descriptors for the pivot images. Continuing this example, a decoder takes the pivot images and descriptors and enables a generative machine learning model to generate a set of images based on the pivot images and descriptors, which are then combined to reconstruct the video or otherwise generate the video. In this way, key conceptual elements of the video are preserved and generated by the decoder or its components (e.g., the generative machine learning model), while the amount of data transferred between the encoder and decoder is reduced.

[0020] Generally, compression techniques rely on eliminating or reducing redundant data and are limited by various constraints, including video and audio fidelity. Furthermore, the focus on maintaining fidelity limits the effectiveness of compression techniques. Moreover, video compression is necessary for sharing or otherwise transmitting video over the internet, as it reduces the amount of data required to stream or send video to viewers, while network bandwidth is a finite resource. One way to address this problem is by using video encoding formats (e.g., video compression formats), which are content representation formats used to store or transmit digital video content (e.g., in data files or bitstreams). Typically, these formats use video compression algorithms such as Discrete Cosine Transform (DCT) coding or motion compensation.

[0021] However, these video compression techniques degrade the quality of the original video, resulting in visual artifacts such as blockiness, pixelation, blurring, and ringing. Furthermore, these artifacts affect the viewing experience and the accuracy of video analysis. In addition, video compression increases the computational resource usage (e.g., processor utilization) required for video encoding and decoding, which reduces the performance and efficiency of the devices involved in video encoding and decoding. Video compression can also introduce errors or distortions during transmission over networks such as the Internet due to data loss, corrupted video frames, and / or synchronization issues between audio and video. Finally, the motion complexity and / or texture of the video content degrades the efficiency and quality of compressed video.

[0022] In light of this, the embodiments discussed herein provide a technical solution to the shortcomings and limitations of existing technologies associated with video compression. In one embodiment, a generative machine learning model reconstructs or otherwise generates a video based on data that preserves key concepts and / or elements of the original video. In one embodiment, encoding of the compressed video is performed by selecting at least a subset of the video's frames (e.g., pivot images) and causing a machine learning model to generate natural language descriptors for the video frames between that subset and / or the subset of frames. In one embodiment, a quality metric of the compressed video and / or the video displayed to a viewer (e.g., a video generated by a decoder) assesses the preservation of video concepts and / or elements. In one example, the determination of the quality metric is based on the amount or degree to which the reconstructed video conveys ideas and / or maintains the narrative coherence of the original video.

[0023] More specifically, video compression tools include encoders that generate compressed video (e.g., data files, data objects, data streams, etc.) based on the video. This compressed video is then reconstructed by a decoder executed by a user device for display to a viewer. In one example, a generative machine learning model is used to generate a descriptor for a pivot image extracted from the video, and the video is reconstructed based on this descriptor and the pivot image. As used herein, a “generative machine learning model” refers to a machine learning model of various types and / or combinations that generates text, images, or other data based on input. Example generative machine learning models include LLMs (e.g., GPT-4, LLaMA-2, Bard) and diffusion models (e.g., DALL-E2, StableDiffusion, and Midjourney). In one example, an encoder extracts frames from a video and uses a generative machine learning model to generate natural language descriptions of the frames, natural language descriptions of objects within the frames, and / or natural language descriptions of inter-frame relationships.

[0024] For clarity, assume the video to be compressed consists of a static background and a soccer ball moving along a path on that background. In this example, the encoder extracts every ten frames of the video and causes a first generative machine learning model to generate descriptors (e.g., natural language descriptions of the soccer ball and its position) for each extracted frame. In this example, the extracted frames are selected based on time intervals (e.g., every ten frames), although other algorithms can be used to select the video frames to be extracted, as described in more detail below. Continuing with this example, the extracted frames (e.g., pivot images) and their corresponding descriptors are provided to a decoder, which reconstructs the original video based on the extracted frames and descriptors. For example, a second machine learning model modifies the extracted frames based on the descriptors to depict the position of the soccer ball moving along a path on the background.

[0025] In one embodiment, the encoder includes one or more machine learning models that take video as input, extract pivot images, and generate descriptors. In one example, a first machine learning model detects changes between video frames (e.g., changes to objects between video frames, changes to background or location within the video, changes to the concept or tone conveyed by the video, etc.) and selects a pivot image at least in part based on the detected changes. Returning to the example above, the first machine learning model selects a specific frame as the pivot image based on the detection of a second soccer ball. Continuing with this example, the first machine learning model or another machine learning model can then generate a descriptor for that change. For example, "The second soccer ball appears at pixel location (x:240, y:-123)". In other embodiments, a large language model (LLM) generates descriptions of the video frames, and the selection of pivot images is performed based on the descriptions generated by the LLM. For example, a pivot image is selected based on a new object in a specific frame as described by the LLM.

[0026] In one embodiment, the decoder includes one or more generative machine learning models that take pivot images (e.g., frames extracted from a video) and descriptors (e.g., natural language descriptions of the pivot images, frames not extracted from the video, other components of the video, and / or concepts, themes, or other information associated with the video) as input and generate a set of images. In one example, this set of images generated by the generative models is combined to reconstruct the original video compressed by the encoder. For example, by acquiring the compressed video at least from the encoder or other device (e.g., a server computer system running storage services of a computing resource service provider) and having the decoder reconstruct the video, a viewer can stream the video over a network (e.g., the Internet).

[0027] In one embodiment, the generative machine learning model included in the decoder generates the set of images by modifying the pivot image at least based on a descriptor. In one example, the pivot image includes an image of a flamingo standing in water, and the corresponding descriptor describes the flamingo taking off. Continuing with this example, based on this input, the generative model generates a set of images including the flamingo taking off. Furthermore, in various embodiments, the generative model generates the set of images by modifying the pivot image at least (e.g., adding noise to the pivot image and denoising the pivot image to generate new images). In other embodiments, the generative model generates entirely new images for the set of images. For example, the generative model uses the pivot image and the descriptor as a basis to generate frames for a reconstructed video. In some embodiments, a mask is used to remove objects from the pivot image, and a background is generated for the generative model and the objects to be modified by the generative model.

[0028] Specific embodiments that improve video compression performance include streaming video. This is because various embodiments implement techniques that use generative machine learning models to generate compressed video data and then reconstruct the original video based on that compressed video data. Compression techniques are typically limited by the need to maintain audio and video fidelity. Furthermore, compression techniques may introduce unwanted artifacts and / or errors into the compressed video data, requiring additional computational resources for reconstruction, and the amount of data that can be removed is also limited. A significantly more efficient alternative is to employ at least one generative model capable of reconstructing the video based on a relatively small subset of video frames (compared to the entire video and text data).

[0029] Some embodiments have the technical effect of reducing the computational resource consumption required to stream or otherwise transmit video data over a network. As discussed above, video data constitutes a large portion of internet traffic. However, compression techniques do little to reduce the amount of computational resources required for video streaming. As discussed herein, some embodiments allow for algorithm selection and / or pivot image extraction, which significantly reduces the amount of data that needs to be transmitted while maintaining video fidelity and video concept. In this way, streaming video over a network (such as the internet) requires fewer computational resources and less network bandwidth.

[0030] Additionally, some embodiments have the technical effect of improving coding quality metrics. Currently, compression metrics focus on pixel-level fidelity. With respect to some existing methods that allow video data to be compressed, the resulting video fidelity is lower than the original video. However, the embodiments described herein allow video data to be reconstructed with the same or higher fidelity. Therefore, by allowing the focus of the metric to be directed towards maintaining and / or conveying the conceptual elements of the original video in the reconstructed video, some embodiments have the technical effect of improving quality metrics.

[0031] TurningFigure 1 , Figure 1 This is a schematic diagram of an operating environment 100 in which one or more embodiments of the present disclosure may be implemented. It should be understood that the arrangements and other arrangements described herein are illustrated by way of example only. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, and functional groups, etc.) may be used in addition to or in lieu of the arrangements and elements shown, and some elements may be omitted entirely for clarity. Furthermore, many of the elements described herein are functional entities that may be implemented as discrete or distributed components, or may be implemented in combination with other components, and may be implemented in any suitable combination and location. The various functions performed by one or more entities described herein may be performed by hardware, firmware, and / or software. For example, some functions may be performed by a processor that executes instructions stored in memory, as referenced in [reference]. Figure 7 Further description.

[0032] It should be understood that Figure 1 The operating environment 100 shown is an example of a suitable operating environment. Among other components not shown, the operating environment 100 includes a user device 102, a video compression tool 104, a computing resource service provider 120, and a network 106. Figure 1 Each component shown can be implemented via any type of computing device, such as a combination Figure 7 One or more computing devices 700 are described. These components can communicate with each other via a network 106, which can be wireless, wired, or both. Network 106 may include multiple networks or networks of networks, but for the sake of simplicity, it is shown without obscuring aspects of this disclosure. By way of example, network 106 may include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks (such as the Internet), and / or one or more private networks. Network 106 includes wireless telecommunication networks, such as base stations, communication towers, or even access points, and components (and other components) may provide wireless connectivity. Networking environments are common in offices, enterprise-wide computer networks, intranets, and the Internet. Accordingly, network 106 is not described in detail.

[0033] It should be understood that, within the scope of this disclosure, any number of devices, servers, and other components may be employed within operating environment 100. Each component may include a single device or multiple devices collaborating in a distributed environment. For example, video compression tool 104 includes multiple server computer systems collaborating in a distributed environment to perform the operations described in this disclosure. In embodiments, video compression tool 104 is provided as a service of computing resource service provider 120 or otherwise implemented.

[0034] User equipment 102 can be any type of computing device capable of being operated by an entity (e.g., an individual or organization) and acquires data from video compression tool 104 and / or computing resource service provider 120 (e.g., from a data repository), which may be facilitated by computing resource service provider 120. In various embodiments, user equipment 102 may access or otherwise display video 126B. For example, application 108 includes a video streaming application that includes a decoder 128 that acquires compressed video data from video compression tool 104 and displays video 126B to one or more viewers.

[0035] In some implementations, user equipment 102 is combined with Figure 7 The type of computing device described. By way of example and not limitation, user device 102 may be embodied as a personal computer (PC), laptop computer, mobile device, smartphone, tablet computer, smartwatch, wearable computer, personal digital assistant (PDA), global positioning system (GPS) or device, video player, handheld communication device, gaming device or system, entertainment system, in-vehicle computer system, embedded system controller, remote control, home appliance, consumer electronics device, workstation, any combination of the above devices, or any other suitable device.

[0036] User equipment 102 may include one or more processors and one or more computer-readable media. The computer-readable media may also include computer-readable instructions executable by one or more processors. In an embodiment, the instructions are generated by one or more applications (such as...) Figure 1 The application 108 shown is used to illustrate this. For simplicity, application 108 is referred to as a single application, but in reality, its functionality can be implemented by one or more applications.

[0037] In various embodiments, application 108 includes any application capable of facilitating information exchange between user device 102 and video compression tool 104. For example, application 108 may operate as a user interface to a streaming service provided by a computing resource service provider. In some implementations, application 108 includes a web application that can run in a web browser and may be hosted at least partially on the server side of operating system environment 100. Additionally or alternatively, application 108 may include dedicated applications, such as those supported by user device 102 and decoder 128. In some cases, application 108 is integrated into the operating system (e.g., as a service). Therefore, the term "application" is contemplated broadly herein.

[0038] For example, in a cloud-based implementation, application 108 is used to interface with functionality implemented by video compression tool 104 and / or a computing resource service provider. In some embodiments, components or portions of video compression tool 104 are implemented on user equipment 102 or other systems or devices. Therefore, it should be understood that in some embodiments, application 108 is provided via multiple devices arranged in a distributed environment that collectively provide the functionality described herein. Additionally, other components, not shown, may also be included within the distributed environment.

[0039] In various embodiments, the computing resource service provider 120 includes a plurality of computing devices that provide a multi-tenant environment, wherein (e.g., user-operated) computing devices are provided with access to the computing resources of the computing resource service provider 120. In one example, the computing devices operated by the computing resource service provider 120 include combinations of... Figure 7 The described type of computing device. In other examples, the computing device operated by computing resource service provider 120 includes [devices / devices]. Figure 8 The described cloud computing architecture type. Furthermore, in this embodiment, the computing resource service provider 120 provides various services that can be used to access computing resources (e.g., server computer systems, network devices, storage devices, etc.). For example, the services provided by the computing resource service provider 120 include computing services, storage services, video streaming services, networking services, or other services that allow computing devices to access computing resources. In this embodiment, a video compression tool 104 is provided as a service of the computing resource service provider 120.

[0040] like Figure 1 As illustrated, video compression tool 104 generates compressed video data based on video 126A. In embodiments, video compression tool 104 and / or encoder 124 of video compression tool 104 use computing resources of computing resource service provider 120 to perform compression operation 132. In one example, compression operation 132 includes generating pivot image 142 and descriptor 144 based at least in part on video 126A. In embodiments, compression operation 132 includes other operations (such as eliminating redundant data, combining data, storing data, or other operations used to generate compressed video data). In one example, compression operation 132 includes generating a data object (e.g., a data file in an archive file format such as .ZIP) that includes pivot image 142 and descriptor 144.

[0041] In an embodiment, video 126A includes an electronic representation of motion visual images in the form of encoded digital data. In one example, video 126A includes a series of images that generate an electronic representation of motion visual images on a display device (e.g., the display device of user equipment 102) while being displayed rapidly in succession (e.g., 30 frames per second). In an embodiment, video 126A is captured and / or stored in an uncompressed format (e.g., High-Level Video Coding [H.264] or Moving Picture Experts Group-4 [MPEG-4]).

[0042] In various embodiments, video 126A is streamed or otherwise transmitted via network 106. For example, streaming video 126A is a process of transmitting video data in real-time or near real-time over the Internet, which allows a viewer (e.g., a user) to watch video 126A on user equipment 102 (e.g., without needing to download the entire video 126A before watching). In embodiments, video 126A is streamed via network 106 from a physical and / or virtual server operated by computing resource service provider 120 to user equipment 102 to deliver audio and video elements using various protocols such as Hypertext Transfer Protocol (HTTP), Transmission Control Protocol / Internet Protocol (TCP / IP), and / or Hypertext Markup Language (HTML)).

[0043] In an embodiment, video compression tool 104 compresses video 126A before streaming or otherwise transmitting the data to user equipment 102. For example, encoder 124 encodes or otherwise generates compressed video data based at least in part on video 126A being streamed to user equipment 102. As described herein, in an embodiment, compressed video data includes pivot image 142 and descriptor 144. Furthermore, in some embodiments, compressed video data may include additional data besides pivot image 142 and descriptor 144. For example, compressed video data may include data identifying the video source, video author, encoder 124, one or more machine learning models used to generate the compressed video data, or other data that may be used by decoder 128 to generate, display, or otherwise process the compressed video data.

[0044] In various embodiments, pivot image 142 comprises a subset of frames and / or images from video 126A. In one example, pivot image 142 is extracted from video 126A at least in part based on multiple frames. Continuing with this example, encoder 124 extracts every ten frames of video 126A. In other embodiments, one or more machine learning models are used to determine keyframes and / or conceptual elements of video 126A in order to determine the pivot image. In one example, an object detection model (e.g., Scale Invariant Feature Transform [SIFT], Convolutional Neural Network [CNN], Video Object Detection [VOD], Region-Based Convolutional Neural Network [R-CNN], Single Detector [SSD], Detection Transformer [DETR], etc.) is used to detect objects in images (e.g., frames) of video 126A.

[0045] Continuing this example, encoder 124 selects pivot image 142 based at least in part on objects detected in a specific image within video 126A. For example, encoder 124 selects that specific image as pivot image if the position of an object changes, the number of objects changes, the size or shape of an object changes, the color of an object changes, or any other modification is made to the objects within the image. In various embodiments, the outputs of object detection models from multiple images of video 126A are compared. For example, encoder 124 compares the outputs of object detection models for consecutive frames of video 126A to select pivot image 142. In another example, encoder 124 compares the outputs of object detection models for every ten frames of video 126A to select pivot image 142.

[0046] In other embodiments, a large language model (LLM) generates a description of the images in video 126A, and encoder 124 selects a pivot image based at least in part on the output of the LLM. In one example, images (e.g., frames) are extracted from video 126A and provided as input to the LLM. In such an embodiment, the LLM then outputs a natural language description of image 126. Continuing with this example, encoder 124 then selects a pivot image 142 based at least in part on the output of the LLM. For example, encoder 124 selects the pivot image based on the length of the output, the number of objects described in the output, the concepts described in the output, the actions described in the output, or other attributes of the output. In an embodiment, the output of the LLM is compared for multiple frames. For example, if the output of the LLM includes a description of a new object or is longer than a previous output, the encoder selects the corresponding frame as the pivot image. In other embodiments, the input to the LLM may include part or all of video 126A. For example, the LLM may generate descriptions of scenes or other discrete portions of the video.

[0047] In some embodiments, a combination of machine learning models and / or algorithms may be used to determine the pivot images 142. In one example, an object detection model may be used to process every ten frames of video 126A. In another example, if the object detection model detects a modification to an object within a video frame, an LLM is used to generate a description of that frame. Various combinations of machine learning models may be used to determine a set of pivot images 142 such that the concepts included in video 126A are captured by the pivot images 142.

[0048] In embodiments, descriptor 144 includes data that can be provided to decoder 128 or its components (e.g., generative machine learning model 130) to generate or otherwise reconstruct video 126B. In one example, descriptor 144 is generated at least in part based on video 126A and / or pivot image 142. In embodiments, an LLM or other machine learning model (e.g., GPT, bidirectional encoder representation from a transformer [BERT], CNN, etc.) takes video 126A and / or pivot image 142 as input and outputs a natural language description, which is used by encoder 124 to generate descriptor 144. For example, after extracting a particular pivot image, encoder 124 provides that particular pivot image as input to LLM, which then generates a cue (e.g., a natural language description) that guides the generative machine learning model of decoder 128 toward the image. In other examples, video 126A and pivot image 142 are provided to LLM, and LLM generates descriptor 144 at least in part based on frames of video 126A between pivot images 142. In a specific example, video 126A includes a background with two trees and a cat running between them; pivot image 142 includes a first image of the cat on the first tree and a second image of the cat on the second tree. Continuing with this specific example, descriptor 144 includes a description of the conceptual elements of video 126A (e.g., the cat running from the first tree to the second tree). In various embodiments, descriptor 144 is modified or otherwise used to generate a set of cues for generative model 130.

[0049] In various embodiments, descriptor 144 includes a natural language description of the corresponding pivot image. Furthermore, the level and / or amount of the generated description can be variable based on various aspects of the system, such as the required amount of compression, the method of selecting the pivot image, the type of generative machine learning model used, the number of generative machine learning models used, or other aspects of environment 100. In one example, descriptor 144 includes the position and / or motion of an object in pivot image 142. In another example, descriptor 144 includes a description of video 126A between consecutive pivot images. In embodiments, encoder 124 modifies the output from an LLM or other machine learning model to generate descriptor 144. For example, the natural language description of pivot image 142 generated by an LLM or other machine learning model is modified or otherwise used to generate a cue for a text-to-image model (e.g., DALL-E 2).

[0050] In various embodiments, the generative model 130 included in the decoder 128 takes the pivot image 142 and descriptor 144 as input and generates a plurality of images, at least in part, based on the input. These images are combined to generate video 126B (e.g., the plurality of images are used as frames of video 126B). In one example, the generative machine learning model 130 includes a neural network that takes images and natural language as input and generates images. In various embodiments, the generative machine learning model 130 is trained to modify or otherwise use the pivot image 142 to generate frames of video 126B by recreating or otherwise reconstructing video 126A by moving objects within the pivot image 142 at least based on descriptor 144. In this manner, the quality metric associated with the encoder 124 and / or decoder 128 includes a measurement of the similarity between the reconstructed video 126B and the original video 126A (e.g., the degree to which video 126B maintains the conceptual elements of video 126A).

[0051] See now Figure 2 The diagram depicts a block diagram of an example system 200 including an encoder 224 and a decoder 228, the encoder 224 generating compressed video data 208 at least partially based on video 226A, and the decoder 228 generating video 226B or otherwise reconstructing video 226B at least partially based on the compressed video data 208. The illustrated encoder 224 uses a machine learning model that extracts a pivot image 242 from video 226A and generates a descriptor 244. In one example, the pivot image 242 and the descriptor are included in the compressed video data 208. The illustrated decoder 228 includes a generative model 230 that takes the compressed video data as input and generates video 226B.

[0052] In some embodiments, video 226A is obtained from a streaming service or other service of computing resource service provider 220. In one example, video 226A includes previously recorded multimedia data (e.g., recorded audio and video). In another example, video 226A is streamed in real time or otherwise recorded (e.g., live video broadcasting). In embodiments, video 226A is obtained from a storage device via a network, such as from a storage service that transmits data over the Internet. In some embodiments, video 226A is maintained within the service of computing resource service provider 220. Furthermore, in some examples, computing resource service provider 220 provides encoder 224 and / or machine learning model 202 as a service.

[0053] In one embodiment, encoder 224 generates compressed video data 208. For example, using computing resources provided by computing resource service provider 220, encoder 224 extracts a set of pivot images 242 from video 226A. In one embodiment, pivot images 242 are selected from a subset of frames in video 226. For example, pivot images 242 are selected from every ten frames, every two frames, or other subsets of frames in the set of frames in video 226A. In other embodiments, machine learning model 202 determines the set of pivot images 242.

[0054] In various embodiments, machine learning model 202 includes one or more machine learning models trained to perform various tasks in conjunction with this disclosure. For example, machine learning model 202 includes detecting objects within frames of video 226A as described above. Figure 1 The described object detection models (e.g., SIFT, CNN, R-CNN, SSD, DETR, etc.). Continuing this example, machine learning model 202 selects pivot images 242 based at least in part on objects within frames of video 226A. As described above... Figure 1 As described, pivot image 242 includes frames and / or images from video 226A, which include conceptual elements of video 226A. In an embodiment, a particular frame of video 226A is selected as the pivot image because machine learning model 202 detects a modification of an object or several objects in a particular frame relative to at least one other frame. Overview of Artificial Intelligence (AI) Systems

[0055] Artificial intelligence (AI) systems refer to AI computing environments or architectures that include the infrastructure and components supporting the development, training, and deployment of AI models. They provide developers with the necessary hardware, software, and frameworks to create and run AI applications. AI systems can be cloud-based AI solutions that utilize cloud computing infrastructure to develop, train, deploy, and manage AI models and applications. AI models can specifically refer to generative AI models, which are designed to generate new data or content that is similar to, or in some cases completely different from, the data on which the AI ​​model was trained.

[0056] Artificial intelligence systems can include transformer models capable of performing complex neural language processing tasks. Transformer models—also known as large language models (LLMs)—have applications across a wide range of industries. An LLM is a trained deep learning model capable of recognizing, summarizing, translating, predicting, and generating content using very large datasets. LLMs, like other types of generative AI models, are associated with training and inference phases. In the training phase, the model is taught to learn patterns, relationships, and knowledge from the training dataset; and the inference phase involves making predictions, classifying, or generating outputs for real-world tasks or queries.

[0057] Unlike convolutional neural networks (CNNs), which are typically used for image tasks and rely primarily on convolutional operations, transformer models are based on a simple generalized matrix multiplication (GEMM) task, which can be further decomposed into performing a dot product operation on two vectors. While CNN architectures are generally computationally intensive with a relatively small number of parameters, transformer models are the opposite—a large number of parameters but a relatively small number of computations. The performance bottleneck of LLM architectures may lie in memory throughput and capacity rather than the computational engine.

[0058] The transformer model operates in conjunction with memory accesses to retrieve the weight matrix and vector (input vector or partial results from previous stages of the model) from memory and multiply them. This is correct for the model's attention sublayer, FFN (feedforward network), sublayers, and finally the embedding layer. Since vector-matrix multiplication is actually composed of many vector-vector multiplications (dot products), it can be said that most memory accesses are used to read two vectors in order to perform a dot product operation on them. Thus, reading the complete vector is inefficient.

[0059] Thus, the transducer model (also referred to as the “generative AI model” in this paper) requires computational resources, including processors and memory for the training and inference phases. Generative AI models operate with different types of processors, such as central processing units (CPUs) or graphics processing units (GPUs), whose architectures can include multi-core CPUs or parallel processors (including GPUs and TPUs). Memory can be used to store model parameters and intermediate data for the training and inference phases. Memory requirements can depend on the size and architecture of the generative AI model. In an illustrative manner, an LLM can support the inference phase, which involves using the trained model to make predictions, draw conclusions, or generate outputs based on input data or patterns learned during the model training phase. During the inference phase, the LLM can use DRAM (Dynamic Random Access Memory) to store various components and data used for inference. The LLM can store its pre-trained model parameters (e.g., weights and biases of neural network layers) in DRAM, and when new inputs are provided for inference, the model accesses these parameters from DRAM to make predictions.

[0060] The inference phase can be divided into two phases: a prompting phase and an autoregressive phase. The prompting phase may involve receiving and processing a batch of new lexical units as part of the same inference process. The prompting phase can operate based on a key-value (KV) caching technique, where a KV cache is created for the lexical units in the batch. During the prompting phase, the input is resolved. The autoregressive phase may involve using the model to generate lexical units one by one, based on previous lexical units and relying on reading the KV cache of previously processed lexical units, and adding only new data for new lexical units to the KV cache. This autoregressive phase includes the model generating a response to the input from the prompting phase.

[0061] Back Figure 2 In various other embodiments, other types of machine learning models may be used alone or in combination to determine a set of pivot images 242. For example, frames of video 226A are provided to the LLM as input, and natural language descriptions of the frames are generated. In various embodiments, the natural language descriptions of the frames (e.g., by encoder 224) are used to determine the pivot images 242. For example, encoder 224 selects a particular frame as a pivot image based at least in part on the number of objects described in the natural language description of the particular frame, the number of objects described in the natural language description of the particular frame relative to at least one other frame, the length of the natural language description, a comparison of the natural language description of the particular frame with at least one other frame, or other attributes of the natural language description of the particular frame.

[0062] In various embodiments, once the pivot image 242 is extracted or otherwise selected, the encoder 224 generates a descriptor 244. (As described above...) Figure 1As described, descriptor 244 includes a natural language description of the pivot image 242 to be provided to generative model 230. In one example, pivot image 242 is provided as input to machine learning model 202, and machine learning model 202 outputs descriptor 244. Continuing with this example, machine learning model 202 includes an LLM that generates prompts for generative model 230 such that generative model 230 is able to reconstruct video 226B at least partially based on pivot image 242.

[0063] In various embodiments, the compressed video data 208 is stored by a computing resource service provider 220 or other entity (e.g., an edge network device). Furthermore, in embodiments, the compressed video data 208 is provided to a computing device (such as those described above). Figure 1 The user equipment 102 described above executes a decoder 228. In one example, when acquiring compressed video data 208, the decoder 228 causes the generative model 230 to generate a set of images, which the decoder 228 combines to generate video 226B. (As described above...) Figure 1 As described, the generative model 230 includes various machine learning models (such as GPT, LLM, diffusion models, neural networks, or other machine learning models) for generating images based at least in part on images (e.g., pivot images) and corresponding descriptions (e.g., descriptors).

[0064] In various embodiments, generative model 230 generates output 210 at least partially based on image 206 and mask 204. In one example, image 206 includes a pivot image, and mask 204 includes a layer and / or set of pixels that overlay or otherwise hide objects in the pivot image. Continuing with this example, generative model 230 then uses mask 204 to move objects within image 206 to generate output 210. In other embodiments, generative model 230 is trained to generate output 210 at least partially based on the pivot image without using mask 204. In one example, generative model 230 takes a pivot image including two trees and a cat as input, such as... Figure 2 As shown, the position of the cat is moved based at least in part on the position indicated in descriptor 244. In this embodiment, generative model 230 is trained to modify or otherwise use pixels or other elements of pivot image 242 to generate images (e.g., reconstruct frames of video) based at least in part on natural language descriptions included in descriptor 244.

[0065] In various embodiments, generative model 230 is trained with video data, pivot images, and descriptors to generate similar data. In one example, noise or other random or pseudo-random data is added to the training data (e.g., a forward diffusion process), and generative model 230 is trained by at least removing noise (e.g., a backward diffusion process) to recreate or otherwise reconstruct the training data. In various embodiments, once generative model 230 is trained, a sampling process is used to generate output 210. In one example, output 210 is generated by at least providing Gaussian noise or other random or pseudo-random sampled noise to generative model 230, which then performs denoising.

[0066] In various embodiments, descriptor 244 is provided as a cue to generative model 230, and the text encoder of generative model 230 maps the cue to a representation space, wherein the representation space includes image encoding that captures the semantic information of the cue. In this embodiment, the image decoder of generative model 230 randomly generates images that include visual components corresponding to the semantic information.

[0067] In one example, generative model 230 reconstructs a video by adding noise to the pivot image 242 and denoising the resulting image (e.g., the noisy pivot image 242) at least in part based on semantic or other information included in the descriptor 244. Continuing this example, generative model 230 continues the process to generate reconstructed video frames that can be displayed to a viewer. In an embodiment, generative model 230 is at least partially based on the decoder 228 or other entities (e.g., in conjunction with the above). Figure 1 The user equipment 102 described herein provides a frame rate to generate a set of frames for reconstructing the video.

[0068] exist Figure 2 In the example illustrated, where pivot image 242 includes two trees and descriptor 244 describes the movement of a cat between the two trees, generative model 230 generates multiple frames (e.g., output 210) where successive frames, as indicated in descriptor 244, depict the cat moving between the two trees (e.g., using the denoising process described above). Furthermore, in some embodiments, mask 204 is used to remove objects from pivot image 242 before adding noise and performing the denoising process to generate frames for the reconstructed video.

[0069] Figure 3 This is a flowchart illustrating a method 300 for compressing video data for display to a user according to at least one embodiment. For example, method 300 may be... Figure 1The video compression tool 104 and / or decoder 128 execute this. Each block of the methods 300, 400, and 500 described herein, or any other method, includes a computational process that is executed using any combination of hardware, firmware, and / or software. For example, various functions can be performed by a processor that executes instructions stored in memory. These methods can also be embodied as computer-usable instructions stored on a computer storage medium. For example, the methods can be provided as standalone applications, services, or managed services (standalone or in combination with another managed service), or plug-ins to another product.

[0070] As shown in box 302, method 300 implements the system's video acquisition. (Combined with the above...) Figure 1 As described, in various embodiments, video is acquired by an encoder to generate compressed video data. In one example, the encoder is operated by a streaming service that transmits compressed video data to a user device for display to a viewer. At block 304, the system implementing method 300 determines the pivot image to be included in the compressed video data based at least in part on the video. For example, the encoder determines frames in the video that convey conceptual information associated with the video. As described above, several algorithmic techniques can be used to determine or otherwise select the pivot image. In one example, a subset of video frames is sampled at least in part based on a time interval and / or the number of frames. In other examples, a machine learning model is used to select the pivot image based at least in part on the number of objects included in the frame.

[0071] At box 306, the system implementing method 300 generates descriptors at least in part based on pivot images and / or video. In one example, an LLM or other machine learning model generates descriptors at least in part based on pivot images. In embodiments, the descriptor includes a natural language description of the pivot image. In other embodiments, the descriptor includes structured data (e.g., a JavaScript object representation [JSON] file) indicating properties of objects within the pivot image (e.g., position, movement, size, shape, etc.). For example, the descriptor includes text-based data (e.g., data, pseudocode, source code, etc.) provided to a generative model of the encoder to enable the generative model to reconstruct the video. In embodiments, descriptors are generated for video and / or portions thereof (e.g., frames between pivot images).

[0072] At box 308, the system implementing method 300 generates compressed video data based at least in part on the pivot image and descriptor. In one example, the pivot image and descriptor are combined into a single archived data object. In some examples, additional information and / or data are included in the compressed video data. At box 310, the system implementing method 300 transmits the compressed data to an endpoint. In one example, a user device requests video, and in response, compressed video data is transmitted to the user device.

[0073] At box 312, the system implementing method 300 provides pivot images and descriptors to a generative machine learning model. As described above, according to various embodiments, the decoder acquires the pivot images and descriptors and generates a set of output images using a generative machine learning model (e.g., a diffusion model). At box 314, the system implementing method 300 acquires the output of the machine learning model. In one example, the pivot images comprise frames from a video, and the descriptors describe the movement of objects or other conceptual elements between the pivot images, while the generative machine learning model takes the pivot images and descriptors as input and outputs images comprising the movement of objects or other conceptual elements indicated in the pivot images and descriptors. In this example, the video is reconstructed from the set of images output by the generative model.

[0074] At box 316, the system implementing method 300 generates video based at least in part on the output of a generative machine learning model. For example, the decoder generates video by at least combining the output images obtained from the generative machine learning model. In some examples, the decoder generates video once the generative machine learning model has generated all the output images. In other examples, the decoder generates and displays (e.g., streams) video while the generative machine learning model is still generating output based at least in part on the pivot image and descriptor.

[0075] Figure 4 This is a flowchart illustrating a method 400 for generating a pivot image and descriptor for compressed video data according to at least one embodiment. For example, method 400 may be derived from... Figure 1 The encoder 124 of the video compression tool 104 is executed. As shown at block 402, the system implementing method 300 acquires video frames. In various embodiments, video is acquired, and then individual frames of the video (e.g., video frames) are extracted and / or processed. At block 404, the system implementing method 400 determines and / or detects objects in the video frames. For example, as described above, an object detection model takes video frames as input and outputs data associated with objects in the video frames (e.g., labels, confidence intervals, markers, names, type information, etc.).

[0076] At box 406, the system implementing method 400 determines whether there is a change in state among the objects depicted in the video frame. For example, the system implementing method 400 compares the number of objects detected in the previous frame with the number of objects detected in this video frame. Additionally, in another example, the system implementing method 400 detects other state changes, such as position, size, shape, orientation, or other aspects of the object that may convey conceptual information in the video. If no change in state is detected, the system implementing method 400 proceeds to box 408. At box 408, the system implementing method 400 determines whether there is an additional video frame in the video. If an additional video frame exists (e.g., the video has not yet terminated), the system implementing method 400 continues method 400 at box 402 and acquires the next video frame. If no additional video frame is present in the video, method 400 proceeds to box 412, as described below.

[0077] Returning to box 406 above, if a change in state is detected, the system implementing method 400 proceeds to box 410 and selects the video frame as the pivot image. For example, the video frame is extracted and stored as the pivot image. At box 412, the system implementing method 400 generates a descriptor. As described above, in various embodiments, the descriptor includes textual data (such as natural language descriptions) associated with the pivot image and / or video. For example, the pivot image is provided to an LLM that generates natural language descriptions of the pivot image and / or conceptual elements connected to or otherwise associated with the pivot image. At box 414, the system implementing method 400 provides the pivot image and descriptor. For example, the pivot image and descriptor are transmitted to a decoder performed by a user equipment.

[0078] Figure 5 This is a flowchart illustrating a method 500 for reconstructing video, at least in part, based on compressed video data, according to at least one embodiment, for displaying to a viewer. For example, method 500 may be derived from... Figure 1 The decoder 128 is executed. As shown in box 502, the system implementing method 500 obtains the pivot image and descriptor of the generative machine learning model. (As described above...) Figure 1 As described, in various embodiments, the compressed video data includes a set of pivot images and descriptors, which are provided as input to a generative machine learning model. In one example, the generative machine learning model generates an output image by modifying the pivot images at least partially based on the descriptors.

[0079] At box 504, the system implementing method 500 acquires the output of the generative machine learning model. At box 506, if additional pivot images and / or descriptors exist, the system implementing method 500 returns to box 502 and continues with method 500. However, if no additional pivot images and / or descriptors exist, the system implementing method 500 proceeds to box 508. At box 508, the system implementing method 500 generates a video based at least in part on the output of the generative machine learning model. For example, a set of images generated as the output of the generative model is combined to reconstruct the video. Other embodiments

[0080] In some embodiments, a computerized system (such as the computerized system described in any of the embodiments above) includes a memory component and a processing device coupled to the memory component for performing operations. The operations include acquiring a video comprising a plurality of images and selecting a pivot image from the plurality of images. The operations may further include: providing at least the pivot image as input to a first machine learning model, causing the first machine learning model to generate a descriptor at least partially based on the pivot image, wherein the descriptor includes a linguistic description of the pivot image; and providing the pivot image and the descriptor to a decoder. Advantageously, in this manner, these embodiments of the present disclosure enable improved compression rates for video data while maintaining sufficient fidelity and conceptual information in the video. Furthermore, in this manner, as described herein, the embodiments reduce the network traffic and computing resources required for streaming video services and / or video transmission, thereby allowing these computing resources to be used for other tasks.

[0081] In any combination of the above embodiments of the computerized system, the pivot image depicts the conceptual elements of the video.

[0082] In any combination of the above embodiments of the computerized system, the operation further includes selecting a pivot image, and further includes: selecting a pivot image from the plurality of images based on a change detected between two or more images in a plurality of images by a second machine learning model.

[0083] In any combination of the above embodiments of the computerized system, the change includes modifications to objects depicted in two or more images, detected by a second machine learning model.

[0084] In any combination of the above embodiments of the computerized system, the operation further includes detecting additional objects relative to at least one of two or more images by a second machine learning model.

[0085] In any combination of the above embodiments of the computerized system, the operation further includes prompting a first machine learning model to describe conceptual elements of the video relative to a pivot image and at least one of a plurality of images.

[0086] In any combination of the above embodiments of the computerized system, the operation further includes: at the decoder, providing at least a pivot image and a descriptor as a first input to a third machine learning model, causing the third machine learning model to generate a reconstructed video, wherein the third machine learning model uses at least a portion of the pivot image and the descriptor to output a second plurality of images, which are combined to generate the reconstructed video.

[0087] In any combination of the above embodiments of the computerized system, the first machine learning model includes a large language model, the second machine learning model includes a neural network, and the third machine learning model includes a diffusion model.

[0088] In other embodiments, a non-transitory computer-readable medium storing executable instructions embodied thereon causes the processing device to perform operations when executed by the processing device. The operations include acquiring a pivot image from video and causing a machine learning model to generate a descriptor based at least in part on the pivot image, the descriptor providing a natural language description of the pivot image. The operations may also include generating a compressed data object including the descriptor and the pivot image, and providing the compressed data object to an endpoint via a network. Advantageously, in this manner, these embodiments of the present disclosure enable increased compression rates of video data while maintaining sufficient fidelity and conceptual information in the video. Furthermore, in this manner, as described herein, the embodiments reduce the network traffic and computing resources required for streaming video services and / or video transmission, thereby allowing these computing resources to be used for other tasks.

[0089] In any combination of the above embodiments of the medium, the operation further includes: causing a decoder executed by the endpoint to generate a reconstructed video by providing at least a descriptor and a pivot image as input to the generative model.

[0090] In any combination of the above embodiments of the medium, the generative model generates intermediate frames between the pivot image and the second pivot image in the reconstructed video, at least in part based on descriptors.

[0091] In any combination of the above embodiments of the medium, the operation further includes: sampling frames of the video within a time interval.

[0092] In any combination of the above embodiments of the medium, the operation further includes: causing a second machine learning model to determine that the pivot image includes conceptual elements of the video.

[0093] In any combination of the above embodiments of the medium, the operation further includes: causing the machine learning model to generate a second descriptor, the second descriptor including a second natural language description of the relationship between a set of pivot images and at least one other pivot image obtained from the video, wherein the pivot image in the at least one other pivot image is provided as input to the machine learning model.

[0094] In any combination of the above embodiments of the medium, the machine learning model includes a large language model (LLM).

[0095] In any combination of the above embodiments of the medium, the operation further includes: causing the machine learning model to generate a second natural language description of a frame of the video, and selecting the frame as a pivot image based at least in part on the second natural language description.

[0096] In other embodiments, a method is provided. This method includes acquiring a descriptor and a pivot image, the descriptor comprising a natural language description associated with a pivot image generated by a first machine learning model extracted from a video, and enabling a second machine learning model to generate a reconstructed video based at least in part on the pivot image and the descriptor. Advantageously, in this manner, these embodiments of the present disclosure enable increased compression rates of video data while maintaining sufficient fidelity and conceptual information in the video. Furthermore, in this manner, as described herein, embodiments reduce the network traffic and computing resources required for streaming video services and / or video transmission, thereby allowing these computing resources to be used for other tasks.

[0097] In any combination of the above embodiments of the method, the descriptor further includes a second natural language description of the object within the pivot image.

[0098] In any combination of the above embodiments of the method, the method includes causing a second machine learning model to generate a reconstructed video, and further includes causing the second machine learning model to reconstruct a first version of the video.

[0099] In any combination of the above embodiments of the method, the method includes causing a second machine learning model to generate a reconstructed video, and further includes combining a plurality of images generated by the second machine learning model at least in part based on pivot images and descriptors. Example computing environment

[0100] Various implementations have been described; now, several example computing environments suitable for implementing embodiments of this disclosure are described, including... Figure 6 , Figure 7 and Figure 8 The example computing devices and example distributed computing environments are shown in the text. Figure 6This is a block diagram of a language model 600 (e.g., a BERT model or a generative pre-trained transformer [GPT]-4 model) according to some embodiments, which makes specific predictions (e.g., answering questions) using specific inputs. In one embodiment, language model 600 corresponds to machine learning model 202 described herein. For example, model 600 represents or includes, as described above... Figure 1 and Figure 2 The functionality described by machine learning model 202 or generative models 130 and 230. In various embodiments, language model 600 includes one or more encoder and / or decoder blocks 606 (or any of their converters or portions thereof).

[0101] First, the natural language corpus of input 601 (e.g., various WIKIPEDIA English vocabulary or BooksCorpus) is converted into lexical units, then into feature vectors, and embedded into input embedding 602 to derive the meaning of each natural language word (e.g., English semantics) during pre-training. In some embodiments, to understand English, language model 600 ingests corpus documents such as textbooks, journals, blogs, social media feeds, etc.

[0102] In some embodiments, unlike existing Long Short-Term Memory (LSTM) models, each word or character in input(s)601 is mapped to input embeddings 602 in parallel or simultaneously. Input embeddings 602 map words to feature vectors representing those words. However, the same word (e.g., “apple”) may have different meanings in different sentences (e.g., phone to fruit). This is why a position encoder 604 can be implemented. The position encoder 604 provides a vector of contextual information about a word (e.g., “apple”) based on its position in a sentence. For example, for the message “I just sent a document,” since “I” is at the beginning of the sentence, the embodiment could indicate a position closer to “just” in the embedding than to “document.” Some embodiments use sine / cosine functions to generate the position encoder vector using the following two example formulas:

[0103] After passing one or more inputs 601 through input embedding 602 and applying position encoder 604, the output is a word embedding feature vector, which encodes positional information or context based on position encoder 604. These word embedding feature vectors are then passed to one or more encoder and / or decoder blocks 606, where they pass through multi-head attention layers 606-1 and feedforward layers 606-2. Multi-head attention layers 606-1 are generally responsible for focusing on or processing certain portions of the feature vectors representing specific parts of one or more inputs 601 by generating attention vectors. For example, in a question-answering system, multi-head attention layer 606-1 determines that the i-th word (or a specific word in a sentence) is used to answer the question or is related to other words in the same or other blocks, and its output is an attention vector. For each word, some embodiments generate an attention vector that captures the contextual relationships between other words or other character sequences in the same sentence. For a given word, some embodiments calculate a weighted average of other words containing that given word (e.g., other words in the same line or block) or otherwise aggregate their attention vectors to compute a final attention vector.

[0104] In some embodiments, the single-head attention mechanism has abstract vectors Q, K, and V that extract different components of a particular word. These vectors are used to compute the attention vector for each word using the following formula (3):

[0105] For multi-head attention, there are multiple weight matrices W. q W k and W v Therefore, there are multiple attention vectors Z for each word. However, the neural network may expect only one attention vector per word. Accordingly, another weighting matrix W z This is used to ensure that the output is still an attention vector for each word. In some embodiments, after layers 606-1 and 606-2, some form of normalization (e.g., batch normalization and / or layer normalization) is performed to smooth the loss surface, making it easier to optimize while using a larger learning rate.

[0106] Layers 606-3 and 606-4 represent residual connection layers and / or normalization layers, where the normalization layers re-center, rescale, or normalize the data along the feature dimensions. Feedforward layer 606-2 is a feedforward neural network applied to each attention vector in the attention vectors output by the multi-head attention layer 606-1. Feedforward layer 606-2 transforms the attention vectors into a form that can be processed by the next encoder block, or a form that can be predicted at 608. For example, assuming the document includes the first natural language sequence "The deadline is...", then one or more encoder / decoder blocks 606 predict that the next natural language sequence will be a specific date or a specific word based on previous documents containing the same or similar language as the first natural language sequence.

[0107] In some embodiments, one or more encoder / decoder blocks 606 include pre-training to learn language (pre-training) and making corresponding predictions. In some embodiments, fine-tuning is unnecessary because some embodiments perform cue engineering or cue learning. Pre-training is performed to understand language, while fine-tuning is performed to learn a specific task, such as learning to answer a set of questions (in a question-answering [QA] system).

[0108] In some embodiments, by training synchronously or simultaneously on two unsupervised tasks (masked language modeling [MLM] and next sentence prediction [NSP]), one or more encoder / decoder blocks 606 learn the language and context for words during pre-training. Regarding input and output, at the pre-training stage, the natural language corpus of input 601 can be various historical documents, such as textbooks, journals, and magazines, to output predicted natural language characters in 608 (prediction or prompting engineering is not performed at runtime at this time). One or more example encoder / decoder blocks 606 take sentences, paragraphs, or sequences (e.g., included in one or more inputs 601) where random words are replaced with masks. The goal is to output the value or meaning marked by the masks. For example, if a line reads “Please [mask] this document as soon as possible,” the predicted value for “mask” is “send.” This helps one or more encoder / decoder blocks 606 understand the bidirectional context of sentences, paragraphs, or lines in the document. In the case of NSP, one or more encoder / decoder blocks 606 take two or more elements (such as sentences, lines, or paragraphs) as input and, for example, determine whether the second sentence in the document actually follows (e.g., whether it is directly after) the first sentence in the document. This helps the encoder / decoder blocks 606 understand the context over all elements in the document, not just the context within a single element. By using both methods together, the encoder / decoder blocks 606 achieve a good understanding of natural language.

[0109] In some embodiments, during pre-training, the input to encoder / decoder block 606 is a set (e.g., two) of masked sentences (for sentences that have one or more masks), alternatively, these sentences may be partial strings or paragraphs. In some embodiments, each word is represented as a lexical, and some of the lexical elements are masked. Each lexical is then converted into a word embedding (e.g., 602). On the output side is a binary output for the prediction of the next sentence. For example, if the masked sentence 2 immediately follows (e.g., directly) the masked sentence 1, the component might output 1. The output is a word feature vector corresponding to the output of the machine learning model function. Therefore, the number of input word feature vectors is the same as the number of output word feature vectors.

[0110] In some embodiments, the initial embedding (e.g., input embedding 602) consists of three vectors: a lexical embedding, a segment or contextual question embedding, and a positional embedding. In some embodiments, the following functions occur during the pre-training phase. The lexical embedding is a pre-trained embedding. The segment embedding is a sentence number (which includes one or more inputs [s] 601), which is encoded as a vector (e.g., first sentence, second sentence, etc., assuming a top-down, right-to-left approach). The positional embedding is a vector representing the position of a specific word in such a sentence, which can be generated by the positional encoder 604. When these three embeddings are added or concatenated together, an embedding vector is generated, which is used as input to one or more encoder / decoder blocks 606. Since all vectors are fed into one or more encoder / decoder blocks 606 simultaneously, and the language model needs to preserve some order, the segment embedding and positional embedding are used for temporal ordering.

[0111] In pre-training, the output is typically a binary value C (for NSP) and various word vectors (for MLM). For training, the loss (e.g., cross-entropy loss) is minimized. In some embodiments, all feature vectors are of the same size and are generated simultaneously. Thus, each word vector can be passed to a fully connected hierarchical output with the same number of neurons as the number of words in the vocabulary.

[0112] In some embodiments, after pre-training, encoder / decoder blocks(s) 606 perform cue engineering or fine-tuning on various QA datasets by converting different QA formats into a unified sequence-to-sequence format. For example, some embodiments perform QA tasks by adding new question-answering heads or encoder / decoder blocks, much like adding a masked language model head (during pre-training) for performing an MLM task, except that this task is part of the cue engineering or fine-tuning. This includes encoder / decoder blocks(s) 606 that process the input (e.g., Figure 1The pivot image 142 and / or descriptor 144 in the text are used to make predictions and generate prompt responses, as indicated in 604. In some embodiments, prompt engineering is the process of developing and optimizing text prompts for a language model to achieve a desired output. In other words, prompt engineering includes the process of mapping prompts (e.g., questions) to their corresponding outputs (e.g., answers) for training. For example, if a user asks the model to generate a poem about people fishing on a lake, it is expected that it will generate a different poem each time. The user can then label the output or answer from best to worst. Such labels are input to the model to ensure that the model is giving answers that are closer to human or optimal, while attempting to minimize the worst answer (e.g., via reinforcement learning). In some embodiments, the “prompt” described herein includes one or more of the following: a request (e.g., a question or instruction [e.g., “write a poem”]), target content, and one or more examples as described herein.

[0113] In some embodiments, additionally or alternatively, input 601 includes other inputs, such as Figures 1 to 5 The input to the machine learning model is described herein. In the illustrative example, the prediction of the output represents a descriptor for a pivot image, pivot image set, video, or video segment derived from the initial prompts and contextual information described herein. For example, the prediction can be generative text, such as a natural language description of an image or image set, a generative answer to a question, machine-translated text, or other generative text. As an alternative to prompting engineering, certain embodiments of the input (or derived from...) Figures 1 to 5 The inputs or prompts sent or received by the machine learning model described herein represent inputs provided to the encoder / decoder module 608 at runtime or after model 600 has been trained, tested, and deployed. Similarly, in these embodiments, the predictions in output 608 represent predictions made at runtime or after model 600 has been trained, tested, and deployed.

[0114] refer to Figure 7 An example computing device is provided and is generally referred to as computing device 700. Computing device 700 is merely an example of a suitable computing environment and is not intended to impose any limitation on the scope or functionality of the embodiments of this disclosure, nor should computing device 700 be construed as having any dependency or requirement on any one or a combination of the illustrated components.

[0115] The embodiments of this disclosure are described in the general context of computer code or machine-usable instructions (including computer-usable or computer-executable instructions, such as program modules), which are executed by a computer or other machine (such as a smartphone, tablet PC or other mobile device, server, or client device). Generally, a program module (including routines, programs, objects, components, data structures, etc.) refers to code that performs a specific task or implements a specific abstract data type. Embodiments of this disclosure are practiced in a variety of system configurations, including mobile devices, consumer electronics, general-purpose computers, more specialized computing devices, and so on. Embodiments of this disclosure are also practiced in distributed computing environments, where tasks are performed by remote processing devices linked via a communication network. In a distributed computing environment, program modules may reside on local and remote computer storage media (including memory storage devices).

[0116] Some embodiments include end-to-end software systems that operate within the system components described herein to manipulate computer hardware to provide system functionality. At a low level, the hardware processor typically executes instructions selected from a machine language (also known as machine code or native) instruction set specific to a given processor. The processor recognizes native instructions and performs corresponding low-level functions, such as those related to logic, control, and memory operations. Low-level software written in machine code can provide more complex functionality for higher-level software. Accordingly, in some embodiments, computer-executable instructions include any software, including low-level software written in machine code, higher-level software (such as application software), and any combination thereof. In this regard, system components can manage resources and provide services for system functionality. Any other variations and combinations are contemplated within the embodiments of this disclosure.

[0117] refer to Figure 7 The computing device 700 includes a bus 710 that is directly or indirectly coupled to the following devices: a memory 712, one or more processors 714, one or more presentation components 716, one or more input / output (I / O) ports 718, one or more I / O components 720, and an illustrative power supply 722. In one example, bus 710 represents one or more buses (such as an address bus, a data bus, or a combination thereof). Although for clarity, Figure 7 The various boxes are shown with lines, but in reality, these boxes represent logical components, not necessarily actual components. For example, presentation components include display devices (such as I / O components). Furthermore, the processor has memory. The inventors recognize this as the nature of the art and reiterate... Figure 7The illustrations are for informational purposes only and are intended to illustrate example computing devices that may be used in conjunction with one or more embodiments of this disclosure. No distinction is made between categories such as “workstation,” “server,” “laptop,” or “handheld device,” as all of these are envisioned in… Figure 7 Within the scope and refer to "Computing Devices".

[0118] Computing device 700 typically includes a variety of computer-readable media. Computer-readable media can be any available medium accessible to computing device 700, and includes volatile and non-volatile media, removable media, and non-removable media. By way of example, and not limitation, computer-readable media includes computer storage media and communication media. Computer storage media includes volatile and non-volatile media, removable media, and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to: random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage devices, magnetic cartridges, magnetic tape, disk storage devices or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible by computing device 700. Computer storage media does not include the signal itself. Communication media typically embodies computer-readable instructions, data structures, program modules, or other data in the form of modulated data signals (such as carrier waves or other transmission mechanisms), and includes any information delivery medium. The term "modulated data signal" means a signal in which one or more characteristics are set or altered in such a way that information is encoded in the signal. By way of example, and not limitation, communication media include wired media (such as wired networks or direct wired connections) and wireless media (such as sound waves, radio frequency (RF), infrared, and other wireless media). Any combination of the above should be included within the scope of computer-readable media.

[0119] Memory 712 includes computer storage media in the form of volatile memory and / or non-volatile memory. In one example, the memory may be removable, non-removable, or a combination thereof. Hardware devices include, for example, solid-state memory, hard disk drives, and optical disk drives. Computing device 700 includes one or more processors 714 that read data from various entities, such as memory 712 or I / O components 720. As used herein and in one example, the term "processor" or "a processor" refers to more than one computer processor. For example, the term "processor" (or "a processor") refers to at least one processor, which may be a physical processor or a virtual processor, such as a computer processor on a virtual machine. The term "processor" (or "a processor") may also refer to multiple processors, each of which may be a physical processor or a virtual processor, such as a multiprocessor system, distributed processing or distributed computing architecture, cloud computing system, or parallel processing performed by more than a single processor. Furthermore, the various operations implemented or performed by processors as described herein may also be performed by more than one processor.

[0120] One or more presentation components 716 present data indications to a user or other device. Presentation components include, for example, display devices, speakers, printing components, vibration components, etc.

[0121] I / O port 718 allows computing device 700 to be logically coupled to other devices, including I / O components 720, some of which are built-in. Illustrative components include microphones, joysticks, game controllers, satellite antennas, scanners, printers, or wireless devices. I / O components 720 can provide a natural user interface (NUI) that processes user-generated air gestures, voice, or other physiological input. In some cases, the input is transmitted to appropriate network elements for further processing. NUI can implement any combination of voice recognition, touch and stylus recognition, facial recognition, biometric recognition, in-screen and near-screen gesture recognition, air gestures, head and eye tracking, and touch recognition associated with a display on computing device 700. In one example, computing device 700 is equipped with a depth camera for gesture detection and recognition, such as a stereo camera system, an infrared camera system, a red-green-blue (RGB) camera system, and combinations thereof. Additionally, computing device 700 may be equipped with an accelerometer or gyroscope that enables motion detection. The output of an accelerometer or gyroscope can be provided to the display of a computing device 700 to render immersive augmented reality or virtual reality.

[0122] Some embodiments of computing device 700 include one or more radio modules 724 (or similar wireless communication components). The radio module transmits and receives radio or wireless communications. For example, computing device 700 is a wireless terminal adapted to receive communications and media over various wireless networks. Computing device 700 can communicate with other devices via wireless protocols such as Code Division Multiple Access (CDMA), Global System for Mobile Communications (GSM), or Time Division Multiple Access (TDMA). In one embodiment, radio communication can be a short-range connection, a long-range connection, or a combination of short-range and long-range wireless telecommunication connections. In various embodiments, the references to the “short” and “long” types of connections do not refer to a spatial relationship between two devices. Rather, the references to short-range and long-range generally refer to different categories or types of connections (e.g., primary and secondary connections). By way of example, and not limitation, a short-range connection includes a Wi-Fi® connection to a device providing access to a wireless communication network (e.g., a mobile hotspot), such as a wireless local area network (WLAN) connection using the 802.11 protocol; a Bluetooth connection to another computing device is a second example of a short-range connection or near-field communication connection. By way of example, and not limitation, long-distance connections may include connections using one or more of the following protocols: Code Division Multiple Access (CDMA), General Packet Radio Service (GPRS), Global System for Mobile Communications (GSM), Time Division Multiple Access (TDMA), and the 802.16 protocol.

[0123] Now for reference Figure 8 An example distributed computing environment 800 in which the implementation of this disclosure can be adopted is provided illustratively. In particular, Figure 8 A high-level architecture of an example cloud computing platform 810 is shown, which can host a technology solution environment or a portion thereof (e.g., a data hosting environment). It should be understood that the arrangement and other arrangements described herein are illustrated by way of example only. For example, many of the elements described herein can be implemented as discrete or distributed components, or in combination with other components, and in any suitable combination and location, as described above. Other arrangements and elements may be used in addition to those shown (e.g., machines, interfaces, functions, sequences, and functional groupings) or alternatives to those arrangements and elements.

[0124] The data center can support a distributed computing environment 800, which includes a cloud computing platform 810, racks 820, and nodes 830 (e.g., computing devices, processing units, or blades) located within the racks 820. This technical solution environment can be implemented using the cloud computing platform 810, which runs cloud services across different data centers and geographical regions. The cloud computing platform 810 can implement a structure controller 840 component for providing and managing the allocation, deployment, upgrades, and management of cloud services. Typically, the cloud computing platform 810 is used to store data or run service applications in a distributed manner. The cloud computing platform 810 in the data center can be configured to host and support the operation of endpoints for specific service applications. In one example, the cloud computing platform 810 can be a public cloud, a private cloud, or a dedicated cloud.

[0125] Node 830 may be provided with host 850 (e.g., operating system or runtime environment), which runs a defined software stack on node 830. Node 830 may also be configured to perform specific functions (e.g., compute node or storage node) within cloud computing platform 810. Node 830 is allocated to run one or more parts of a tenant's service application. A tenant can refer to a customer utilizing the resources of cloud computing platform 810. The service application components of cloud computing platform 810 supporting a specific tenant may be referred to as multi-tenant infrastructure or multi-tenancy. About Figure 8 The terms “service application,” “application,” or “service” are used interchangeably and broadly refer to any software or software component that runs on or accesses storage and computing devices within a data center.

[0126] When node 830 is supporting more than one individual service application, node 830 is partitioned into virtual machines (e.g., virtual machine 852 and virtual machine 854). Physical machines can also run individual service applications simultaneously. Virtual machines or physical machines can be configured as individualized computing environments supported by resources 860 (e.g., hardware and software resources) in the cloud computing platform 810. It is conceivable that resources can be configured for specific service applications. Furthermore, each service application can be divided into functional parts, such that each functional part can run on a separate virtual machine. In the cloud computing platform 810, multiple servers can be used to run service applications and perform data storage operations in a cluster. In one embodiment, these servers perform data operations independently but appear as a single device to the outside world, referred to as a cluster. Each server in the cluster can be implemented as a node.

[0127] In some embodiments, client device 880 is linked to a service application in cloud computing platform 810. Client device 880 can be any type of computing device, such as a reference... Figure 1The user equipment 102 described herein, and the client equipment 880, can be configured to issue commands to the cloud computing platform 810. In embodiments, the client equipment 880 communicates with the service application via a Virtual Internet Protocol (IP) and a load balancer or other means of directing communication requests to a designated endpoint in the cloud computing platform 810. Certain components of the cloud computing platform 810 communicate with each other via a network (not shown), which includes, but is not limited to, one or more local area networks (LANs) and / or wide area networks (WANs). Additional structural and functional features of embodiments of the technical solution

[0128] As the various components utilized have been identified herein, it should be understood that any number of components and arrangements can be employed to achieve the desired functionality within the scope of this disclosure. For example, for clarity of concept, components in the embodiments depicted in the figures are shown in lines. Other arrangements of these and other components can also be implemented. For example, although some components are depicted as single components, many elements described herein can be implemented as discrete or distributed components, or in combination with other components, and in any suitable combination and location. Some elements may be omitted entirely. Furthermore, the various functions performed by one or more entities as described herein can be implemented by hardware, firmware, and / or software, as described below. For example, various functions can be performed by a processor executing instructions stored in memory. Thus, other arrangements and elements may be used in addition to or in lieu of those arrangements and elements shown (e.g., machines, interfaces, functions, sequences, and functional groups).

[0129] The embodiments described in the following paragraphs can be combined with one or more of the specific alternatives described. In particular, the claimed embodiments may include alternative references to more than one other embodiment. The claimed embodiments may specify further limitations on the claimed subject matter.

[0130] For the purposes of this disclosure, the word "including" has the same broad meaning as the word "comprising," and the word "access" includes "receiving," "referencing," or "retrieval." Furthermore, the word "communication" has the same broad meaning as "receiving" or "transmitting" facilitated by a software or hardware bus, receiver, or transmitter using the communication medium described herein. Additionally, unless otherwise stated, words such as "a" and "an" include both singular and plural forms. Thus, for example, the limitation of "an feature" is satisfied when one or more features are present. Furthermore, the term "or" includes conjunction, disjunction, and both (therefore, a or b includes a or b, and a and b).

[0131] As used herein, the term "set" can be used to refer to an ordered (i.e., sequential) or unordered (i.e., non-sequential) collection of objects (or elements), such as machines (e.g., computer devices), physical and / or logical addresses, graph nodes, graph edges, functions, etc. As used herein, a set can include... N There are elements, among which N It can be any positive integer. In other words, the set can include... 1 , 2 , 3 … N 1 object and / or element, of which N It is a positive integer with no upper limit. Therefore, as used in this article, the set does not include the empty set (i.e., the empty set), which does not contain any elements (e.g., for the empty set, ...). N =0). A set may contain only a single element. In other embodiments, a set may include many elements that are significantly larger than one, two, three, or billions of elements. A set may be an infinite set or a finite set. Some sets may include discrete objects (e.g., the set of natural numbers). Other sets may include contiguous objects (e.g., the set of real numbers). In some embodiments, a “collection of objects” that is not an empty set of objects may be referred to interchangeably as “one or more objects” or “at least one object,” where the term “object” can represent any object or element that can be included in the collection. Therefore, the phrases “one or more objects” and “at least one object” may be used interchangeably to refer to a collection of objects that is not an empty set of objects. A collection of objects that includes at least two objects may be referred to as “multiple objects.”

[0132] As used herein and in one example, the term "subset" is a set included in another set. A subset can be (but is not required to be) a proper or strict subset of another set in which it is included. That is, if set B is a subset of set A, then in some embodiments, set B is a proper or strict subset of set A. In other embodiments, set B is a subset of set A, but not a proper or strict subset of set A. For example, set A and set B can be equal sets, and set B can be called a subset of set A. In such embodiments, set A can also be called a subset of set B. If the intersection of two sets is an empty set, then the two sets can be disjoint sets.

[0133] As used herein, the terms "application" or "app" are used interchangeably to refer to any software-based program, package, or product executable via one or more (physical or virtual) computers or devices. An application can be a collection of software products that, when executed, provide one or more computing and / or data services to an end user. In some embodiments, an application can refer to a collection of applications that can be executed together to provide one or more computing and / or data services. Applications included in a collection of applications can be executed sequentially, in parallel, or in any combination thereof. Execution of multiple applications, including a single application, can be interleaved. For example, an application may include a first application and a second application. Execution of the application may include sequential execution of the first and second applications, or parallel execution of the first and second applications. In other embodiments, execution of the first and second applications may be interleaved.

[0134] For the purposes of the detailed discussion above, embodiments of this disclosure are described with reference to computing devices or distributed computing environments; however, the computing devices and distributed computing environments depicted herein are merely non-limiting examples. Furthermore, the terms "computer system" and "computing system" are used interchangeably herein, and therefore a computer system is not limited to a single computing device, nor does a computing system necessarily require multiple computing devices. Rather, various aspects of the embodiments of this disclosure can be implemented on a single computing device or multiple computing devices, as described herein. Additionally, components can be configured to perform novel aspects of the embodiments, wherein the term "configured for" can mean "programmed to" perform a specific task or implement a specific abstract data type using code. Furthermore, while embodiments of this disclosure generally relate to the technical environments and schematic diagrams described herein, it should be understood that the described techniques can be extended to other implementation contexts.

[0135] Many different arrangements of the various components depicted, and those not shown, are possible without departing from the scope of the following claims. Embodiments of this disclosure have been described in an illustrative and not limiting manner. Alternative embodiments will become apparent to the reader of this disclosure upon reading it, and as a result of reading it. Alternative ways of implementing the foregoing can be accomplished without departing from the scope of the following claims. Certain features and sub-combinations are useful and can be used without reference to other features and sub-combinations, and these features and sub-combinations are contemplated within the scope of the claims.

Claims

1. A system comprising: Memory components; as well as A processing device coupled to the memory component, the processing device being configured to perform operations including: Acquire a video containing multiple images; Select a pivot image from the plurality of images; By providing at least the pivot image as input to a first machine learning model, the first machine learning model generates a descriptor at least partially based on the pivot image, wherein the descriptor includes a linguistic description of the pivot image; and The pivot image and the descriptor are provided to the decoder.

2. The system of claim 1, wherein the pivot image depicts conceptual elements of the video.

3. The system according to claims 1 and 2, wherein selecting the pivot image further comprises: The pivot image is selected from the plurality of images based on changes detected between two or more images using a second machine learning model.

4. The system according to claims 1-3, wherein the change includes modifications to the objects depicted in the two or more images, detected by the second machine learning model.

5. The system of claims 1-4, wherein the change includes the detection by the second machine learning model of an additional object relative to at least one of the two or more images.

6. The system according to claims 1-5, wherein generating the descriptor by the first machine learning model further comprises: The first machine learning model describes the conceptual elements of the video relative to the pivot image and at least one other image among the plurality of images.

7. The system of claims 1-6, wherein the processing device further performs the following operations: at the decoder, by providing at least the pivot image and the descriptor as first inputs to a third machine learning model, causing the third machine learning model to generate a reconstructed video, wherein the third machine learning model uses at least a portion of the pivot image and the descriptor to output a second plurality of images, the second plurality of images being combined to generate the reconstructed video.

8. The system according to claims 1-7, wherein the first machine learning model comprises a large language model, the second machine learning model comprises a neural network, and the third machine learning model comprises a diffusion model.

9. A non-transitory computer-readable medium storing executable instructions embodied thereon, the executable instructions causing the processing device to perform operations when executed by a processing device, the operations including: Obtain pivot images from video; The machine learning model generates descriptors based at least in part on the pivot image, the descriptors providing a natural language description of the pivot image; Generate a compressed data object including the descriptor and the pivot image; as well as The compressed data object is provided to the endpoint via the network.

10. The medium of claim 9, wherein the medium further stores executable instructions that cause the processing device to perform the following operations: cause a decoder executed by the endpoint to generate a reconstructed video by providing at least the descriptor and the pivot image as input to a generative model.

11. The medium according to claims 9 and 10, wherein the generative model generates intermediate frames in the reconstructed video between the pivot image and the second pivot image based at least in part on the descriptor.

12. The medium according to claims 9-11, wherein acquiring the pivot image further comprises: Frames of the video are sampled within a time interval.

13. The medium according to claims 9-12, wherein acquiring the pivot image further comprises: The second machine learning model determines that the pivot image includes conceptual elements of the video.

14. The medium of claims 9-13, wherein the medium further stores executable instructions that cause the processing device to perform the following operations: causing the machine learning model to generate a second descriptor, the second descriptor including a second natural language description of the relationship between the set of pivot images and at least one other pivot image acquired from the video, wherein the pivot image in the at least one other pivot image is provided as input to the machine learning model.

15. The medium according to claims 9-14, wherein acquiring the pivot image further comprises: The machine learning model generates a second natural language description of the frames in the video, and selects the frames as the pivot images based at least in part on the second natural language description.

Images