Modifying video content

Abstract

Systems and techniques for modifying video data are described herein. For example, a method for modifying video data is provided. The method can include obtaining first tokens based on a first frame of the video data, where each of the first tokens comprises a feature vector corresponding to a respective position within the first frame of the video data; obtaining second tokens based on a second frame of the video data, where each of the second tokens comprises a feature vector corresponding to a respective position within the second frame of the video data; determining a destination token from the first tokens; determining a candidate token from the second tokens based on respective relationships between the candidate token and the destination token; merging the candidate token with the destination token, resulting in a modified second token; and processing the modified second token using a diffusion model.

Description

Technical Field

[0001] This disclosure relates in general to processing video content. For example, aspects of this disclosure include systems and techniques for modifying video content. Background Technology

[0002] Video content can be modified by altering frames of image data (e.g., modifying one frame of a video at a time). In some cases, machine learning models can be trained to modify video content. For example, a latent diffusion model can be used to modify frames of video data (e.g., replacing objects in a frame of video data with different objects). Summary of the Invention

[0003] The following is a simplified summary of the invention relating to one or more aspects disclosed herein. Therefore, this summary should not be considered an exhaustive overview relating to all conceived aspects, nor should it be considered to identify key or decisive elements relating to all conceived aspects or to depict the scope associated with any particular aspect. Accordingly, the following summary presents certain concepts in a simplified form relating to one or more aspects of the mechanisms disclosed herein, preceding the detailed description presented below.

[0004] Systems and techniques for modifying video data are described. According to at least one example, a method for modifying video data is provided. The method includes: obtaining a first token based on a first frame of video data, wherein each of the first tokens includes a feature vector corresponding to a corresponding position within the first frame of video data; obtaining a second token based on a second frame of video data, wherein each of the second tokens includes a feature vector corresponding to a corresponding position within the second frame of video data; determining a destination token from the first tokens; determining a candidate token from the second tokens based on a correspondence between the candidate token and the destination token; merging the candidate token and the destination token to generate a modified second token; and processing the modified second token using a diffusion model.

[0005] In another example, an apparatus for modifying video data is provided. The apparatus includes: one or more memories; and one or more processors (e.g., circuit-configured) coupled to the one or more memories. The one or more processors are configured to: obtain a first token based on a first frame of video data, wherein each of the first tokens includes a feature vector corresponding to a corresponding position within the first frame of video data; obtain a second token based on a second frame of video data, wherein each of the second tokens includes a feature vector corresponding to a corresponding position within the second frame of video data; determine a destination token from the first tokens; determine a candidate token from the second tokens based on a correspondence between the candidate tokens and the destination token; merge the candidate tokens with the destination tokens to produce a modified second token; and process the modified second token using a diffusion model.

[0006] In another example, a non-transitory computer-readable medium is provided having instructions stored thereon, which, when executed by one or more processors, cause the one or more processors to: obtain a first token based on a first frame of video data, wherein each of the first tokens includes a feature vector corresponding to a corresponding position within the first frame of video data; obtain a second token based on a second frame of video data, wherein each of the second tokens includes a feature vector corresponding to a corresponding position within the second frame of video data; determine a destination token from the first tokens; determine a candidate token from the second tokens based on a correspondence between the candidate token and the destination token; merge the candidate token with the destination token to produce a modified second token; and process the modified second token using a diffusion model.

[0007] In another example, an apparatus for modifying video data is provided. The apparatus includes: components for obtaining a first token based on a first frame of video data, wherein each of the first tokens includes a feature vector corresponding to a corresponding position within the first frame of video data; components for obtaining a second token based on a second frame of video data, wherein each of the second tokens includes a feature vector corresponding to a corresponding position within the second frame of video data; components for determining a destination token from the first tokens; components for determining a candidate token from the second tokens based on a corresponding relationship between the candidate token and the destination token; components for merging the candidate token and the destination token to generate a modified second token; and components for processing the modified second token using a diffusion model.

[0008] In another example, a method for modifying video data is provided. The method includes: obtaining a plurality of tokens, the plurality of tokens comprising a corresponding set of tokens for each of a plurality of frames of video data; identifying a destination token from the plurality of tokens; determining a candidate token from the plurality of tokens based on a correspondence between the candidate token and the destination token; merging the candidate token with the destination token to generate a modified second token; and processing the modified second token using a diffusion model.

[0009] In another example, an apparatus for modifying video data is provided. The apparatus includes: one or more memories; and one or more processors (e.g., circuit-configured) coupled to the one or more memories. The one or more processors are configured to: obtain a plurality of tokens, the plurality of tokens comprising a corresponding set of tokens for each of a plurality of frames of the video data; identify a destination token from the plurality of tokens; determine a candidate token from the plurality of tokens based on a correspondence between the candidate token and the destination token; merge the candidate token with the destination token to generate a modified second token; and process the modified second token using a diffusion model.

[0010] In another example, a non-transitory computer-readable medium is provided having instructions stored thereon, which, when executed by one or more processors, cause the one or more processors to: obtain a plurality of tokens, the plurality of tokens comprising a corresponding set of tokens for each of a plurality of frames of the video data; identify a destination token from the plurality of tokens; determine a candidate token from the plurality of tokens based on a correspondence between the candidate token and the destination token; merge the candidate token with the destination token to produce a modified second token; and process the modified second token using a diffusion model.

[0011] In another example, an apparatus for modifying video data is provided. The apparatus includes: components for obtaining a plurality of tokens, the plurality of tokens including a corresponding set of tokens for each of a plurality of frames of video data; components for identifying a destination token from the plurality of tokens; components for determining a candidate token from the plurality of tokens based on a corresponding relationship between the candidate token and the destination token; components for merging the candidate token with the destination token to generate a modified second token; and components for processing the modified second token using a diffusion model.

[0012] In another example, a method for modifying image data is provided. The method includes: obtaining tokens based on the image data, wherein each of the tokens includes a feature vector corresponding to a corresponding location within the image data; determining a destination token from the tokens; obtaining a segmentation mask based on the image data; determining a candidate token from the tokens based on a correspondence between the candidate token and the destination token and based on the segmentation mask; merging the candidate token and the destination token to generate a modified token; and processing the modified token using a diffusion model.

[0013] In another example, an apparatus for modifying image data is provided. The apparatus includes: one or more memories; and one or more processors (e.g., circuit-configured) coupled to the one or more memories. The one or more processors are configured to: obtain tokens based on the image data, wherein each of the tokens includes a feature vector corresponding to a corresponding location within the image data; determine a destination token from the tokens; obtain a segmentation mask based on the image data; determine a candidate token from the tokens based on a correspondence between the candidate token and the destination token and based on the segmentation mask; merge the candidate token and the destination token to generate a modified token; and process the modified token using a diffusion model.

[0014] In another example, a non-transitory computer-readable medium is provided having instructions stored thereon, which, when executed by one or more processors, cause the one or more processors to: obtain tokens based on image data, wherein each of the tokens includes a feature vector corresponding to a corresponding location within the image data; determine a destination token from the tokens; obtain a segmentation mask based on the image data; determine a candidate token from the tokens based on a correspondence between the candidate token and the destination token and based on the segmentation mask; merge the candidate token with the destination token to produce a modified token; and process the modified token using a diffusion model.

[0015] In another example, an apparatus for modifying image data is provided. The apparatus includes: components for obtaining tokens based on the image data, wherein each of the tokens includes a feature vector corresponding to a corresponding position within the image data; components for determining a destination token from the tokens; components for obtaining a segmentation mask based on the image data; components for determining a candidate token from the tokens based on a corresponding relationship between the candidate token and the destination token and based on the segmentation mask; components for merging the candidate token and the destination token to generate a modified token; and components for processing the modified token using a diffusion model.

[0016] In another example, a method for modifying image data is provided. The method includes: identifying a first portion and a second portion of image data based on a segmentation mask; processing the first portion of the image data using a diffusion model to generate a modified first portion of the image data; and generating modified image data based on the modified first portion and the second portion of the image data.

[0017] In another example, an apparatus for modifying image data is provided. The apparatus includes: one or more memories; and one or more processors (e.g., circuit-configured) coupled to the one or more memories. The one or more processors are configured to: identify a first portion and a second portion of image data based on a segmentation mask; process the first portion of the image data using a diffusion model to generate a modified first portion of the image data; and generate modified image data based on the modified first portion and the second portion of the image data.

[0018] In another example, a non-transitory computer-readable medium is provided having instructions stored thereon, which, when executed by one or more processors, cause the one or more processors to: identify a first portion and a second portion of image data based on a segmentation mask; process the first portion of the image data using a diffusion model to generate a modified first portion of the image data; and generate modified image data based on the modified first portion and the second portion of the image data.

[0019] In another example, an apparatus for modifying image data is provided. The apparatus includes: components for identifying a first portion and a second portion of image data based on a segmentation mask; components for processing the first portion of the image data using a diffusion model to generate a modified first portion of the image data; and components for generating modified image data based on the modified first portion and the second portion of the image data.

[0020] In another example, a method for modifying image data is provided. The method includes: identifying a first portion and a second portion of image data based on a segmentation mask; processing the first portion of the image data using a diffusion model to generate a modified first portion of the image data; and combining the modified first portion and the second portion of the image data to produce modified image data.

[0021] In another example, an apparatus for modifying image data is provided. The apparatus includes: one or more memories; and one or more processors (e.g., circuit-configured) coupled to the one or more memories. The one or more processors are configured to: identify a first portion and a second portion of image data based on a segmentation mask; process the first portion of the image data using a diffusion model to generate a modified first portion of the image data; and combine the modified first portion and the second portion of the image data to produce modified image data.

[0022] In another example, a non-transitory computer-readable medium is provided having instructions stored thereon, which, when executed by one or more processors, cause the one or more processors to: identify a first portion and a second portion of image data based on a segmentation mask; process the first portion of the image data using a diffusion model to generate a modified first portion of the image data; and combine the modified first portion and the second portion of the image data to produce modified image data.

[0023] In another example, an apparatus for modifying image data is provided. The apparatus includes: components for identifying a first portion and a second portion of image data based on a segmentation mask; components for processing the first portion of the image data using a diffusion model to generate a modified first portion of the image data; and components for combining the modified first portion and the second portion of the image data to produce modified image data.

[0024] In another example, a method for modifying image data is provided. The method includes: identifying a first portion and a second portion of image data based on a segmentation mask; processing the first portion of the image data using a first number of diffusion steps of a diffusion model to generate a partially modified first portion of the image data; and processing the partially modified first portion and the second portion of the image data using a second number of diffusion steps of the diffusion model to generate modified image data.

[0025] In another example, an apparatus for modifying image data is provided. The apparatus includes: one or more memories; and one or more processors (e.g., in a circuit configuration) coupled to the one or more memories. The one or more processors are configured to: identify a first portion and a second portion of image data based on a segmentation mask; process the first portion of the image data using a first number of diffusion steps of a diffusion model to generate a partially modified first portion of the image data; and process the partially modified first portion and the second portion of the image data using a second number of diffusion steps of the diffusion model to generate modified image data.

[0026] In another example, a non-transitory computer-readable medium is provided having instructions stored thereon, which, when executed by one or more processors, cause the one or more processors to: identify a first portion and a second portion of image data based on a segmentation mask; process the first portion of the image data using a first number of diffusion steps of a diffusion model to generate a partially modified first portion of the image data; and process the partially modified first portion and the second portion of the image data using a second number of diffusion steps of the diffusion model to generate modified image data.

[0027] In another example, an apparatus for modifying image data is provided. The apparatus includes: components for identifying a first portion and a second portion of image data based on a segmentation mask; components for processing the first portion of the image data using a first number of diffusion steps of a diffusion model to generate a partially modified first portion of the image data; and components for processing the partially modified first portion and the second portion of the image data using a second number of diffusion steps of the diffusion model to generate modified image data.

[0028] In some aspects, one or more of the devices described herein are, may be part of, or may include: mobile devices (e.g., mobile phones or so-called "smartphones," tablet computers, or other types of mobile devices), extended reality devices (e.g., virtual reality (VR) devices, augmented reality (AR) devices, or mixed reality (MR) devices), vehicles (or computing devices, components, or systems of vehicles), smart or connected devices (e.g., Internet of Things (IoT) devices), wearable devices, personal computers, laptop computers, video servers, televisions (e.g., network-connected televisions), robotic devices or systems, or other devices. In some aspects, each device may include one image sensor (e.g., a camera) or multiple image sensors (e.g., multiple cameras) for capturing one or more images. In some aspects, each device may include one or more displays for displaying one or more images, notifications, and / or other displayable data. In some aspects, each device may include one or more speakers, one or more light-emitting devices, and / or one or more microphones. In some aspects, each device may include one or more sensors. In some cases, the one or more sensors may be used to determine the location of the device, the state of the device (e.g., tracking state, operating state, temperature, humidity level and / or another state) and / or for other purposes.

[0029] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to define the scope of the claimed subject matter. This subject matter should be understood with reference to the appropriate portions of the entire specification, any or all drawings, and each claim.

[0030] The foregoing and other features and aspects will become more apparent from the following description, claims and accompanying drawings. Attached Figure Description

[0031] The following description, with reference to the accompanying drawings, details exemplary examples of this application: Figure 1 This is a block diagram illustrating an example specific implementation of a system that may include a central processing unit (CPU) configured to perform one or more of the functions described herein. Figure 2 It includes two sets of images that illustrate the forward diffusion process (which is fixed) and the reverse diffusion process (which is learned) of the diffusion model. Figure 3 This is a diagram illustrating how a diffusion model can be used, according to some aspects of this disclosure, to distribute diffusion data from initial data to noise in the forward diffusion direction; Figure 4This is a diagram illustrating a U-Net architecture for a diffusion model according to some aspects of this disclosure; Figure 5 This is a block diagram of an example potential diffusion model 500 illustrating the steps of a potential diffusion process according to various aspects of this disclosure; Figure 6 Includes two images to illustrate examples of modified image data or modified video data; Figure 7 Includes two images to illustrate examples of modified image data or modified video data; Figure 8 Includes multiple circles representing tokens of an image; Figure 9A This is a block diagram illustrating an example image / video modification system according to various aspects of this disclosure; Figure 9B This is a block diagram illustrating an example image / video modification system according to various aspects of this disclosure; Figure 10 Includes three frames representing tokens according to various aspects of this disclosure; Figure 11 Examples of images processed through various steps of the diffusion process according to various aspects of this disclosure; Figure 12 This is a block diagram illustrating an example system of a diffusion-mixing process that can be realized according to various aspects of this disclosure; Figure 13 This is a block diagram illustrating an example system that can realize the diffusion-mixing process; Figure 14 This is a flowchart illustrating examples of processes for modifying video data according to various aspects of this disclosure; Figure 15 This is a flowchart illustrating examples of processes for modifying video data according to various aspects of this disclosure; Figure 16 This is a flowchart illustrating examples of processes for modifying image data according to various aspects of this disclosure; Figure 17 This is a flowchart illustrating examples of processes for modifying image data according to various aspects of this disclosure; Figure 18 This is a flowchart illustrating examples of processes for modifying image data according to various aspects of this disclosure; Figure 19 This is a block diagram illustrating examples of deep learning neural networks that can be used to implement a perception module and / or one or more verification modules according to various aspects of this disclosure; Figure 20This is a block diagram illustrating examples of convolutional neural networks (CNNs) according to various aspects of this disclosure; and Figure 21 This is a block diagram illustrating an example computing device architecture that can implement the various technologies described herein. Detailed Implementation

[0032] Certain aspects of this disclosure are provided below. Some of these aspects may be applied independently, and some may be applied in combination, as will be apparent to those skilled in the art. Specific details are set forth in the following description for purposes of explanation in order to provide a thorough understanding of the various aspects of this application. However, it will be apparent that various aspects may be practiced without these specific details. The accompanying drawings and descriptions are not intended to be limiting.

[0033] The following description provides only exemplary aspects and is not intended to limit the scope, applicability, or configuration of this disclosure. Rather, the following description of exemplary aspects will provide those skilled in the art with descriptions that can be used to implement the exemplary aspects. It should be understood that various changes can be made to the function and arrangement of the elements without departing from the scope of this application as set forth in the appended claims.

[0034] The terms “exemplary” and / or “example” are used herein to mean “serving as an example, instance, or illustration.” Any aspect described herein as “exemplary” and / or “example” is not necessarily to be construed as superior to or better than other aspects. Similarly, the term “aspects of this disclosure” does not require that all aspects of this disclosure include the features, advantages, or modes of operation discussed.

[0035] Diffusion-based video editing techniques have achieved impressive quality and can transform input videos by, for example, replacing objects and attributes of interest or compositing frames according to a desired style. However, such solutions typically require high computational costs to preserve temporal consistency in the generated frames, such as through diffusion inversion within a self-attention module and / or cross-frame interaction.

[0036] This paper describes systems and techniques for incorporating several methods to achieve efficiency when modifying image data (e.g., video content, such as one or more frames of a video). According to some aspects, machine learning models (e.g., neural network models) may include a spatiotemporal token merging mechanism to fuse redundant tokens during inference by the machine learning model, which can improve the speed of spatiotemporal attention (e.g., in zero-shot techniques). Additionally or alternatively, in some cases, the machine learning model may include a saliency-based merging mechanism that provides a trade-off between the ratio of retained tokens in the foreground and background regions of a frame (e.g., a video frame), thereby allowing adaptive resource allocation. In some examples, the computational cost of the machine learning model can be further reduced by completely avoiding the diffusion process on certain regions of the frame (e.g., the background region of the frame), which may be unnecessary when editing other parts of the frame (e.g., the shape or attributes of foreground objects).

[0037] Various aspects of this application will be described below with reference to the accompanying drawings.

[0038] Figure 1 An example implementation of system 100 is illustrated, which may include a central processing unit (CPU 102) (which may be a multi-core CPU) configured to perform one or more of the functions described herein. Parameters or variables (e.g., neural signals and synaptic weights), system parameters associated with computing devices (e.g., a weighted neural network), task information, and other information may be stored in a memory block associated with the neural processing unit (NPU 108), a memory block associated with the CPU 102, a memory block associated with the graphics processing unit (GPU 104), a memory block associated with the digital signal processor (DSP 106), memory 116, and / or may be distributed across multiple blocks. Instructions executed at the CPU 102 may be loaded from the program memory associated with the CPU 102 or may be loaded from memory 116.

[0039] System 100 may also include additional processing blocks tailored for specific functions, such as GPU 104, DSP 106, connectivity engine 118 (which may include fifth-generation (5G) connectivity, fourth-generation LTE (4G LTE) connectivity, Wi-Fi connectivity, USB connectivity, Bluetooth connectivity, etc.), and multimedia processor 112 capable of, for example, detecting and recognizing gestures. In one specific implementation, the NPU is implemented in CPU 102, DSP 106, and / or GPU 104. System 100 may also include one or more sensor processors 114, one or more image signal processors (ISP 110), and / or navigation engine 120, which may include a global positioning system. In some examples, sensor processor 114 may be associated with or connected to one or more sensors for providing sensor input to sensor processor 114. For example, the one or more sensors and sensor processor 114 may be provided in the same computing device, coupled to the same computing device, or otherwise associated with the same computing device.

[0040] System 100 may be implemented as a system-on-a-chip (SoC). System 100 may be based on an Advanced Reduced Instruction Set Computer (RISC) machine (ARM) instruction set. System 100 and / or its components may be configured to perform machine learning techniques according to various aspects of this disclosure discussed herein. For example, system 100 and / or its components may be configured to implement machine learning models (e.g., quantized trained machine learning models) as described herein and / or according to various aspects of this disclosure.

[0041] Machine learning (ML) can be considered a subset of artificial intelligence (AI). ML systems can include algorithms and statistical models that computer systems can use to perform various tasks through pattern-dependent inference without explicit instructions. An example of an ML system is a neural network (also known as an artificial neural network), which can include groups of interconnected artificial neurons (e.g., neuron models). Neural networks can be used in a variety of applications and / or devices, such as image and / or video decoding, image analysis and / or computer vision applications, Internet Protocol (IP) cameras, Internet of Things (IoT) devices, autonomous vehicles, service robots, and more.

[0042] Individual nodes in a neural network mimic biological neurons by taking input data and performing simple operations on that data. The results of these simple operations on the input data are selectively passed to other neurons. Weights are associated with each vector and node in the network, and these values ​​constrain how the input data relates to the output data. For example, the input data of each node can be multiplied by its corresponding weight value, and the products can be summed. The sum of the products can be adjusted with optional biases, and activation functions can be applied to the results to produce the node's output signal or "output activation" (sometimes called a feature map or activation map). The weights can initially be determined by an iterative stream of training data through the network (e.g., weights are established during training phases where the network learns how to identify a particular category based on the characteristics of its typical input data).

[0043] There are different types of neural networks, such as Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), Generative Adversarial Networks (GANs), Multilayer Perceptron (MLP) neural networks, Transformer Neural Networks, and Diffusion-based Neural Networks. For example, a Convolutional Neural Network (CNN) is a feedforward artificial neural network. A CNN may comprise a collection of artificial neurons, each possessing a receptive field (e.g., a localized region of the input space) and collectively tiling the input space. RNNs work on the principle of storing the layer's output and feeding that output back to the input to help predict the layer's outcome. A GAN is a generative neural network that learns patterns in the input data so that the neural network model can generate new synthetic outputs, which may reasonably come from the original dataset. A GAN may comprise two neural networks operating together: a generative neural network that generates the synthetic output and a discriminative neural network that evaluates the authenticity of the output. In an MLP neural network, data is fed into the input layer, and one or more hidden layers provide an abstraction level to the data. The output layer can then be predicted based on this abstract data.

[0044] Deep learning (DL) is an example of machine learning techniques and can be considered a subset of ML. Many DL methods are based on neural networks, such as RNNs or CNNs, and utilize multiple layers. Using multiple layers in a deep neural network allows for the progressive extraction of higher-level features from a given raw data input. For example, the output of the first layer of artificial neurons becomes the input of the second layer, the output of the second layer becomes the input of the third layer, and so on. The layers located between the input and output of the entire deep neural network are often called hidden layers. Hidden layers learn (e.g., are trained) by transforming intermediate inputs from previous layers into slightly more abstract and complex representations that can be provided to subsequent layers until the final or desired representation is obtained as the final output of the deep neural network.

[0045] As noted above, neural networks are examples of machine learning systems and can include an input layer, one or more hidden layers, and an output layer. Data is provided from input nodes in the input layer, processed by hidden nodes in one or more hidden layers, and output is produced by output nodes in the output layer. Deep learning networks typically include multiple hidden layers. Each layer of a neural network can include a feature map or activation map, which can include artificial neurons (or nodes). Feature maps can include filters, kernels, etc. Nodes can include one or more weights used to indicate the importance of nodes in one or more layers. In some cases, deep learning networks may have a series of many hidden layers, where earlier layers are used to determine simple and low-level properties of the input, and later layers build a hierarchy of more complex and abstract properties.

[0046] Deep learning architectures can learn hierarchical structures of features. For example, if presented with visual data, the first layer can learn to recognize relatively simple features in the input stream, such as edges. In another example, if presented with auditory data, the first layer can learn to recognize spectral power at specific frequencies. The second layer, taking the output of the first layer as input, can learn to recognize combinations of features, such as simple shapes in visual data or combinations of sounds in auditory data. For example, higher layers can learn to represent complex shapes in visual data or words in auditory data. Even higher layers can learn to recognize common visual objects or spoken phrases.

[0047] Deep learning architectures perform particularly well when applied to problems with a natural hierarchical structure. For example, the classification of motorized vehicles can benefit from first learning to identify features such as wheels, windshields, and others. These features can then be combined in different ways at higher levels to identify cars, trucks, and airplanes.

[0048] Neural networks can be designed to have multiple connectivity patterns. In feedforward networks, information is passed from lower layers to higher layers, where each neuron in a given layer communicates with neurons in higher layers. As described above, hierarchical representations can be built in successive layers of a feedforward network. Neural networks can also have recurrent or feedback (also known as top-down) connections. In recurrent connections, the output from a neuron in a given layer can be passed to another neuron in the same layer. Recurrent architectures can help identify patterns across more than one block of input data that is sequentially delivered to the neural network. Connections from neurons in a given layer to neurons in lower layers are called feedback (or top-down) connections. Networks with many feedback connections can be helpful when the recognition of higher-level concepts can aid in discerning specific lower-level features of the input.

[0049] Figure 2Two sets of images 200 are provided, illustrating the forward diffusion process (which is fixed) and the reverse diffusion process (which is learned) of the diffusion model. For example... Figure 2 As shown in the forward diffusion process, noise 204 is gradually added to the first set of images 202 at different time steps in a total of T time steps (e.g., forming a Markov chain), thereby generating a series of noisy samples X1 to X2. T .

[0050] From a training perspective, the diffusion model acquires an image and slowly adds noise to it to destroy the information within the image. In some respects, the noise 204 is Gaussian noise. Each time step can correspond to... Figure 2 Each consecutive image in the first set of images 202 shown. Figure 2 The initial image X0 is an image of a cat. Noise 204 is added to each image (corresponding to noisy samples X1 to X). T This causes the pixels in each image to gradually spread out until the final image (corresponding to sample X) is reached. T This essentially matches the noise distribution. For example, by adding noise, as the time step increases, each data sample X1 to X... T Gradually losing its distinguishable features, it eventually leads to the final sample X T Equivalent to the target noise distribution, such as a unit variance, zero Gaussian distribution. .

[0051] The second set of images, 206, illustrates the reverse diffusion process, where X... T It is the starting point for noisy images (e.g., images with Gaussian noise). The diffusion model can be trained to reverse the diffusion process (e.g., by training model p). θ- (x t-1 | x t This generates new data. In some respects, the diffusion model can be trained by finding the inverse Markov transformation that maximizes the likelihood of the training data. By traversing backward along the time-step chain, the diffusion model can generate new data. For example, as... Figure 2 As shown, inverse diffusion is then performed to generate X0 as an image of the cat. In other cases, the input and output data may vary depending on the task for which the diffusion model was trained.

[0052] As noted above, the diffusion model is trained to denoise or restore the original image X0 in a progressive process, as shown in the second set of images 206. In some respects, the neural network of the diffusion model can be trained to denoise or restore the original image X0 in a progressive process. t-1 Restore X in the case t As shown in the following example equation:

[0053] The diffusion nucleus can be defined as follows: definition

[0054] Sampling can be defined as follows:

[0055] In some cases, Value scheduling (also known as noise scheduling) is designed to make and .

[0056] The diffusion model runs iteratively to progressively generate the input image X0. In one example, the model may have twenty steps. However, in other examples, the number of steps can vary.

[0057] Figure 3 This is an illustration of how a diffusion model can be used to distribute diffusion data from initial data to noise in the forward diffusion direction, based on some aspects. Note that the initial data q(X0) is detailed in the initial stage of the diffusion process. An illustrative example of data q(X0) is... Figure 2 The image shown is the initial image of the flowers in the vase. As the diffusion model iterates and sampling noise is iteratively added to the data from t=0 to t=T, as... Figure 3 As shown, the data becomes noisier and may eventually result in pure noise (e.g., in q(X)). T ) place). Figure 3 The example illustrates the progress of the data and how the data spreads along with noise during the forward diffusion process.

[0058] In some respects, the distribution of diffuse data (e.g., such as...) Figure 3 (As shown) can be as follows: .

[0059] In the above equation, Indicates the distribution of diffusion data. Indicates the joint distribution. This represents the distribution of the input data, and It is a diffusion kernel. In this respect, the model can be improved by first sampling... And then sample To sample (This can be called ancestor sampling). The diffusion kernel takes input and returns a vector or other data structure as output.

[0060] The following is an overview of the training and sampling algorithms for the diffusion model. The training algorithm may include the following steps: repeat Perform gradient descent steps on the following expression Until convergence The sampling algorithm may include the following steps: for Finish return

[0061] Figure 4 This is an illustration of a U-Net architecture 400 used for a diffusion model, based on some aspects. An initial image 402 (e.g., flowers in a vase) is provided to the U-Net architecture 400, which includes a series of Residual Network (ResNet) blocks and self-attention layers to represent the network. (x t The U-Net architecture 400 also includes a fully connected layer 410. In some cases, the time representation 412 can be a sinusoidal position embedding or a random Fourier feature. Noise output 408 from the forward diffusion process is also shown.

[0062] U-Net architecture 400 includes, for example Figure 4 The contraction path 404 and expansion path 406 are shown, giving it a U-shaped architecture. The contraction path 404 can be a convolutional network comprising repeated convolutional layers (which apply convolutional operations), each followed by a rectified linear unit (ReLU) and max-pooling operation. When processing an image (e.g., image 402) during the contraction path 404, the spatial information of image 402 is reduced as features are generated. The expansion path 406 combines features and spatial information through a series of up-convolutions and cascading with high-resolution features from the contraction path 404. Some layers can be self-attention layers, which explicitly model complete contextual information by leveraging global interactions between semantic features at the encoder ends.

[0063] Latent diffusion models (also known as stable diffusion models) introduce a diffusion process into the latent space of machine learning models (e.g., variational autoencoder (VAE) neural networks), thereby making the machine learning models more efficient and enabling high-resolution image synthesis. For example, the encoder of a VAE (… -decoder ) can be trained to capture by Given a low-dimensional latent distribution, such that This can be achieved by training a U-Net (e.g., Figure 4 The U-Net architecture 400 (which may include ResNet blocks and attention modules in some cases) is used to predict the noise introduced during the forward diffusion process to formulate the aforementioned denoising process in the latent space, which optimizes the objective given below:

[0064] here, It is the scheduler Introducing a noise-free latent value in each step Total noise, It is the diffusion time step The corresponding part at that location has a noise latent value, and This involves conditioning (e.g., embedding text prompts provided as input). Utilizing predictive noise. , can Within a step Iteratively apply denoising diffusion implicit model (DDIM) sampling to recover the original latent data distribution. Such as in the following:

[0065] in These are the parameters of the noise dispatcher.

[0066] When using Steady Diffusion (SD) for video generation or editing, a key factor is ensuring the temporal consistency of the generated frames relative to one or more previous frames in the video. In addition to modifications to the U-Net model (such as temporal attention and 2+1D convolutions), it also facilitates denoising that relies on control signals and / or DDIM inversion to begin with a set of relevant noise latent values.

[0067] Figure 5 This is a block diagram illustrating the steps of a potential diffusion process that can be implemented according to various aspects of this disclosure, representing an example potential diffusion model 500. The potential diffusion model 500 can modify video or image data. The potential diffusion model 500 can be an example of a video (or image) editing system. The principles described herein can be applied to potential diffusion models (such as...) Figure 4 , Figure 5Implemented (and / or used) in the potential diffusion model illustrated in the example and / or other diffusion models.

[0068] Figure 6 Includes two images to illustrate examples of modified image data or modified video data. For example, Figure 6 Image 602, which may be a frame of video data, can be provided as input to an image editing or video editing system (e.g., Figure 5 Potential diffusion model 500). Figure 6 Includes a target cue 608 that can be provided together with image 602 to an image editing or video editing system. Figure 6 It also includes image 606, which can be a frame of video data that can be output as an image editing or video editing system. The image editing or video editing system can modify image 602 according to target cues 608 to generate image 606. Figure 6 An example of shape editing is shown, in which the shape of an object in image 602 is modified in image 606.

[0069] Figure 7 Includes two images to illustrate examples of modified image data or modified video data. For example, Figure 7 This includes an image 702, which may be a frame of video data. Image 702 can be provided as input to an image editing or video editing system (e.g., Figure 5 Potential diffusion model 500). Figure 7 Includes a target cue 708 that can be provided together with image 702 to an image editing or video editing system. Figure 7 It also includes image 706, which can be a frame of video data that can be output as an image editing or video editing system. The image editing or video editing system can modify image 702 to generate image 706 based on target cues 708. Figure 7 An example of attribute editing is shown, in which the attributes of an object in image 702 are modified in image 706.

[0070] Video editing systems (e.g., Figure 5 The latent diffusion model 500 can take video (e.g., including image 602 or image 702) and / or one or more images and target cues (e.g., target cues 608 or target cues 708) as input. The target cues describe how the user wants the output video or image to look. Figure 6 The example illustrates shape editing, where a Jeep is shown driving in image 602. The goal cue 608 is to transform the Jeep into a Porsche. Attribute editing preserves the shape of an object but changes one of its styles or adds to the object's properties. For example, Figure 7 This demonstrates shape editing. In Figure 7In the image 702, the target cue 708 is to transform the swan in the image 702 into a Swarovski crystal swan with a red beak, swimming in the river near the wall and bushes.

[0071] Diffusion models have become widely used as generative artificial intelligence (AI) / machine learning (ML) solutions for generating images and videos. Diffusion models for video editing work as follows: A latent diffusion model (LDM) can be pre-trained on data. Given a text prompt, an LDM typically generates a single image. A stable diffusion model is an example of an LDM. Given an image model, an LDM forces the creation of frames that are temporarily consistent with each other, such that they represent a video stream.

[0072] For example, a video may include a background and objects that move coherently and naturally within the frames of the video. Several techniques can be used to advance these potential diffusion models to achieve temporal consistency. One technique is called diffusion inversion. Another technique is called conditioning. Conditioning involves inputting additional control signals that are temporarily relevant. Conditioning can review all frames to create the next frame. Additionally or alternatively, cross-frame operation can be enabled to facilitate temporal consistency. Enabling cross-frame operation can be applied with or without inversion and / or conditioning.

[0073] Video modification techniques are computationally expensive. For example, editing images and / or videos (e.g., to modify the shape or properties of an object) can be computationally expensive. In this disclosure, the term computationally expensive can refer to computational operations that may take time and / or consume power. Modifying a single image may take several seconds. The computational cost of video modification may prevent these types of models (e.g., LDM) from modifying video data in real time.

[0074] A diffusion model operates on a series of diffusion steps, for example, in a chain of diffusion steps (e.g., applying a unit model of the latent variables 20 to 50 times to noise). A chain of diffusion steps can be computationally expensive. One reason for this is that the unit model is based on a self-attention operation, which can be computationally expensive. In other words, a latent diffusion model (LDM) iteratively runs a denoising model (e.g., between 20 and 50 diffusion steps). Each step may include a computationally expensive self-attention operation.

[0075] In some cases, machine learning models may output tokens based on processed frames. In this disclosure, the term "token" may refer to a feature vector of a specific portion of the input. In the context of a video frame, a token may refer to a feature vector comprising values ​​representing visual characteristics of a portion or region of the video frame (e.g., a location within the video frame). When self-attention is performed, tokens interact with other tokens (e.g., the self-attention operation processes the values ​​of various tokens). In this context, a token can be a pixel in the representation space of a cell. A token can represent a spatial location in the latent space of a cell. As noted above, self-attention is an expensive operation. If redundant tokens exist for an image (e.g., tokens that look very similar to each other), the redundant tokens carry redundant information. Redundant tokens can potentially be removed from the self-attention operation, making self-attention more efficient.

[0076] Token merging is a technique that can be used to remove redundant tokens. For example, similar tokens can be merged into a single token because they carry redundant information. Tokens that are not similar to other tokens remain unmerged. Token merging can speed up LDM by approximately 2x (or more) by reducing the number of tokens.

[0077] Figure 8 This includes multiple circles representing tokens that represent three images (e.g., three consecutive frames of video data). Specifically, Figure 8 The representation includes tokens for three image frames (frames 802, 806, and 810), with nine tokens per frame. Each token may be or may include a feature vector comprising feature values ​​representing the corresponding position in the frame. Tokens are exemplified relative to the corresponding position in a particular frame. For example, token 804 may be a feature vector comprising feature values ​​representing the upper-left portion of frame 802.

[0078] As noted above, token merging can be used to remove redundant tokens. For example, token merging can include one or more destination tokens for each frame identifier (e.g., through random sampling). Figure 8 In the example, for simplicity, one destination token per frame is illustrated, including token 804 for frame 802, token 808 for frame 806, and token 812 for frame 810. In other examples, more destination tokens may be identified. Destination tokens will be retained (e.g., not removed during token merging). Token merging may include measuring the distance of all other tokens to the destination token (e.g., determining the cosine similarity between each token and the destination token). Tokens similar to the destination token (e.g., based on a similarity threshold) may be merged (e.g., through average pooling). Therefore, in Figure 8In the illustrated case, for frame 802, there are five merged tokens similar to destination token 804. The merged tokens are merged into destination token 804 through average pooling. The merged tokens can then be removed. This leaves some tokens that are not similar to the destination token and will not be merged; these can be referred to as unmerged tokens. After this operation, the destination token and the unmerged tokens are retained. The merged tokens are removed from further computation. Figure 8 In the illustrated example, four tokens are retained per frame. This speeds up computation. Therefore, token merging saves computational resources, making image and / or video editing using LDM less computationally expensive.

[0079] Token merging can be applied to one or more (e.g., each) of the self-attention modules in an LDM. For example, an LDM may include a U-Net architecture with an encoder and a decoder (e.g., as shown in the image). Figure 5 The potential diffusion model is illustrated in 500. Self-attention modules can exist throughout the encoder and decoder. Token merging can be applied to one or more of the self-attention modules. For example, each unit block of an LDM may include a self-attention module, a cross-attention module, and some residual convolutions. For all diffusion steps, token merging can be applied to each layer before each self-attention module.

[0080] When applying stable diffusion to video, a self-attention module can apply self-attention to tokens within each frame. A self-attention module can function in a similar manner to a cross-frame attention module (e.g., a module that applies cross-attention across multiple frames), such as when a query token comes from one frame, but the key and value come from another. This differs from performing cross-attention on text tokens (e.g., text hints). The cross-frame attention nature of a self-attention module is not depicted in the accompanying figures. Additionally or alternatively, token merging can be applied when performing cross-frame attention.

[0081] LDMs can use tokens by projecting them using linear projection. In some cases, there might be layers (of the LDM) that take tokens and output queries, keys, and values ​​for each token. An LDM can use as many queries, keys, and values ​​as tokens. For example, tokens can be projected, and for each input token, there could be one query, one key, and one value. In such examples, the cost of computing self-attention is quadratic with respect to the number of tokens.

[0082] Token merging (ToMe) can be implemented as a zero-sample plugin for Visual Transformers (ViTs), used to reduce computational requirements by removing redundant tokens. In contrast to token pruning methods, ToMe introduces a lightweight merging mechanism based on token similarity. For example, tokens with the highest cosine similarity to any other token are grouped together to save computation (and improve throughput). ToMe can be used for stable diffusion-based text-to-image generation, as well as techniques such as token demerging and grid-based token sampling.

[0083] For example, a potential token representation can be provided. Correspondingly, Indicates batch, Indicates the number of frames. Indicates the number of tokens, and Indicates the number of channels. Number of tokens. It can be further expressed as This represents the height (e.g., the number of pixels in the vertical direction) and the width (e.g., the number of pixels in the horizontal direction). To calculate the similarity between tokens, the tokens can be divided into two sets, including destination tokens. Heyuan Token . The destination token in the data can be sampled to be evenly distributed across each frame, such as by using a size-based method. A two-dimensional (2D) grid is used to select the sampling index.

[0084] From integers and Parameterization. Here, rand( . ) is a pseudo-random integer generator, parameterized by an upper bound. The remaining tokens are grouped frame by frame to form .

[0085] Next, the source token and destination token The similarity between them can be calculated for each frame, such as the following:

[0086] in This represents the cosine similarity between two vector sets. Finally, it represents the vector with the highest similarity to any destination token within the frame. The source tokens are merged (via average pooling), essentially matching the most redundant source tokens with their corresponding destinations. Such merged tokens are... Indicates that the unmerged token is... This indicates that what remains after the merge operation is [x].dst , x unm The set of [key-value pairs]. ToMe is typically applied before the attention block, where computational savings can be maximized. It can be applied only to key-value pairs, or to all query-key-value triples. In both cases, the indexes computed in Equation 1 should be shared among the key-value pairs. When ToMe is applied to triples, the output representation should be demerged after the attention operation to preserve the original shape (or resolution), especially for generative tasks. This is done via token demerging, which simply copies the merged tokens to their original locations based on the same set of indexes.

[0087] The system and techniques described herein enable three-dimensional (3D) token merging. In some cases, 3D token merging provides a token reduction technique for images and / or videos that can be implemented with Lead-to-Me token merging (ToMe), in which case 3D token merging may be referred to herein as 3D ToMe. ToMe can be implemented for a visual transformer (ViT) in the image domain. A naive extension would apply ToMe individually for each frame. There are at least two limitations to this approach: (1) it does not account for temporal redundancy, and (2) it does not maximize the information retained after merging. In the case of video input, redundant information exists in the timeline (e.g., across time from frame to frame), such as at common frame rates (e.g., 25-30 frames per second (fps)). If ToMe is applied individually to each frame, it will still retain some tokens per frame, which may be unnecessary. Conversely, if some temporal tokens (e.g., tokens with the same information across time) can be discarded, the potential results include a higher reduction rate (and better latency). Furthermore, the destination token index sampled for ToMe does not need to be the same for each frame. If different randomizations are used for each frame (see Equation 1), the 3D token merging technique described in this paper can preserve different information fragments. 3D ToMe addresses the aforementioned limitations of ToMe. First, according to the 3D ToMe technique, the system can define a method for merging tokens of size ( Destination token () The spatiotemporal grid for sampling is as follows:

[0088] From integers Parameterization. This allows the system reduction to be preserved as The number of tokens, in this case, the system can allocate them according to each time window. Instead of sampling every frame, this is done by controlling the time window. There may be trade-offs in terms of time redundancy. The remaining tokens are considered... The system can utilize two options to calculate the similarity between the source and the destination, including: (1) Window Time Search (WTS), where the similarity is calculated separately for each time window. (1) Similarity between tokens within a frame, or (2) Global Time Search (GTS), where similarity is calculated once between all tokens across all frames.

[0089] Compared to window-time search, each source can only match destinations within the same time window, thus providing more control over time merging, for example, based on the following:

[0090] For global temporal search, each source can be matched to any destination within the entire spatiotemporal volume, such as based on the following:

[0091] In some cases, to maximize the information retained across frames, different randomized indices can be computed to sample the destination token for each time window. Similar to 2D ToMe, if 3D ToMe is applied to query-key-value triples, the same demerging operation can be performed after the attention layer.

[0092] Figure 9A The diagram on the left illustrates an example image / video modification system (e.g., System 900A) (e.g., a latent diffusion model (LDM)). Figure 9A It also includes an example layer unfolded diagram of the encoder layer of LDM, which includes a 3D token merging module. Figure 9A It also includes a view of the 3D token merging module's operations on the tokens in three example frames.

[0093] Figure 9A The leftmost column (column 902) illustrates foreground-only diffusion, where the latent (foreground) portion undergoes all diffusion steps and the latent (background) portion skips the first T_fg step. Regarding Figure 12 and Figure 13 Additional details are provided regarding how to make a portion of the potential value skip at least some diffusion steps.

[0094] Figure 9A The second column from the left (column 904) illustrates a typical configuration of a layer within the leftmost Unet. The second column from the left (column 904) also illustrates where 3D token merging is applied (e.g., before spatiotemporal attention).

[0095] Figure 9AThe third column from the left (column 906) illustrates the difference between the original token merging (top) and the 3D token merging (bottom). Fewer destination tokens are present in the 3D token merging example. Further token merging across frames saves computational resources. About Figure 10 Additional details regarding token merging are provided.

[0096] Figure 9A The rightmost column (column 908) illustrates saliency-based token merging. For example, when calculating cosine similarity between tokens, foreground / background information (e.g., segmentation map) is used to artificially reduce the weight of similarity among foreground tokens, thereby reducing the probability that they will be merged into the destination tokens. This helps preserve the quality of the foreground regions.

[0097] Figure 10 This includes representation tokens for three frames (frames 1002, 1012, and 1022). A destination token (e.g., token 1014) is selected for all three frames based on 3D token merging. Then, the merged tokens (e.g., tokens 1006, 10016, and 1026) are identified across all three frames (e.g., based on the cosine similarity between the tokens of all three frames and a single destination token). Figure 10 In the illustrated example, there are two unmerged tokens per frame. This illustrates a significant reduction in the number of tokens, resulting in a significant saving in computational resources.

[0098] 3D token merging can leverage temporal redundancy. Furthermore, 3D token merging allows for a flexible trade-off between spatial and temporal reduction for the same cost. 3D token merging allows for more relevant tokens in a larger pool (e.g., a pool including tokens from multiple frames). Further 3D token merging enables long-term temporal attention, which might otherwise be infeasible.

[0099] 3D token merging can involve tokens from different frames interacting with each other during the merging phase. Figure 10 In this process, all tokens from three different frames are placed into the same pool from which a single destination token is sampled. Within this pool, similarity is determined and token merging is performed. 3D token merging leverages temporal redundancy between frames. 3D token merging also uses a larger token pool to potentially find tokens related to the destination token. This allows for a higher probability of finding merged tokens similar to the destination token. By doing so, 3D token merging enables models that rely on temporal attention to operate with reasonable latency. Without 3D token merging, such models would be too slow for many applications. 3D token merging can be applied to self-attention modules, allowing these modules to operate without frame isolation.

[0100] There are several ways to select the frames to which 3D token merging is applied. As another example, a pool of any number of frames from the video data (e.g., 8 or 16 frames) can be selected. After 3D token merging of the pool, a subsequent pool of frames can be selected and 3D merged. As another example, frames can be 3D merged within a sliding window. As yet another example, all frames from the video data can be 3D merged simultaneously.

[0101] Figure 9B This is a block diagram illustrating an example system 900B capable of performing 3D token merging according to various aspects of this disclosure. For example, system 900B may obtain frames 912, 914, and 916. Frames 912, 914, and 916 may be frames of video data (e.g., consecutive frames).

[0102] The token extractor 918 of system 900B can generate token 920 based on frame 912, token 922 based on frame 914, and token 924 based on frame 916. Each of tokens 920, 922, and 924 may be or may include multiple tokens. Each token may be or may include a feature vector corresponding to a corresponding position within the corresponding frame.

[0103] The token merger 926 of system 900B can determine the destination token from the tokens of one of frames 912, 914, or 916. For example, the token merger 926 can determine the destination token from token 922. In some aspects, the token merger 926 can randomly select the destination token.

[0104] Furthermore, token merger 926 can determine candidate tokens from tokens 920, 922, and 924. In some aspects, token merger 926 can determine candidate tokens based on the relationship between candidate tokens and destination tokens. For example, token merger 926 can determine candidate tokens based on the cosine distance between candidate tokens and destination tokens.

[0105] Furthermore, token merger 926 can merge candidate tokens with destination tokens. Token merger 926 can generate token 928. Token 928 may include modified tokens. Token 928 may include tokens based on each of frames 912, 914, and 916.

[0106] Token 928 can be provided to diffusion model 930, and diffusion model 930 can generate frame 932 based on token 928. Frame 932 may include multiple frames. Diffusion model 930 can be... Figure 9A An example of a diffusion model in column 902.

[0107] System 900B may obtain (e.g., video data) a group of frames; the group of frames includes frames 912, 914, and 916. In some aspects, system 900B may select a group of frames from the frames of the video data. In some aspects, after generating frame 932 based on the group of frames, system 900B may select another group of frames from the video data and process that other group of frames to generate another frame based on that other group of frames.

[0108] In some respects, a frame group can be a frame pool comprising frames 912, 914, and 916. For example, system 900B can divide frames of video data into pools and process frames in a given pool one at a time. In such a case, after processing a frame pool comprising frames 912, 914, and 916, system 900B can process another frame pool comprising three additional frames (e.g., frames after frame 916 in the video data).

[0109] In some respects, the group can be a sliding frame window. For example, after processing a frame window that includes frames 912, 914, and 916, the system 900B can process a frame window that includes frames 914, 916, and additional frames (e.g., frames after frame 916).

[0110] Additionally or alternatively, system 900B may implement saliency-based merging. For example, regardless of whether system 900B processes multiple frames (e.g., frames 912, 914, and 916) together (e.g., based on 3D token merging), system 900B may implement saliency-based token merging by merging at least token 920 based on segmentation mask 936.

[0111] In some aspects, system 900B may include segmenter 934, which may generate segmentation mask 936 based on one or more of frames 912, 914, and / or 916. Alternatively, system 900B may obtain segmentation mask 936 from another source (e.g., a segmenter external to system 900B). Segmentation mask 936 may be or may include foreground-background segmentation, saliency segmentation, and / or a cross-attention map from the latent representation.

[0112] In any case, token merger 926 can merge tokens based on segmentation mask 936. Where system 900B implements 3D token merging and significance-based token merging, token merger 926 can merge tokens 920, 922, and 924 based on segmentation mask 936. In any case, token merger 926 can merge tokens based on segmentation mask 936. Where system 900B implements significance-based token merging without implementing 3D token merging, token merger 926 can merge token 920 based on segmentation mask 936.

[0113] To merge tokens (e.g., tokens 920, 922, and / or 924) based on segmentation mask 936, token merger 926 may weight the correspondence between candidate tokens and destination tokens based on corresponding portions of the segmentation mask. This weighting makes it less likely that tokens corresponding to salient portions of image data, such as those identified by the segmentation mask, will be identified as candidate tokens.

[0114] Alternatively, candidate tokens can be determined based on the relationship between the destination token and the candidate token. This relationship can be determined as cosine similarity. Candidate tokens can also be determined based on a similarity threshold. For example, candidate tokens can be determined based on the relationship between the candidate token and the destination token satisfying a similarity threshold.

[0115] According to several aspects, the systems and techniques described herein can perform saliency-based token merging. For example, in generative modeling, users often focus more on salient regions of the generated image / video. This can be a single or multiple foreground objects in a given input. Particularly for video editing, the input often includes a prominent foreground, for which any inconsistencies (e.g., flickering) in the edited output are highlighted during viewing. Saliency-based token merging could involve allocating more computational budget to salient regions to preserve better quality in such regions without necessarily sacrificing the quality of other regions.

[0116] Significance-based token merging can be achieved through Token Merging (ToMe). ToMe may involve sampling a subset of the most informative tokens. Significance-based token merging may involve forcing a higher percentage of such tokens to originate from the salient region within the same frame.

[0117] As previously described, which tokens are merged is based on the similarity between the calculated source and destination tokens. Saliency-based token merging may disregard the similarity of source tokens corresponding to salient regions. This is done based on saliency masks. For example, a saliency mask may be obtained as an oracle segmentation map or obtained within an oracle segmentation map, defined in the dataset itself, included in a segmentation map extracted by an off-the-shelf model (e.g., a foreground-background segmentation mask or a saliency-based segmentation mask), and / or may be a cross-attention map (corresponding to a specific keyword) within the latent representation of U-Net itself.

[0118] Based on predefined (but controllable) significance strength Significance-based token merging can reweight such similarities, thereby forcing corresponding source tokens not to merge. Given a binary significance mask... The binary saliency mask is obtained through the source token index. Resampling is then performed, and this is used to calculate salient similarity (SSim), as shown below.

[0119] in This refers to the saliency strength. For simplicity, the input for the similarity calculation is omitted here. The merge operation remains unchanged, but it is now based on the updated similarity. A lower η corresponds to a higher number of tokens sampled from the salient region. It's important to note that salience only applies to the tokens retained. ,However Sampling is still based on 3D meshes. This means that tokens will be sampled from non-salient areas (e.g., the background) to achieve reasonable quality in the overall generation.

[0120] For example, in video editing, there may be a well-defined foreground. In many cases, only the foreground needs to be changed (e.g., by shape or attribute editing). Many image / video editing applications internally generate segmentation masks (e.g., Oracle segmentation masks, cross-attention maps from latent representations). Saliency-based token merging encourages merging to occur on background tokens. Merging tokens in background regions can save significant computational resources in areas (of the image or frame) that the user might not be interested in. Saliency-based token merging artificially reduces the similarity of foreground tokens, thus forcing them to remain unmerged. This can lead to adaptive resource allocation (e.g., spending more computation on salient foreground regions).

[0121] Figure 11 This is an example of an image processed through various steps of the diffusion process. Figure 11 The example illustrates destination tokens and unmerged tokens. The default sampling example illustrates destination tokens and unmerged tokens scattered across the image, including both foreground and background. The saliency-based sampling example illustrates that, based on saliency-based token merging, many unmerged tokens are located on foreground objects, and very few merged tokens are located on foreground objects. Token merging in the background can degrade image quality in the background. The absence of merged tokens on foreground objects allows the foreground objects to maintain their image quality.

[0122] There is the same number of destination tokens and unmerged tokens between the default sampling example and the saliency-based sampling example. Therefore, token merging will have the same cost in both cases because the total number of tokens remains unchanged. However, in the case of saliency-based sampling, token merging may reduce the image quality of the reconstructed background, while in the case of default sampling, the reduction in image quality may occur in both the background and foreground.

[0123] Saliency-based token merging encourages merging to occur in the background. It also encourages placing unmerged tokens in the foreground region. For example, saliency-based token merging might place unmerged tokens in the foreground region. Unmerged tokens will simply remain as is; no merging will occur on them. Unmerged tokens will be safely propagated to layers and self-attention. However, destination tokens and merged tokens may be affected by token merging, which could impact the quality of the output image and / or video data.

[0124] Many video editing methods have internally available segmentation masks that can be easily reused for saliency-based token merging. Saliency-based token merging weights the similarity when calculating the similarity of all tokens relative to the destination token. For example, the similarity between each token and the destination token can be determined. Saliency-based token merging determines whether the token being evaluated is in the foreground. For foreground tokens, similarity can be reduced using a scaling factor. This will automatically prevent those tokens from being merged.

[0125] Based on several aspects, the systems and techniques described in this paper can perform diffusion hybridization. In some cases, cross-frame attention (sparse or dense) layers can be the slowest component in a neural network model. However, after applying 3DToMe with a large reduction rate, the Residual Network (ResNet) block emerges as a new bottleneck, being more than twice as slow as the attention block. In a diffusion pipeline based on a full transformer, there may be even more reductions within the same frame. However, since convolutions depend on spatial structure (which is shuffled after ToMe), different strategies may be needed to make inference even faster.

[0126] For shape / attribute editing settings, the focus is on local changes to the input frame (typically, on the foreground object) compared to style editing which focuses on global changes. For example, the edit output for regions of interest (RoIs) is surprisingly good if all uninteresting latent values ​​(e.g., the background) are masked. This means that in such settings, the diffusion pipeline does not need to rely on contextual information at all. Processing only the RoIs will result in further computational resource savings. To put this into practice, diffusion blending uses two components: (1) sampling the ROIs while being compatible with attention layers and convolutional layers, and (2) blending the ROIs with non-ROIs to produce a consistent output.

[0127] Sampling RoIs by selecting latent values ​​corresponding to RoI masks (e.g., saliency masks, segmentation masks, etc.), which can be irregularly shaped, can be computationally efficient compared to sampling RoIs directly. Simultaneously, the 3D spatiotemporal structure should be preserved after sampling to ensure the output is compatible with convolutions. Variants of convolutions that support irregular shapes exist (e.g., deformable, sparse, and partial); however, such implementations are not as optimized (or fast) as convolutions with regular shapes. Furthermore, for shape editing, if only RoIs are sampled and the output is generated, non-overlapping regions may exist between the source RoI and the target. This can become problematic when later replacing and blending non-RoI regions. Therefore, a simpler sampling strategy—cropping regular bounding boxes around the RoI as input—is used. In practice, this works well and provides a reasonable speedup compared to more computationally efficient strategies, and makes blending easier.

[0128] In some cases, systems and techniques can mix RoIs with non-RoIs. For example, when feeding only RoIs through a U-Net editing pipeline, non-RoI regions can be replaced at the output to achieve the desired reconstruction. A naive option would be to simply paste the generated RoIs in situ in the original frame. This can be done in latent space or pixel space, with the former allowing greater flexibility. However, there are two problems with the naive option: (1) the denoising process of U-Net typically introduces a global color shift compared to the original input (even for shape / attribute edits where this is particularly undesirable), and (2) the boundary between RoIs and non-RoIs becomes prominent. When pasting in latent space (before VAE decoding), these problems can be (1) addressed by using the U-Net output with RoI regions. and VAE encoder output To avoid this, the distribution statistics (mean and standard deviation) between the two regions are normalized for the RoI. This is because the generated shapes within the RoI may contain statistical information that is not necessarily expected to be considered (e.g., swans). (The duck's color changes), therefore, in RoI calculations... It may be desirable to use a saliency mask. This is used to mask source and target objects. Note that these statistics are aggregated only across space (not across batches, times, or channels). Normalization can be performed to update the U-Net output. Then paste it into the VAE encoder latent value. Above, as in the following,

[0129] in It's a padding operation that allows the RoI to fit in place within the original frame, and It is an extended version of the saliency mask, which makes the boundary between RoI and non-RoI smoother—which solves the above problem to some extent (2). Blending operations (e.g., Poisson blending) can also be performed in pixel space to further address the problem (2).

[0130] In some cases, the system and techniques can perform diffusion mixing. For example, to improve the above operation, diffusion mixing can be applied to perform mixing during intermediate diffusion steps (rather than after all T diffusion steps). Here, for The noise reduction process can be completed in one step. Execute above to obtain Next, the results can be normalized, pasted, and then fully cropped in an area with similar noise. (That is, in-situ mixing in the potential of RoI + non-RoI), as described above. Then, The remainder of the diffusion step can be performed on the resulting full cut. This can be obtained using denoised diffusion implicit model (DDIM) inversion in an inversion-based editing pipeline without significant additional cost. Diffusion borrowing can be represented as follows:

[0131] This technology allows non-RoI potential to be assessed only through... A diffusion step instead of This allows for the creation of more faithful non-RoI reconstructions in the generated edit sequences, while also saving computation (and latency) spent on them.

[0132] For shape and / or attribute editing, background processing may not be necessary at all. Additionally or alternatively, there may be no loss of foreground reconstruction fidelity due to the lack of background context. Therefore, diffusion blending may include feeding only the foreground latent value into the diffusion process. The result may be fewer tokens to run LDM on, potentially leading to higher inference speeds.

[0133] For example, in a latent diffusion model, there may exist a latent space within the encoder where the latent diffusion process occurs. The diffusion process can be, or may include, an iterative application of a U-Net that occurs over 20 to 50 steps. The resulting latent values ​​can be refined. The refined latent values ​​can be decoded to generate an image. Depending on the diffusion mixing, the expensive latent diffusion process can be skipped for the background portion of the image and applied to the foreground portion. Thus, the foreground portion can be modified by the latent diffusion process, while the background portion remains unchanged.

[0134] Diffusion blending can use a segmentation mask (e.g., the same segmentation mask used for saliency-based token merging). The segmentation mask may come from a cross-attention module or an off-the-shelf saliency model. The latent values ​​of the image can be split (e.g., split into salient and non-salient parts based on the segmentation mask). The image can be cropped to generate a rectangular portion of the image that includes the salient parts of the image (e.g., foreground objects). The cropped portion of the image can undergo a latent diffusion process. The cropped portion can be modified based on the input image and text cue to generate updated foreground latent values. The background portion of the image can bypass the latent diffusion process. The modified cropped portion and background portion can then be recombined before decoding. Diffusion blending reduces the cost of diffusion because diffusion operates on a much smaller image that does not include the background.

[0135] For example, Figure 12 This is a block diagram illustrating a system 1200 capable of performing a diffusion-mixing process according to various aspects of this disclosure. For example, encoder 1210 may extract features (z0) (e.g., latent representations or “latent values”) from input image 1202. Encoder 1210 may be a neural network encoder (e.g., a backbone neural network) trained to extract features from an image. System 1200 may crop (e.g., using encoder 1210) a rectangular foreground portion 1204 from image 1202. For example, system 1200 may crop the foreground portion (represented as...) of feature z0. The corresponding part. In some examples, part 1204 may be selected based on part 1204 associated with the region of interest (e.g., based on a text cue indicating a swan as the region of interest, not shown). Additionally or alternatively, the part may be selected based on a segmentation mask (such as a foreground-background segmentation mask). System 1200 may process the cropped part 1204 using a latent diffusion process 1206. For example, latent diffusion process 1206 may receive a text cue 1208 and may transform an object in the cropped part 1204 (e.g., a foreground object such as a swan) into a duck based on the cue, thereby producing a modified cropped part 1207. The decoder 1212 of system 1200 may then compare the modified cropped part 1207 with the remaining part 1205 of the image part (e.g., the background) (represented as Combined to generate output image 1214.

[0136] Alternatively, portion 1204 and the remaining portion 1205 may not be mixed after the entire diffusion process 1206, but rather mixed at some point within the diffusion process 1206. For example, for some steps of the diffusion process (e.g., the final step or stage), the latent value of the remaining portion 1205 may enter the latent diffusion process 1206. In such an example, the latent value of the remaining portion 1205 may only undergo the final few diffusion steps (e.g., the last or final five diffusion steps), instead of the latent value of the remaining portion 1205 undergoing all the diffusion steps of the latent diffusion process 1206 (e.g., 20 diffusion steps). For example, if the latent diffusion process has 20 diffusion steps, 15 diffusion steps will be performed on the latent value of the selected portion 1204. The remaining portion 1205 of the image may be injected, and the final five diffusion steps will be performed on the latent value of the entire image. In this way, remixing can be performed midway through the diffusion process.

[0137] For example, Figure 13 This is a block diagram illustrating a system 1300 capable of implementing a diffusion-blending process according to various aspects of the present disclosure. For example, a rectangular portion can be cropped from an image. The portion can undergo several diffusion steps of a potential diffusion process. The potential diffusion process can receive text prompts and transform the portion (e.g., a foreground object such as a swan) into a duck based on the prompts. After the portion has undergone multiple diffusion steps, the modified portion can be combined with the remainder of the image. The combined image can then undergo further diffusion steps of the diffusion process.

[0138] The techniques described in this paper (specifically 3D token merging, saliency-based token merging, and diffusion mixing) can be operated individually and / or together. For example, multiple frames of video data can be modified by using 3D token merging to merge tokens across frames during a diffusion process. Merging tokens across frames can save computational resources.

[0139] Furthermore, before performing the diffusion process, the foreground portion of each image can be identified (e.g., based on a saliency segmentation mask). The foreground portion can be a rectangle cropped from the image. The foreground portion can be processed by the diffusion process, and the background process can discard at least some diffusion steps. Processing the foreground portion and discarding at least some diffusion steps is an example of diffusion mixing. Skipping processing on portions of the image saves computational resources.

[0140] Furthermore, when tokens are merged (e.g., within a cropped portion of an image, within a full image frame, across multiple frames, and / or across multiple cropped portions of multiple corresponding images), weights can be modified based on the segmentation map (e.g., according to saliency-based token merging). Modifying weights based on the segmentation mask can induce more token merging in the background portion of the image and leave the foreground portion with fewer token mergings. Diffusion blending can generate a rectangular foreground portion of the image. Saliency-based token merging can identify pixels within the rectangular foreground portion as salient or insignificant and adjust the weights within the rectangular foreground portion accordingly.

[0141] In some cases, the machine learning systems or networks described in this paper (e.g., such as...) Figure 5 Training of one or more of the potential diffusion models 500 and various other machine learning networks can be performed using online training, offline training, and / or various combinations of online and offline training. In some cases, online may refer to a period of time during which input data (e.g., received sensor data, received images, etc.) is processed, for example, to perform video content modification processing implemented by the systems and techniques described herein. In some examples, offline may refer to a period of idle time or a period during which no input data is processed. Additionally, offline may be based on one or more time conditions (e.g., after a certain amount of time has elapsed, such as a day, a week, a month, etc.) and / or may be based on various other conditions, such as network and / or server availability, and various other conditions. In some aspects, offline training of a machine learning model (e.g., a neural network model) may be performed by a first device (e.g., a server device) to generate a pre-trained model, and a second device may receive the trained model from the second device. In some cases, a second device (e.g., a mobile device, XR device, vehicle or vehicle system / component, or other device) may perform online (or on-device) training of a pre-trained model to further adapt or tune the model's parameters.

[0142] Figure 14 This is a flowchart illustrating an example process 1400 for modifying video data according to various aspects of this disclosure. One or more operations of process 1400 may be performed by a computing device (or apparatus) or a component of such computing device (e.g., chipset, codec, etc.). The computing device may be a mobile device (e.g., a mobile phone), a network-connected wearable device (such as a watch), an extended reality (XR) device (such as a virtual reality (VR) device or an augmented reality (AR) device), a vehicle or a component or system of a vehicle, a desktop computing device, a tablet computing device, a server computer, a robotic device, and / or any other computing device having the resource capability to perform process 1400. One or more operations of process 1400 may be implemented as software components that execute and run on one or more processors.

[0143] At box 1402, the computing device (or one or more components thereof) may obtain a first token based on a first frame of video data, wherein each of the first tokens includes a feature vector corresponding to a corresponding position within the first frame of the video data. For example, Figure 9B System 900B can obtain the token 922 of frame 914 of video data. For example, system 900B can obtain the token 922 of frame 914 of video data. Figure 9A The token is illustrated and labeled t0 in the bottom portion of column 906. As another example, system 900B can obtain... Figure 10 The token of frame 1012.

[0144] At box 1404, the computing device (or one or more components thereof) may obtain a second token based on a second frame of video data, wherein each of the second tokens includes a feature vector corresponding to a corresponding position within the second frame of video data. For example, Figure 9B System 900B can obtain the token 924 of frame 916 of video data. For example, system 900B can obtain the token 924 of frame 916 of video data. Figure 9A The token, denoted as t1, is illustrated in the bottom portion of column 906. As another example, system 900B can obtain... Figure 10 The token of frame 1002.

[0145] At box 1406, the computing device (or one or more components thereof) may determine the destination token from the first token. For example, system 900B may determine that some of the tokens obtained at box 1402 are destination tokens. For example, system 900B may determine that... Figure 9A The darker gray token, illustrated and labeled t0 at the bottom of column 906, is the destination token. As another example, system 900B can determine... Figure 10 Token 1014 is a destination token.

[0146] In some respects, the destination token can be randomly determined from the first token. For example, Figure 9B The token merger 926 can (at box 1406) randomly select a destination token from the tokens obtained at box 1402.

[0147] At box 1408, a computing device (or one or more components thereof) may determine a candidate token from a second set of tokens based on the correspondence between the candidate token and the destination token. For example, token merger 926 may determine that... Figure 9A At least some of the lighter gray tokens, illustrated and labeled t1 in the bottom portion of column 906, are candidate tokens. As another example, token merger 926 can determine... Figure 10 Tokens 1006, 1016 and / or 1026 are candidate tokens.

[0148] In some respects, the correspondence between candidate tokens and destination tokens can be based on the cosine distance between them. For example, token merger 926 can determine the correspondence between candidate tokens and destination tokens based on the cosine distance between them determined at box 1406. Figure 9A At least some of the lighter gray tokens, illustrated and labeled t1 in the bottom portion of column 906, are candidate tokens. As another example, token merger 926 may determine the candidate tokens based on the cosine distance between token 1014 and tokens 1006, 1016, and / or 1026. Figure 10 Tokens 1006, 1016 and / or 1026 are candidate tokens.

[0149] At box 1410, a computing device (or one or more components thereof) can merge a candidate token with a destination token to produce a modified second token. For example, token merger 926 can... Figure 9A At least some of the lighter gray tokens, illustrated and labeled t1 in the bottom portion of column 906, are associated with at least one destination token (e.g., Figure 9A At least one of the darker gray tokens (indicated and marked as t0 in the bottom portion of column 906) can be merged. As another example, token merger 926 can merge token 1006, token 1016 and / or token 1026 with token 1014.

[0150] At box 1412, the computing device (or one or more components thereof) may use a diffusion model to process the modified second token. For example, Figure 9B The diffusion model 930 can handle tokens 928 (e.g., tokens included in the merge at box 1410).

[0151] In some aspects, to process the modified second token, the computing device (or one or more components thereof) may process the unmerged tokens of the second token and not process merged candidate tokens. For example, when processing tokens, diffusion model 930 may process unmerged tokens (such as the unmerged token at box 1410) and not process merged tokens (such as the merged token at box 1410). For example, diffusion model 930 may process tokens 1008, 1018, and 1028 and not process tokens 1006, 1016, and 1026.

[0152] Figure 9A and Figure 9B Columns 904 and 906 illustrate concepts related to process 1400. Additionally, Figure 8 and Figure 10 The concepts related to process 1400 are illustrated.

[0153] Figure 15 This is a flowchart illustrating example process 1500 for modifying video data according to various aspects of this disclosure. One or more operations of process 1500 may be performed by a computing device (or apparatus) or a component of such computing device (e.g., chipset, codec, etc.). The computing device may be a mobile device (e.g., a mobile phone), a network-connected wearable device (such as a watch), an extended reality (XR) device (such as a virtual reality (VR) device or an augmented reality (AR) device), a vehicle or a component or system of a vehicle, a desktop computing device, a tablet computing device, a server computer, a robotic device, and / or any other computing device having the resource capability to perform process 1500. One or more operations of process 1500 may be implemented as software components that execute and run on one or more processors.

[0154] At box 1502, the computing device (or one or more components thereof) may obtain tokens based on image data, wherein each token includes a feature vector corresponding to a corresponding location within the image data. For example, Figure 9B System 900B can obtain the token 920 of frame 912 of video data. For example, system 900B can obtain the token 920 of frame 912 of video data. Figure 9A The tokens are listed and marked in column 908.

[0155] At box 1504, the computing device (or one or more components thereof) can determine the destination token from the tokens. For example, system 900B can determine that some of the tokens obtained at box 1502 are destination tokens. For example, system 900B can determine that some of the tokens in token 920 are destination tokens. As another example, system 900B can determine that some of the tokens in column 908 are destination tokens.

[0156] In some respects, the destination token can be randomly determined from the first token.

[0157] At box 1506, a computing device (or one or more components thereof) may obtain a segmentation mask based on image data. For example, Figure 9B The system 900B can obtain a segmentation mask of 936.

[0158] In some respects, the segmentation mask may be based on at least one of the following: foreground-background segmentation; saliency segmentation; or cross-attention map derived from the latent representation.

[0159] At box 1508, a computing device (or one or more components thereof) may determine candidate tokens from the tokens based on the corresponding relationship between candidate tokens and destination tokens and based on a segmentation mask. For example, token merger 926 may determine candidate tokens from tokens 920 based on the relationship between candidate tokens and destination tokens (e.g., determined at box 1504).

[0160] In some respects, the relationship between candidate tokens and destination tokens can be based on the cosine distance between them.

[0161] In some aspects, in order to determine candidate tokens from tokens based on the correspondence between candidate tokens and destination tokens and based on a segmentation mask, a computing device (or one or more components thereof) may weight the correspondence between candidate tokens and destination tokens based on corresponding portions of the segmentation mask. For example, in Figure 9A In column 908, based on the foreground mask (which is an example of a split mask), some tokens are marked as not to be merged (e.g., marked as unmerged tokens).

[0162] In some respects, the weighting makes it less likely that a token corresponding to a significant portion of image data, such as that identified by a segmentation mask, will be identified as a candidate token.

[0163] In some respects, candidate tokens can be further based on similarity thresholds.

[0164] At box 1510, a computing device (or one or more components thereof) may merge a candidate token with a destination token to produce a modified token. For example, token merger 926 may merge token 920 to generate token 928.

[0165] At box 1512, the computing device (or one or more components thereof) may use a diffusion model to process modified tokens. For example, Figure 9B The diffusion model 930 can process token 928 to generate frame 932.

[0166] In some respects, in order to process the modified second token, the computing device (or one or more components thereof) may process the unmerged token of the second token and not process the merged candidate token.

[0167] In some respects, image data may be or may include frames of video data. The computing device (or one or more components thereof) may repeat process 1500 for additional frames of video data.

[0168] Figure 9A Column 908 illustrates concepts related to process 1500. Additionally, Figure 11 The concepts related to process 1500 are illustrated.

[0169] Figure 16This is a flowchart illustrating an example process 1600 for modifying image data according to various aspects of this disclosure. One or more operations of process 1600 may be performed by a computing device (or apparatus) or a component of such computing device (e.g., chipset, codec, etc.). The computing device may be a mobile device (e.g., a mobile phone), a network-connected wearable device (such as a watch), an extended reality (XR) device (such as a virtual reality (VR) device or an augmented reality (AR) device), a vehicle or a component or system of a vehicle, a desktop computing device, a tablet computing device, a server computer, a robotic device, and / or any other computing device having the resource capability to perform process 1600. One or more operations of process 1600 may be implemented as software components that execute and run on one or more processors.

[0170] At box 1602, the computing device (or one or more components thereof) may identify a first portion and a second portion of the image data based on a segmentation mask. For example, diffusion model 930 may include... Figure 12 System 1200 and / or Figure 13 System 1300. System 1200 or System 1300 can obtain indications of portions of an image. The portions of the image can be associated with regions of interest and / or based on segmentation masks (such as foreground-background masks).

[0171] In some respects, the segmentation mask may be based on at least one of the following: foreground-background segmentation; saliency segmentation; or cross-attention map derived from the latent representation.

[0172] In some respects, a computing device (or one or more components thereof) may crop a portion from an image. In some respects, the portion may be rectangular.

[0173] At box 1604, the computing device (or one or more components thereof) may use a diffusion model to process a first portion of the image data to generate a modified first portion of the image data. For example, system 1200 or system 1300 may process portions of the image through several steps of a diffusion process.

[0174] At box 1606, a computing device (or one or more components thereof) may generate modified image data based on a modified first portion of the image data and a second portion of the image data. For example, system 1200 or system 1300 may generate an image based on a portion of the image modified at box 1606 and the remainder of the image.

[0175] In some aspects, to generate modified image data, a computing device (or one or more components thereof) may combine a modified first portion of the image data with a second portion of the image data to produce modified image data. For example, Figure 12The system 1200 can combine the modified portion of an image with the remaining portion of the image.

[0176] In some aspects, in order to combine a modified first portion of image data with a second portion of image data, a computing device (or one or more components thereof) may mix pixels from the modified first portion of image data and the second portion of image data.

[0177] In some aspects, a first portion of the image data is processed through a first number of diffusion steps in a diffusion model to generate a modified first portion of the image data. Furthermore, to generate the modified image data, a computing device (or one or more components thereof) may use a second number of diffusion steps in the diffusion model to process both the modified first portion and the second portion of the image data to generate the modified image data. For example, Figure 13 The system 1300 can process the portion through a first number of diffusion steps, and then process the partially processed portion and the remainder of the image through a second number of diffusion steps.

[0178] Figure 17 This is a flowchart illustrating example process 1700 for modifying image data according to various aspects of this disclosure. One or more operations of process 1400 may be performed by a computing device (or apparatus) or a component of such computing device (e.g., chipset, codec, etc.). The computing device may be a mobile device (e.g., a mobile phone), a network-connected wearable device (such as a watch), an extended reality (XR) device (such as a virtual reality (VR) device or an augmented reality (AR) device), a vehicle or a component or system of a vehicle, a desktop computing device, a tablet computing device, a server computer, a robotic device, and / or any other computing device having the resource capability to perform process 1700. One or more operations of process 1700 may be implemented as software components that execute and run on one or more processors.

[0179] At box 1702, the computing device (or one or more components thereof) may identify a first portion and a second portion of the image data based on a segmentation mask.

[0180] In some respects, the segmentation mask may be based on at least one of the following: foreground-background segmentation; saliency segmentation; or cross-attention map derived from the latent representation.

[0181] In some aspects, a first portion of the image data may be cropped from the image data. In some aspects, the first portion of the image data may be or may include a rectangular portion cropped from the image data.

[0182] At box 1704, the computing device (or one or more components thereof) may use a diffusion model to process a first portion of the image data to generate a modified first portion of the image data.

[0183] At box 1706, a computing device (or one or more components thereof) may combine a modified first portion of the image data and a second portion of the image data to produce modified image data.

[0184] In some aspects, in order to combine a modified first portion of image data with a second portion of image data, a computing device (or one or more components thereof) may mix pixels from the modified first portion of image data and the second portion of image data.

[0185] In some respects, image data may be or may include frames of video data. The computing device (or one or more components thereof) may repeat process 1700 for additional frames of video data.

[0186] Figure 9A Column 904 illustrates concepts related to process 1700. Additionally, Figure 12 The concepts related to process 1700 are illustrated.

[0187] Figure 18 This is a flowchart illustrating an example process 1800 for modifying image data according to various aspects of this disclosure. One or more operations of process 1800 may be performed by a computing device (or apparatus) or a component of such computing device (e.g., chipset, codec, etc.). The computing device may be a mobile device (e.g., a mobile phone), a network-connected wearable device (such as a watch), an extended reality (XR) device (such as a virtual reality (VR) device or an augmented reality (AR) device), a vehicle or a component or system of a vehicle, a desktop computing device, a tablet computing device, a server computer, a robotic device, and / or any other computing device having the resource capability to perform process 1800. One or more operations of process 1800 may be implemented as software components that execute and run on one or more processors.

[0188] At box 1802, the computing device (or one or more components thereof) may identify a first portion and a second portion of the image data based on a segmentation mask.

[0189] In some respects, the segmentation mask may be based on at least one of the following: foreground-background segmentation; saliency segmentation; or cross-attention map derived from the latent representation.

[0190] In some aspects, a first portion of the image data may be cropped from the image data. In some aspects, the first portion of the image data may be or may include a rectangular portion cropped from the image data.

[0191] At box 1804, the computing device (or one or more components thereof) may use a first number of diffusion steps of the diffusion model to process a first portion of the image data to generate a partially modified first portion of the image data.

[0192] At box 1806, the computing device (or one or more components thereof) may use a second number of diffusion steps of the diffusion model to process the partially modified first portion of the image data and the second portion of the image data to generate modified image data.

[0193] In some respects, image data may be or may include frames of video data. The computing device (or one or more components thereof) may repeat process 1800 for additional frames of video data.

[0194] Figure 9A Column 904 illustrates concepts related to process 1800. Additionally, Figure 13 The concepts related to process 1800 are illustrated.

[0195] In some examples, as previously noted, the methods described herein (e.g., Figure 14 Process 1400 Figure 15 Process 1500 Figure 16 Process 1600 Figure 17 Process 1700 Figure 18 The process 1800 and / or other methods described herein may be performed wholly or partially by a computing device or apparatus. In one example, one or more methods of the process may be performed by... Figure 1 System 100 Figure 5 Potential diffusion model 500 Figure 9A The system 900A, or another system or device, may execute the method. In another example, one or more of the methods (e.g., process 1400, process 1500, process 1600, process 1700, process 1800 and / or other methods described herein) may be performed by... Figure 21 The computing device architecture 2100 shown is implemented wholly or in part. For example, it has Figure 21 The computing device of the computing device architecture 2100 shown may include system 100, potential diffusion model 500 and / or Figure 9AThe system 900A comprises or is included in components thereof, and enables the operation of processes 1400, 1500, 1600, 1700, 1800, and / or other processes described herein. In some cases, the computing device or apparatus may include various components such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other components configured to perform the steps of the processes described herein. In some examples, the computing device may include a display, a network interface configured to communicate and / or receive data, any combination thereof, and / or other components. The network interface may be configured to communicate and / or receive Internet Protocol (IP) based data or other types of data.

[0196] A component capable of implementing a computing device in a circuit. For example, the component may include electronic circuitry or other electronic hardware, and / or may be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuits (e.g., a microprocessor, graphics processing unit (GPU), digital signal processor (DSP), central processing unit (CPU), and / or other suitable electronic circuitry), and / or may include computer software, firmware, or any combination thereof for performing the various operations described herein, and / or may be implemented using computer software, firmware, or any combination thereof for performing the various operations described herein.

[0197] Processes 1400, 1500, 1600, 1700, 1800, and / or other processes described herein are illustrated as logic flowcharts, whose operations represent sequences of operations that can be implemented in hardware, computer instructions, or combinations thereof. In the context of computer instructions, each operation represents a computer-executable instruction stored on one or more computer-readable storage media that, when executed by one or more processors, performs the described operation. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc., that perform a particular function or implement a particular data type. The order in which operations are described is not intended to be construed as limiting, and any number of described operations can be combined in any order and / or in parallel to implement a process.

[0198] Additionally, processes 1400, 1500, 1600, 1700, 1800, and / or other processes described herein may be executed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that executes jointly on one or more processors, by hardware, or a combination thereof. As noted above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program comprising multiple instructions executable by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0199] Figure 19 This is an exemplary example of a Neural Network 1900 (e.g., a deep learning neural network), which can be used to implement machine learning-based image generation, feature segmentation, implicit neural representation generation, rendering, classification, object detection, image recognition (e.g., face recognition, object recognition, scene recognition, etc.), feature extraction, authentication, gaze detection, gaze prediction, and / or automation. For example, a Neural Network 1900 could be... Figure 4 U-Net architecture 400, Figure 5 Potential diffusion model 500 Figure 9A diffusion model Figure 12 Diffusion models of potential diffusion processes Figure 13 An example of a diffusion model that represents a potential diffusion process.

[0200] Input layer 1902 includes input data. In one exemplary example, input layer 1902 may include data representing an image and / or text prompt. Neural network 1900 includes multiple hidden layers 1906a, 1906b through 1906n. Hidden layers 1906a, 1906b through 1906n comprise "n" hidden layers, where "n" is an integer greater than or equal to one. Multiple hidden layers can be made to include as many layers as needed for a given application. Neural network 1900 further includes an output layer 1904 that provides the output produced by the processing performed by hidden layers 1906a, 1906b through 1906n. In one exemplary example, output layer 1904 may provide data latent values ​​and / or data.

[0201] The neural network 1900 may be or may include a multi-layer neural network with interconnected nodes. Each node may represent a piece of information. The information associated with these nodes is shared between different layers, and each layer retains the information while processing it. In some cases, the neural network 1900 may include a feedforward network, in which case there are no feedback connections in which the network's output is fed back into itself. In some cases, the neural network 1900 may include a recurrent neural network, which may have loops that allow information to be carried across nodes when reading input.

[0202] Information can be exchanged between nodes via node-to-node interconnects between layers. Nodes in input layer 1902 can activate the node set in the first hidden layer 1906a. For example, as shown, each input node in input layer 1902 is connected to each node in the first hidden layer 1906a. Nodes in the first hidden layer 1906a can transform the information of each input node by applying an activation function to the input node information. The information derived from this transformation can then be passed to nodes in the next hidden layer 1906b, activating those nodes, which can then perform their own specified functions. Example functions include convolution, upsampling, data transformation, and / or any other suitable function. The output of hidden layer 1906b can then activate nodes in the next hidden layer, and so on. The output of the last hidden layer 1906n can activate one or more nodes in output layer 1904, at which the output is provided. In some cases, although a node in neural network 1900 (e.g., node 1908) is shown as having multiple output lines, the node has a single output and is shown as all lines output from the node representing the same output value.

[0203] In some cases, each node or the interconnection between nodes may have weights, which are a set of parameters derived from the training of the neural network 1900. Once the neural network 1900 is trained, it can be referred to as a trained neural network, which can be used to perform one or more operations. For example, the interconnection between nodes may represent a piece of information about what the interconnected nodes have learned. The interconnection may have tunable numerical weights that can be tuned (e.g., based on the training dataset), allowing the neural network 1900 to adapt to the input and learn as more and more data is processed.

[0204] The neural network 1900 can be pre-trained to process features from the data in the input layer 1902 using different hidden layers 1906a, 1906b to 1906n in order to provide an output through the output layer 1904. In an example where the neural network 1900 is used to identify features in an image, the neural network 1900 can be trained using training data that includes both images and labels, as described above. For example, training images can be input into the network, where each training image has a label indicating a feature in the image (for a feature segmentation machine learning system) or a label indicating the category of activity in each image. In an example where object classification is used for illustrative purposes, the training images may include images of the number 2, in which case the label of the image may be [0 0 1 0 0 0 0 0 0 0].

[0205] In some cases, the Neural Network 1900 can use a training process called backpropagation to adjust the weights of its nodes. As noted above, the backpropagation process includes forward pass, loss function, back pass, and weight update. For each training iteration, forward pass, loss function, back pass, and parameter update are performed. For each set of training images, this process can be repeated up to a specific number of iterations until the Neural Network 1900 is trained well enough to accurately tune the weights of each layer.

[0206] For an example of identifying objects in an image, the forward pass may include passing a training image through a neural network 1900. The weights are initially randomized before training the neural network 1900. As an illustrative example, the image may include a numerical array representing the pixels of the image. Each number in the array may include a value from 0 to 255 describing the intensity of the pixel at that location in the array. In one example, the array may include a 28×28×3 numerical array with 28 rows and 28 columns of pixels and 3 color components (such as red, green, and blue, or lightness and two chroma components, etc.).

[0207] As noted above, for the first training iteration of the Neural Network 1900, the output will likely include values ​​due to the weights being randomly selected during initialization without prioritizing any particular class. For example, if the output is a vector with probabilities that an object includes different classes, the probability values ​​for each class may be equal or at least very similar (e.g., 0.1 for each of ten possible classes). Using the initial weights, the Neural Network 1900 cannot determine low-level features and therefore cannot make an accurate determination of what the object's classification might be. A loss function can be used to analyze the error in the output. Any suitable loss function can be defined, such as cross-entropy loss. Another example of a loss function is mean squared error (MSE), which is defined as... The loss can be set to equal E.总和 The value of .

[0208] For the first training image, the loss (or error) will be high because the actual value will be significantly different from the predicted output. The goal of training is to minimize the loss so that the predicted output matches the training labels. The Neural Network 1900 performs backpropagation by determining which inputs (weights) contribute most to the network's loss and can adjust the weights to reduce and eventually minimize the loss. The derivative of the loss with respect to the weights (denoted as...) can be calculated. dL / dW ,in W These are the weights at a specific layer, used to determine the weights that contribute the most to the network's loss. After calculating the derivative, a weight update can be performed by updating all the weights of the filter. For example, weights can be updated so that they change in the opposite direction of the gradient. A weight update can be represented as... ,in w Indicates weight, w i Let represent the initial weights, and η represent the learning rate. The learning rate can be set to any suitable value, where a high learning rate includes larger weight updates, while a lower value indicates smaller weight updates.

[0209] Neural Network 1900 can include any suitable deep network. An example includes a Convolutional Neural Network (CNN), which includes an input layer and an output layer, with multiple hidden layers between them. The hidden layers of a CNN include a series of convolutional layers, non-linear layers, pooling layers (for downsampling), and fully connected layers. Neural Network 1900 can include any other deep network besides CNNs, such as autoencoders, deep belief networks (DBNs), recurrent neural networks (RNNs), etc.

[0210] Figure 20 This is an exemplary example of a Convolutional Neural Network (CNN) 2000. The input layer 2002 of the CNN 2000 includes data representing an image or frame. For example, the data may include a numerical array representing pixels of an image, where each number in the array includes a value from 0 to 255 describing the intensity of the pixel at that location in the array. Using the previous example from above, the array may include a 28×28×3 numerical array with 28 rows and 28 columns of pixels and 3 color components (e.g., red, green, and blue, or lightness and two chroma components, etc.). The image can be passed through a convolutional hidden layer 2004, an optional non-linear activation layer, a pooling hidden layer 2006, and a fully connected layer 2008 (which may be hidden) to obtain the output at the output layer 2010. Although... Figure 20Only one hidden layer from each hidden layer is shown in the diagram, but those skilled in the art will understand that multiple convolutional hidden layers, non-linear layers, pooling hidden layers, and / or fully connected layers may be included in CNN 2000. As previously described, the output may indicate a single category of an object, or may include probabilities that best describe the category of an object in an image.

[0211] The first layer of CNN 2000 can be a convolutional hidden layer 2004. The convolutional hidden layer 2004 analyzes the image data input from layer 2002. Each node in the convolutional hidden layer 2004 is connected to a node (pixel) region of the input image called a receptive field. The convolutional hidden layer 2004 can be thought of as one or more filters (each filter corresponding to a different activation or feature map), where each convolutional iteration of the filter is a node or neuron in the convolutional hidden layer 2004. For example, the region of the input image covered by the filter at each convolutional iteration will be the filter's receptive field. In an exemplary example, if the input image consists of a 28×28 array, and each filter (and its corresponding receptive field) is a 5×5 array, then there will be 24×24 nodes in the convolutional hidden layer 2004. Each connection between a node and its receptive field learns weights, and in some cases, learns an overall bias, allowing each node to learn to analyze its specific local receptive field in the input image. Each node in the convolutional hidden layer 2004 will have the same weights and biases (called shared weights and shared biases). For example, the filter has a weight (digital) array and the same depth as the input. For the image frame example, the filter would have a depth of 3 (based on the three color components of the input image). An exemplary example of the filter array size is 5×5×3, corresponding to the size of the receptive field of a node.

[0212] The convolutional nature of the convolutional hidden layer 2004 stems from the fact that each node of the convolutional layer is applied to its corresponding receptive field. For example, the filters of the convolutional hidden layer 2004 can begin at the top left corner of the input image array and can convolve around the input image. As noted above, each convolutional iteration of the filter can be considered as a node or neuron of the convolutional hidden layer 2004. In each convolutional iteration, the filter value is multiplied by a corresponding number of the original pixel values ​​of the image (e.g., a 5×5 filter array is multiplied by a 5×5 array of input pixel values ​​at the top left corner of the input image array). The multiplications from each convolutional iteration can be summed to obtain the sum of that iteration or node. The process then continues at the next location in the input image based on the receptive field of the next node in the convolutional hidden layer 2004. For example, the filter can move a step size (called stride) to the next receptive field. The stride can be set to 1 or any other suitable amount. For example, if the stride is set to 1, the filter will move 1 pixel to the right in each convolutional iteration. Processing the filter at each unique location in the input volume produces a number representing the filter result at that location, thus determining a sum value for each node of the convolutional hidden layer 2004.

[0213] The mapping from the input layer to the convolutional hidden layer 2004 is called an activation map (or feature map). An activation map includes values ​​for each node representing the filter results at each location in the input volume. Activation maps can include arrays containing various sums of values ​​produced by the filter for each iteration of the input volume. For example, if a 5×5 filter is applied to each pixel of a 28×28 input image (with a stride of 1), the activation map would consist of a 24×24 array. The convolutional hidden layer 2004 can include several activation maps to identify multiple features in the image. Figure 20 The example shown includes three activation maps. Using these three activation maps, the convolutional hidden layer 2004 can detect three different types of features, each of which is detectable across the entire image.

[0214] In some examples, nonlinear hidden layers can be applied after the convolutional hidden layer 2004. Nonlinear layers can be used to introduce nonlinearity into a system that has already computed linear operations. An exemplary example of a nonlinear layer is the Corrected Linear Unit (ReLU) layer. The ReLU layer applies the function f(x) = max(0, x) to all values ​​in the input volume, which changes all negative activations to 0. Therefore, ReLU can add nonlinearity to the CNN 2000 without affecting the receptive field of the convolutional hidden layer 2004.

[0215] A pooling hidden layer 2006 can be applied after the convolutional hidden layer 2004 (and, when used, after a non-linear hidden layer). The pooling hidden layer 2006 is used to simplify the information in the output of the convolutional hidden layer 2004. For example, the pooling hidden layer 2006 can take each activation map output from the convolutional hidden layer 2004 and use a pooling function to generate a condensed activation map (or feature map). Max pooling is an example of a function performed by the pooling hidden layer. The pooling hidden layer 2006 can use other forms of pooling functions, such as average pooling, L2 norm pooling, or other suitable pooling functions. Pooling functions (e.g., max pooling filters, L2 norm filters, or other suitable pooling filters) are applied to each activation map included in the convolutional hidden layer 2004. Figure 20 In the example shown, three pooling filters are used to convolve the three activation maps in the hidden layer 2004.

[0216] In some examples, max pooling can be used by applying a max pooling filter (e.g., of size 2×2) with a stride (e.g., equal to the dimension of the filter, such as stride 2) to the activation map output from convolutional hidden layer 2004. The output from the max pooling filter includes the maximum number in each sub-region of the filter convolution. Using a 2×2 filter as an example, each unit in the pooling layer summarizes a region of 2×2 nodes from the previous layer (each node being a value in the activation map). For example, four values ​​(nodes) in the activation map will be analyzed by the 2×2 max pooling filter at each iteration of the filter, with the maximum of the four values ​​being output as the "maximum" value. If such a max pooling filter is applied to an activation filter of 24×24 nodes from convolutional hidden layer 2004, the output from pooling hidden layer 2006 will be an array of 12×12 nodes.

[0217] In some examples, L2 norm pooling filters may also be used. L2 norm pooling filters involve calculating the square root of the sum of squares of the values ​​in a 2×2 region (or other suitable region) of the activation map (instead of calculating the maximum value as done in max pooling), and using the calculated value as the output.

[0218] Pooling functions (e.g., max pooling, L2 norm pooling, or other pooling functions) determine whether a given feature is found anywhere in a region of an image and discard the exact localization information. This can be done without affecting the results of feature detection because once a feature has been found, its exact location is less important than its approximate location relative to other features. Max pooling (and other pooling methods) offers the benefit of having far fewer pooling features, thus reducing the number of parameters required in subsequent layers of a CNN 2000.

[0219] The final connected layer in the network is a fully connected layer that connects each node from the pooling hidden layer 2006 to each output node in the output layer 2010. Using the example above, the input layer comprises 28×28 nodes encoding the pixel intensity of the input image, the convolutional hidden layer 2004 comprises 3×24×24 hidden feature nodes based on applying a 5×5 local receptive field (for filtering) to three activation maps, and the pooling hidden layer 2006 comprises a layer based on applying a max-pooling filter to a 2×2 region on each of the three feature maps. Extending this example, the output layer 2010 may comprise ten output nodes. In such an example, each node of the 3×12×12 pooling hidden layer 2006 is connected to each node of the output layer 2010.

[0220] The fully connected layer 2008 takes the output of the previous pooling hidden layer 2006 (which should represent an activation map of high-level features) and determines the features most relevant to a particular class. For example, the fully connected layer 2008 can determine the high-level features most relevant to a particular class and may include weights (nodes) for those high-level features. The product between the weights of the fully connected layer 2008 and the pooling hidden layer 2006 can be computed to obtain the probabilities for different classes. For example, if the CNN 2000 is used to predict that the object in an image is a person, there will be high values ​​in the activation map representing the high-level features of a person (e.g., two legs, a face at the top of the object, two eyes at the top left and top right of the face, a nose in the middle of the face, a mouth at the bottom of the face, and / or other features common to people).

[0221] In some examples, the output from output layer 2010 may include an M-dimensional vector (M=10 in the previous example). M indicates the number of classes from which CNN 2000 must choose when classifying objects in an image. Other example outputs may also be provided. Each number in the M-dimensional vector represents the probability that an object belongs to a certain class. In an exemplary example, if the 10-dimensional output vector representing objects of ten different classes is [0 0 0.05 0.8 0 0.15 0 0 0 0], then the vector indicates a 5% probability that the image is an object of the third class (e.g., a dog), an 80% probability that the image is an object of the fourth class (e.g., a person), and a 15% probability that the image is an object of the sixth class (e.g., a kangaroo). The probability of a class can be considered as the confidence level that an object is part of that class.

[0222] Figure 21An example computing device architecture 2100 is illustrated, illustrating example computing devices capable of implementing the various technologies described herein. In some examples, the computing device may include a mobile device, a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a video server, a vehicle (or a computing device of a vehicle), or other devices. For example, computing device architecture 2100 may include, implement, or be included in any one or all of system 100, device 600, device 800, and / or image processing system 1100. Additionally or alternatively, computing device architecture 2100 may be configured to perform process 1000 and / or other processes described herein.

[0223] The components of computing device architecture 2100 are shown to communicate electrically with each other using a connection 2112, such as a bus. The example computing device architecture 2100 includes a processing unit (CPU or processor) 2102 and a computing device connection 2112 that couples various computing device components, including computing device memories 2110 (such as read-only memory (ROM) 2108 and random access memory (RAM) 2106), to the processor 2102.

[0224] The computing device architecture 2100 may include a cache of high-speed memory that is directly connected to, very close to, or integrated as part of the processor 2102. The computing device architecture 2100 may copy data from memory 2110 and / or storage device 2114 to cache 2104 for fast access by the processor 2102. In this way, the cache can provide performance improvements by avoiding latency for the processor 2102 while waiting for data. These and other modules may control or be configured to control the processor 2102 to perform various actions. Other computing device memory 2110 may also be available. Memory 2110 may include various different types of memory with different performance characteristics. The processor 2102 may include any general-purpose processor and hardware or software services configured to control the processor 2102 (such as services 12116, 2118, and 32120 stored in storage device 2114), as well as dedicated processors in which software instructions are incorporated into the processor design. The processor 2102 may be a self-contained system comprising multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors can be symmetric or asymmetric.

[0225] To enable user interaction with computing device architecture 2100, input device 2122 can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, voice input, etc. Output device 2124 can also be one or more of a variety of output mechanisms known to those skilled in the art, such as a display, projector, television, speaker device, etc. In some instances, multi-mode computing devices allow users to provide multiple types of input to communicate with computing device architecture 2100. Communication interface 2126 typically controls and manages user input and computing device output. There are no limitations on operation on any particular hardware arrangement, and therefore the underlying features here can be easily replaced to obtain improved hardware or firmware arrangements as they are developed.

[0226] Storage device 2114 is a non-volatile memory and may be a hard disk or other type of computer-readable medium capable of storing computer-accessible data, such as magnetic tape cassettes, flash memory cards, solid-state memory devices, digital multifunction disks, magnetic tape cartridges, random access memory (RAM) 2106, read-only memory (ROM) 2108, and hybrid forms thereof. Storage device 2114 may include services 2116, 2118, and 2120 for controlling processor 2102. Other hardware or software modules are envisioned. Storage device 2114 may be connected to computing device connection 2112. In one aspect, a hardware module performing a particular function may include software components for performing that function stored in a computer-readable medium connected to necessary hardware components such as processor 2102, connection 2112, output device 2124, etc.

[0227] With reference to a given parameter, property, or condition, the term "substantially" may mean that a person skilled in the art would understand that a given parameter, property, or condition is satisfied with a small degree of variance (such as, for example, within acceptable manufacturing tolerances). For example, depending on the specific parameter, property, or condition that is substantially satisfied, the parameter, property, or condition may be satisfied at least 90%, at least 95%, or even at least 99%.

[0228] Various aspects of this disclosure are applicable to any suitable electronic device (such as a security system, smartphone, tablet, laptop, vehicle, drone, or other device) that includes or is coupled to one or more active depth sensing systems. Although devices having or coupled to a light projector are described below, various aspects of this disclosure are applicable to devices having any number of light projectors and are therefore not limited to any particular device.

[0229] The term "device" is not limited to one or a specific number of physical objects (such as a smartphone, a controller, a processing system, etc.). As used herein, a device can be any electronic device having one or more parts that implement at least some parts of this disclosure. Although the following description and examples use the term "device" to describe various aspects of this disclosure, the term "device" is not limited to a specific configuration, type, or number of objects. Additionally, the term "system" is not limited to multiple components or specific aspects. For example, a system may be implemented on one or more printed circuit boards or other substrates and may have movable or static components. Although the following description and examples use the term "system" to describe various aspects of this disclosure, the term "system" is not limited to a specific configuration, type, or number of objects.

[0230] Specific details are provided in the foregoing description to provide a thorough understanding of the aspects and examples presented herein. However, those skilled in the art will understand that these aspects can be practiced without these specific details. For clarity, in some cases, the technology may be presented as comprising individual functional blocks, including functional blocks comprising devices, device components, steps or routines in methods embodied in software or a combination of hardware and software. Additional components may be used in addition to those shown in the figures and / or described herein. For example, circuits, systems, networks, processes and other components may be shown as components in block diagram form to avoid obscuring these aspects in unnecessary detail. In other cases, well-known circuits, processes, algorithms, structures and techniques may be shown without unnecessary detail to avoid obscuring the aspects.

[0231] Various aspects described above can be presented as processes or methods, depicted as flowcharts, diagrams, data flow graphs, structure diagrams, or block diagrams. Although flowcharts can describe operations as sequential processes, many operations within an operation can be executed in parallel or concurrently. Furthermore, the order of operations can be rearranged. A process terminates when its operations are completed, but a process may have additional steps not included in the accompanying diagrams. A process can correspond to a method, function, procedure, subroutine, subroutine, etc. When a process corresponds to a function, its termination may correspond to the function returning to the calling function or the main function.

[0232] The processes and methods described in the examples above can be implemented using stored computer-executable instructions or computer-executable instructions otherwise obtainable from a computer-readable medium. Such instructions may include, for example, instructions and data that configure, cause or otherwise configure, a general-purpose computer, special-purpose computer, or processing device to perform a function or group of functions. The portion of the computer resources used may be accessible via a network. Computer-executable instructions may be, for example, binary files, intermediate format instructions (such as assembly language), firmware, source code, etc.

[0233] The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or carrying instructions and / or data. Computer-readable media may include non-transitory media in which data can be stored and which do not include carrier waves and / or transient electronic signals propagating wirelessly or over a wired connection. Examples of non-transitory media include, but are not limited to, magnetic disks or magnetic tapes, optical storage media (such as compact discs (CDs) or digital versatile discs (DVDs)), flash memory, magnetic disks or optical disks, USB devices equipped with non-volatile memory, network storage devices, any suitable combinations thereof, etc. Computer-readable media may store code and / or machine-executable instructions thereon, which may represent procedures, functions, subroutines, programs, routines, subroutines, modules, software packages, classes, or any combination of instructions, data structures, or program statements. Code segments may be coupled to other code segments or hardware circuitry by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, independent variables, parameters, data, etc., can be transmitted, forwarded, or sent through any suitable means, including memory sharing, message passing, token passing, network sending, etc.

[0234] In some respects, computer-readable storage devices, media, and memories may include cables or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer-readable storage media explicitly exclude media such as energy, carrier signals, electromagnetic waves, and the signals themselves.

[0235] Devices implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented as software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) for performing necessary tasks may be stored in a computer-readable or machine-readable medium. A processor may perform the necessary tasks. Typical examples of form factors include laptops, smartphones, mobile phones, tablet devices, or other small form factor personal computers, personal digital assistants, rack-mounted devices, standalone devices, etc. The functionality described herein may also be embodied in peripheral devices or interlocking cards. By further example, such functionality may also be implemented on circuit boards of different chips or different processes executed on a single device.

[0236] Instructions, media for delivering such instructions, computing resources for executing them, and other structures for supporting such computing resources are example components for providing the functionality described in this disclosure.

[0237] In the foregoing description, aspects of this application have been described with reference to their specific aspects, but those skilled in the art will recognize that this application is not limited thereto. Therefore, although illustrative aspects of this application have been described in detail herein, it is to be understood that various inventive concepts may be embodied and adopted in various other ways, and the appended claims are not intended to be construed as including these variations unless limited by prior art. The various features and aspects of the applications described above may be used individually or in combination. Furthermore, without departing from the broader scope of this specification, aspects may be used in any number of environments and applications beyond those described herein. Therefore, the specification and drawings should be considered illustrative rather than restrictive. For illustrative purposes, the methods are described in a particular order. It should be understood that, in alternative aspects, the methods may be performed in a different order than described.

[0238] Those skilled in the art will understand that the less than ("<") and greater than (">") symbols or terms used herein may be replaced with less than or equal to ("≤") and greater than or equal to ("≥") symbols without departing from the scope of this description.

[0239] When a component is described as being “configured” to perform certain operations, such configuration can be achieved, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., microprocessors or other suitable electronic circuits) to perform the operations, or any combination thereof.

[0240] The phrase “coupled to” means any component that is physically connected directly or indirectly to another component, and / or any component that communicates directly or indirectly with another component (e.g., connected to another component via a wired or wireless connection and / or other suitable communication interface).

[0241] Claim language or other languages ​​that state "at least one of" and / or "one or more of" in a set indicate that one member of the set or multiple members of the set (in any combination) satisfy the claim. For example, claim language stating "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In another example, claim language stating "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, A and B and C, or any repetition is information or data (e.g., A and A, B and B, C and C, A and A and B, etc.), or any other ordering, repetition, or combination of A, B, and C. The language "at least one of" and / or "one or more of" in a set does not limit the set to the items listed in the set. For example, the language of a claim stating "at least one of A and B" or "at least one of A or B" may refer to A, B, or A and B, and may additionally include items not listed in the set of A and B. The phrases "at least one" and "one or more" are used interchangeably herein.

[0242] Claims using phrases such as "at least one processor, the at least one processor being configured to," "at least one processor being configured to," "one or more processors, the one or more processors being configured to," or "one or more processors being configured to," or other languages, indicate that one or more processors (in any combination) are capable of performing associated operations. For example, a claim using the phrase "at least one processor, the at least one processor being configured to: X, Y, and Z" means that a single processor can be used to perform operations X, Y, and Z; or that multiple processors are each assigned a specific subset of tasks to perform operations X, Y, and Z, such that the multiple processors together perform X, Y, and Z; or that a group of multiple processors work together to perform operations X, Y, and Z. In another example, a claim using the phrase "at least one processor, the at least one processor being configured to: X, Y, and Z" could mean that any single processor can perform only at least one subset of operations X, Y, and Z.

[0243] When referring to one or more elements that perform functions (e.g., steps of a method), one element may perform all functions, or more than one element may jointly perform these functions. When more than one element jointly performs these functions, each function does not need to be performed by every single element (e.g., different functions may be performed by different elements), and / or each function does not need to be performed by only one element as a whole (e.g., different elements may perform different sub-functions of a function). Similarly, when referring to one or more elements configured to cause another element (e.g., a device) to perform functions, one element may be configured to cause another element to perform all functions, or more than one element may be jointly configured to cause another element to perform these functions.

[0244] When referring to an entity that performs or is configured to perform functions (e.g., steps of a method) (e.g., any entity or device described herein), the entity may be configured to cause one or more elements (individually or collectively) to perform those functions. One or more components of the entity may include at least one memory (or one or more memories), at least one processor (or one or more processors), at least one communication interface, another component configured to perform one or more of those functions, and / or any combination thereof. When referring to an entity that performs functions, the entity may be configured to cause one component to perform all functions, or to cause more than one component to perform those functions collectively. When the entity is configured to cause more than one component to perform those functions collectively, each function does not need to be performed by every single component (e.g., different functions may be performed by different components), and / or each function does not need to be performed by only one component as a whole (e.g., different components may perform different sub-functions of a function).

[0245] The various exemplary logic blocks, modules, circuits, and algorithm steps described in conjunction with the aspects disclosed herein can be implemented as electronic hardware, computer software, firmware, or combinations thereof. To clearly illustrate this interchangeability between hardware and software, various exemplary components, blocks, modules, circuits, and steps have been broadly described above in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in different ways for each specific application, but such specific implementation decisions should not be construed as departing from the scope of this application.

[0246] The techniques described herein can also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques can be implemented in any of a variety of devices, such as general-purpose computers, wireless communication devices (mobile phones), or integrated circuit devices with multiple uses, including applications in wireless communication devices (mobile phones) and other devices. Any feature described as a module or component can be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, these techniques can be implemented at least in part by a computer-readable data storage medium comprising program code including instructions that, when executed, perform one or more of the methods described above. The computer-readable data storage medium can form part of a computer program product, which may include packaging materials. The computer-readable medium may include memory or data storage media, such as random access memory (RAM) (such as synchronous dynamic random access memory (SDRAM)), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, etc. Additionally or alternatively, the technology may be implemented at least in part by a computer-readable communication medium that carries or conveys program code in the form of instructions or data structures that can be accessed, read and / or executed by a computer, such as propagated signals or waves.

[0247] The program code can be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such processors can be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor; however, in alternatives, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Therefore, as used herein, the term "processor" may refer to any of the foregoing structures, any combination of the foregoing structures, or any other structure or means suitable for implementing the techniques described herein.

[0248] The exemplary aspects of this disclosure include: Aspect 1. An apparatus for modifying video data, the apparatus comprising: one or more memories configured to store the video data; and one or more processors coupled to the one or more memories and configured to: obtain a first token based on a first frame of the video data, each of the first tokens including a feature vector corresponding to a corresponding position within the first frame of the video data; obtain a second token based on a second frame of the video data, each of the second tokens including a feature vector corresponding to a corresponding position within the second frame of the video data; determine a destination token from the first tokens; determine a candidate token from the second tokens based on a correspondence between the candidate tokens and the destination token; merge the candidate tokens with the destination tokens to generate a modified second token; and process the modified second token using a diffusion model.

[0249] Aspect 2. The apparatus according to aspect 1, wherein the destination token is randomly determined from the first token.

[0250] Aspect 3. The apparatus according to any one of Aspect 1 or 2, wherein the correspondence between the candidate token and the destination token is based on the cosine distance between the candidate token and the destination token.

[0251] Aspect 4. The apparatus according to any one of Aspects 1 to 3, wherein, in order to process the modified second token, the one or more processors are configured to process unmerged tokens in the second token and not process the merged candidate tokens.

[0252] Aspect 5. The apparatus according to any one of aspects 1 to 4, the apparatus further comprising at least one camera configured to capture the first frame and the second frame of the video data.

[0253] Aspect 6. An apparatus for modifying video data, the apparatus comprising: one or more memories configured to store the video data; and one or more processors coupled to the one or more memories and configured to perform operations including: obtaining a plurality of tokens, the plurality of tokens comprising a corresponding set of tokens for each of a plurality of frames of the video data; identifying a destination token from the plurality of tokens; determining a candidate token from the plurality of tokens based on a correspondence between the candidate token and the destination token; merging the candidate token with the destination token to generate a modified second token; and processing the modified second token using a diffusion model.

[0254] Aspect 7. The apparatus according to aspect 6, wherein the plurality of frames of the video data comprises a frame pool of the video data, and wherein the one or more processors are configured to repeat the operation for a further frame pool of the video data.

[0255] Aspect 8. The apparatus according to aspect 7, wherein the plurality of frames of the video data includes a sliding frame window of the video data, and wherein the one or more processors are configured to repeat the operation for the sliding frame window of the video data.

[0256] Aspect 9. The apparatus according to any one of Aspects 6 to 8, wherein the plurality of frames of the video data comprises all frames of the video data.

[0257] Aspect 10. The apparatus according to any one of aspects 6 to 9, the apparatus further comprising at least one camera configured to capture the plurality of frames of the video data.

[0258] Aspect 11. An apparatus for modifying image data, the apparatus comprising: one or more memories configured to store the image data; and one or more processors coupled to the one or more memories and configured to perform operations including: obtaining tokens based on the image data, wherein each of the tokens includes a feature vector corresponding to a corresponding location within the image data; determining a destination token from the tokens; obtaining a segmentation mask based on the image data; determining the candidate tokens from the tokens based on a correspondence between the candidate tokens and the destination tokens and based on the segmentation mask; merging the candidate tokens with the destination tokens to generate a modified token; and processing the modified tokens using a diffusion model.

[0259] Aspect 12. The apparatus according to aspect 11, wherein the segmentation mask is based on at least one of: foreground-background segmentation; saliency segmentation; or a cross-attention map from a latent representation.

[0260] Aspect 13. The apparatus according to any one of Aspects 11 or 12, wherein, in order to determine the candidate token from the tokens based on the correspondence between the candidate token and the destination token and based on the segmentation mask, the one or more processors are configured to: weight the correspondence between the candidate token and the destination token based on the corresponding portion of the segmentation mask.

[0261] Aspect 14. The apparatus according to aspect 13, wherein the weighting makes it less likely that a token corresponding to a significant portion of the image data as identified by the segmentation mask will be identified as a candidate token.

[0262] Aspect 15. The apparatus according to any one of Aspects 11 to 14, wherein the candidate token is further based on a similarity threshold.

[0263] Aspect 16. The apparatus according to any one of Aspects 11 to 15, wherein the correspondence between the candidate token and the destination token is based on the cosine distance between the candidate token and the destination token.

[0264] Aspect 17. The apparatus according to any one of Aspects 11 to 16, wherein the image data comprises frames of video data, and wherein the one or more processors are configured to repeat the operation for additional frames of the video data.

[0265] Aspect 18. The apparatus according to any one of aspects 11 to 17, the apparatus further comprising at least one camera configured to capture the image data.

[0266] Aspect 19. An apparatus for modifying image data, the apparatus comprising: one or more memories configured to store the image data; and one or more processors coupled to the one or more memories and configured to: identify a first portion of the image data and a second portion of the image data based on a segmentation mask; process the first portion of the image data using a diffusion model to generate a modified first portion of the image data; and combine the modified first portion of the image data and the second portion of the image data to produce modified image data.

[0267] Aspect 20. The apparatus according to aspect 19, wherein the segmentation mask is based on at least one of: foreground-background segmentation; saliency segmentation; or a cross-attention map from a latent representation.

[0268] Aspect 21. The apparatus according to any one of Aspects 19 or 20, wherein, in order to combine the modified first portion of the image data and the second portion of the image data, the one or more processors are configured to mix pixels from the modified first portion of the image data and the second portion of the image data.

[0269] Aspect 22. The apparatus according to any one of aspects 19 to 21, wherein the first portion of the image data is cropped from the image data.

[0270] Aspect 23. The apparatus according to any one of aspects 19 to 22, wherein the first portion of the image data includes a rectangular portion cropped from the image data.

[0271] Aspect 24. The apparatus according to any one of aspects 19 to 23, the apparatus further comprising at least one camera configured to capture the image data.

[0272] Aspect 25. An apparatus for modifying image data, the apparatus comprising: one or more memories configured to store the image data; and one or more processors coupled to the one or more memories and configured to: identify a first portion and a second portion of the image data based on a segmentation mask; process the first portion of the image data using a first number of diffusion steps of a diffusion model to generate a partially modified first portion of the image data; and process the partially modified first portion and the second portion of the image data using a second number of diffusion steps of the diffusion model to generate modified image data.

[0273] Aspect 26. The apparatus according to aspect 25, further comprising at least one camera configured to capture the image data.

[0274] Aspect 27. A method for modifying video data, the method comprising: obtaining a first token based on a first frame of video data, wherein each of the first tokens includes a feature vector corresponding to a corresponding position within the first frame of video data; obtaining a second token based on a second frame of video data, wherein each of the second tokens includes a feature vector corresponding to a corresponding position within the second frame of video data; determining a destination token from the first tokens; determining a candidate token from the second tokens based on a correspondence between the candidate token and the destination token; merging the candidate token and the destination token to generate a modified second token; and processing the modified second token using a diffusion model.

[0275] Aspect 28. The method according to aspect 27, wherein the destination token is randomly determined from the first token.

[0276] Aspect 29. The method according to any one of Aspects 27 or 28, wherein the corresponding relationship between the candidate token and the destination token is based on the cosine distance between the candidate token and the destination token.

[0277] Aspect 30. The method according to any one of Aspects 27 to 29, wherein processing the modified second token includes processing the unmerged tokens in the second token and not processing the merged candidate tokens.

[0278] Aspect 31. A method for modifying video data, the method comprising: obtaining a plurality of tokens, the plurality of tokens including a corresponding set of tokens for each of a plurality of frames of video data; identifying a destination token from the plurality of tokens; determining a candidate token from the plurality of tokens based on a correspondence between the candidate token and the destination token; merging the candidate token with the destination token to generate a modified second token; and processing the modified second token using a diffusion model.

[0279] Aspect 32. The method according to aspect 31, wherein the plurality of frames of the video data comprises a frame pool of the video data, and wherein the method is repeated for a further frame pool of the video data.

[0280] Aspect 33. The method according to any one of Aspects 31 or 32, wherein the plurality of frames of the video data includes a sliding frame window of the video data, and wherein the method is repeated for the sliding frame window of the video data.

[0281] Aspect 34. The method according to any one of Aspects 31 to 33, wherein the plurality of frames of the video data includes all frames of the video data.

[0282] Aspect 35. A method for modifying image data, the method comprising: obtaining tokens based on the image data, wherein each of the tokens includes a feature vector corresponding to a corresponding location within the image data; determining a destination token from the tokens; obtaining a segmentation mask based on the image data; determining the candidate token from the tokens based on a correspondence between the candidate token and the destination token and based on the segmentation mask; merging the candidate token and the destination token to generate a modified token; and processing the modified token using a diffusion model.

[0283] Aspect 36. The method according to aspect 35, wherein the segmentation mask is based on at least one of: foreground-background segmentation; saliency segmentation; or a cross-attention map from a latent representation.

[0284] Aspect 37. The method according to any one of Aspects 35 or 36, wherein determining the candidate token from the tokens based on the correspondence between the candidate token and the destination token and based on the segmentation mask further comprises: weighting the correspondence between the candidate token and the destination token based on the corresponding portion of the segmentation mask.

[0285] Aspect 38. The method according to aspect 37, wherein the weighting makes it less likely that a token corresponding to a significant portion of the image data as identified by the segmentation mask will be identified as a candidate token.

[0286] Aspect 39. The method according to any one of Aspects 35 to 38, wherein the candidate token is further based on a similarity threshold.

[0287] Aspect 40. The method according to any one of Aspects 35 to 39, wherein the correspondence between the candidate token and the destination token is based on the cosine distance between the candidate token and the destination token.

[0288] Aspect 41. The method according to any one of Aspects 35 to 40, wherein the image data comprises frames of video data, and the method is repeated for additional frames of the video data.

[0289] Aspect 42. A method for modifying image data, the method comprising: identifying a first portion of the image data and a second portion of the image data based on a segmentation mask; processing the first portion of the image data using a diffusion model to generate a modified first portion of the image data; and combining the modified first portion of the image data and the second portion of the image data to produce modified image data.

[0290] Aspect 43. The method according to aspect 42, wherein the segmentation mask is based on at least one of: foreground-background segmentation; saliency segmentation; or a cross-attention map from the latent representation.

[0291] Aspect 44. The method according to any one of Aspects 42 or 43, wherein combining the modified first portion and the second portion of the image data comprises a mixture of pixels from the modified first portion and the second portion of the image data.

[0292] Aspect 45. The method according to any one of aspects 42 to 44, wherein the first portion of the image data is cropped from the image data.

[0293] Aspect 46. The method according to any one of aspects 42 to 44, wherein the first portion of the image data includes a rectangular portion cropped from the image data.

[0294] Aspect 47. A method for modifying image data, the method comprising: identifying a first portion of the image data and a second portion of the image data based on a segmentation mask; processing the first portion of the image data using a first number of diffusion steps of a diffusion model to generate a partially modified first portion of the image data; and processing the partially modified first portion of the image data and the second portion of the image data using a second number of diffusion steps of the diffusion model to generate modified image data.

[0295] Aspect 48. A non-transitory computer-readable storage medium having instructions stored thereon, which, when executed by one or more processors, cause the one or more processors to perform any one of aspects 27 to 47.

[0296] Aspect 49. An apparatus for providing virtual content for display, the apparatus comprising one or more components for performing operations according to any one of aspects 27 to 47.

[0297] Aspect 50. An apparatus for modifying video data, the apparatus comprising: one or more memories configured to store the video data; and one or more processors coupled to the one or more memories and configured to: obtain a first token based on a first frame of the video data, each of the first tokens including a feature vector corresponding to a corresponding position within the first frame of the video data; obtain a second token based on a second frame of the video data, each of the second tokens including a feature vector corresponding to a corresponding position within the second frame of the video data; determine a destination token from the first tokens; determine a candidate token from the second tokens based on a correspondence between the candidate tokens and the destination token; merge the candidate tokens with the destination tokens to generate a modified second token; and process the modified second token using a diffusion model.

[0298] Aspect 51. The apparatus according to aspect 50, wherein the correspondence between the candidate token and the destination token is based on the cosine distance between the candidate token and the destination token.

[0299] Aspect 52. The apparatus according to any one of Aspects 50 to 51, wherein, in order to process the modified second token, the one or more processors are configured to process unmerged tokens in the second token and not process the merged candidate tokens.

[0300] Aspect 53. The apparatus according to any one of Aspects 50 to 52, wherein the one or more processors are configured to: identify a first frame group of the video data, the first frame group including a first frame of the video data and a second frame of the video data; identify an additional frame group of the video data; determine an additional modified token based on the additional frame group of the video data; and process the additional modified token using the diffusion model.

[0301] Aspect 54. The apparatus according to aspect 53, wherein the additional frame group of video data includes a frame pool.

[0302] Aspect 55. The apparatus according to any one of aspects 53 to 54, wherein the additional frame group of video data includes a sliding frame window.

[0303] Aspect 56. An apparatus for modifying image data, the apparatus comprising: one or more memories configured to store the image data; and one or more processors coupled to the one or more memories and configured to: obtain tokens based on the image data, wherein each of the tokens includes a feature vector corresponding to a corresponding location within the image data; determine a destination token from the tokens; obtain a segmentation mask based on the image data; determine the candidate tokens from the tokens based on a correspondence between the candidate tokens and the destination tokens and based on the segmentation mask; merge the candidate tokens with the destination tokens to generate a modified token; and process the modified tokens using a diffusion model.

[0304] Aspect 57. The apparatus according to aspect 56, wherein the segmentation mask is based on at least one of: foreground-background segmentation; saliency segmentation; or a cross-attention map from a latent representation.

[0305] Aspect 58. The apparatus according to any one of Aspects 56 to 57, wherein, in order to determine the candidate token from the tokens based on the correspondence between the candidate token and the destination token and based on the segmentation mask, the one or more processors are configured to: weight the correspondence between the candidate token and the destination token based on the corresponding portion of the segmentation mask.

[0306] Aspect 59. The apparatus according to aspect 58, wherein the weighting makes it less likely that a token corresponding to a significant portion of the image data as identified by the segmentation mask will be identified as a candidate token.

[0307] Aspect 60. The apparatus according to any one of Aspects 56 to 59, wherein the candidate token is further based on a similarity threshold.

[0308] Aspect 61. The apparatus according to any one of Aspects 56 to 60, wherein the correspondence between the candidate token and the destination token is based on the cosine distance between the candidate token and the destination token.

[0309] Aspect 62. The apparatus according to any one of aspects 56 to 61, wherein the image data comprises frames of video data, and wherein the one or more processors are configured to: determine an additional modified token based on additional frames of the video data; and process the additional modified token using the diffusion model.

[0310] Aspect 63. An apparatus for modifying image data, the apparatus comprising: one or more memories configured to store the image data; and one or more processors coupled to the one or more memories and configured to: identify a first portion and a second portion of the image data based on a segmentation mask; process the first portion of the image data using a diffusion model to generate a modified first portion of the image data; and generate modified image data based on the modified first portion and the second portion of the image data.

[0311] Aspect 64. The apparatus according to aspect 63, wherein, in order to generate the modified image data, the one or more processors are configured to combine the modified first portion of the image data and the second portion of the image data to generate the modified image data.

[0312] Aspect 65. The apparatus according to any one of aspects 63 to 64, wherein: the first portion of the image data is processed by a first number of diffusion steps of the diffusion model to generate the modified first portion of the image data; and in order to generate the modified image data, the one or more processors are configured to process the modified first portion of the image data and the second portion of the image data using a second number of diffusion steps of the diffusion model to generate the modified image data.

[0313] Aspect 66. The apparatus according to any one of aspects 63 to 65, wherein the segmentation mask is based on at least one of: foreground-background segmentation; saliency segmentation; or a cross-attention map from a latent representation.

[0314] Aspect 67. The apparatus according to any one of aspects 63 to 66, wherein, in order to combine the modified first portion of the image data and the second portion of the image data, the one or more processors are configured to mix pixels from the modified first portion of the image data and the second portion of the image data.

[0315] Aspect 68. The apparatus according to any one of aspects 63 to 67, wherein the first portion of the image data is cropped from the image data.

[0316] Aspect 69. The apparatus according to any one of aspects 63 to 68, wherein the first portion of the image data includes a rectangular portion cropped from the image data.

[0317] Aspect 70. A method for modifying video data, the method comprising: obtaining a first token based on a first frame of the video data, wherein each of the first tokens includes a feature vector corresponding to a corresponding position within the first frame of the video data; obtaining a second token based on a second frame of the video data, wherein each of the second tokens includes a feature vector corresponding to a corresponding position within the second frame of the video data; determining a destination token from the first tokens; determining a candidate token from the second tokens based on a correspondence between the candidate tokens and the destination tokens; merging the candidate tokens with the destination tokens to generate a modified second token; and processing the modified second token using a diffusion model.

[0318] Aspect 71. The method according to aspect 70, wherein the corresponding relationship between the candidate token and the destination token is based on the cosine distance between the candidate token and the destination token.

[0319] Aspect 72. The method according to any one of Aspects 70 to 71, wherein processing the modified second token includes processing the unmerged tokens in the second token and not processing the merged candidate tokens.

[0320] Aspect 73. The method according to any one of Aspects 70 to 72, the method further comprising: identifying a first frame group of the video data, the first frame group including a first frame of the video data and a second frame of the video data; identifying an additional frame group of the video data; determining an additional modified token based on the additional frame group of the video data; and processing the additional modified token using the diffusion model.

[0321] Aspect 74. The method according to aspect 73, wherein the additional frame group of video data includes a frame pool.

[0322] Aspect 75. The method according to any one of Aspects 73 to 74, wherein the additional frame group of video data includes a sliding frame window.

[0323] Aspect 76. A method for modifying image data, the method comprising: obtaining tokens based on the image data, wherein each of the tokens includes a feature vector corresponding to a corresponding location within the image data; determining a destination token from the tokens; obtaining a segmentation mask based on the image data; determining the candidate token from the tokens based on a correspondence between the candidate token and the destination token and based on the segmentation mask; merging the candidate token and the destination token to generate a modified token; and processing the modified token using a diffusion model.

[0324] Aspect 77. The method according to aspect 76, wherein the segmentation mask is based on at least one of: foreground-background segmentation; saliency segmentation; or a cross-attention map from the latent representation.

[0325] Aspect 78. The method according to any one of Aspects 76 to 77, wherein determining the candidate token from the tokens based on the correspondence between the candidate token and the destination token and based on the segmentation mask comprises weighting the correspondence between the candidate token and the destination token based on the corresponding portion of the segmentation mask.

[0326] Aspect 79. The method according to aspect 78, wherein the weighting makes it less likely that a token corresponding to a significant portion of the image data as identified by the segmentation mask will be identified as a candidate token.

[0327] Aspect 80. The method according to any one of Aspects 76 to 79, wherein the candidate token is further based on a similarity threshold.

[0328] Aspect 81. The method according to any one of Aspects 76 to 80, wherein the correspondence between the candidate token and the destination token is based on the cosine distance between the candidate token and the destination token.

[0329] Aspect 82. The method according to any one of Aspects 76 to 81, wherein the image data comprises frames of video data, the method further comprising: determining an additional modified token based on additional frames of the video data; and processing the additional modified token using the diffusion model.

[0330] Aspect 83. A method for modifying image data, the method comprising: identifying a first portion of the image data and a second portion of the image data based on a segmentation mask; processing the first portion of the image data using a diffusion model to generate a modified first portion of the image data; and generating modified image data based on the modified first portion of the image data and the second portion of the image data.

[0331] Aspect 84. The method according to aspect 83, wherein generating the modified image data includes combining the modified first portion of the image data and the second portion of the image data to generate the modified image data.

[0332] Aspect 85. The method according to any one of Aspects 83 to 84, wherein: the first portion of the image data is processed by a first number of diffusion steps of the diffusion model to generate the modified first portion of the image data; and generating the modified image data includes processing the modified first portion of the image data and the second portion of the image data using a second number of diffusion steps of the diffusion model to generate the modified image data.

[0333] Aspect 86. The method according to any one of Aspects 83 to 85, wherein the segmentation mask is based on at least one of: foreground-background segmentation; saliency segmentation; or a cross-attention map from a latent representation.

[0334] Aspect 87. The method according to any one of aspects 83 to 86, wherein combining the modified first portion and the second portion of the image data comprises a mixture of pixels from the modified first portion and the second portion of the image data.

[0335] Aspect 88. The method according to any one of aspects 83 to 87, wherein the first portion of the image data is cropped from the image data.

[0336] Aspect 89. The method according to any one of aspects 83 to 88, wherein the first portion of the image data includes a rectangular portion cropped from the image data.

[0337] Aspect 90. A non-transitory computer-readable storage medium having instructions stored thereon, the instructions causing the at least one processor, when executed, to perform any one of aspects 70 to 89.

[0338] Aspect 91. An apparatus for providing virtual content for display, the apparatus comprising one or more components for performing operations according to any one of aspects 70 to 89.

Claims

1. An apparatus for modifying video data, the apparatus comprising: One or more memories, the one or more memories being configured to store the video data; and One or more processors, said one or more processors being coupled to said one or more memories and configured to: A first token is obtained based on a first frame of the video data, wherein each of the first tokens includes a feature vector corresponding to a corresponding position within the first frame of the video data; A second token is obtained based on a second frame of video data, wherein each of the second tokens includes a feature vector corresponding to a corresponding position within the second frame of video data; Determine the destination token from the first token; The candidate token is determined from the second token based on the corresponding relationship between the candidate token and the destination token; The candidate token is merged with the destination token to generate a modified second token; as well as The modified second token is processed using a diffusion model.

2. The apparatus of claim 1, wherein the correspondence between the candidate token and the destination token is based on the cosine distance between the candidate token and the destination token.

3. The apparatus according to claim 1, wherein, In order to process the modified second token, the one or more processors are configured to process unmerged tokens in the second token and not process merged candidate tokens.

4. The apparatus of claim 1, wherein the one or more processors are configured to: A first frame group is identified by the video data, the first frame group including the first frame and the second frame of the video data; Additional frame groups that identify the video data; The additional modified token is determined based on the additional frame group of the video data; as well as The diffusion model is used to process the additional modified token.

5. The apparatus of claim 4, wherein the additional frame group of video data comprises a frame pool.

6. The apparatus of claim 4, wherein the additional frame group of video data includes a sliding frame window.

7. The apparatus of claim 1, wherein the one or more processors are configured to: The modified second token is used to generate an output image for display using the diffusion model.

8. The apparatus of claim 7, further comprising a display configured to display the output image.

9. The apparatus of claim 1, further comprising at least one camera configured to capture the first frame and the second frame of the video data.

10. An apparatus for modifying image data, the apparatus comprising: One or more memories, the one or more memories being configured to store the image data; and One or more processors, said one or more processors being coupled to said one or more memories and configured to: Tokens are obtained based on image data, wherein each of the tokens includes a feature vector corresponding to a corresponding position within the image data; Determine the destination token from the token; The segmentation mask is obtained based on the image data; The candidate token is determined from the tokens based on the correspondence between the candidate token and the destination token, and based on the segmentation mask. The candidate token is merged with the destination token to generate a modified token; as well as The modified token is processed using a diffusion model.

11. The apparatus of claim 10, wherein the segmentation mask is based on at least one of the following: Foreground-background segmentation; Significant segmentation; or Cross-attention graph derived from latent representation.

12. The apparatus according to claim 10, wherein, In order to determine the candidate token from the tokens based on the correspondence between the candidate token and the destination token and based on the segmentation mask, the one or more processors are configured to: The corresponding relationship between the candidate token and the destination token is weighted based on the corresponding part of the segmentation mask.

13. The apparatus of claim 12, wherein the weighting makes it less likely that a token corresponding to a significant portion of the image data as identified by the segmentation mask will be identified as a candidate token.

14. The apparatus of claim 10, wherein the candidate token is further based on a similarity threshold.

15. The apparatus of claim 10, wherein the correspondence between the candidate token and the destination token is based on the cosine distance between the candidate token and the destination token.

16. The apparatus of claim 10, wherein the image data comprises frames of video data, and wherein the one or more processors are configured to: Determine the additional modified token based on the additional frames of the video data; and The diffusion model is used to process the additional modified token.

17. The apparatus of claim 10, wherein the one or more processors are configured to: The modified token is used to generate an output image for display using the diffusion model.

18. The apparatus of claim 17, further comprising a display configured to display the output image.

19. The apparatus of claim 10, further comprising at least one camera configured to capture the image data.

20. An apparatus for modifying image data, the apparatus comprising: One or more memories, the one or more memories being configured to store the image data; and One or more processors, said one or more processors being coupled to said one or more memories and configured to: The first part and the second part of the image data are identified based on a segmentation mask; The first portion of the image data is processed using a diffusion model to generate a modified first portion of the image data; as well as Modified image data is generated based on the modified first portion of the image data and the second portion of the image data.

21. The apparatus according to claim 20, wherein, In order to generate the modified image data, the one or more processors are configured to combine the modified first portion of the image data and the second portion of the image data to produce the modified image data.

22. The apparatus according to claim 20, wherein: The first portion of the image data is processed through a first number of diffusion steps of the diffusion model to generate the modified first portion of the image data; as well as In order to generate the modified image data, the one or more processors are configured to process the modified first portion and the second portion of the image data using a second number of diffusion steps of the diffusion model to generate the modified image data.

23. The apparatus of claim 20, wherein the segmentation mask is based on at least one of the following: Foreground-background segmentation; Significant segmentation; or Cross-attention graph derived from latent representation.

24. The apparatus according to claim 20, wherein, In order to combine the modified first portion and the second portion of the image data, the one or more processors are configured to mix pixels from the modified first portion and the second portion of the image data.

25. The apparatus of claim 20, wherein the first portion of the image data is cropped from the image data.

26. The apparatus of claim 20, wherein the first portion of the image data includes a rectangular portion cropped from the image data.

27. The apparatus of claim 20, further comprising at least one camera configured to capture the image data.

28. The apparatus of claim 20, further comprising a display configured to display the modified image data.

Images