Image compression method based on vector quantization and aligned diffusion transformer

Abstract

The application discloses an image compression method based on vector quantization and alignment diffusion Transformer, comprising the following steps: extracting an initial token sequence of an original image to be compressed, and separating and aggregating the initial token sequence into a semantic token sequence and a source matrix which retains a spatial mapping relationship; performing discrete quantization processing on the semantic token sequence to obtain quantized semantic tokens; performing reverse expansion mapping on the quantized semantic tokens by using the source matrix to restore an initial latent representation; calculating the local spatial variance of the initial latent representation, and generating a spatially adaptive pseudo time step field according to the local spatial variance; generating a latent condition according to the quantized semantic tokens and injecting the latent condition into a diffusion Transformer network as a guide; performing flow matching denoising processing on the initial latent representation based on the pseudo time step field to output a target image. The application improves the representation bottleneck and detail loss problems of traditional quantization, and realizes triple synergistic optimization of high compression efficiency, low decoding delay and high-fidelity visual reconstruction under a low code rate.

Description

Technical Field

[0001] This invention relates to the field of image data processing and compression technology, and in particular to an image compression method based on vector quantization and aligned diffusion Transformer. Background Technology

[0002] With the explosive growth of visual data, efficient image compression technology is of great technical significance in reducing transmission bandwidth consumption and storage medium consumption. Maintaining the complete expression of image features under low bit rate conditions and ensuring high-speed real-time flow of end-to-end encoding and decoding processes are current research directions for the underlying optimization of multimedia communication, low-latency interaction, and massive visual storage systems.

[0003] Current mainstream deep learning image compression schemes are mostly based on continuous latent representation frameworks, combined with autoregressive context models for probability distribution estimation; or they employ traditional discrete representation techniques, retrieving image features from a physical codebook and mapping them into discrete symbol sequences. Meanwhile, some reconstruction efforts are beginning to introduce conventional generative models based on hierarchical downsampling to compensate for visual texture. However, when processing high-resolution images with low bitrates, existing techniques sometimes introduce decoding computational latency due to the serially dependent context prediction in continuous probability estimation. Traditional discrete representation schemes are prone to having many feature vectors idle during model training, resulting in limited overall representation capacity, and the quantization process strips away fine-grained spatial details of the image. Furthermore, conventional generative reconstruction models are limited by the large feature size of shallow latent spaces, incurring considerable computational overhead and making it difficult to achieve high-quality pixel reconstruction while maintaining low latency.

[0004] Existing image compression technologies struggle to balance real-time latency during encoding and decoding, representational capacity of low-level features, and fidelity of visual reconstruction in low-bitrate scenarios. Therefore, there is an urgent need to research an image compression scheme that can overcome the bottleneck of spatial feature loss at low bitrates and possess both efficient compression transfer and low-latency, high-fidelity reconstruction capabilities. Summary of the Invention

[0005] The purpose of this invention is to provide an image compression method based on vector quantization and aligned diffusion Transformer, in order to solve the aforementioned problems in the prior art.

[0006] Technical solution: An image compression method based on vector quantization and aligned diffusion Transformer, comprising:

[0007] Obtain the original image to be compressed and extract the initial token sequence of the original image;

[0008] The initial token sequence is aggregated and separated into a semantic token sequence carrying high-level semantics and a source matrix that preserves the spatial mapping relationship.

[0009] Discrete quantization is performed on the semantic token sequence to obtain quantized semantic tokens;

[0010] By using the source matrix to perform a reverse expansion mapping on the quantized semantic token, an initial latent representation with dense spatial resolution and containing structured noise is recovered.

[0011] Calculate the local spatial variance of the initial latent representation and generate a spatially adaptive pseudo-time step field based on the local spatial variance;

[0012] Generate potential conditions based on quantized semantic tokens;

[0013] Latent conditions are injected into the diffuse Transformer network as guidance, and flow matching denoising is performed on the initial latent representation based on a spatially adaptive pseudo-time step field to output the reconstructed target image.

[0014] Furthermore, the initial token sequence is aggregated, separating it into a semantic token sequence carrying high-level semantics and a source matrix that preserves the spatial mapping relationship, including:

[0015] The feature similarity between tokens in the initial token sequence is calculated based on the attention mechanism.

[0016] The initial token sequence is clustered and merged based on feature similarity to obtain a semantic token sequence with decoupled location information, and clustering routing information is generated during the clustering and merging process.

[0017] Record the clustering routing information into the source matrix.

[0018] Furthermore, the semantic token sequence is discretized and quantized to obtain quantized semantic tokens, including:

[0019] Semantic token sequences are processed using a lookup-free vector quantization mechanism;

[0020] During the quantization process, the sign function is used to determine the numerical polarity of continuous feature vectors in the semantic token sequence, so as to map the continuous feature vectors into discrete binary features.

[0021] The binary feature combination is converted into the corresponding integer set index, and the integer set index is used as the quantized semantic token.

[0022] Furthermore, by performing a reverse extension mapping on the quantized semantic tokens using the source matrix, an initial latent representation with dense spatial resolution and containing structured noise is recovered, including:

[0023] The source matrix is ​​used as a spatial routing index to extract mapping association features;

[0024] Based on the spatial mapping relationship of the mapping association feature records, the discrete features corresponding to the quantized semantic tokens are redistributed and expanded to the feature space layout that matches the initial token sequence, thus constructing the initial latent representation.

[0025] Furthermore, the local spatial variance of the initial latent representation is calculated, and a spatially adaptive pseudo-time step field is generated based on the local spatial variance, including:

[0026] Evaluate local spatial variance to measure the spatial uncertainty of the initial latent representation in different regions;

[0027] The local spatial variance is projected onto the target latent feature dimension, and the projected variance data is converted into a pixel-level or feature block-level spatial-temporal step feature map through a preset monotonic function, resulting in a spatially adaptive pseudo-temporal step field.

[0028] Furthermore, latent conditions are generated based on quantized semantic tokens and injected into the diffuse Transformer network as guidance, including:

[0029] Extract the latent feature projection vector of the derived self-quantized semantic token and construct the latent conditions;

[0030] By replacing the text cue guidance branch in the diffusion Transformer network with latent conditions, text-free conditional control can be performed during the image reasoning and reconstruction phase.

[0031] Furthermore, the initial latent representation is subjected to flow matching denoising processing based on a spatially adaptive pseudo-time step field, including:

[0032] Spatially adaptive pseudo-time step field and initial latent representation are spatially aligned and feature-stitched in a diffuse Transformer network;

[0033] Based on the numerical distribution of the spatially adaptive pseudo-time step field, differentiated denoising intensity features are assigned to different spatial locations of the initial latent representation.

[0034] Based on the differentiated denoising intensity features, the initial latent representation is denoised by performing a forward prediction operation of the flow matching network in one step or less than a preset upper limit of steps.

[0035] Furthermore, the diffused Transformer network is obtained through end-to-end joint optimization pre-training using a two-stage implicit bit rate pruning strategy.

[0036] Furthermore, the pre-training process also includes a co-alignment constraint step for latent conditions:

[0037] Extract pre-defined text feature embeddings that match the image training samples;

[0038] Based on the latent conditions obtained from projection and the preset text feature embeddings, the contrastive co-alignment loss is calculated.

[0039] We utilize contrastive co-alignment loss to constrain model parameters in order to align the semantic consistency of latent conditions with text-guided features during the pre-training phase.

[0040] Furthermore, before extracting the initial token sequence from the original image, the following steps are also included:

[0041] Deep downsampling is performed on the original image using a pre-built downsampling encoder to obtain deep latent feature representations;

[0042] The process of extracting the initial token sequence from the original image involves extracting the deep latent feature representation within its corresponding constant deep latent feature resolution space, thereby reducing the spatial redundancy of the feature sequence.

[0043] Beneficial effects: This invention improves the representation bottleneck and detail loss problem of traditional quantization, and achieves a triple synergistic optimization of high compression efficiency, low decoding latency and high-fidelity visual reconstruction at low bit rates. Attached Figure Description

[0044] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0045] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 A flowchart of an image compression method based on vector quantization and aligned diffusion Transformer provided in an embodiment of this application.

[0047] Figure 2 This is a flowchart illustrating the aggregation process of the initial token sequence provided in an embodiment of this application.

[0048] Figure 3 A flowchart for obtaining a quantized semantic token provided in an embodiment of this application.

[0049] Figure 4 This is a flowchart illustrating the reverse expansion mapping of quantized semantic tokens using a source matrix, provided as an embodiment of this application.

[0050] Figure 5A flowchart for calculating the local spatial variance of the initial potential representation provided in an embodiment of this application.

[0051] Figure 6 A flowchart for generating potential conditions based on quantized semantic tokens, provided for embodiments of this application.

[0052] Figure 7 A flowchart illustrating the two-stage implicit bit rate pruning strategy provided in this application embodiment. Detailed Implementation

[0053] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0054] Furthermore, in the embodiments of this application, the words "exemplary," "for example," etc., are used to indicate that they are examples, illustrations, or descriptions. Any embodiment or design scheme described as "exemplary" in the embodiments of this application should not be construed as being better or more advantageous than other embodiments or design schemes. Specifically, the use of the term "exemplary" is intended to present the concept in a concrete manner.

[0055] Example 1

[0056] This embodiment provides an image compression method based on vector quantization and aligned diffusion Transformer, such as Figure 1 As shown, the method includes the following steps:

[0057] Step 101: Obtain the original image to be compressed and extract the initial token sequence of the original image.

[0058] As an optional implementation, the original image to be compressed can be obtained by reading image data from an edge acquisition terminal or a cloud storage server. The original image can be conventional RGB three-channel image data. Further, the initial token sequence of the original image is extracted using a basic feature extraction network. A standard convolutional neural network layer can be used to perform deep downsampling processing on the original image; for example, in this embodiment, the downsampling factor is set to 32. The spatial feature map obtained after feature extraction is then flattened, outputting an initial token sequence of length L. This process is a forward spatial feature dimensionality reduction operation, used to remove redundant spatial representations of the original pixels, and does not involve a progressive noise degradation process.

[0059] Step 102: Aggregate the initial token sequence to separate it into a semantic token sequence carrying high-level semantics and a source matrix that preserves the spatial mapping relationship.

[0060] Accordingly, aggregation is used to compress lengthy feature sequences to accommodate low bit rates. Specifically, by calculating the similarity between feature elements within the initial token sequence, spatially adjacent or similarly represented tokens are clustered and merged. After aggregation, the original feature sequence of length L is reduced to a semantic token sequence of length K, where K is less than L. Simultaneously, the connections and affiliations generated during the clustering and merging process are fully preserved and constructed as a source matrix. The source matrix is ​​specifically represented as a sparse matrix or association index table with dimensions K rows and L columns, whose internal matrix elements record the affiliation weights or connection mappings of each token in the original sequence when merged into a specific semantic cluster center. Through this explicit separation operation, high-level semantic features and low-level spatial details are separately preserved, achieving structural decoupling.

[0061] Step 103: Perform discrete quantization on the semantic token sequence to obtain quantized semantic tokens.

[0062] Furthermore, the semantic token sequence, represented by continuous floating-point values, is mapped to a discrete symbol space. Specifically, a predefined quantization mapping operator is applied to each continuous feature vector in the semantic token sequence, nonlinearly converting it into a discrete index value in a finite set of integers, thereby obtaining a quantized semantic token. Discrete quantization processing can operate independently of the continuous probability distribution properties of the image data, allowing the output discrete token to be adapted to the subsequent lossless entropy coding module, compressing the overall transmission bitstream to the limit of low bitrate. The quantized semantic token and the source matrix together constitute the compressed bitstream, which is sent to the decoding end via the transmission channel. At the decoding end, the quantized semantic token and the source matrix are parsed and extracted from the received compressed bitstream for subsequent reconstruction processing.

[0063] Step 104: Use the source matrix to perform a reverse extension mapping on the quantized semantic token to recover the initial latent representation with dense spatial resolution and containing structured noise.

[0064] Specifically, at the start of the image decoding and reconstruction stage, the compact discrete features need to be restored to their original large-dimensional spatial layout. Specifically, the source matrix accompanying the bitstream transmission is used as a high-dimensional dynamic routing table or spatial weight allocation mask to perform a reverse allocation operation on quantized semantic tokens of length K. The reverse expansion mapping, referring to the membership relationships recorded in the source matrix, redistributes, diffuses, and fills the discrete features back into a dense resolution spatial grid of length L according to their corresponding weights. The resulting initial latent representation not only aligns with the initial token sequence in terms of feature dimensions, but its internal data structure also includes structured distortion noise introduced by the preceding compressed discrete quantization operation.

[0065] Step 105: Calculate the local spatial variance of the initial latent representation and generate a spatially adaptive pseudo-time step field based on the local spatial variance.

[0066] Meanwhile, local spatial variance is used to quantitatively measure the level of uncertainty of the initial latent representation in different spatial regions, or the complexity of texture corruption. Specifically, the variance value can be calculated within the feature channel dimension or a preset local spatial sliding window. Further, through a preset differentiable feature projection function and a nonlinear monotonically increasing function, the local spatial variance corresponding to each pixel block or feature block is transformed into denoising time step parameters of varying magnitudes, thereby constructing a spatially adaptive pseudo-time step field with the same spatial size as the entire image. Under this mapping mechanism, local regions with larger variances indicate higher uncertainty caused by structured noise contamination, and therefore will be mapped and assigned relatively larger numerical indices in the pseudo-time step field.

[0067] Step 106: Generate potential conditions based on quantized semantic tokens.

[0068] Optionally, the latent conditions are used as the internal core driving and control signals for subsequent generation of large models. Specifically, a specialized latent condition projection operator is constructed to extract and derive multidimensional continuous conditional feature projection vectors from the quantized semantic tokens received at the decoding end, which serve as latent conditions.

[0069] Step 107: Inject latent conditions into the diffuse Transformer network as guidance, and perform flow matching denoising on the initial latent representation based on the spatially adaptive pseudo-time step field to output the reconstructed target image.

[0070] As an example, the Diffused Transformer network operates in a constant deep latent feature resolution space to reduce the redundant computational overhead of traditional shallow network structures.

[0071] To achieve this configuration, this embodiment employs a multi-layer Transformer encoder architecture. The network contains NL cascaded Transformer blocks, each consisting of a self-attention layer, a cross-attention layer, and a feedforward network layer in sequence. The self-attention layer handles the internal feature interactions of the initial latent representation, while the cross-attention layer receives latent conditions as key-value pair inputs. The hidden dimension of each layer is set to dh, and the number of attention heads is n. _h Those skilled in the art can determine the specific values ​​of the above-mentioned architecture parameters based on actual computing resources and image resolution requirements using conventional model scaling optimization methods.

[0072] Specifically, the text prompt input in the conventional pre-trained generative model is completely replaced with latent conditions, and this latent condition is injected into the network's cross-attention mechanism layer as a guiding signal. Simultaneously, a spatially adaptive pseudo-time step field is concatenated with the initial latent representation along the channel dimension and input into the network. The network applies targeted denoising and restoration gains to local regions of the input features based on the specific numerical values ​​of the pseudo-time step field at each spatial location, thereby completing the denoising logic through a single-step or fewer-step forward prediction operation. After stream matching denoising, the feature sequence, originally introduced by discretization quantization into structured noise, is effectively restored and smoothed. Finally, a conventional image decoding layer is used to restore the denoised latent features, outputting a high-fidelity reconstructed target image.

[0073] Example 2

[0074] In one possible implementation, the initial token sequence is aggregated, separating it into a semantic token sequence carrying high-level semantics and a source matrix that preserves the spatial mapping relationship, such as... Figure 2 As shown, it includes the following steps:

[0075] Step 201: Calculate the feature similarity between tokens in the initial token sequence based on the attention mechanism.

[0076] In one aspect, when calculating feature similarity, an initial token sequence is loaded, with its resolution dimension represented as the product of length L and channel dimension d. A linear transformation matrix is ​​used to project the initial token sequence into a query vector matrix, a key vector matrix, and a value vector matrix. The query vector matrix and the key vector matrix are both L-row products of channel dimension d. Next, matrix multiplication is performed on the transposes of the query and key vector matrices to calculate the dot product correlation value between any two tokens. To maintain numerical stability, the dot product correlation value is divided by the square root of the channel dimension d to obtain a normalized dot product matrix. A normalized exponential function is applied to the normalized dot product matrix to output an attention feature correlation matrix with dimensions L rows and L columns. Each element in this matrix represents the degree of correlation between two tokens at the corresponding position in the feature space, thus determining the feature similarity between tokens in the initial token sequence.

[0077] In some alternative implementations, the method for calculating feature similarity may also employ a cosine similarity metric operator to calculate the cosine value of the angle between two feature vectors, or use the reciprocal of the Euclidean distance to characterize the feature proximity between tokens.

[0078] Step 202: Cluster and merge the initial token sequence according to feature similarity to obtain a semantic token sequence with decoupled location information, and generate clustering routing information during the clustering and merging process.

[0079] Preferably, feature similarity is used to perform redundancy compression on the sequence. Specifically, the initial token sequence of length L is divided into a first token subset and a second token subset, each of length L / 2. Based on the feature similarity index recorded in the attention feature association matrix, for each token in the first token subset, the corresponding target with the highest similarity in the second token subset is searched, establishing several pairs of associated tokens. All established associated token pairs are sorted in descending order of similarity value, and the top M pairs with the highest similarity are selected for feature fusion. The feature fusion operation can specifically be a weighted sum or average of two related feature vectors, thereby merging the two tokens into a new feature node. By continuously performing the merging operation a preset number of times, the initial sequence of length L is compressed into a semantic token sequence containing K independent feature nodes. Since the spatial coordinate information of each feature node has been separated from the ontology feature vector during feature fusion or mean calculation, a semantic token sequence with decoupled positional information is obtained. The typical range for the number of target semantic tokens K to be retained can be set between L / 4 and L / 32.

[0080] In some alternative implementations, the cluster merging execution strategy can also introduce density-based clustering algorithms or adopt the bipartite graph maximum matching criterion to filter the token sets to be merged, in order to adapt to the dimensionality reduction requirements of different image texture complexities.

[0081] Step 203: Record the clustering routing information into the source matrix.

[0082] Furthermore, to enable the decoding end to reverse-engineer the dense spatial layout, the clustering trajectory needs to be tracked and recorded in real time at the encoding end. Specifically, an initial state source matrix is ​​initialized, with a structure of K rows and L columns, forming a two-dimensional grid matrix. When a predetermined token in the initial token sequence is assigned to or merged into the i-th cluster center node in the semantic token sequence, the corresponding routing association weight is written to the grid intersection node in the i-th row and j-th column of the initial state source matrix. If no merging occurs or there is no affiliation, the value at the corresponding intersection node is set to 0. Finally, by traversing all merging paths throughout the entire clustering merging cycle, all clustering routing information is recorded in the source matrix as matrix elements. The source matrix as a whole presents a high-dimensional sparse matrix structure, used to store the two-dimensional spatial mapping topology of high-level semantic features in the original image, providing a routing index reference for subsequent demerging and recovery at the decoding end.

[0083] In some alternative implementations, the routing information recorded in the source matrix can also be converted into a one-dimensional indexed linked list data structure or a compressed set of non-zero element triples to reduce channel occupancy in the transmission bandwidth.

[0084] According to one aspect of this application, the process of clustering and merging the initial token sequence also includes dynamic code rate adjustment, specifically including:

[0085] Optionally, the bandwidth status parameters of the current communication network or the preset storage capacity limit command can be received in real time.

[0086] Meanwhile, the image compression system can be configured within a hardware terminal with network interaction capabilities. In network transmission mode, the data interface module monitors the round-trip time, packet loss rate, and channel throughput of the network channel in real time, and calculates and generates the current bandwidth status parameters of the communication network accordingly. In local storage mode, the control main chip reads the pre-stored allocation quota in the configuration chip and obtains the preset storage capacity limit instruction. The bandwidth status parameters and storage capacity limit instruction serve as external boundary constraint variables, provided in real time to the compression control center as the basis for subsequent dynamic adjustment of the image bitrate.

[0087] In some alternative implementations, the frequency of receiving parameters can be dynamically adjusted by receiving externally input image perception quality level instructions as constraint parameters, or by detecting the current remaining battery power.

[0088] Optionally, the number of target semantic tokens retained in the clustering and merging operation can be dynamically adjusted according to bandwidth status parameters or storage capacity limit instructions to achieve adaptive control of the output image compression bitrate.

[0089] As an optional implementation, the output bitrate of the image is affected by the volume of transmitted features, while the volume of the semantic token sequence is determined by the number of tokens. Specifically, the control center sets a unified decision-making process. In the first execution path, when the bandwidth status parameter is lower than a preset bandwidth threshold or the storage capacity limit instruction is lower than a preset capacity threshold, the number of target semantic tokens retained in the clustering and merging operation is dynamically adjusted. Furthermore, the value of the target semantic token retention number K is reduced, and feature merging is performed during the clustering and merging process to shorten the output feature sequence and reduce the output image compression bitrate. In the second execution path, when the bandwidth status parameter is greater than or equal to a preset bandwidth threshold and the storage capacity limit instruction is greater than or equal to a preset capacity threshold, the number of target semantic tokens retained in the clustering and merging operation is dynamically adjusted. Specifically, the value of the target semantic token retention number K is increased, the proportion of clustering and merging is reduced, the integrity of high-resolution image features is improved, and adaptive control of the output image compression bitrate is achieved.

[0090] In some alternative implementations, the adjustment range of the target semantic token retention quantity K can be implemented using a proportional-integral-derivative control algorithm to achieve a smooth and gradual transition, avoiding abrupt changes in image compression quality due to network jitter.

[0091] Example 3

[0092] In one possible implementation, the combined process of aggregating the initial token sequence and discretely quantizing the semantic token sequence is performed using any one of the following: token encoding mode, random order generation mode, or token-by-token autoregressive prediction mode.

[0093] Accordingly, during the actual operation of the image compression system, due to differences in real-time constraints and hardware architectures across different application scenarios, the compression system is configured with a unified decision-making mechanism. The control node determines the direction of the target data stream based on the type of hardware computing resources currently mounted on the system and the latency threshold configuration parameters.

[0094] In the first execution path, when the system is detected to have hardware acceleration resources that meet the concurrency requirements, the process of obtaining the quantized semantic token and the source matrix adopts the token encoding mode.

[0095] Specifically, this mode performs static global feedforward mapping computation by loading a pre-built end-to-end visual tagger network. The end-to-end visual tagger network can include an encoder structure based on alternating cascades of multilayer perceptrons and multi-head self-attention mechanisms. The tagger encoding mode is implemented by performing parallel feature extraction operations based on a feedforward neural network on the initial input token sequence.

[0096] Simultaneously, since the spatial features of the entire image have been flattened and loaded, the self-attention mechanism module within the network can simultaneously evaluate the global dependencies of all feature nodes. During this forward propagation, the clustering layer within the network calculates feature similarity based on the global feature distribution and simultaneously outputs the compressed continuous feature sequence and the source matrix used to record the merging trajectory during the dimensionality reduction phase. Further, the quantization layer receives this continuous feature sequence and performs parallel discrete polarity mapping on the continuous vectors using a pre-configured lookup-free quantization operator, outputting a quantized semantic token with discrete integer index attributes.

[0097] As an optional implementation, a token-based encoding mode is employed to avoid the serial computation latency caused by the token-by-token prediction mechanism in the autoregressive model. In the aforementioned end-to-end forward mapping operation, the data flow and processing actions of all tokens are decoupled and parallel in the spatial dimension. Utilizing the concurrent processing array of the computing device, the system can obtain the full discretization result through a single complete forward inference computation, controlling the end-to-end latency overhead of the encoding / decoding framework. To evaluate the processing efficiency performance of this token-based encoding mode, it can be calculated using a defined computing resource mapping model, the expression of which is as follows:

[0098] T _encode =L*C_ ops / P_ compute ;

[0099] Among them, T _encode The processing latency during the encoding stage is represented by L, where L is the length of the initial input token sequence, and C is the input token sequence length. _ops P is the operation instruction constant required for single-token processing. _compute This represents the concurrent computation throughput metric of the invoked processor. As shown in the expression above, when the processor's concurrent computation throughput metric is high, the latency overhead corresponding to this encoding mode decreases monotonically.

[0100] In some alternative implementations, the end-to-end visual tagger network executing the tagger encoding pattern can further integrate tensor processing core operators at the hardware level. Furthermore, by merging clustering compression operations and discrete polarity mapping operations in the network into a single execution node at the computation graph level through a system compiler, memory read / write access frequency and data transfer overhead can be reduced, improving the cache hit rate when processing high-resolution images.

[0101] Example 4

[0102] In one possible implementation, when using a random order generation mode, a pre-configured random order generator is used to generate discrete tokens with relevant positional instructions as quantized semantic tokens based on the feature distribution of the initial token sequence; and a source predictor is used to recover the positional space details based on the quantized semantic tokens and their carried relevant positional instructions to construct the source matrix.

[0103] Optionally, within the unified decision-making mechanism, a second execution path parallel to the aforementioned encoding mode is provided. When the monitoring module detects that the computing terminal is equipped with a tensor computation core optimized for autoregressive generation computation, or receives an external input control instruction requiring the retention of independent generation control weights for the local structure of the image, the data flow direction is switched.

[0104] Specifically, the system loads a pre-configured random sequence generator. This generator employs an undirected sequence modeling architecture, distinct from the linear order of fixed raster scans. Internally, it maintains a state walk matrix and performs autoregressive prediction of non-fixed traversal paths based on the gradient of the feature information entropy distribution. Within each sequence generation cycle, the network layers output feature vectors and simultaneously calculate the relevant positional instructions bound to these vectors through a positional bias network layer. These positional instructions are represented as continuous vectors carrying coordinate bias information and scale dimension, used to label the relative coordinate system of the currently generated discrete markers in the original two-dimensional physical space distribution. The set of discrete sequences generated via non-deterministic paths and carrying coordinate bias information is output as quantized semantic tokens. By introducing a random sequence mechanism, the model increases the diversity of sampling paths when dealing with regions with complex textures.

[0105] Furthermore, since the random generation process lacks the dense spatial mapping relationship of a static marker, the system calls and runs an independent source predictor. The source predictor receives quantized semantic tokens and embedded relevant positional instructions as input feature variables. This predictor performs a backward inference calculation of the spatial dimension, calculating the correlation score between the relevant positional instructions on each discrete marker and all spatial coordinate nodes on the target-size grid by parsing and decoding them, thereby establishing a probabilistic mapping link between the low-dimensional sequence features and the dense two-dimensional grid. The alignment probability is calculated using exponential normalization, as shown in the following expression:

[0106] P _align =exp(M _pos *V _dir ) / Σ(exp(M _pos *V _dir ));

[0107] Among them, P _align To calculate the obtained spatial alignment probability matrix, M _pos V is the position embedding matrix generated based on the target dense spatial grid initialization. _dir The system parses and extracts directional feature vectors from relevant location instructions. Through the aforementioned probability calculation logic, the source predictor determines the specific coordinate set that each discrete marker corresponds to in the full-size grid. Based on the determination result, the system initializes a two-dimensional table structure in memory, writes the extracted mapping trajectory parameters into the table, and outputs the source matrix that establishes the two-dimensional routing association state. Through this type of generation and reverse prediction coupling loop, the discretely generated token sequence is re-bound to the physical space structure.

[0108] In some alternative implementations, the random sequence generator may load a visually saliency feature mask extracted by a preprocessing operator as a priori bias before performing the sequence prediction task. Based on the distribution gradient of pixel intensity in the visually saliency feature mask, computational priority is assigned to high-frequency texture regions with high information density. Under this condition, the predictor adjusts the initial weight distribution of the state walk matrix, prioritizing the generation of discrete markers covering high-frequency regions.

[0109] Example 5

[0110] In one possible implementation, when using a token-by-token autoregressive prediction mode, based on the initial token sequence, quantized semantic tokens are generated sequentially according to the grating order, and a source matrix is ​​constructed according to the spatial coordinates of each token in the grating order; wherein, during the prediction process, a key-value cache compression mechanism is used to remove redundancy from the historical key-value cache, thereby reducing the computational overhead of sequence prediction.

[0111] Specifically, when the control node detects that the system is operating in a restricted edge device environment with streaming computing power and has the requirement for temporal causal computation, a third execution path is triggered. The token-by-token autoregressive prediction mode refers to a working paradigm based on a causal autoregressive architecture, predicting and generating discrete tokens position-by-position in sequence. Each token represents the independent token unit generated sequentially in the sequence prediction link. The system loads an autoregressive prediction network configured with a causal attention mask. The raster order specifically refers to the traversal generation path of the feature sequence on a two-dimensional spatial grid, starting from the initial spatial coordinates (0,0) in row 0, advancing column by column from left to right along the current row, jumping to the starting column of the next row after completing a single row traversal, until reaching the final coordinates of the last row and last column to end the traversal operation. When performing the token-by-token autoregressive prediction operation, the prediction network receives the previously generated token sequences as context input parameters, and through the mapping calculation between the internal multilayer perception module and the causal attention module, outputs the feature probability distribution corresponding to the current spatial coordinate node. Furthermore, greedy sampling is performed on the feature probability distribution to extract continuous feature vectors, and a pre-defined quantization mapping operator is invoked to convert them into discrete-state quantized semantic tokens. This node-by-node recursive generation method maintains the causal logical coherence of the image spatial structure.

[0112] Furthermore, in the long-sequence raster order prediction process, as the number of generation steps increases, the storage volume of historical feature states within the autoregressive network expands linearly, leading to increased memory bandwidth usage and deteriorated computational latency. Under these conditions, a key-value caching and compression mechanism needs to be deployed within the prediction network. Specifically, the key vectors and value vectors generated by the network's attention layer in each computation are extracted and written into a dynamically long cache queue. In each new prediction step, the cache controller initiates state evaluation logic to perform redundancy calculation on the historical nodes in the cache queue. As a calculation rule, the attention weight score between the current query vector and each historical key vector in the cache queue is calculated as a basic metric for feature relevance. To implement the compression and culling operation, a cache retention score calculation expression is constructed:

[0113] S _keep =W _attn *γ+1 / (D _space +1);

[0114] Each item is a normalized dimensionless score indicator. _keep To retain scores in the cache, W _attn To calculate the extracted attention weight score, γ is a preset association weight coefficient, and D _space This represents the Euclidean distance between the historical node and the currently generated node on the 2D raster mesh. The scores of all historical nodes are calculated and output using the formula described above.

[0115] Furthermore, the score array is compared with a preset retention threshold. Key-value pairs below the threshold are identified as redundant data and removed from the cache queue, freeing up the corresponding memory address space. This pruning and updating of the cache queue limits the lower limit of the historical feature capacity required for single-step prediction, thereby eliminating redundant computations in the sequence prediction process.

[0116] In some alternative implementations, the key-value cache compression mechanism can also be replaced by a token-merging-based downsampling strategy. Specifically, multiple historical key-value pairs that are spatially adjacent and have a feature cosine similarity greater than a preset merging threshold are extracted. Their feature values ​​are then added together using an average pooling operator, and the average value is calculated to merge them into a single macro key-value pair. This compresses the physical volume of the cache queue without hard truncation of historical features.

[0117] Example 6

[0118] In one possible implementation, the semantic token sequence is discretized and quantized to obtain quantized semantic tokens, such as... Figure 3 As shown, it includes the following steps:

[0119] Step 601: The semantic token sequence is processed using a lookup-free vector quantization mechanism.

[0120] On the other hand, traditional lookup table-based vector quantization methods require maintaining a physical codebook matrix containing a large number of discrete feature vectors in memory. During network training and inference, sometimes only a small number of feature vectors are frequently retrieved and activated, while most vectors remain idle and unused, leading to low codebook resource utilization. To circumvent this physical limitation, the system incorporates a lookup table-free vector quantization mechanism.

[0121] In practice, the quantization module receives the long sequence of data passed from the preprocessing stage, eliminating the need for nearest neighbor distance search and feature matching calculations between the feature vector and the massive physical codebook table. The scalar quantizer independently performs nonlinear discrete mapping operations on each channel dimension of the feature vector. This approach eliminates the Euclidean distance calculation process in high-dimensional space, reduces the memory access load of the encoding module during runtime, and provides a data interface for establishing deterministic discrete space mappings.

[0122] Step 602: During the quantization process, the sign function is used to determine the numerical polarity of the continuous feature vectors in the semantic token sequence, so as to map the continuous feature vectors into discrete binary features.

[0123] As an optional implementation, for the semantic token sequence input to the quantization module, the system extracts the i-th feature vector in a continuous floating-point state. This feature vector itself contains N-dimensional channel feature components. During polarity conversion, an internally fixed nonlinear sign function is called as a determiner for each dimension of this feature vector.

[0124] Specifically, the determiner is configured with a fixed threshold boundary based on a zero value. When the value of the read channel feature component is less than zero, it is mapped to a set negative polarity state identifier; when the value of the channel feature component is greater than or equal to zero, it is mapped to a set positive polarity state identifier. The feature data, originally distributed in a continuous floating-point space, is truncated and restricted to a data space containing only two discrete states: positive and negative polarity. The calculation expression for the above discretization process is as follows:

[0125] z _q =-1*I(z _in <0)+1*I(z _in ≥0);

[0126] Among them, z _q Let z be the discretized binary feature vector, I be the state determination indicator function, and z be the binary feature vector. _in The input quantization module uses the continuous feature vector components, with constants -1 and 1 corresponding to the set negative and positive polarity state identifiers, respectively. After the above conversion calculation, the output is a discrete binary feature containing only the positive and negative state identifier data.

[0127] In some alternative implementations, the fixed threshold boundary for determining numerical polarity is not limited to absolute zero. Depending on the brightness distribution conditions of the image, a dynamic drift offset based on the channel mean can be introduced to shift and correct the input features before performing the above sign determination, thereby adapting to feature aggregation phenomena under different lighting conditions.

[0128] Step 603: Convert the binary feature combination into the corresponding integer set index, and use the integer set index as the quantized semantic token.

[0129] Accordingly, the discrete states scattered across various channels are encoded into globally unique codes that can be recognized by the entropy encoder, requiring a merging and compression process. The system extracts the positive and negative polarity identifiers across all N dimensions from the binary features output after polarity determination, and applies exponential shift-accumulation logic to calculate representative scalar integers. The expression for this transformation process is as follows:

[0130] Index val =Σ _j (2 (j-1) *I(z_j >0));

[0131] Among them, Index val To calculate the corresponding integer set index value, j represents the current extracted feature channel dimension index variable, Σ _j This represents the summation operator for channel variable j, accumulating it step by step from the first dimension to the Nth dimension. (j-1) The basis scaling weights assigned to the binary feature state in the j-th dimension, I(z_ j >0) is an indicator function for determining whether the binary feature component in the j-th dimension is in a positive polarity state.

[0132] Furthermore, by applying the aforementioned exponential weighted cumulative mapping operation, the multidimensional binary feature vector of length N is compressed into a single integer value through dimensionality reduction mapping. This integer value is output as a quantized semantic token for bitstream transmission. Simultaneously, the above conversion mechanism implicitly establishes a high-dimensional virtual feature codebook without occupying storage space; the physical capacity limit of this virtual codebook is constrained to a power of 2. The image compression scheme built upon this mechanism reduces the redundancy overhead of physical codebook storage and the risk of training collapse while maintaining the ability to discretize structural features.

[0133] Example 7

[0134] In one possible implementation, the quantized semantic tokens are back-extended and mapped using the source matrix to recover the initial latent representation with dense spatial resolution and containing structured noise, such as... Figure 4 As shown, it includes the following steps:

[0135] Step 701: Use the source matrix as a spatial routing index to extract mapping association features.

[0136] As an optional implementation, the decoder performs a routing parsing operation upon receiving the compressed bitstream. Specifically, it reads and parses the source matrix carried in the bitstream. The source matrix is ​​structurally represented as a sparse weighted grid or a two-dimensional mapping table, recording the flow trajectory of feature clustering and merging during the encoding stage. The decoder extracts the set of coordinates of non-zero elements from the source matrix and transforms it into a directed mapping association feature. This feature defines the addressing path for data migration from a low-dimensional, compact semantic space to a high-dimensional, dense physical space. Obtaining this addressing path is fundamental for subsequent spatial data recovery, ensuring that the feature reverse allocation process has definite spatial constraints.

[0137] Step 702: Based on the spatial mapping relationship of the mapping associated feature records, the discrete features corresponding to the quantized semantic tokens are redistributed and expanded to the feature space layout that matches the initial token sequence to construct the initial latent representation.

[0138] Accordingly, after obtaining the addressing path, the system initiates the anti-merging and reorganization calculation module to execute the feature recovery logic. The system loads a quantized semantic token of length K. Based on the mapping association features, spatial copying and reallocation operations are performed on each discrete feature vector. If the source matrix indicates that multiple high-dimensional spatial coordinates are merged into the same low-dimensional semantic token during the encoding stage, then during the decoding stage, the discrete features corresponding to the low-dimensional semantic token are synchronously copied and filled into the memory nodes corresponding to the multiple high-dimensional spatial coordinates. From the perspective of matrix operations, the integer set index corresponding to the quantized semantic token needs to be reversed and restored to a binary feature vector matrix Z of length K and dimension d. _Kq Based on this, the calculation expression for the above reverse expansion is as follows:

[0139] y _init =S _trans *Z _Kq ;

[0140] Among them, y _init To compute the generated initial latent representation, S _trans Z is the transpose matrix obtained by performing matrix transpose operation on the source matrix. _Kq This is the binary feature vector matrix obtained by restoring the integer index of the quantized semantic token.

[0141] Furthermore, after the aforementioned matrix mapping and feature allocation operations, the feature sequence in the low-dimensional state is expanded and restored into a two-dimensional feature sequence layout with dense spatial resolution. Since the data source for this reconstruction operation is compressed and polarity-truncated discrete features, the initial latent representation generated, while restoring the macroscopic spatial topology, internally contains structured noise formed by the accumulation of quantization errors. This feature distribution, possessing full-size spatial resolution and carrying structured noise features, will serve as the initial physical state for the downstream flow matching network to perform the denoising task.

[0142] In some optional implementations, to smooth out the numerical step phenomenon of neighboring pixel blocks caused by the copying and filling of discrete features, the anti-merging and reorganization calculation module can call a preset local mean operator or bilinear interpolation operator after performing the redistribution to perform spatial transition processing on the expanded feature space layout, thereby adjusting the data gradient between neighboring grid nodes.

[0143] Example 8

[0144] In one possible implementation, before extracting the initial token sequence of the original image, the process includes: performing deep downsampling processing on the original image using a pre-built downsampling encoder to obtain a deep latent feature representation; the process of extracting the initial token sequence of the original image is to extract the deep latent feature representation within its corresponding constant deep latent feature resolution space, thereby reducing the spatial redundancy of the feature sequence.

[0145] Specifically, the Diffusion Transformer network, when processing high-resolution images, often faces the technical constraint of quadratic growth in computational complexity. To control the computational load, the system employs a deep downsampling architecture. After acquiring the original image, a downsampling encoder built based on multi-layer convolutional residual blocks is invoked to perform spatial dimension compression mapping on the original pixel data. After downsampling, the image resolution scale is reduced in both the horizontal and vertical directions, typically configured to 1 / 32 of the original image size. This constructs a deep latent space with a 32x downsampling. Subsequent feature extraction, quantization aggregation, and diffusion generation computations are all confined to this constant deep latent feature resolution space. This feature space scale setting avoids the inefficient computational allocation problem caused by traditional U-Net-based methods that only operate at shallow resolutions.

[0146] According to one aspect of this application, the local spatial variance of the initial latent representation is calculated, and a spatially adaptive pseudo-time step field is generated based on the local spatial variance, such as... Figure 5 As shown, it includes:

[0147] Step 801: Calculate the local spatial variance by analyzing the statistical feature numerical distribution within a preset local spatial window of the initial latent representation, in order to measure the spatial uncertainty characteristics of the initial latent representation in different regions.

[0148] As an optional implementation, the initial latent representation output from the anti-merging and recombining network carries the structural and textural information of the image's underlying layers, along with distortion noise induced by discrete quantization operations. To quantitatively identify the degree of data distortion in different regions, local spatial variance is introduced as an evaluation metric. Specifically, a two-dimensional sliding window of a set size is used to traverse the initial latent representation. For each feature block or pixel-level node, the feature values ​​of each channel component within it are extracted, and the variance distribution parameter of this set of values ​​is statistically analyzed. This parameter objectively characterizes the internal divergence of the initial latent representation. The larger the calculated variance value, the denser the texture information in the corresponding spatial region or the more unpredictable the quantization truncation error, thus measuring the spatial uncertainty characteristics of that region.

[0149] Step 802: Project the local spatial variance onto the target latent feature dimension, and convert the projected variance data into a pixel-level or feature block-level spatial-temporal step feature map through a preset monotonic function to obtain a spatially adaptive pseudo-temporal step field.

[0150] Accordingly, since the extracted local spatial variance is independent of the channel dimension of the feature map itself in terms of dimensionality, spatial dimensionality reduction and mapping alignment operations are required. The system calls a differentiable feature projection function to linearly project the original variance data, making its dimensionality conform to the input settings of the subsequent flow matching network. Furthermore, a nonlinear monotonically increasing function based on exponential distribution parameters is applied to convert the projected and aligned variance values ​​into time step parameters representing the denoising iteration process. The calculation expression of the above mapping mechanism is as follows:

[0151] t _field =F _mono (P _θ (σ _local ));

[0152] Among them, t _field To calculate the obtained spatial-temporal step feature map, F _mono P is a pre-defined monotonically increasing function. _θ For feature projection operations with trainable parameters, σ _local Let t be the local spatial variance of the input. Based on the monotonically increasing mathematical correlation property, high-frequency detail regions of the image with higher variance will be mapped and assigned a larger time step parameter, and the final output t will be... _field This is the spatially adaptive pseudo-time step field corresponding to the two-dimensional grid layout.

[0153] Preset monotonic function F _mono It can be in the following form:

[0154] t = sigmoid(β*x + b);

[0155] Where β is a trainable scaling parameter, b is a bias parameter, and x is the variance after projection. Differentiable feature projection function P _θ It consists of a single-layer linear transformation that projects the dimension of the local variance from a scalar to a vector space that matches the dimension of the feature channels. Those skilled in the art can use conventional model training methods in the field to perform end-to-end optimization of the above-mentioned trainable parameters.

[0156] According to one aspect of this application, stream matching denoising is performed on the initial latent representation based on a spatially adaptive pseudo-time step field, including:

[0157] Optionally, the spatially adaptive pseudo-time step field and the initial latent representation are spatially aligned and concatenated in a diffuse Transformer network to obtain composite input features.

[0158] Specifically, after obtaining the index map representing spatial uncertainty, it needs to be injected into the denoising control loop. The system extracts a spatially adaptive pseudo-time step field and aligns it pixel-by-pixel with the initial latent representation carrying structured noise in the spatial coordinate dimension. After alignment, a splicing operator is executed in the channel dimension of the tensor data. The spliced ​​composite feature data is uniformly transmitted to the input of the diffuse Transformer network, so that the model can read the time step state corresponding to each pixel node in each round of calculation for subsequent spatial feature updates.

[0159] Optionally, based on the numerical distribution of the spatially adaptive pseudo-time step field in the composite input features, differentiated denoising intensity features are assigned to different spatial locations of the initial latent representation.

[0160] During the process of feature propagation layer by layer in the network, the bias gain distribution of the normalization module in the adaptive layer is adjusted using the local time step parameter introduced by feature concatenation. Since the values ​​in the pseudo-time step field of spatial adaptation are non-uniformly discrete, the computational gain of the network at different physical coordinate points is differentially allocated.

[0161] Specifically, for high-frequency texture damage areas with larger time step values, the network will activate a higher proportion of nonlinear channel features, thereby giving the area a stronger denoising intensity feature; while for smooth background areas with smaller time step values, the network will converge the activation threshold of the feature channels, giving a relatively weaker denoising intensity feature, thus avoiding excessive smearing of smooth areas caused by a globally uniform denoising mode.

[0162] Optionally, based on the differentiated denoising intensity features, the initial latent representation is denoised by performing a single-step or fewer-than-preset-maximum-step forward prediction operation of the flow matching network to obtain the denoised latent features.

[0163] Furthermore, due to the adoption of spatially adaptive conditional input, the model can perform deterministic inference with extreme acceleration based on the Eulerian flow matching framework. The network outputs residual displacement predictions with the same dimension as the input, and combines them with the time-step state to calculate the final sharp features. The single-step flow matching operation expression is as follows:

[0164] y _target =y _init -V _net (y _init ,t _field );

[0165] Among them, y _target The clear latent features output after noise reduction; y _init The initial latent representation of the input carrying structured noise; t _field For spatially adaptive pseudo-time step field; V _net The forward residual prediction operation performed by the Diffusion Transformer network based on the flow matching framework outputs a displacement estimate from the noisy input to the clean feature. Specifically, the network output of this displacement estimate implicitly includes a scaling factor for the pseudo-time step field. Through this computational logic, the system performs spatial non-uniformity correction and denoising on the initial latent representation in a single forward pass or in forward operations with fewer than three iterations.

[0166] In some alternative implementations, the accelerated reconstruction strategy of performing single-step flow matching can be replaced by a second-order Runge-Kutta solver, which can obtain a smoother solution trajectory and perform convergence correction on complex multimodal edges, provided that computing power allows.

[0167] Example 9

[0168] In one possible implementation, latent conditions are generated based on quantized semantic tokens and injected into the diffuse Transformer network as guidance, such as... Figure 6 As shown, it includes the following steps:

[0169] Step 901: Extract the latent feature projection vector of the derived self-quantized semantic token and construct the latent conditions.

[0170] As an optional implementation, conventional large-capacity diffusion generative networks typically invoke a pre-deployed visual language model during the inference phase to extract textual prompts from external input and generate conditional guidance sequences. Image compression terminals are often deployed in edge devices with limited storage capacity and computing power; invoking a visual language model with a large number of parameters increases memory usage and latency overhead per inference cycle. To reduce this dependence on external data and the operating environment, the system extracts quantized semantic tokens arriving at the decoder's memory and uses them as the sole data source for endogenous conditional generation. Specifically, the discrete integer index sequence contained in the quantized semantic tokens is read. By retrieving a lightweight one-dimensional mapping matrix pre-installed within the decoder, the discrete integer index sequence is converted into a feature embedding vector with a continuous value range.

[0171] Furthermore, a latent conditional projection operator configured using a multilayer perceptron or linear feedforward network is invoked to perform channel-dimensional upscaling and alignment calculations on the feature embedding vector. After the aforementioned linear projection transformation operation, the system outputs a latent feature projection vector, which matches the feature dimension scale of the subsequent diffusion Transformer network's cross-attention layer. This multidimensional vector set carries the high-level image semantic attributes that have been clustered and truncated at the encoding end, and the system outputs and defines it as latent conditional.

[0172] In some alternative implementations, the latent conditional projection operator can be replaced by a one-dimensional convolutional block configured with residual bypasses. By configuring the convolutional kernel width, correlation compensation computation between locally adjacent discrete tokens can be introduced during the feature projection dimensionality upscaling stage.

[0173] Step 902: Replace the text cue guidance branch in the diffusion Transformer network with latent conditions to perform text-free conditional control during the image reasoning and reconstruction phase.

[0174] Accordingly, after obtaining the internally derived feature vectors, they need to be connected to the diffusion denoising control loop to replace the language prior. In the structure of the diffusion Transformer network, the cross-attention mechanism layer reserves input data ports for receiving text prompt feature embeddings. In the current stage, the control chip cuts off and shields the original text prompt extraction and guidance branch data paths in the network structure, and allocates and routes the generated potential conditions to all cross-attention mechanism layers in the network system, as well as the guidance signal input of the adaptive layer normalization module.

[0175] Furthermore, in the actual image inference and reconstruction stage, the adaptive layer normalization module reads the latent conditions, performs linear regression calculations to obtain the scale and translation parameters required for the affine transformation, and uses these parameters to perform distribution modulation operations on the latent features transmitted in the backbone network. Simultaneously, the cross-attention mechanism layer calculates the corresponding key and value matrices from the received latent conditions using a linear projection matrix, and then performs multi-head attention feature interaction operations with the query matrix from the backbone feature extraction path. Through the spatial reconstruction of the above data flow path, the model does not need to receive externally input descriptive text or trigger the loading of the language model during the computation cycle of generative inference. It only relies on the semantic feature data derived from the internal mapping of the compressed bitstream to complete the conditional control of image reconstruction quality.

[0176] In some optional implementations, if an external control system issues a reference cue vector for secondary image editing, a floating-point weight allocation parameter can be configured within the network. Based on this parameter, a weighted fusion feature vector of the latent conditions and the reference cue vector is calculated, and this weighted fusion feature vector is injected into the cross-attention mechanism layer as an auxiliary guiding term.

[0177] Example 10

[0178] In one possible implementation, the diffuse Transformer network is pre-trained based on an image training sample set and obtained through end-to-end joint optimization using a two-stage implicit bitrate pruning strategy.

[0179] Specifically, the image training sample set can be a publicly available large-scale natural image dataset (such as the ImageNet training set or other high-quality natural image sets of equivalent size). Training images are preprocessed by randomly cropping them to a uniform resolution before being input into the network. Deep generative large models face problems of feature space collapse and gradient divergence under low bit rate constraints when performing end-to-end image compression training.

[0180] Accordingly, to address the parameter oscillations encountered by the aforementioned network during convergence, an offline training server is configured to perform model training based on a two-stage implicit bitrate pruning strategy. This strategy establishes a closed-loop tuning feedback mechanism for parameter optimization. By dynamically adjusting the weights of various penalty coefficients in the training loss function, the original single optimization objective is decoupled into two independent optimization cycles: feature representation information construction and perceptual quality improvement. This achieves a balanced tuning between image reconstruction distortion and compressed bitstream length in the deep latent feature space.

[0181] According to one aspect of this application, an end-to-end joint optimization pre-training is performed using a two-stage implicit bit rate pruning strategy, such as... Figure 7 As shown, it includes:

[0182] Step 1001: In the first training phase, a smaller pre-configured rate-distortion tradeoff weight is used to relax the entropy constraint of the feature representation. Based on the image training sample set, the initial network is trained to preserve basic features, and a basic training network model is obtained.

[0183] Specifically, in the first training phase, the system loads a preset image training sample set as the input signal. To prevent the network from losing high-frequency texture information due to bit rate penalties in the early stages of training, the training algorithm is configured with pre-configured rate-distortion tradeoff weights within a small numerical range. In this embodiment, the pre-configured rate-distortion tradeoff weight is set to 0.2 in the first phase. Under this weight configuration, the network reduces the constraint on the entropy capacity of the latent variable space, allowing a large number of intermediate feature vectors to be retained through forward propagation. When iteratively updating the network parameters using the Adam optimizer, the basic loss function for the first phase is calculated, and the expression for the calculation of the basic loss function for the first phase is as follows:

[0184] Loss _stage1 =λ _base *R _loss +D _loss +L _align ;

[0185] Among them, Loss _stage1 The calculation result of the basic loss function output in the first stage, λ _base For the pre-configured rate-distortion tradeoff weights, R _loss To calculate the obtained bit rate loss value, D _loss To measure the acquired overall distortion error value, L _align This is used to calculate the obtained contrastive co-alignment loss value. Where R... _loss Specifically, the probability distribution of discrete tokens is obtained by calling the existing cross-entropy calculation operator to evaluate them; its standard operation mechanism will not be explained in detail here.

[0186] Step 1002: In the second training phase, based on the basic training network model, a larger pre-configured rate-distortion tradeoff weight is used to gradually tighten the bit rate constraint, and a pre-configured adversarial target is introduced to jointly calculate the adversarial loss. The network model is fine-tuned by using a weighted combination of adversarial loss and rate-distortion loss to improve the perceptual quality, resulting in a diffusion Transformer network.

[0187] Furthermore, after completing the basic feature-preserving training, the network inherits the model parameters output from the first training stage and then enters the second training stage. In this stage, the system adjusts the control variables, configuring the pre-configured rate-distortion tradeoff weights to a larger numerical range. In this embodiment, this larger weight is set to 4.0 in the second stage. By increasing the tradeoff weights, the model will increase the penalty for bit rate loss during loss calculation backpropagation, performing implicit bit rate pruning and forcing the network to remove redundant token indices that contribute little to visual reconstruction. Simultaneously, the system attaches a pre-configured adversarial objective module composed of a multi-scale discriminator network to the training loop. The discriminator network receives the output reconstructed image and real image samples, calculating the classification score. The generator network calculates and updates the gradient with the optimization objective of deceiving the discriminator network, thereby constructing the adversarial loss. The calculation expression for the second-stage fine-tuning loss function is as follows:

[0188] Loss _stage2 =λ _target *R _loss +D _loss +L _align +λ _adv *L _adv ;

[0189] Among them, Loss _stage2 The calculation result of the loss function output in the second stage is λ. _target To compensate for the larger pre-configuration rate distortion after adjustment, λ _adv For the pre-configured adversarial loss weight hyperparameters, L _adv This is to calculate the obtained adversarial loss value.

[0190] During the two-stage training process described above, those skilled in the art can use conventional model training methods in the field to optimize the parameters of the network model. Specifically, this includes using the Adam optimizer or the AdamW optimizer to perform gradient descent. The hyperparameters (such as learning rate, batch size, and number of iterations) during the training process can be determined by conventional hyperparameter tuning methods.

[0191] According to one aspect of this application, the fine-tuning training of the network model to improve perceptual quality further includes:

[0192] Optionally, a joint distortion loss metric model is constructed, which includes a pixel-level mean square error calculator and a perceptual similarity distance calculator based on a pre-trained feature extraction network. The joint distortion loss metric model is used to evaluate the comprehensive distortion difference between the reconstructed image output by the network model performing forward inference on the image training samples and the real image corresponding to the image training samples, and the comprehensive distortion difference is used as a penalty constraint for fine-tuning the training.

[0193] Furthermore, to obtain the aforementioned comprehensive distortion error value, an error evaluation benchmark covering different observation dimensions needs to be established. The system instantiates a joint distortion loss measurement model in memory. The pixel-level mean square error calculator within this model independently calculates the pixel color difference dispersion between the output image and the real image at corresponding coordinate points using existing mean square error algorithms. Simultaneously, the perceptual similarity distance calculator calls a perceptual model based on a fixed-parameter network structure to extract feature map mapping sequences of two images across multiple hidden channels. It then outputs perceptual similarity indices and depth image structure texture similarity indices by calculating the cosine distance or normalized Euclidean distance between feature tensors. The pixel-level error and feature-level distance are linearly summed by their respective pre-configured weight coefficients to output the comprehensive distortion difference. Through multi-dimensional error penalty feedback, the model is guided to reconstruct texture edges that conform to the visual sampling distribution under limited bitrate conditions.

[0194] In some alternative implementations, when the computing resources of the offline training server are limited or when it is necessary to increase the throughput of a single iteration cycle, the system can implement a degradation metric scheme. Specifically, this involves disabling or disabling the perceptual similarity distance calculator based on the pre-trained feature extraction network in the joint distortion loss metric model, and retaining only the output of the pixel-level mean square error calculator as the sole penalty metric.

[0195] According to one aspect of this application, the pre-training process also includes co-alignment constraints for latent conditions:

[0196] Optionally, a preset text feature embedding that matches the image training samples is extracted; a contrastive co-alignment loss is calculated based on the latent conditions obtained by projection and the preset text feature embedding; the contrastive co-alignment loss is accumulated into the total training loss, and the contrastive co-alignment loss contained in the total training loss is used to constrain the model parameters so as to align the semantic consistency of the latent conditions and the text-guided features during the pre-training stage.

[0197] Accordingly, since the external text prompt features are removed in the subsequent online inference stage, an equivalent mapping relationship between the two needs to be established during the offline training stage. In the training sequence, the system equips each input image training sample with a pre-defined text feature embedding extracted by an external language processing model. During network forward propagation, latent conditions generated by the latent projection operator are extracted. The system employs a contrastive learning mechanism to calculate the spatial distance between the latent conditions and the paired pre-defined text feature embeddings. Simultaneously, text features corresponding to other image samples within the batch are selected as the negative sample pair set. The calculation expression for this contrastive co-alignment loss is as follows:

[0198] L _align =-1*log(exp((C _lat *C _text_pos) / τ) / Σ _j (exp((C _lat *C _text_j ) / τ)));

[0199] Among them, L _align To calculate the obtained contrastive co-alignment loss, C _lat C is the latent conditional vector obtained by projection. _text_pos For embedding positively preset text features that match the current sample, C _text_j Let τ be the embedded feature of the j-th contrastive text stored in the batch memory, and Σ be a preset temperature constant used to scale the dot product result of the feature vectors. _j This is an indicator for the summation operation performed on all candidate text feature sets in the current batch, where log is the scalar of the natural logarithm operation and exp is the scalar of the exponential mapping operation.

[0200] According to the above loss formula, the larger the dot product score between the latent condition and the positively matched text embedding, and the smaller the dot product score with the unmatched negative sample, the lower the calculated error penalty value. This alignment error term is added to the network's backpropagation path to constrain the iterative update direction of the guiding parameter matrix, thereby allowing the latent features derived from the compressed representation to take over the control of cross-modal features within the generation network.

[0201] This application employs a spatially adaptive pseudo-time step field based on local spatial variance mapping, combined with text-independent latent conditional projection, to guide the flow matching network to perform single-step or few-step forward prediction in the deep latent space. This solves the decoding latency problem caused by continuous probability estimation and shallow diffusion architecture, reduces multi-step iteration and the computational stacking of large feature maps, and keeps the computational latency of generating reconstruction within the range that meets the needs of interactive communication.

[0202] Meanwhile, to address the issues of codebook idleness and limited representation capacity caused by traditional discrete quantization, a lookup-free vector quantization mechanism based on sign functions is adopted. By performing polarity determination and exponential shift combination on feature vectors, the continuous space is mapped to a set of virtual integer indices, reducing the maintenance cost of large-scale physical codebooks and ensuring the basic representation dimension under ultra-low code rates while improving the utilization rate of discrete features.

[0203] Furthermore, an attention-based decoupled coding and high-dimensional routing reverse reconstruction network was constructed. By separating the source matrix that records the clustering trajectories, and using it as a routing index at the decoding end to perform spatial expansion and feature redistribution, discrete subject semantics are backfilled into a dense resolution layout with high fidelity without increasing additional bandwidth load, preserving the underlying structural topology of the image.

[0204] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0205] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0206] It should be noted that the various specific technical features described in the above embodiments can be combined in any suitable manner without contradiction. To avoid unnecessary repetition, the present invention will not describe the various possible combinations separately.

Claims

1. An image compression method based on vector quantization and aligned diffusion Transformer, characterized in that, include: Obtain the original image to be compressed and extract the initial token sequence of the original image; The initial token sequence is aggregated and separated into a semantic token sequence carrying high-level semantics and a source matrix that preserves the spatial mapping relationship. Discrete quantization is performed on the semantic token sequence to obtain quantized semantic tokens; By using the source matrix to perform a reverse expansion mapping on the quantized semantic token, an initial latent representation with dense spatial resolution and containing structured noise is recovered. Calculate the local spatial variance of the initial latent representation and generate a spatially adaptive pseudo-time step field based on the local spatial variance; Generate potential conditions based on quantized semantic tokens; Latent conditions are injected into the diffuse Transformer network as guidance, and flow matching denoising is performed on the initial latent representation based on a spatially adaptive pseudo-time step field to output the reconstructed target image.

2. The method according to claim 1, characterized in that, The initial token sequence is aggregated and separated into a semantic token sequence carrying high-level semantics and a source matrix that preserves the spatial mapping relationship, including: The feature similarity between tokens in the initial token sequence is calculated based on the attention mechanism. The initial token sequence is clustered and merged based on feature similarity to obtain a semantic token sequence with decoupled location information, and clustering routing information is generated during the clustering and merging process. Record the clustering routing information into the source matrix.

3. The method according to claim 1, characterized in that, Discrete quantization is performed on the semantic token sequence to obtain quantized semantic tokens, including: Semantic token sequences are processed using a lookup-free vector quantization mechanism; During the quantization process, the sign function is used to determine the numerical polarity of continuous feature vectors in the semantic token sequence, so as to map the continuous feature vectors into discrete binary features. The binary feature combination is converted into the corresponding integer set index, and the integer set index is used as the quantized semantic token.

4. The method according to claim 1, characterized in that, By performing a reverse extension mapping on the quantized semantic token using the source matrix, an initial latent representation with dense spatial resolution and containing structured noise is recovered, including: The source matrix is ​​used as a spatial routing index to extract mapping association features; Based on the spatial mapping relationship of the mapping association feature records, the discrete features corresponding to the quantized semantic tokens are redistributed and expanded to the feature space layout that matches the initial token sequence, thus constructing the initial latent representation.

5. The method according to claim 1, characterized in that, Calculate the local spatial variance of the initial latent representation and generate a spatially adaptive pseudo-time step field based on the local spatial variance, including: Evaluate local spatial variance to measure the spatial uncertainty of the initial latent representation in different regions; The local spatial variance is projected onto the target latent feature dimension, and the projected variance data is converted into a pixel-level or feature block-level spatial-temporal step feature map through a preset monotonic function, resulting in a spatially adaptive pseudo-temporal step field.

6. The method according to claim 1, characterized in that, Potential conditions are generated based on quantized semantic tokens and injected into the diffused Transformer network as bootstrapping, including: Extract the latent feature projection vector of the derived self-quantized semantic token and construct the latent conditions; By replacing the text cue guidance branch in the diffusion Transformer network with latent conditions, text-free conditional control can be performed during the image reasoning and reconstruction phase.

7. The method according to claim 1, characterized in that, A spatially adaptive pseudo-time step field is used to perform flow matching denoising on the initial latent representation, including: Spatially adaptive pseudo-time step field and initial latent representation are spatially aligned and feature-stitched in a diffuse Transformer network; Based on the numerical distribution of the spatially adaptive pseudo-time step field, differentiated denoising intensity features are assigned to different spatial locations of the initial latent representation. Based on the differentiated denoising intensity features, the initial latent representation is denoised by performing a forward prediction operation of the flow matching network in one step or less than a preset upper limit of steps.

8. The method according to claim 1, characterized in that, The Diffusion Transformer network is obtained through end-to-end joint optimization pre-training using a two-stage implicit bit rate pruning strategy.

9. The method according to claim 8, characterized in that, The pre-training process also includes a co-alignment constraint step for latent conditions: Extract pre-defined text feature embeddings that match the image training samples; Based on the latent conditions obtained from projection and the preset text feature embeddings, the contrastive co-alignment loss is calculated. We utilize contrastive co-alignment loss to constrain model parameters in order to align the semantic consistency of latent conditions with text-guided features during the pre-training phase.

10. The method according to claim 1, characterized in that, Before extracting the initial token sequence from the original image, the following is also included: Deep downsampling is performed on the original image using a pre-built downsampling encoder to obtain deep latent feature representations; The process of extracting the initial token sequence from the original image involves extracting the deep latent feature representation within its corresponding constant deep latent feature resolution space, thereby reducing the spatial redundancy of the feature sequence.

Images