Image translation and generation method based on spatio-temporal alignment discrete diffusion bridge

Abstract

The application provides an image translation and generation method based on a space-time alignment discrete diffusion bridge, and relates to the technical fields of computer vision and image generation. The method first constructs a discrete label representation of a source image and a target image, obtains a discrete label sequence corresponding to the target image and the source image, and constructs a conditional input representation corresponding to the source image; then constructs a discrete bridge type forward diffusion process based on mixed absorption and information guide mask scheduling, mixes the discrete label corresponding to the target image with a mask label, and determines the priority of the discrete label according to the information density; further constructs a discrete diffusion bridge reverse process, decodes step by step based on the prediction confidence according to the sampling distribution of the target image, and resamples the un-decoded area from the mixed absorption state to obtain a generated image label sequence; finally, the bridge type discrete diffusion model is trained by using a cross entropy loss acting only on the absorbed area.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and image generation technology, and in particular to an image translation and generation method based on a spatiotemporally aligned discrete diffusion bridge. Background Technology

[0002] Image translation and generation are core research directions at the intersection of computer vision and artificial intelligence, and key supporting technologies for scenarios such as digital image processing, multimedia content creation, and intelligent visual interaction. Image translation aims to achieve accurate mapping and conversion between images of different visual domains, styles, and attributes, such as colorization of black and white images, semantic conversion of aerial images, artistic style transfer, and image dehazing and enhancement. Image generation, on the other hand, focuses on building entirely new images from scratch that conform to visual logic, are detailed, and have a consistent style, based on existing data, text instructions, or reference samples.

[0003] In traditional image processing, image translation and generation largely rely on conventional methods such as manually designed features, parameter optimization, template matching, and texture mapping. These methods require manual extraction of shallow features such as edges, textures, and colors from images, and then use preset rules and mathematical transformations to complete image conversion and generation. They are only suitable for simple, single visual tasks, and their processing scenarios are relatively limited. When faced with complex natural scenes, multi-semantic targets, and high-resolution images, traditional methods struggle to capture the deep semantic information and structural features of images. The generated and translated images generally suffer from problems such as missing details, blurred edges, stylistic distortion, and semantic misalignment. Furthermore, they have poor generalization ability and cannot adapt to diverse and personalized visual needs, thus limiting their application scope.

[0004] Discrete diffusion models are a class of generative models that perform progressive masking and reconstruction in a discrete latent space. The standard discrete diffusion model progressively absorbs the target image labels into a pure mask state through a forward process, and then progressively reconstructs the original image content from the pure mask state through a backward process. Compared to autoregressive generative models, discrete diffusion models possess parallel label prediction capabilities; compared to continuous diffusion models, they are more suitable for modeling in a unified discrete visual label space.

[0005] However, when the standard discrete diffusion model is directly applied to image translation and generation tasks, a significant spatiotemporal mismatch exists. Spatially, the standard forward process gradually absorbs the target image into a pure mask state, completely discarding the structural priors in the source image. This makes the model more akin to regenerating the target content from semantic blanks during backward inference, rather than transforming it based on the source image, thus hindering the maintenance of structural and identity consistency. Temporally, the training phase uses random, unordered masks, while the inference phase typically employs a confidence-based progressive decoding strategy, first generating simple regions and then complex regions. The inconsistency between the training and inference trajectories makes it difficult for the model to effectively learn the transformation logic of complex regions.

[0006] Therefore, a novel discrete diffusion image translation and generation method is needed to address the issues of not being able to retain prior spatial structure and inconsistencies between training and inference time trajectories, while maintaining the advantages of parallel decoding in discrete diffusion models. Summary of the Invention

[0007] The technical problem to be solved by the present invention is to address the shortcomings of the prior art by providing an image translation and generation method based on a spatiotemporally aligned discrete diffusion bridge, thereby realizing image translation and generation.

[0008] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0009] On one hand, this invention provides an image translation and generation method based on a spatiotemporally aligned discrete diffusion bridge. Based on the standard discrete diffusion model, it constructs a bridge-type forward diffusion process based on hybrid absorption and information-guided mask scheduling, as well as a discrete diffusion bridge reverse process, to obtain a bridge-type discrete diffusion model, thereby realizing image translation and generation. The method includes the following steps:

[0010] Step S1: Construct discrete label representations of the source image and the corresponding target image, obtain discrete label sequences corresponding to the target image and the source image, and construct the conditional input representation corresponding to the source image;

[0011] Step S2: Construct a discrete bridge forward diffusion process based on hybrid absorption and information-guided mask scheduling, mix the discrete markers corresponding to the target image with the mask markers, and determine the priority of the discrete markers according to the information density;

[0012] Step S3: Construct a discrete diffusion bridge reverse process, based on the sampling distribution of the target image, decode step by step according to the prediction confidence, and resample the undecoded region from the mixed absorption state to obtain the generated image label sequence;

[0013] Step S4: Train the bridged discrete diffusion model using cross-entropy loss that only applies to the absorbed region to achieve image editing, style transfer, image restoration, modality conversion, or text-to-image generation.

[0014] Furthermore, the specific method of step S1 is as follows:

[0015] Step S11: Construct a discrete label representation of the target image to obtain the discrete label sequence corresponding to the target image;

[0016] First, the target image is input into an image discrete tagger for encoding to obtain the discrete visual tag sequence corresponding to the target image. ;

[0017] Step S12: Construct a discrete label representation of the source image to obtain the discrete label sequence corresponding to the source image;

[0018] For tasks that have a source image as input, the source image is also input into the image discrete labeler to obtain the discrete label sequence corresponding to the source image. ;

[0019] Step S13: Construct the conditional input representation corresponding to the source image;

[0020] For image translation and generation tasks controlled by text or task instructions, text prompts, editing instructions, style descriptions, or task category information are encoded as conditional input representations. ; This condition input This represents the conditional control information used in the bridge-type discrete diffusion model.

[0021] Furthermore, the specific method of step S2 is as follows:

[0022] Step S21: Construct the absorption form of the standard discrete diffusion of the target image; Under the absorption form of the standard discrete diffusion of the target image, the discrete markers corresponding to the target image are gradually absorbed into the pure mask state during the forward process;

[0023] Step S22: Construct the mixed absorption state of the target image;

[0024] Based on the absorption form of the standard discrete diffusion of the target image, a mixed absorption state of the target image is constructed, which takes the following form:

[0025] ;

[0026] in, Represents the target image. The mixed absorption state at each position, Represents the target image. The source image markers corresponding to each location, Indicates the proportion of source image markers injected. Indicates a mask marker;

[0027] Step S23: Based on the mixed absorption state of the target image, construct a bridge-type forward transfer process for the target image;

[0028] After constructing the mixed absorption state of the target image, the target image is then processed. The bridge forward transfer at each position is defined as:

[0029] ;

[0030] in, Represents the conditional probability distribution. Indicates time Next target image The marking status of each position, Represents the target image. The original target markers at each location, Represents the category distribution of the target image labels; For a moment Next A binary mask indicating a variable at each position; when When, the original marker of the target image is preserved at this position; when At that time, this position is absorbed into a mixed absorption state. ;

[0031] Step S24: Construct a pixel-level information density map of the target image;

[0032] Step S25: Based on the pixel-level information density map of the target image, construct an information-guided mask scheduling mechanism to determine the final sampling distribution of the target image, and then obtain the mask sequence;

[0033] Furthermore, the specific method for constructing the pixel-level information density map of the target image in step S24 is as follows:

[0034] For image translation tasks with a source image, a pixel-level information density map of the target image is constructed based on the differences between the target image and the source image:

[0035] ;

[0036] in, This is a pixel-level information density map of the target image. Represents the target image. One channel, Indicates the source image number 1 One channel, Indicates the number of channels in the target image and the source image;

[0037] For the task of generating text-to-target image from a passive image, the local variance of the target image is used to describe the regional structural complexity. The pixel-level information density map of the target image is as follows:

[0038] ;

[0039] in, Represents the grayscale image of the target image. This represents the local average pooling operator, where k represents the pooling factor of the target image.

[0040] Furthermore, the specific method of step S25 is as follows:

[0041] After downsampling the pixel-level information density map M to the discrete marker resolution, the marker-level information content map is obtained. Then, the information content map of the label level. The probability distribution is normalized to an information-guided probability distribution and then weighted and fused with a standard uniform distribution to obtain the final sampling probability distribution of the target image.

[0042] ;

[0043] in, Represents the target image. The sampling probability at each location, Represents the target image. The probability of information guidance at each location. Represents a uniform distribution probability. Indicates the randomness control parameter;

[0044] Based on the sampling probability distribution of the target image, at time... Select the marker positions to be absorbed to construct the time. mask sequence .

[0045] Further, the specific method of step S3 is as follows: Step S31: Based on the bridge forward diffusion process, construct the initial state of the bridge reverse process to generate the image;

[0046] The endpoint state of the target image in the bridge-type forward diffusion process is used as the initial state of the generated image; for the generated image, the [missing information - likely a specific image or process]... The initial state of the markers at each position. Obtained by sampling the mixed absorption state of the target image:

[0047] ;

[0048] Step S32: Construct the predicted probability conditional distribution of discrete image labels generated by the bridge-type reverse process;

[0049] At each reverse time step, based on the current state of the generated image The discrete label sequence y corresponding to the source image and the conditional input representation Predictive conditional probability distribution of discrete image labels ,in, Indicates the parameters of the discrete diffusion model;

[0050] Step S33: Calculate candidate labels and prediction confidence;

[0051] For the currently undecoded number At each of the given locations, the candidate label with the highest probability is selected from the predicted conditional probability distribution of the discrete labels in the generated image. :

[0052] ;

[0053] Simultaneously calculate candidate labels Corresponding prediction confidence level :

[0054] ;

[0055] in, Indicates candidate tokens The corresponding prediction confidence level;

[0056] Step S34: Construct a bridge-based reverse transfer based on prediction confidence;

[0057] Based on the predicted confidence at each position, positions with a predicted confidence greater than or equal to a set threshold are directly decoded at the current time, while positions with a predicted confidence less than the set threshold continue to retain the bridge absorption state.

[0058] The reverse transition form of the bridge-type discrete diffusion model is:

[0059] ;

[0060] in, This represents a bridge-shaped marker obtained by resampling from a mixed absorption state;

[0061] Step S35: Perform multi-step bridge-type reverse generation to obtain the final output generated image;

[0062] Repeat steps S32 to S34 until all positions are decoded to obtain the discrete label sequence of the final generated image; then, input the discrete label sequence of the generated image into the image decoder to recover the final output generated image.

[0063] Furthermore, the specific method of step 4 is as follows:

[0064] Step S41: Construct training objectives for the absorbed regions and determine the loss function;

[0065] During the training of the bridge-type discrete diffusion model, the supervised loss is calculated only for the position of absorption at the current time, and its loss function is... for:

[0066] ;

[0067] in, Represents the expected value of conditional probability, when When , it means that the i-th position has been absorbed and needs to be included in the loss calculation; when The time interval indicates that the i-th position still retains the original target label and is not included in the loss calculation at the current time.

[0068] Step S42: Perform bridged discrete diffusion model training;

[0069] During training, the target image and source image are first encoded into discrete label sequences, and then a bridge-type forward diffusion state is constructed according to step S2. Then Discrete label sequence corresponding to the source image and conditional input The data are input into the discrete diffusion model to predict the discrete label sequence of the generated image, and the parameters of the discrete diffusion model are updated according to the loss function described in step S41.

[0070] Step S43: Use the trained bridge-type discrete diffusion model to complete image translation and generation;

[0071] After the input source image and condition information are encoded in step S1, a bridge-type initial state is constructed, and then the bridge-type reverse generation process described in step S3 is executed to finally output the generated image.

[0072] On the other hand, this application proposes a computer-readable storage medium storing executable instructions that, when executed, cause a processor to perform the image translation and generation method based on a spatiotemporally aligned discrete diffusion bridge.

[0073] Thirdly, this application proposes a computer program product, including a computer program or instructions that, when executed by a processor, implement the image translation and generation method based on a spatiotemporally aligned discrete diffusion bridge.

[0074] The beneficial effects of adopting the above technical solution are as follows: The image translation and generation method based on spatiotemporally aligned discrete diffusion bridge provided by this invention establishes a spatiotemporally consistent bridge-type generation trajectory within the framework of a discrete diffusion model. On the one hand, it utilizes a hybrid absorption mechanism to explicitly preserve the prior structure of the source image in the diffusion latent space, improving the structure preservation capability in image translation tasks. On the other hand, it uses an information-guided mask scheduling mechanism to coordinate the absorption order during training with the decoding order during inference, enhancing the discrete diffusion model's ability to model semantic changes in complex regions. Furthermore, during the reverse process, state resampling through hybrid absorption ensures that undecoded regions are always constrained by the source image anchor points, thereby facilitating the acquisition of high-quality image output results under conditions of fewer sampling steps. Attached Figure Description

[0075] Figure 1 This is a schematic diagram of the structure of the image translation and generation method based on spatiotemporally aligned discrete diffusion bridge provided in an embodiment of the present invention;

[0076] Figure 2 This is a schematic diagram of the amount of information calculated by the discrete diffusion bridge for four source domain graphs according to an embodiment of the present invention, wherein (a) is for the first source domain graph, (b) is for the second source domain graph, (c) is for the third source domain graph, and (d) is for the fourth source domain graph;

[0077] Figure 3 This is a schematic diagram of the test results of the bridge-type discrete diffusion model provided in the embodiment of the present invention on the modal translation task, wherein (a) is the input image under the condition of fog + snow, and (b) is the input image under the condition of low light + fog + rain;

[0078] Figure 4 This diagram illustrates the test results of the bridge-type discrete diffusion model provided in this embodiment of the invention on four different input images in an image restoration task. Detailed Implementation

[0079] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0080] Combination Figures 1-4 This embodiment provides an image translation and generation method based on a spatiotemporally aligned discrete diffusion bridge. Based on the standard discrete diffusion model, it constructs a bridge-type forward diffusion process based on hybrid absorption and information-guided mask scheduling, as well as a discrete diffusion bridge reverse process, to obtain a bridge-type discrete diffusion model and realize image translation and generation. Specifically, it includes the following steps:

[0081] Step S1: Construct discrete label representations of the source image and the corresponding target image, obtain discrete label sequences corresponding to the target image and the source image, and construct the conditional input representation corresponding to the source image;

[0082] Step S11: Construct a discrete label representation of the target image to obtain the discrete label sequence corresponding to the target image;

[0083] First, the target image is input into an image discrete tagger for encoding to obtain the discrete visual tag sequence corresponding to the target image. ;in, This represents the sequence of markers for the target image on a discrete visual vocabulary, with each position corresponding to a discrete visual marker. In this way, a continuous image space can be mapped to a unified discrete marker space, facilitating subsequent modeling within a discrete diffusion framework. In this embodiment, the image discrete marker is an image word segmenter.

[0084] Step S12: Construct a discrete label representation of the source image to obtain the discrete label sequence corresponding to the source image;

[0085] For tasks involving source image input, such as image editing, style transfer, image restoration, modality transformation, and super-resolution, the source image is also input into the image discrete labeler to obtain the discrete label sequence corresponding to the source image. ;in, This represents the sequence of labels for the source image on a discrete visual vocabulary. This source image label sequence will be used as a spatial anchor in subsequent mixed absorption states to preserve the structural priors of the input image during the generation process.

[0086] Step S13: Construct the conditional input representation corresponding to the source image;

[0087] For image translation and generation tasks controlled by text or task instructions, text prompts, editing instructions, style descriptions, or task category information are encoded as conditional input representations. ; where the condition input This represents the conditional control information used in the bridge-type discrete diffusion model. In image editing tasks, This can be used to edit text descriptions; in style transfer tasks, It can be a target style description; in text-to-image generation tasks... This can be a text prompt. When the task does not depend on explicit text conditions, It can also be empty. By introducing conditional input representation, the method of this invention can be uniformly applied to various image translation and generation scenarios.

[0088] Step S2: Construct a discrete bridge forward diffusion process based on hybrid absorption and information-guided mask scheduling, mix the discrete markers corresponding to the target image with the mask markers, and determine the priority of the discrete markers according to the information density;

[0089] Step S21: Construct the absorption form of the standard discrete diffusion of the target image;

[0090] In the standard discrete diffusion model, the discrete markers corresponding to the target image are gradually absorbed into a pure mask state during the forward propagation process. The absorption form is as follows:

[0091] ;

[0092] in, Represents the conditional probability distribution. Indicates time Next target image The marking status of each position, Represents the target image. The original target markers at each location, This represents a time-increasing noise scheduling function. Indicates a mask marker, This represents the category distribution of the target image labels.

[0093] According to this absorption method, as the diffusion time increases, more and more discrete markers corresponding to the target image are replaced with pure mask markers. Although this process is applicable to unconditional generation, it completely discards the spatial structure information of the source image in image translation tasks.

[0094] Step S22: Construct the mixed absorption state of the target image;

[0095] To preserve the structural prior of the source image, a mixed absorption state of the target image is constructed, which takes the following form:

[0096] ;

[0097] in, Represents the target image. The mixed absorption state at each position, Represents the target image. The source image markers corresponding to each location, Indicates the source image marker injection ratio. When When the value is greater than zero, the mixed absorption state of the target image simultaneously contains both source image markers and mask marker information. This means that the state after the discrete markers corresponding to the target image are absorbed is no longer a pure mask, but a bridging state that retains certain structural cues from the source image. This design allows the discrete diffusion model to still perceive the source image content at the forward diffusion endpoint, helping to improve the structure preservation capability in image translation tasks.

[0098] Step S23: Construct a bridge-type forward transfer process for the target image;

[0099] Based on the mixed absorption state of the target image, after constructing the mixed absorption state of the target image, the target image is then processed. The bridge forward transfer at each position is defined as:

[0100] ;

[0101] in, For a moment Next A binary mask indicating a variable at each position; when When, the original marker of the target image is preserved at this position; when At that time, this position is absorbed into a mixed absorption state. .

[0102] Through this bridge-type forward transfer method, the endpoint of the forward diffusion changes from a pure mask state to a bridge-type absorption state containing the prior of the source image, thereby establishing an explicit generation bridge from the source image to the target image.

[0103] Step S24: Construct a pixel-level information density map of the target image;

[0104] To ensure that the diffusion order during the training phase of the discrete diffusion model is consistent with the generation order during the inference phase, it is necessary to further construct an information-guided mask scheduling mechanism. Therefore, the pixel-level information density map of the target image is constructed first. .

[0105] For image translation tasks with a source image, a pixel-level information density map of the target image can be constructed based on the differences between the target image and the source image:

[0106] ;

[0107] in, This is a pixel-level information density map of the target image. Represents the target image. One channel, Indicates the source image number 1 One channel, This represents the number of channels in the target image and the source image. The greater the difference between the target image and the source image, the stronger the semantic change in that region, the higher the translation difficulty, and therefore the higher its information density.

[0108] For text-to-target image generation tasks from source images, the local variance of the target image can be used to describe the region structural complexity. The pixel-level information density map of the target image can be written as:

[0109] ;

[0110] in, Represents the grayscale image of the target image. This represents the local average pooling operator, where k represents the pooling factor of the target image. The larger the local variance of the target image, the more complex the texture and structure of that region, and the richer the information it contains.

[0111] Step S25: Based on the pixel-level information density map of the target image, construct an information-guided mask scheduling mechanism to determine the final sampling distribution of the target image, and then obtain the mask sequence.

[0112] After downsampling the pixel-level information density map M to the discrete marker resolution, the marker-level information content map is obtained. Then, the information content map of the label level. The probability distribution is normalized to an information-guided probability distribution and then weighted and fused with a standard uniform distribution to obtain the final sampling probability distribution of the target image.

[0113] ;

[0114] in, Represents the target image. The sampling probability at each location, Represents the target image. The probability of information guidance at each location. Represents a uniform distribution probability. This represents the randomness control parameter.

[0115] Based on this sampling distribution, at time Select the marker positions to be absorbed to construct the time. mask sequence In this way, high-information regions are preferentially absorbed at earlier times, while low-information regions are absorbed at later times, thus forming a difficulty-aware bridge-like diffusion trajectory. This scheduling method enables the discrete diffusion model to more closely approximate the generation order of "recovering simple regions first, then complex regions" during the inference phase during the training phase.

[0116] Step S3: Construct a discrete diffusion bridge inverse process. Based on the sampling distribution of the target image, decode step by step according to the prediction confidence, and resample the undecoded regions from the mixed absorption state to obtain the generated image label sequence, including the following steps:

[0117] Step S31: Construct the initial state for the bridged reverse process to generate the image;

[0118] At the start of reverse generation, the endpoint state of the target image in the bridge-type forward diffusion process is used as the initial state of the generated image; for the generated image... The initial state of the markers at each position. Obtained by sampling the mixed absorption state of the target image:

[0119] ;

[0120] That is, the reverse process does not start from a global pure mask state, but from a bridge state containing structural information of the source image. This allows the discrete diffusion model to retain the source image anchor point in the initial stage, thereby avoiding the reverse generation from completely deviating from the input image structure.

[0121] Step S32: Construct the predicted probability conditional distribution of discrete image labels generated by the bridge-type reverse process;

[0122] At each reverse time step, the discrete diffusion model is based on the current state of the generated image. The discrete label sequence y corresponding to the source image and the conditional input representation Predictive conditional probability distribution of discrete image labels ,in, This represents the parameters of the discrete diffusion model. Through the predicted conditional probability distribution, the discrete diffusion model can provide candidate results and their confidence levels for target labels at each location that has not yet been recovered, providing a basis for subsequent stepwise decoding.

[0123] Step S33: Calculate candidate labels and prediction confidence;

[0124] For the currently undecoded number For each position, the candidate label with the highest probability is first selected from the predicted conditional probability distribution of the discrete labels in the generated image. :

[0125] ;

[0126] Simultaneously calculate candidate labels Corresponding prediction confidence level :

[0127] ;

[0128] in, Indicates candidate tokens The corresponding prediction confidence level. In this way, the discrete diffusion model can distinguish which positions already have sufficiently high certainty and which positions still need to retain bridge anchor points for further inference.

[0129] Step S34: Construct a bridge-based reverse transfer based on prediction confidence;

[0130] Based on the predicted confidence at each location, locations with a predicted confidence greater than or equal to a set threshold are directly decoded at the current time, while locations with a predicted confidence less than the set threshold continue to retain the bridge absorption state; the back transition form of the bridge discrete diffusion model is as follows:

[0131] ;

[0132] in, This represents a bridged marker obtained by resampling from a mixed absorption state.

[0133] The key to this reverse update method is that, for locations that have not yet been decoded, it does not simply reset them to pure masks, but rather returns them to a mixed state that includes prior information from the source image. In this way, the discrete diffusion model can repeatedly utilize the structural information of the source image throughout the entire reverse inference process, progressively refining complex regions.

[0134] Step S35: Perform multi-step bridge-type reverse generation to obtain the final output generated image.

[0135] Steps S32 to S34 are repeated until decoding is complete at all positions, resulting in a discrete label sequence for the final generated image. This discrete label sequence is then input into the image decoder to recover the final output generated image.

[0136] Because the method of this invention adopts a unified mechanism of bridge-type forward absorption and bridge-type backward generation, it can achieve good generation results in image editing, style transfer, restoration and reconstruction, modality mapping and text-to-image generation tasks, and maintain high quality with fewer sampling steps.

[0137] Step S4: Train the bridged discrete diffusion model using cross-entropy loss that only applies to the absorbed region to achieve image editing, style transfer, image restoration, modality conversion, or text-to-image generation;

[0138] Step S41: Construct training objectives for the absorbed regions and determine the loss function;

[0139] During the training of the bridge-type discrete diffusion model, the supervised loss is calculated only for the position of absorption at the current time, and its loss function is... for:

[0140] ;

[0141] in, Represents the expected value of conditional probability, when When , it means that the i-th position has been absorbed and needs to be included in the loss calculation; when The time interval indicates that the i-th position retains the original target label and is not included in the loss calculation at the current time step. By supervising only the absorbed region, the discrete diffusion model can more effectively learn its ability to recover the target image content from the bridge state.

[0142] Step S42: Perform bridged discrete diffusion model training;

[0143] During training, the target image and source image are first encoded into discrete label sequences, and then a bridge-type forward diffusion state is constructed according to step S2. Then Discrete label sequence corresponding to the source image and conditional input The inputs are combined into the discrete diffusion model to predict the discrete label sequence of the generated image, and the parameters of the discrete diffusion model are updated according to the loss function described in step S41. Through this training method, the discrete diffusion model can simultaneously learn how to recover the content of the target image using the structural anchor points and conditional information of the source image.

[0144] Step S43: Use the trained bridge-type discrete diffusion model to complete image translation and generation;

[0145] In practical applications, after encoding the input source image and conditional information in step S1, a bridge-type initial state is constructed, and then the bridge-type reverse generation process described in step S3 is executed, ultimately outputting the generated image. This method can be used for image editing tasks, where the source image is the image to be edited and the conditional input is the editing command; it can also be used for style transfer, image restoration, modality conversion, and super-resolution tasks; for text-to-image generation tasks, an explicit source image does not need to be input, and only the conditional control information is retained to complete the generation. Therefore, the method of this invention can be uniformly adapted to various image translation and generation scenarios.

[0146] Through the above steps, this invention establishes a spatiotemporally consistent bridged generation trajectory within the framework of a discrete diffusion model, thus obtaining a bridged discrete diffusion model. On one hand, a hybrid absorption mechanism is used to explicitly preserve the source image structure prior in the diffusion latent space, improving the structure preservation capability in image translation tasks. On the other hand, an information-guided mask scheduling mechanism is used to coordinate the absorption order during training with the decoding order during inference, enhancing the discrete diffusion model's ability to model semantic changes in complex regions. Furthermore, during the reverse process, hybrid absorption state resampling ensures that undecoded regions are always constrained by the source image anchor points, thereby facilitating the acquisition of high-quality output results under conditions of fewer sampling steps. Qualitative test results on modal translation and image restoration tasks are as follows: Figures 3-4 As shown. By Figure 3 As shown, in modal translation tasks, the method of this invention can more effectively complete the mapping between the source modality and the target modality. While ensuring the overall structural correspondence, it generates a result image with target modal features and outperforms existing methods in terms of edge contours, region correspondence, and local detail representation. Figure 4 As shown, in image restoration tasks, the method of the present invention can more accurately restore the texture and structural information of degraded areas, reduce the problem of detail loss caused by blurring, noise or defects, and make the restoration results superior to existing methods in terms of visual quality and structural integrity.

[0147] Example 2:

[0148] This embodiment proposes a computer-readable storage medium that stores executable instructions. When these instructions are executed, if they are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium.

[0149] The computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the image translation and generation method based on spatiotemporally aligned discrete diffusion bridge described in the various embodiments of this application.

[0150] The aforementioned storage media include: flash memory, hard disks, multimedia cards, card-type memory (e.g., SD (Secure Digital Memory Card) or DX (Memory Data Register, MDR) memory), random access memory (RAM), static random-access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic storage, disks, optical discs, servers, APP (Application) application stores, and other media capable of storing program verification codes. These media store computer programs, which, when executed by a processor, can implement the various steps of the image translation and generation method based on spatiotemporal aligned discrete diffusion bridges described above.

[0151] Example 3:

[0152] This embodiment proposes a computer program product, including a computer program or instructions, which, when executed by a processor, implements the image translation and generation method based on a spatiotemporally aligned discrete diffusion bridge.

[0153] Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a computer program product.

[0154] The various embodiments in this application are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0155] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the present invention.

Claims

1. An image translation and generation method based on spatiotemporally aligned discrete diffusion bridges, characterized in that, Based on the standard discrete diffusion model, a bridge-type forward diffusion process and a discrete diffusion bridge reverse process based on hybrid absorption and information-guided mask scheduling are constructed to obtain the bridge-type discrete diffusion model, which realizes image translation and generation, including the following steps: Step S1: Construct discrete label representations of the source image and the corresponding target image, obtain discrete label sequences corresponding to the target image and the source image, and construct the conditional input representation corresponding to the source image; Step S2: Construct a discrete bridge forward diffusion process based on hybrid absorption and information-guided mask scheduling, mix the discrete markers corresponding to the target image with the mask markers, and determine the priority of the discrete markers according to the information density; Step S3: Construct a discrete diffusion bridge reverse process, based on the sampling distribution of the target image, decode step by step according to the prediction confidence, and resample the undecoded region from the mixed absorption state to obtain the generated image label sequence; Step S4: Train the bridged discrete diffusion model using cross-entropy loss that only applies to the absorbed region to achieve image editing, style transfer, image restoration, modality conversion, or text-to-image generation.

2. The image translation and generation method based on spatiotemporally aligned discrete diffusion bridge according to claim 1, characterized in that, The specific method for step S1 is as follows: Step S11: Construct a discrete label representation of the target image to obtain the discrete label sequence corresponding to the target image; First, the target image is input into an image discrete tagger for encoding to obtain the discrete visual tag sequence corresponding to the target image. ; Step S12: Construct a discrete label representation of the source image to obtain the discrete label sequence corresponding to the source image; For tasks that have a source image as input, the source image is also input into the image discrete labeler to obtain the discrete label sequence corresponding to the source image. ; Step S13: Construct the conditional input representation corresponding to the source image; For image translation and generation tasks controlled by text or task instructions, text prompts, editing instructions, style descriptions, or task category information are encoded as conditional input representations. ; This condition input This represents the conditional control information used in the bridge-type discrete diffusion model.

3. The image translation and generation method based on spatiotemporally aligned discrete diffusion bridge according to claim 2, characterized in that, The specific method for step S2 is as follows: Step S21: Construct the absorption form of the standard discrete diffusion of the target image; Under the absorption form of the standard discrete diffusion of the target image, the discrete markers corresponding to the target image are gradually absorbed into the pure mask state during the forward process; Step S22: Construct the mixed absorption state of the target image; Based on the absorption form of the standard discrete diffusion of the target image, a mixed absorption state of the target image is constructed, which takes the following form: ； in, Represents the target image. The mixed absorption state at each position, Represents the target image. The source image markers corresponding to each location, Indicates the proportion of source image markers injected. Indicates a mask marker; Step S23: Based on the mixed absorption state of the target image, construct a bridge-type forward transfer process for the target image; After constructing the mixed absorption state of the target image, the target image is then processed. The bridge forward transfer at each position is defined as: ； in, Represents the conditional probability distribution. Indicates time Next target image The marking status of each position, Represents the target image. The original target markers at each location, Represents the category distribution of the target image labels; For a moment Next A binary mask indicating a variable at each position; when When, the original marker of the target image is preserved at this position; when At that time, this position is absorbed into a mixed absorption state. ; Step S24: Construct a pixel-level information density map of the target image; Step S25: Based on the pixel-level information density map of the target image, construct an information-guided mask scheduling mechanism to determine the final sampling distribution of the target image, and then obtain the mask sequence.

4. The image translation and generation method based on spatiotemporally aligned discrete diffusion bridge according to claim 3, characterized in that, The specific method for constructing the pixel-level information density map of the target image in step S24 is as follows: For image translation tasks with a source image, a pixel-level information density map of the target image is constructed based on the differences between the target image and the source image: ； in, This is a pixel-level information density map of the target image. Represents the target image. One channel, Indicates the source image number 1 One channel, Indicates the number of channels in the target image and the source image; For the task of generating text-to-target image from a passive image, the local variance of the target image is used to describe the regional structural complexity. The pixel-level information density map of the target image is as follows: ； in, Represents the grayscale image of the target image. This represents the local average pooling operator, where k represents the pooling factor of the target image.

5. The image translation and generation method based on spatiotemporally aligned discrete diffusion bridge according to claim 4, characterized in that, The specific method for step S25 is as follows: After downsampling the pixel-level information density map M to the discrete marker resolution, the marker-level information content map is obtained. ; Then, the information content map at the tag level... The probability distribution is normalized to an information-guided probability distribution and then weighted and fused with a standard uniform distribution to obtain the final sampling probability distribution of the target image. ； in, Represents the target image. The sampling probability at each location, Represents the target image. The probability of information guidance at each location. Represents a uniform distribution probability. Indicates the randomness control parameter; Based on the sampling probability distribution of the target image, at time... Select the marker positions to be absorbed to construct the time. mask sequence .

6. The image translation and generation method based on spatiotemporally aligned discrete diffusion bridge according to claim 5, characterized in that, The specific method for step S3 is as follows: Step S31: Based on the bridge forward diffusion process, construct the initial state of the image generated by the bridge backward process; The endpoint state of the target image in the bridge-type forward diffusion process is used as the initial state of the generated image; for the generated image, the [missing information - likely a specific image or process]... The initial state of the markers at each position. Obtained by sampling the mixed absorption state of the target image: ； Step S32: Construct the predicted probability conditional distribution of discrete image labels generated by the bridge-type reverse process; At each reverse time step, based on the current state of the generated image The discrete label sequence y corresponding to the source image and the conditional input representation Predictive conditional probability distribution of discrete image labels ,in, Indicates the parameters of the discrete diffusion model; Step S33: Calculate candidate labels and prediction confidence; For the currently undecoded number At each of the given locations, the candidate label with the highest probability is selected from the predicted conditional probability distribution of the discrete labels in the generated image. : ； Simultaneously calculate candidate labels Corresponding prediction confidence level : ； in, Indicates candidate tokens The corresponding prediction confidence level; Step S34: Construct a bridge-based reverse transfer based on prediction confidence; Based on the predicted confidence at each position, positions with a predicted confidence greater than or equal to a set threshold are directly decoded at the current time, while positions with a predicted confidence less than the set threshold continue to retain the bridge absorption state. The reverse transition form of the bridge-type discrete diffusion model is: ； in, This represents a bridge-shaped marker obtained by resampling from a mixed absorption state; Step S35: Perform multi-step bridge-type reverse generation to obtain the final output generated image; Repeat steps S32 to S34 until all positions are decoded to obtain the discrete label sequence of the final generated image; then, input the discrete label sequence of the generated image into the image decoder to recover the final output generated image.

7. The image translation and generation method based on spatiotemporally aligned discrete diffusion bridge according to claim 6, characterized in that, The specific method for step 4 is as follows: Step S41: Construct training objectives for the absorbed regions and determine the loss function; During the training of the bridge-type discrete diffusion model, the supervised loss is calculated only for the position of absorption at the current time, and its loss function is... for: ； in, Represents the expected value of conditional probability, when When , it means that the i-th position has been absorbed and needs to be included in the loss calculation; when The time interval indicates that the i-th position still retains the original target label and is not included in the loss calculation at the current time. Step S42: Perform bridged discrete diffusion model training; During training, the target image and source image are first encoded into discrete label sequences, and then a bridge-type forward diffusion state is constructed according to step S2. Then Discrete label sequence corresponding to the source image and conditional input The data are input into the discrete diffusion model to predict the discrete label sequence of the generated image, and the parameters of the discrete diffusion model are updated according to the loss function described in step S41. Step S43: Use the trained bridge-type discrete diffusion model to complete image translation and generation; After the input source image and condition information are encoded in step S1, a bridge-type initial state is constructed, and then the bridge-type reverse generation process described in step S3 is executed to finally output the generated image.

8. A computer-readable storage medium storing executable instructions for performing the image translation and generation method based on a spatiotemporally aligned discrete diffusion bridge as described in any one of claims 1-7, characterized in that, When the instruction is executed, it causes the processor to perform the image translation and generation method based on a spatiotemporally aligned discrete diffusion bridge.

9. A computer program product for executing the image translation and generation method based on spatiotemporally aligned discrete diffusion bridge as described in any one of claims 1-7, characterized in that, This includes a computer program or instructions that, when executed by a processor, implement the image translation and generation method based on a spatiotemporally aligned discrete diffusion bridge.

Images