A simulation sample generation and training data set augmentation system for white light endoscopic gastrointestinal endoscopic images

Abstract

The present application belongs to the technical field of generative artificial intelligence and computer vision, and discloses a two-stage image synthesis and training data set augmentation system for digestive tract endoscopic images under a white light endoscope, a first-stage image repair subsystem, and an effective polyp generation network, which can perform bidirectional region editing after training, that is, can synthesize a simulated target region with consistent anatomy within a given mask, and can also reconstruct normal background tissue texture in the image at the existing synthesized region candidate position. The second-stage semantic anchor mask proposal subsystem uses the trained image repair subsystem, first infers in the direction from the target region to the background to construct a conversion normal donor image evidence base paired with the original target region mask, and then uses DINOv2 self-supervised visual Transformer features and consensus voting to determine anatomically reasonable synthesized region candidate positions, sizes and shapes in the new query image to be augmented. The two subsystems cooperatively realize the full-automatic, context-consistent simulation sample generation and data set augmentation for the digestive tract endoscopic images under the white light endoscope.

Description

Technical Field

[0001] This invention relates to the fields of generative artificial intelligence and computer vision technology, and in particular to a two-stage system and method comprising an endoscopic image restoration subsystem (EPGN) and an automatic semantic anchor mask proposal subsystem. This invention enables fully automated, anatomically plausible simulation sample generation and dataset augmentation for gastrointestinal endoscopic images under white light endoscopy (WLI), thereby solving the long-standing data scarcity bottleneck that has constrained the training of downstream computer vision detection / classification models. Background Technology

[0002] Generative data augmentation has become a major approach to alleviate the long-standing problem of insufficient labeled data in computer vision training samples. In the colonoscopy scenario, target region synthesis based on diffusion models has become a representative paradigm for expanding training corpora without increasing labeling costs. Existing technologies can be broadly classified into three categories, distinguished by the conditional input method of the generative model and the source of the mask during the inference stage. However, each category of methods has known limitations on fidelity, controllability, scalability, or a combination thereof.

[0003] a) The first type is the mask-conditional latent diffusion system, which synthesizes an image containing the target region based on a binary segmentation mask, but does not use text guidance or utilize the surrounding image context. The generated mask is independent of any particular query image, so the candidate location, size, and shape of the synthesized region reflect the statistical prior of the mask generator rather than the anatomical context of the target frame; this type of system lacks a mechanism to transform a realistic background image into an image with a reasonable target region; and its reported downstream segmentation gain is network structure dependent, and may even be negative on some backbone networks.

[0004] b) The second category is text-guided stitching image inpainting systems—represented by existing region generation systems such as Polyp-Gen—which reconstruct the target region on a real background image within a user-provided mask. This type of method injects the mask by extending the first layer of the U-Net convolution to nine input channels and zero-initializing the newly added weights, forcing the model to implicitly learn background preservation, thus producing typical "adhesion" artifacts at the mask boundaries. Single-scale boundary band supervision cannot distinguish between low-frequency illumination mismatches unrelated to the target region and high-frequency texture mismatches relevant to the task. Furthermore, the 77-token CLIP text encoder lacks the ability to express morphologically specific domain-specific text cues.

[0005] c) The third type retains the same Stable-Diffusion-Inpainting U-Net backbone but adds an auxiliary ControlNet-like control branch—recent gastrointestinal endoscopic region repair systems belong to this category—thus inheriting the limitations of the second type in terms of boundary and color consistency. Simultaneously, the number of parameters and inference overhead are nearly doubled, and post-selection using an independent detector makes the enhancement quality dependent on the accuracy of that detector. All three types of methods also share a limitation in mask placement: their mask sources are not conditionalized to the specific anatomical context of the query image.

[0006] This invention overcomes the shortcomings of the three types of methods mentioned above through a coordinated two-stage design. Deterministic latent combination provides a rigorous mathematical guarantee for background preservation, thereby eliminating boundary and color inconsistency artifacts in the second and third types of methods; the maskless channel input architecture avoids reliance on zero-initialized additional input channels and allows direct transfer of pre-trained text-based image weights; in specific embodiments, multi-scale Laplacian pyramid supervision with different frequency band weights can be used to replace the single-scale boundary band supervision in existing technologies; a large language model text encoder with a decoder-only approach and significantly larger embedding dimension and context length is used to replace the CLIP encoder in the second and third types of methods; cross-domain training enables the image inpainting subsystem to have bidirectional editing capabilities, and the semantic anchor mask proposal subsystem utilizes this capability to construct its own target region removal and conversion normal donor database—this mechanism solves the problem of image-independent mask generation in the first type of method, removes the dependence of the third type of method on upstream detectors, and completely eliminates the need for manual mask drawing. No closed-loop process has been disclosed in the prior art where the image inpainting subsystem itself provides donor corpus, and the mask proposal subsystem learns from it. Summary of the Invention

[0007] The purpose of this invention is to provide an integrated two-stage white light endoscopy digestive tract endoscopy image simulation sample generation and dataset amplification system with a closed-loop coupling relationship between two subsystems.

[0008] The present invention achieves the above objectives through a coordinated two-stage design. In the first stage, the image inpainting subsystem, referred to in this paper as the Effective Polyp Generation Network (EPGN), is trained. This subsystem combines deterministic latent space composition for background preservation guarantees, multi-scale supervision with frequency band separation achieved through Laplacian pyramid loss, and a cross-domain masking strategy that endows the subsystem with bidirectional editing capabilities—that is, it can synthesize target regions within a given mask and reconstruct normal tissue at existing synthetic region candidate locations. A second capability is then used in the second stage to construct a database of normal image donors: by calling the trained first-stage subsystem to run along the target region to background direction without any independent region permutation model. The second-stage semantic anchor mask proposal subsystem operates on this database: it utilizes self-supervised visual Transformer features to propose anatomically plausible synthetic region candidate locations, sizes, and shapes in any query image to be augmented, and feeds the resulting candidate masks back to the first-stage subsystem to produce fully automated and anatomically consistent simulated sample generation results. Thus, this invention establishes a closed-loop process, in which the image inpainting subsystem itself provides the donor corpus required for the mask proposal subsystem to learn.

[0009] To achieve the above objectives, the technical solution of the present invention is as follows: a simulation sample generation and training dataset augmentation system for gastrointestinal endoscopic images under white light endoscopy (WLI), wherein the system includes, in the order of operation, a first image inpainting subsystem EPGN and a second semantic anchor mask proposal subsystem; wherein...

[0010] The first image restoration subsystem includes:

[0011] (1.1) Latent coding module, used for targeting High-resolution white light endoscopy input images of the digestive tract. and its corresponding mask The deep compressed residual autoencoder DC-AE (with a spatial downsampling factor of 32 and a latent channel dimension of [missing information]) was pre-trained and frozen. ) Encoding Compressed latent representation;

[0012] (1.2) A latent image inpainting module is used to combine noisy mask latent variables with clean context latent variables according to the binary latent space mask in each training step and denoising step, thereby deterministically preserving the content outside the mask region without providing the mask or context latent variables as explicit input channels of the Transformer.

[0013] (1.3) Cross-domain mask training module, used for each training sample, with probability Construct target region reconstruction sample pairs, or with probability Construct normal tissue reconstruction sample pairs; among which, The sampling probability represents the direction of reconstruction of normal tissue across domains, thereby enabling the image restoration subsystem to achieve bidirectional editing capabilities; and,

[0014] (1.4) A flow matching training module, used to train the diffusion Transformer module according to logit-normal time step sampling and velocity prediction targets, and jointly constrain the denoising process with the latent combined image inpainting module; and,

[0015] The second semantic anchor mask proposal subsystem includes:

[0016] (1.5) A donor database constructed by the first subsystem EPGN, wherein the image containing the target region is input into the first subsystem EPGN and processed along the direction from the target region to the background to generate a corresponding set of transformed normal donor image features, and each donor image is associated with the binary mask of its original target region.

[0017] (1.6) A self-supervised feature extractor based on DINOv2 is used to generate global CLS-token descriptors and local patch-by-patch descriptor sequences from any input image; and a two-level retrieval and consensus voting module, which, for a new query image to be augmented, first retrieves a top-K donor subset through global descriptor cosine similarity, then determines reasonable candidate locations for synthetic regions in the query image by cosine matching between the donor target region patch descriptor and the query patch descriptor, and generates a reasonableness heatmap through multi-donor consensus thresholding; and

[0018] (1.7) An evidence-driven mask sampler is used to sample candidate target region mask locations from a plausibility heatmap and assign a mask size to each location derived from the actual target region size of the top donor voting for that location, apply random shape and size perturbations, and optionally combine multiple target regions in each output mask; and the sampled candidate masks are fed back to a first image inpainting subsystem and generate anatomically plausible white light endoscopic fully automated synthetic training sample images of the digestive tract endoscopy along the background-to-target region direction.

[0019] The latent combined image restoration module of the first subsystem operates in the following combination:

[0020]

[0021] in, For a moment Noisy latent variables; Indicates the input image of the mask. The clean latent code; among which, Indicates encoder, Indicates the pixel space mask The latent space binary mask obtained by strict max pooling corresponding to the autoencoder downsampling factor ensures that any masked pixel in the latent grid cell will mark the entire cell as a mask. The Hadamard product is an element-wise multiplication operation between tensors of the same dimension; as described above. (i.e., input image to mask) The clean latent code obtained by encoding is uniformly referred to as (in the entire text and accompanying drawings of this specification) ( Add a wavy line (to represent the clean latent code of the complete input image) Distinguish between them; This represents the latent tensor obtained through the latent combination and directly input to the Transformer.

[0022] The combination is used for: (a) constructing the Transformer input at each training step; and (b) as an updated remixing step after each Euler denoising step during inference. This combination yields the following rigorous mathematical guarantee: for any... latent position, combined input Equal to Furthermore, this is independent of the Transformer's prediction at that step; by repeatedly applying this combination after each denoising update during inference, the latent representation of the background outside the mask is kept unaltered at the input or update stage of each denoising iteration. The Transformer receives an input latent tensor with the same number of channels as the underlying text-based Transformer, but does not include the pixel space mask, latent space mask, or mask image latent variable-added input channels; the generated region is inferred by the Transformer solely based on the discontinuity between the noise content inside the mask and the clean context content outside the mask, combined with textual conditions.

[0023] The stream matching training module in the first image restoration subsystem employs:

[0024]

[0025]

[0026] in, It is the sigmoid function; For noise; Where z represents the clean latent code corresponding to the complete input image (as opposed to the code representing the masked input image). ); In this context, I represents the unit covariance matrix with the same dimension as the latent variable; The target velocity vector used for flow matching training; Auxiliary random variables sampled from a standard normal distribution are used to obtain the distribution located at [location] after sigmoid transformation. Time steps within the interval The logit-normal timestep distribution provides higher training density at moderate noise levels, within which global structure and detailed textures are formed together.

[0027] The flow offset parameter s is fixed as To distinguish it from the underlying text-based graph model The setting, when When the effective time step distribution is an unbiased logit-normal distribution, then when... At that time, the distribution is biased towards high-noise steps. The reason for using s = 1.0 in image inpainting is because of the contextual latent variables. The global structure has been provided deterministically through the aforementioned combination method to avoid excessive bias in temporal step distribution towards high-noise stages and to enhance texture, lighting, and context consistency training in low- to medium-noise stages. Therefore, the critical training steps are the low- to medium-noise steps; in these steps, the Transformer must fuse the generated content with a clean context, match the lighting and texture on both sides of the mask boundary, and generate fine details. Non-unit flow offsets will systematically downsample these critical steps.

[0028] The flow matching training module uses logit-normal time-step sampling and velocity prediction to train the diffusion Transformer, where the main training relationships include:

[0029]

[0030] in, This represents the sigmoid function; Let represent an auxiliary random variable sampled from a standard normal distribution, and then... Mapping results in the location Continuous time steps of the interval ; Represents the clean latent code of the complete input image; Indicates and Noise variables of the same dimension; This represents the target velocity vector used in training the diffusion Transformer, i.e., from... point to The linear flow matching speed.

[0031] The text condition module supports classifier-free guidance, as follows:

[0032] During training, text embedding Text dropout probability for each training sample Replaced with an empty text embedding obtained by encoding an empty string. During inference, velocity prediction is performed according to the following formula:

[0033]

[0034] in, The classifier-free guided scale has a value range of 1. ; For speed prediction without classifier guidance; The parameter is The velocity prediction function output by the diffusion Transformer takes latent variables, time steps, and text embedding conditions as inputs. Describe text embeddings for the target region. Embedded as empty text; During training Randomly replace with The probability of text being dropped. This guiding scale is lower than the guiding scale used in unconstrained text-based image generation because image inpainting tasks require that text consistency not outweigh contextual consistency with surrounding clean latent variables.

[0035] The donor database for the second subsystem is constructed offline as follows:

[0036] (5.1) For the defined N images containing target regions and its corresponding binary target region mask Processing is required; (among other things) Using lowercase subscript notation, it specifically refers to the first [number]th [item] stored in the donor database. The fixed binary mask corresponding to the amplitude donor, compared with the current input mask of the system as described in claim 2. Distinguishing between them; during the second phase of implementation, each They are used sequentially as input masks for this subsystem. (Provided by the first image inpainting subsystem EPGN) processes the target region along the background editing direction to generate a corresponding set of normal donor images, in which the target region is replaced by reasonable normal tissue. The pre-trained self-supervised feature extractor DINOv2 is used to extract global descriptors and patch-by-patch descriptor tensors for each donor image.

[0037] (5.2) By indexing each patch Detection mask Does the average patch count exceed the coverage threshold? ,according to mean( over patch )> Identify the target region patch set, where Select within the preset coverage threshold range :

[0038]

[0039] And further record each donor in Width and height of the target region bounding box:

[0040]

[0041] (5.3) Save tuples for each donor And stack all global descriptors into a database matrix. To support batch similarity retrieval via a single matrix multiplication; during the construction of the aforementioned donor database, This represents the total number of images containing the target region included in the donor corpus. For donor index; For DINOv2 descriptor dimensions, This represents the total number of patches corresponding to a single image. For patch index, Set the coverage threshold for the target area at the patch level; and These are the horizontal and vertical coordinates of the image pixels, respectively. Indicates donor The binary mask at the pixel position The value at; and The first Width and height of the bounding box of the target region of the data donor (in pixels).

[0042] The descriptor is extracted from the transformed normal donor image, not the original image containing the target region. Since both the donor and query images are background images, the global cosine similarity captures the anatomical context—including mucosal topology, viewpoint, camera-tissue distance, and lighting conditions—rather than just the appearance features of the target region, thus covering the threshold. The selection of this option enables each retained target patch descriptor to encode meaningful target-background transition features while retaining the minimum number of patches for small target regions.

[0043] The two-level retrieval and consensus voting module is used for a given query image. Run as follows:

[0044] (6.1) Global retrieval, calculating the global query descriptor And the cosine similarity of each donor:

[0045]

[0046] In the above formula, This represents the query image to be expanded; Indicates to The global CLS-token descriptor extracted by the DINOv2 feature extractor. Indicates the first The global descriptor of the same origin corresponding to the amplitude donor; That is, querying the image and the first Global cosine similarity between amplitude donors.

[0047] And retain the previous individual donors The set of fine-search donors constituted Used for mask generation

[0048] (6.2) Local patch-level matching and thresholded voting accumulation for each donor and the patch index for each target region Calculate the maximum similarity response of the donor target region patch descriptor relative to all query patch descriptors:

[0049]

[0050] In the above formula, Indicates to Extract the nth element from the obtained patch descriptor matrix Row vector, i.e., the row vector of the query image Descriptors for each patch; Indicates the first The first donor Descriptors for each patch; For vector dot product, It is an L2 norm; donor All target region patches in the query patch The maximum cosine similarity response.

[0051] Subsequently, the voting is accumulated as follows: if the maximum similarity exceeds a local threshold... Then the donor votes on the query patchj:

[0052]

[0053]

[0054]

[0055] In the above formula, Indicates query The number of valid votes obtained, This represents the sum of weighted similarity votes received for this patch. This represents the set of donors who vote for this patch; This is an indicator function; it takes the value 1 if the condition is true, and 0 otherwise. At this point, the two-level retrieval and consensus voting module has completed local voting statistics for each query patch; subsequently, it executes the following cross-donor consensus thresholding step. In the... , In the definition, the summation symbol This refers to the set of fine-search donors defined in (6.1). Summing is performed for each donor n, with the upper and lower bounds of the summation determined by the set The members determined (total) (One donor).

[0056]

[0057] patch mesh heatmap A reasonable heatmap is obtained by bilinear upsampling to image resolution. .

[0058] (6.3) Cross-donor consensus thresholding: for each query patch Only when Time (take) )reserve Otherwise, set it to zero; then, the patch grid heatmap H is upsampled bilinearly to the original image resolution to obtain a reasonable heatmap. A two-level design—that is, using loose max pooling for each donor in (6.2) and strict consensus thresholding across donors in (6.3)—filters out spurious single-donor matches while preserving positions independently identified by multiple anatomical references; and, voting-weighted averaging... It outputs continuous confidence scores instead of binary decisions, thereby supporting downstream mask samplers to perform probabilistic location sampling.

[0059] The donor information-driven mask sampler operates as follows:

[0060] (7.1) Location sampling: The normalized rationality heatmap is denoted as the location sampling distribution. Candidate centers are sampled at the image pixel patch locations according to the category distribution:

[0061]

[0062] Subsequently, the suppression radius around the sampling center was... A circular area is used to prevent clustering during subsequent sampling.

[0063] In the above formula, This represents the coordinates of the sampled candidate mask center in the image pixel coordinate system; Multinomial(·) represents a multinomial distribution; The reasonable heatmap as defined in claim 6 (i.e., a patch grid heatmap upsampled to the original image resolution using bilinear methods); denominator Indicates to The summation is performed over all pixel locations, and the result is normalized to form the position sampling distribution. The center position obtained from the sampling is used as the geometric center of the candidate mask for that time.

[0064] (7.2) Donor information-driven size and shape, and The corresponding patch index is indexed to the donor voting graph. The proposed mask size is inherited from the average target area size of the donors who voted for that location.

[0065]

[0066] In the above formula, and These represent the initial width and initial height of the proposed mask (in pixels), respectively, i.e., the donor voting map. Calculate the arithmetic mean of the width and height of the bounding box of the target region corresponding to each donor; To be with the sampling center The corresponding patch index; The set of donors who vote for this patch as defined in claim 6; | represents a set The cardinality (i.e., the number of donors who voted for this patch), and the range of summation. This indicates that for each donor in the set Summation; and These are the width and height of the bounding box of the nth donor target region, respectively, which are derived from the tuples stored in claim 5 (5.3).

[0067] scale factor Draw from either the reduced or enlarged case with equal probability:

[0068] The probability is 0.5 (reduced).

[0069] The probability is 0.5 (magnified).

[0070] The size perturbation range, the size after perturbation The mask is determined relative to the original target region; where, This is the size scaling factor. Representing an interval Uniform distribution on; and These are the mean width and height of the bounding box of the corresponding donor target region, respectively (the samples are from the tuples saved in (5.3)). The probability that the shape will be randomized to a square; The proportion of the output mask that is designated as a combination of multiple target regions; This refers to the number of target regions extracted when combining multiple target regions.

[0071] Then, the shape is randomized: by probability The mask is forced into a square shape by averaging the dimensions after perturbation.

[0072]

[0073] In the above formula, The dimensions obtained after the aforementioned scale perturbation are, i.e. ,in As a scale factor, This represents the average width and height of the bounding box corresponding to the donor target area; The width and height of the final output mask; where "square-mode" refers to the probability... The triggered shape randomization mode, in which the output size is... Take the upper branch of the piecewise function shown in the above formula, that is, force the mask to be a square to provide the shape prior of the circular (compact) target region; in other cases (i.e. the else branch of the above formula), the perturbed rectangular shape is maintained to reflect the shape diversity of the elongated (extended) target region.

[0074] The proportion in the output mask The mask is specified as a combination of multi-target regions; for these masks, the extraction count is... And under inter-sample suppression conditions, steps (7.1) to (7.2) were repeated for a total of The final combined mask is the element-wise union of all the masks:

[0075]

[0076] In the above formula, This represents the final output composite mask; This refers to the number of target regions extracted when combining multiple target regions as defined above; max(·) is the element-wise maximum value operation, which is equivalent to the element-wise union for binary masks, that is, the candidate masks of each single target region extracted by the above steps (1) to (4) are merged into a single combined mask in the pixel space. The mask size originates from the actual target region size observed by the specific donor voting for this specific position in the consensus voting step, rather than a random prior distribution; thus forming a context-specific size prior that can systematically reflect the observation law of "target region size is related to anatomical context" in the training data. The multi-target region combination option corresponds to the statistical distribution characteristics of multi-target co-occurrence observed in the training data.

[0077] The hardware deployment in the above system includes:

[0078] A single graphics processing unit with at least 4GB of video memory for inference at 512×512 input resolution and at least 24GB of video memory for training, wherein the 512×512 input is DC-AE encoded into a 16×16×32 latent representation;

[0079] A deep compressed residual autoencoder with a spatial downsampling factor of 32 and a latent channel dimension of 32 is kept frozen during image inpainting training;

[0080] A decoder-only large language model text encoder with an embedding dimension of no less than 1024 and a maximum context length of no less than 200 tokens, kept frozen during image inpainting training; and can be optionally executed with 8-bit precision to reduce resource consumption;

[0081] The Diffusion Transformer module contains 28 layers of linear attention Transformer blocks and has 1152 hidden channels. During training for endoscopic image restoration, it is trained with randomly initialized weights and loaded with the completed checkpoints during inference.

[0082] Supports a self-supervised visual DINOv2 feature extractor with a 14-pixel patch size, which remains frozen during the training of the first subsystem and the operation of the second subsystem; for 512×512 input, a cropping / scaling to 512×512 or equivalent effective patch extraction method is used to obtain a 36×36 patch grid.

[0083] A computer-based method for automatically generating simulated samples of gastrointestinal endoscopic images under white light endoscopy includes the following steps:

[0084] Step 1: Receive the gastrointestinal endoscopy query image under white light as the background to be amplified;

[0085] Step 2: Extract global and patch-by-patch self-supervised visual descriptors from the query image;

[0086] Step 3: Retrieve from the pre-built donor database using global cosine similarity. A subset of donors, the donor database being obtained by the image inpainting subsystem running along the target region toward the background;

[0087] Step 4: Calculate the consensus voting rationality heatmap on the query patch by performing local cosine matching between the donor target region descriptor and the query patch descriptor, and retain only the positions that meet the minimum voting threshold.

[0088] Step 5: Sample one or more candidate mask centers from the heatmap, obtain the size of each candidate mask by averaging the target region size of the donors who vote for each specific center, and apply random size perturbation and shape change;

[0089] Step 6: Optionally, multiple candidate masks can be combined into a multi-target region union;

[0090] Step 7: For each candidate mask, (i) encode the query image and its corresponding mask image into a compressed latent representation, (ii) perform latent combination to form the Transformer input, (iii) decode the final latent representation, and (iv) synthesize the decoded image with the original image in pixel space according to the mask to obtain the repaired synthetic training sample output image.

[0091] Compared to existing technologies, the advantages of this invention are as follows: (a) it achieves deterministic background preservation by replacing implicitly learned background preservation with deterministic latent combination; (b) it replaces zero-initialized additional channels with a channelless mask input architecture that allows direct transfer of pre-trained weights; (c) it replaces single-scale boundary supervision with multi-scale loss supervision with band separation and learnable weights; (d) it replaces random mask placement or manual rendering with evidence-driven size inference-based content-aware automatic mask proposal; and (e) it forms a closed-loop process where the image inpainting subsystem provides a donor set for use by the mask proposal subsystem, eliminating the need for an external region permutation model. In specific implementations, compared to relevant prior art baselines, this invention achieves a reduction in Fréchet inception distance and improves mask boundary quality; simultaneously, compared to U-Net-based image inpainting baselines, inference speed is increased by approximately 2.5 times, and the number of parameters is reduced by approximately 32%. In addition, the present invention has the following technical advantages: (f) The compact latent representation generated by 32 times spatial downsampling of DC-AE significantly reduces the number of tokens that the Transformer needs to process, thereby reducing the training memory and computing power requirements; (g) Since the channelless mask input architecture does not introduce additional input channels, the pre-trained weights of the underlying text graph model can be directly transferred without performing an additional zero-initialization learning process on the additional channels. Attached Figure Description

[0092] Figure 1 —The overall two-stage system architecture flowchart shows the first-stage image inpainting subsystem, the donor database construction loop from the first stage to the second stage, the second-stage semantic anchor mask proposal subsystem, and the automatic enhancement loop from the second stage back to the first stage.

[0093] Figure 2 — An end-to-end method flowchart illustrating the steps from receiving the query image to be augmented and the text prompt, through feature extraction, global retrieval, consensus voting, donor information-driven mask sampling, latent combination initialization, iterative stream matching denoising with stepwise remixing, to the final pixel space synthesis. Detailed Implementation

[0094] To make the objectives, technical solutions, and beneficial effects of this invention clearer, the invention will be further described below with reference to the accompanying drawings. The following embodiments are used to illustrate the specific implementation process of the two-stage image synthesis and training dataset augmentation system, and are not intended to limit the scope of protection of this invention.

[0095] Example 1—Training Phase and Donor Database Construction

[0096] This embodiment illustrates the training process of the first image inpainting subsystem EPGN, and the offline construction process of the donor database after training. The purpose of the training phase is to enable the first image inpainting subsystem to simultaneously acquire the ability to synthesize the target region from the background to the target region, and the ability to reconstruct normal tissue from the target region to the background. The donor database construction phase utilizes the trained first image inpainting subsystem to generate converted normal donor images, providing anatomical contextual evidence for the semantic anchor mask proposal in the second phase.

[0097] When training the first image inpainting subsystem, the training corpus includes an image containing the target region, its corresponding binary target region mask, a normal background image, and corresponding text prompts. First, a frozen deep compression autoencoder encodes the complete input image and its corresponding mask image to obtain the complete image latent variables and contextual latent variables. Simultaneously, the pixel-space binary mask is strictly max-pooled according to the autoencoder's spatial downsampling factor to obtain the latent space binary mask. Therefore, in each training step, the noisy latent variables and the clean contextual latent variables are combined according to the following formula and input into the diffusion Transformer:

[0098]

[0099] In the formula, This represents the combined latent tensor input to the diffusion Transformer. This represents a binary mask in the latent space. Represents the noisy latent variable at the current time step. This represents the clean context latent code obtained by encoding the image corresponding to the mask. This indicates element-wise multiplication. During training, this combination relationship allows the model to focus on learning the content of the target region within the mask and its boundary fusion with surrounding tissues, while ensuring that the region outside the mask is always provided by clean contextual latent variables, thus preventing the model from treating background preservation as an additional learning task.

[0100] The cross-domain mask training module constructs two types of sample pairs with different directions during the same training phase. One type of sample pair is used for the background-to-target region direction, i.e., a candidate mask is given within the background tissue or normal region, enabling the model to learn to generate target region content within the candidate region; the other type of sample pair is used for the target region-to-background direction, i.e., a real target region mask is given at the existing target region location, enabling the model to learn to reconstruct normal tissue texture. For the case where an image containing a target region is paired with a random shape mask, the random mask and the real target region mask maintain zero overlap to avoid conflicting supervision signals between the target region content and the normal tissue cues. Through this joint training, the first image inpainting subsystem acquires bidirectional editing capabilities, where the target region-to-background capability forms the basis for subsequent offline construction of the donor database.

[0101] The flow matching training module trains the diffusion Transformer according to the logit-normal timestep sampling and velocity prediction target training described above. This training relationship enables the model to learn the velocity field from noisy latent variables to clean latent representations. Combined with the aforementioned deterministic latent combination relationship, the model mainly optimizes in-mask content generation, boundary transitions, texture continuity, and illumination consistency during training. The text conditional module employs an empty text embedding replacement mechanism during training and supports classifier-free guidance during inference, thereby ensuring that the generated content remains consistent between the target region description, the candidate mask range, and the surrounding clean context.

[0102] After the first image inpainting subsystem is trained, it enters the offline construction phase of the donor database. For each image in the selected corpus containing the target region and its real target region mask, the trained first image inpainting subsystem is invoked to run along the target region to background direction. That is, the image containing the target region is used as the input image, its real target region mask is used as the input mask, and the normal tissue description is used as the text prompt to generate the corresponding transformed normal donor image. Subsequently, the frozen self-supervised visual Transformer feature extractor extracts the global CLS-token descriptor and patch descriptor matrix of each transformed normal donor image, and saves its original target region mask, target region patch index set, and target region bounding box size. Since the target region in the donor image has been replaced by normal tissue, both the donor image and the subsequent query image appear as background images. Therefore, the feature similarity between the two mainly reflects the anatomical context such as mucosal topology, viewpoint, camera-tissue distance, and lighting conditions, rather than the appearance of the target region itself.

[0103] Example 2—End-to-end inference and simulation sample generation after training

[0104] This embodiment illustrates the end-to-end inference and simulation sample generation process after training is completed. Except for the donor database described in Embodiment 1, which has been built offline, the first image inpainting subsystem, diffusion Transformer, self-supervised visual feature extractor, and text conditional module in the following steps all use trained or frozen model parameters and are not updated. The synthetic training sample images output in this embodiment are only for downstream computer vision model training and do not constitute any form of medical diagnostic result, nor are they used to assist in actual medical decision-making.

[0105] During inference, the system first receives a white-light endoscopic gastrointestinal endoscopy query image as the background to be amplified, along with a descriptive text prompt for the target region. The same frozen self-supervised visual feature extractor used in the donor database construction phase then extracts the global descriptor and patch-by-patch descriptor matrix from the query image. Subsequently, the system calculates the global cosine similarity between the query image's global descriptor and the donor database's global descriptor matrix, retaining donors with high similarity to form a refined search set. Since both the donor and query images are background images, this global search step is used to filter donors that are anatomically similar to the query image, rather than filtering images with similar appearances to the target region.

[0106] After the refined search set is determined, the system performs local patch-level matching and cross-donor consensus voting. Using the previous notation, for each donor n in the refined search set and the query patch... The maximum cosine similarity response of the donor target region patch descriptor relative to the query patch is:

[0107]

[0108] In the formula, This indicates that the descriptor of the j-th patch of the query image is being retrieved. This represents the descriptor for the l-th patch of the n-th donor image. Let represent the set of target region patches in the nth donor image determined by the original target region mask, and let cos(·) represent the cosine similarity. This indicates that the donor is a good match for the query patch. The system calculates the maximum local similarity response. When this response satisfies the local similarity condition defined above, the donor votes for the query patch. The system further accumulates the number of valid votes, similarity-weighted votes, and the set of voting donors, and generates a reasonableness heatmap through cross-donor consensus thresholding. This process first uses a relaxed maximum matching within a single donor to improve candidate recall, and then removes false matches caused by a single donor through multi-donor consensus.

[0109] After the candidate locations are determined, the mask size is not directly extracted from a general random prior, but inferred from the boundary size of the donor target region that votes for that location. Using the previous notation, if the patch index corresponding to the sampling center is ĵ, the set of donors voting for that patch is... Then the initial size of the candidate mask is:

[0110]

[0111] In the formula, and These represent the initial width and height of the candidate mask, respectively. This represents the set of donors who vote for the patch corresponding to the sampling location. and Let represent the width and height of the bounding box of the nth donor target region, respectively. The resulting size is inherited from the donor that provides evidence for that specific location, rather than a random size unrelated to the image content. Subsequently, the system obtains the final candidate mask under the size perturbation and shape randomization rules defined above; for samples designated as multi-target region combinations, the position and size sampling is repeated under inter-sample suppression conditions, and multiple candidate masks are merged into a combined mask.

[0112] After the candidate mask is determined, the system inputs the query image and the candidate mask into the trained first image inpainting subsystem EPGN, and performs generation along the background to the target region direction. Specifically, the latent encoding module encodes the query image and its corresponding mask image into a latent representation, and obtains a latent space binary mask from the candidate mask. At each denoising time step, the first image inpainting subsystem still uses the latent combination relationship described in Example 1 to construct the combined latent tensor input to the diffusion Transformer. This combination relationship is an input recombination operation in the inference stage, rather than a new training loss; its function is to forcibly replace the region outside the mask with clean context latent variables in each denoising step, so that the trained diffusion Transformer only generates the target region content within the candidate mask, while keeping the background outside the mask unaltered. Finally, the system decodes the final latent representation back to the pixel space, and performs pixel space synthesis of the decoded image and the original query image according to the candidate mask to obtain a synthesized training sample image of gastrointestinal endoscopy under white light.

[0113] From a system implementation perspective, the first image inpainting subsystem and the second semantic anchor mask proposal subsystem form a closed-loop coupling relationship: after training, the first image inpainting subsystem provides normal donor images that transform the target region to the background direction; the second semantic anchor mask proposal subsystem uses these donor images to determine reasonable candidate positions and sizes in the query image; and the resulting candidate masks are then fed back to the first image inpainting subsystem to generate synthetic samples along the background-to-target region direction. Thus, the training phase, the donor database construction phase, and the inference generation phase are functionally interconnected, but distinct in terms of parameter updates and execution objectives: the training phase updates the parameters of the image inpainting subsystem; the donor database construction phase uses the trained model to generate an evidence base; and the inference phase uses a frozen or trained model to complete automatic mask proposal and simulation sample generation.

[0114] In summary, all the technical contributions of this invention are concentrated in: (i) a deterministic latent space combination method for strictly preserving background content outside the mask region; (ii) a training strategy for bidirectional region editing based on cross-domain training; (iii) a donor database constructed by the image synthesis subsystem itself and a two-level retrieval-consensus voting position proposal mechanism based on this database; and (iv) an evidence-driven synthesis region size inference mechanism. All of the above contributions belong to algorithmic and system innovations in the field of computer vision and generative models. This invention does not advocate any medical diagnostic methods, nor does it advocate any coverage of medical judgments or clinical decisions.

[0115] The above embodiments are not intended to limit the scope of protection of this invention. All equivalent modifications and substitutions made based on the above technical solutions should fall within the scope of protection of the claims of this invention.

Claims

1. A system for generating simulation samples and expanding training datasets for gastrointestinal endoscopic images under white light endoscopy, characterized in that, The system, in operational sequence, includes a first image inpainting subsystem EPGN and a second semantic anchor mask proposal subsystem; wherein... The first image restoration subsystem includes: (1.1) Latent encoding module, used to encode the input image of the gastrointestinal endoscope under white light endoscopy and its corresponding mask into a compressed latent representation; (1.2) Latent combination image inpainting module, used to deterministically combine noisy mask latent variables with clean context latent variables in the training step and the denoising step to preserve background content outside the mask area; (1.3) Cross-domain mask training module, used to construct target region reconstruction sample pairs and normal tissue reconstruction sample pairs, so that the first image restoration subsystem can obtain bidirectional editing capabilities of target region synthesis and normal tissue reconstruction; (1.4) A flow matching training module is used to train the diffusion Transformer module according to the time step sampling and velocity prediction target, and together with the latent combined image inpainting module, constrain the denoising process; The second semantic anchor mask proposal subsystem includes: (1.5) A donor database constructed by the first image restoration subsystem is used to store the transformed normal donor image features obtained by processing from the target region to the background direction and their corresponding original target region masks; (1.6) The feature extraction, two-level retrieval and consensus voting module based on self-supervised visual Transformer is used to retrieve the donor subset based on the global descriptor and determine the location of reasonable candidate target regions in the query image to be augmented based on the local patch descriptor matching, and generate a reasonableness heatmap; (1.7) An evidence-driven mask sampler is used to sample the mask position of candidate target region from the plausibility heatmap, determine the mask size according to the size of the donor target region that votes for the position, and feed the candidate mask back to the first image inpainting subsystem to generate anatomically plausible synthetic training sample images.

2. The system according to claim 1, characterized in that, The latent composite image inpainting module forms the latent tensor input to the Transformer through the following main composite relationships: in, Indicates at time step The combined latent tensor input to the Transformer; Indicates the current training or denoising time step; This represents the latent space binary mask obtained by downsampling from the pixel space mask, where Indicates the first Each potential location belongs to an area that needs to be generated or repaired. This indicates that the location belongs to a contextual background area that needs to remain unchanged; Represents the noisy latent variable at time step t; This represents the clean context latent code obtained by encoding the image corresponding to the mask; The Hadamard product, representing element-wise multiplication between tensors of the same dimension, is a technical application of this composition relation where: when hour, The Transformer receives noisy latent variables at this location and performs target region generation or repair; when hour, That is, the combined input is forced to be a clean context latent code at this position. During the inference process, the combination is re-executed after each denoising update. Therefore, the background latent representation outside the mask is decoupled from the prediction result of the Transformer at each iteration input stage, thereby achieving deterministic background preservation.

3. The system according to claim 1 or 2, characterized in that, The cross-domain mask training module performs sample pair construction in the background-to-target region direction and target region-to-background direction during the same training phase, enabling the first image inpainting subsystem to both generate the target region within a given candidate mask and reconstruct normal tissue at the existing target region location.

4. The system according to claim 1 or 2, characterized in that, The flow matching training module uses logit-normal time-step sampling and velocity prediction to train the diffusion Transformer, where the main training relationships include: in, This represents the sigmoid function; Let represent an auxiliary random variable sampled from a standard normal distribution, and then... Mapping results in the location Continuous time steps of the interval ; Represents the clean latent code of the complete input image; Indicates and Noise variables of the same dimension; This represents the target velocity vector used in training the diffusion Transformer, i.e., from... point to The linear flow matching velocity, the technical rationale for the above relationship is as follows: the model learns to connect the clean latent representation with the velocity field of the noise distribution, and generates the content within the mask through reverse denoising during inference; simultaneously, the latent combination relationship in claim 2 provides a clean context outside the mask in each training or denoising step, enabling the model to focus on learning the content of the target region within the mask and its boundary fusion with the surrounding tissue, and the flow offset parameter can be set as follows: This is to avoid the time step distribution being overly biased towards the high-noise stage and to enhance texture, lighting, and contextual consistency training in the low- to medium-noise stages.

5. The system according to claim 1, characterized in that, The first image inpainting subsystem also includes a text condition module, which performs empty text embedding replacement during training and supports classifier-free guidance during inference, ensuring that the generated target region remains consistent across text descriptions, mask regions, and surrounding clean context.

6. The system according to claim 1, characterized in that, The donor database is constructed offline: the image containing the target region and its binary target region mask are input into the first image inpainting subsystem, and a normal donor image is generated along the direction from the target region to the background; then the global descriptor and local patch descriptor are extracted by the self-supervised visual feature extractor, and donor tuples associated with the original target region mask, the target region patch set, and the target region boundary size are saved.

7. The system according to claim 6, characterized in that, The two-level retrieval and consensus voting module first retrieves a subset of donors from the donor database based on global descriptor similarity. Then, it performs local matching between the donor target region patch descriptor and the query image patch descriptor, and generates a rationality heatmap through cross-donor consensus thresholding to filter out false matches caused by a single donor.

8. The system according to claim 7, characterized in that, The evidence-driven mask sampler uses the normalized rationality heatmap as the location sampling distribution to determine one or more candidate mask centers. The candidate mask size is inferred from the size of the donor target region that votes for the corresponding location, and after size perturbation, shape randomization, and optional multi-target region combination processing, a candidate target region mask is formed to be fed back to the first image restoration subsystem.

9. A computer method implemented using the system according to any one of claims 1 to 8, characterized in that, Includes the following steps: Step 1: Receive the gastrointestinal endoscopy query image under white light endoscopy as the background to be amplified; Step 2: Extract global descriptors and local patch-by-patch descriptors from the query image; Step 3: Retrieve a subset of donors from a pre-built donor database that are similar to the anatomical context of the queried image; Step 4: Generate and threshold a plausibility heatmap based on the local matching results between the donor target region patch descriptor and the query image patch descriptor; Step 5: Sample the center of the candidate mask from the rationality heatmap, and determine the size of the candidate mask according to the target area size of the voting donor; Step 6: Perform size perturbation, shape randomization, and optional multi-target region combination processing on the candidate masks; Step 7: Input the query image and candidate mask into the first image restoration subsystem, and perform latent encoding, latent combination, denoising generation, decoding and pixel space synthesis to obtain a synthetic training sample image of gastrointestinal endoscopy under white light endoscopy.

Images