Basic model construction method for pathological image high-resolution generation based on cell topology guidance

Abstract

The invention provides a basic model construction method for pathological image high-resolution generation based on cell topology guidance, which comprises the following steps of: prompting a large language model based on an interactive learning scheme, and generating synthetic cell arrangement according to real cell arrangement by utilizing a cell arrangement generator; integrating the synthetic cell arrangement as a condition into a potential diffusion model; a high resolution pathology image containing cellular level control is generated through a potential diffusion model. The method has excellent performance in the aspects of authenticity of the synthetic image and retention of diagnostic information, and can significantly improve the performance of lesion subtype classification and segmentation tasks. The method not only enhances the authenticity of medical image synthesis, but also provides a high-quality data enhancement tool for downstream tasks, and promotes the development of artificial intelligence in the field of medical care.

Description

Technical Field

[0001] The present invention relates to the field of digital pathology, and in particular to a method for constructing a basic model for high-resolution generation of pathological images guided by cell topology, and in particular to a deep learning method for generating high-resolution pathological images, which utilizes cell-level conditional control and random sliding inference strategies to enhance the authenticity and diagnostic value of medical image synthesis. Background Art

[0002] In the field of digital pathology, deep learning techniques have demonstrated superior performance in tasks such as cancer classification, prognostic analysis, cell detection and segmentation, and multi-channel immunohistochemistry interpretation. However, the performance of these models is highly dependent on large-scale, high-quality annotated pathology image datasets. Due to the high heterogeneity and complex spatial context of tumors, obtaining sufficient annotated data is extremely difficult. Pathologists need to evaluate panoramic images at multiple scales to ensure robust and reproducible analysis. This costly and time-consuming annotation process leads to data scarcity, severely limiting the performance of learning algorithms and hindering the development of artificial intelligence in healthcare.

[0003] In recent years, visual grounding models have provided novel solutions to this challenge. Visual grounding models are able to generate realistic pathology images, thus enhancing existing datasets.

[0004] However, there are many challenges in synthesizing high-resolution pathology images, including the lack of cell-level annotations resulting in low generation diversity; the complex structure of lesions makes it impossible to generate realistic tissue spatial context, which easily produces inaccurate details and abnormal cell arrangement; and when the diffusion model trained from low-resolution images is sampled at high resolution, the consistency of long-range spatial context may be affected due to distribution shift.

[0005] Most existing pathology image synthesis methods focus on two aspects: one is the generation of accurate representations of small patches based on random or predetermined layouts; the other is the generation of full-field images without the need for specific category conditions. Although these methods can meet the needs to a certain extent, they still have shortcomings in the generation of controllable spatial context at the cellular level. For example, although some methods can generate accurate cell arrangements in small areas, they have difficulty maintaining consistency when expanded to larger areas. Although other methods can generate large-scale images, they ignore the specific conditions of cell types, resulting in the lack of necessary diagnostic information in the generated images. In addition, due to the high resolution and long-range correlation of pathology images, existing diffusion models are also under tremendous pressure in terms of computing resources. Summary of the Invention

[0006] This paper aims to address the scarcity of high-quality annotated data in digital pathology, as well as the problems of insufficient details and poor spatial context preservation of existing models when generating high-resolution pathology images.

[0007] To address these challenges, this paper proposes a method for constructing a foundational model for high-resolution pathology image generation based on cell topology guidance. This method, a novel high-resolution pathology image synthesis method, is called a topology-guided pathology foundational model. This model combines a cell arrangement generator with a randomized sliding inference strategy to achieve high-resolution pathology image synthesis with detailed cellular-level control.

[0008] The present invention comprises the following steps:

[0009] Step 1: Construct a cell arrangement generator. Use the real cell arrangement to prompt the large language model to generate a cell arrangement that conforms to the real distribution characteristics. Step 1 includes the following sub-steps:

[0010] Use cell coordinate points to describe the real cell arrangement, input the text containing all cell coordinates into the large language model and give instructions for synthesizing similar cell coordinate points, and use the large language model to output a synthetic cell arrangement similar to the real cell arrangement.

[0011] To capture the structural characteristics of cell arrangements, including clusters, gaps, and intercellular distances, as well as the distribution dynamics of different cell types, this paper proposes a multi-scale K-function descriptor. By calculating the multi-scale K-function, the descriptor outputs a vectorized cell arrangement structure. This approach can comprehensively describe the local and global structure of cell arrangements.

[0012] The DINO encoder is used to extract one-dimensional token sequences from real and synthetic cell arrangements. The extracted latent vector includes the position information of the cells. The discriminant head is used to process the one-dimensional token sequence on the latent vector. The batch normalization is assisted by the cascaded convolutional layer to obtain the loss function L. d , provides a true / false prediction for each cell arrangement, obtained by computing the distance between the multi-scale K-function vectors of the real and synthetic cell arrangements. a , minimizing the above two losses to guide large language models to optimize the generation of synthetic cell arrangements under multiple prompts.

[0013] Step 2: The selected synthetic cell arrangement is input as a condition into the latent diffusion model to guide the generation of spatial context for pathological images. Step 2 includes the following sub-steps: (2.1) Before training, the selected synthetic cell arrangement is input as a condition into the latent diffusion model. Noise is injected into the latent variable x at time steps t = 1...T through the forward process of the Markov chain. The larger the time step, the more noise there is. The noise-injected result is sampled using the denoising diffusion implicit model. The amount of noise reduction required in the next time step is predicted through the inverse process of the hidden Markov chain to gradually reduce the noise in the latent variable until a clear and noise-free image is finally generated.

[0014] (2.2) Reconstruction loss is used to evaluate the visual similarity between the output image and the input pathological image to ensure the authenticity and consistency of the generated image.

[0015] Step 3: To generate a high-resolution image and ensure smooth transitions between parts to avoid splicing artifacts, the present invention adopts a random sliding inference strategy. Step 3 includes the following sub-steps:

[0016] (3.1) Initialize the large image latent variable Y by sampling from the Gaussian distribution N(0,I) and initialize the first patch starting from the top left corner of Y. Then randomly select subsequent patches to complete discrete time step tracking.

[0017] (3.2) After denoising all patches, the patches are aggregated through the weight map to form a high-resolution latent representation. The final high-resolution latent representation is back-projected back to the pixel space through the decoder to generate a complete high-resolution pathology image.

[0018] Beneficial effects of the present invention

[0019] The present invention initializes large-scale image latent variables and extracts the latent representation of the synthesized cell arrangement as visual conditions through large-scale image synthesis steps based on the cell distribution map, gradually completes the denoising of all patches, and finally forms a high-resolution latent representation to synthesize high-resolution pathological images close to the real images, which can enhance the diagnostic value of the synthesized medical images. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 Schematic diagram of the cell arrangement generator architecture.

[0021] Figure 2 Schematic diagram of the architecture of the latent diffusion model guided by cell alignment.

[0022] Figure 3 Schematic diagram of the random sliding inference strategy architecture.

[0023] Figure 4Comparison of the results of high-resolution pathology image generation guided by various cell arrangements. DETAILED DESCRIPTION

[0024] This paper proposes a novel visual foundation model that aims to synthesize high-resolution pathological images through detailed cellular-level control. The following is a detailed description of the specific implementation:

[0025] Example 1

[0026] Step 1: Use the Cell Arrangement Generator to generate synthetic cell arrangements using an interactive learning scheme to prompt the large language model. This includes the following steps:

[0027] (1.1) Use cell coordinate points to describe the real cell arrangement, convert the real-world cell arrangement into a text entry, which contains the cell location points, and input it into the Tongyi Qianwen API together with the instruction: "This is a pathological cell distribution scatter point distribution location. Please output the cell location scatter points of similar pathological images based on this location text" to generate a synthetic arrangement in the same format as the input text entry.

[0028] (1.2) Cell structure pattern description: The cell arrangement text entry is represented as a scatter plot, with each dot representing a cell. The K function is used to describe the cell arrangement, which can reflect the distribution form at a certain scale. This function is used as a statistical measure to quantify the distribution of points in a given area. Formally, the K function is defined as follows:

[0029]

[0030] Where c and c′ represent any two different cell points, C represents all cells in the cell arrangement scatter plot, K(C) is the normalized value of the expected value E of the points within the radius r, and the value of the radius r is the side length of the square block. The square block is to divide the whole into multiple square blocks with a side length of r when calculating the overall K function. dist(c,c') represents the Euclidean distance between two points, and the normalization factor is the overall cell density λ.

[0031] (1.3) In order to better understand the local cell distribution, we divide each global scatter plot into n local patches according to location, and calculate the single-scale K function for each local patch to obtain the multi-scale K function K l , the expression is:

[0032] K l (C)={K(C1),K(C2),...,K(C n )}

[0033] Where 1, 2, ..., n represent the 1st, 2nd to nth local patches respectively, C represents the cells in the local patch, C1, C2, ..., C n Represent the scattered points in the 1st, 2nd to nth local patches respectively, and extract the scattered points in the local patch from the whole, {C1, C2, ..., C n}∈C is a set of scattered points consisting of all scattered points in the local patch, K(C n ) represents the n Calculate the single-scale K function on each C n Calculate the K function separately and extract the set {K(C1), K(C2), ..., K(C n )} get the multi-scale K function K l , thereby quantifying the local structure of cell layout.

[0034] (1.4) To ensure the similarity between the generated synthetic arrangement and the real arrangement, we introduce a discriminator for adversarial training. The cell arrangement is processed by the DINO encoder and then passed through a series of ViT-S blocks, and finally outputs the true and false predictions through the discriminator head. The loss function L d In the real arrangement p c and synthetic arrangement Wasserstein loss is used for calculation, and the formula is:

[0035]

[0036] D defines the loss L a To measure the distance between the multi-scale K functions between the real and synthetic cell arrangements, we ensure the similarity across scales. c and They represent the real arrangement and the synthetic arrangement respectively, which are the results of calculating K(C) respectively. c and Input the process of the discriminator in this section to judge the real and synthetic arrangement, E represents the cross entropy loss, La is calculated by p c and The L1 similarity between the single-scale K function and the global-scale K function is obtained.

[0037]

[0038] where K l Represents the multi-scale K function in (1.2), K() represents the K function of a single scale for the target as a whole, and n represents dividing the scattered points into n blocks. The final optimization target L o It's L d and L a The linear combination of o=L d +L a .

[0039] (1.6) During the generation process, the cell arrangement generator uses the real arrangement p c , repeatedly prompting the large language model to generate synthetic permutations Minimize L o , select the best generated result.

[0040] Step 2: Input the selected synthetic arrangement as a condition into the latent diffusion model to guide the generation of spatial context of the pathological image. Step 2 includes the following sub-steps:

[0041] (2.1) We designed a cell arrangement association module that uses cell arrangement to generate pathological images. First, the real pathological image is divided into small blocks, and the position information of the cells is extracted. The latent diffusion model is trained with these corresponding pathological images and position information. The input pathological image I is processed by the convolutional neural network (CNN) encoder network to generate embedded features. Specifically, the input pathological image I is processed by the encoder network, specifically by inputting the image into the predefined encoder E to generate the embedded features x0 = E(I) of the reconstruction target. Subsequently, in the latent space, noise is injected into the embedded feature x0 following the time step t through the Markov forward process to obtain the noisy embedded feature x t The process can be expressed as:

[0042]

[0043] where x t represents the image latent variable at time step t, d is the noise term that follows the standard normal distribution, and α t is the noise scheduling parameter controlled by time step t

[0044] (2.2) Design a conditional guided denoising network based on the Transformer U-Net architecture to eliminate noise in the latent features. c For the real cell arrangement, in the denoising process, the real cell arrangement is introduced as conditional information to guide the generation process, making the generation process more controllable. The loss function of the network guides the model to learn the key features of the pathological image by minimizing the mean square error between the generated features and the target features. For the conditional guided denoising process, the reconstruction loss function L is designed. r :

[0045] L r =α t ||x′(x t ,p c )-x0|| 2

[0046] Among them, α t represents the noise scheduling function controlled at time step t, x′(x t ,p c ) represents x′ in the real cell arrangement p c Under guidance, the image latent variable x at time step t t The generated image is x0, which is the embedded feature of the reconstructed target. With the real arrangement p c Similarly, we can use the trained cell arrangement association module to synthesize pathological images.

[0047] Step 3: To generate a high-resolution image and ensure smooth transitions between parts to avoid splicing artifacts, the present invention adopts a random sliding inference strategy. Step 3 includes the following sub-steps:

[0048] (3.1) Use random sliding strategy to synthesize high-resolution images. The high-resolution image can be decomposed into multiple image patches, which are projected into the latent space using encoder E. The large instance is cropped into multiple patches through multiple projections.

[0049] (3.2) A weight map is used to avoid the sharp boundary transition that may be caused by the direct assignment method. The latent vector is weighted according to its distance from the center of the image patch. The weight range is [0.5, 1] to ensure smooth transition and reduce splicing artifacts.

[0050] (3.3) When synthesizing a high-resolution image, we first initialize the large image latent variable and extract the synthetic cell arrangement as the visual condition. Starting from the top left corner, we gradually denoise all patches to form a high-resolution latent representation, and then obtain the final high-resolution synthesized image.

[0051] This paper proposes a novel high-resolution pathology image synthesis method to address the scarcity of high-quality annotated data, the lack of detailed images generated by existing models, and poor preservation of spatial context in digital pathology. This paper utilizes a cell arrangement generator and a random sliding inference strategy to achieve high-resolution pathology image synthesis with detailed cellular-level control. Specifically, the cell arrangement generator uses an interactive learning scheme to prompt a large language model to generate synthetic cell arrangements. A multi-scale cross-K function descriptor captures structural features, ensuring that the generated cell arrangements are realistic and reliable. These synthetic cell arrangements are used as conditional inputs into a latent diffusion model to guide the generation of high-resolution images. Furthermore, the paper employs a random sliding inference strategy to aggregate discrete low-resolution samples into a single high-resolution representation, ensuring smooth transitions between image components and avoiding stitching artifacts. Experimental results demonstrate that this method excels in synthesizing images with high fidelity and preserving diagnostic information, significantly improving performance in lesion subtype classification and segmentation tasks. This method not only enhances the fidelity of medical image synthesis but also provides a high-quality data augmentation tool for downstream tasks, promoting the development of artificial intelligence in healthcare.

[0052] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the scope of protection of the present invention. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the essence and scope of the technical solutions of the present invention.

Claims

1. A method for generating high-resolution pathological images based on cell topology guidance, characterized by: The method comprises the following steps, Based on an interactive learning scheme to prompt a large language model, a cell arrangement generator is used to generate synthetic cell arrangements based on real cell arrangements; integrating the synthetic cell arrangement as a condition into a potential diffusion model; Generate high-resolution pathology images with cellular-level control through latent diffusion models.

2. The method according to claim 1, wherein: The cell arrangement generator is used to generate a synthetic cell arrangement according to the real cell arrangement. Using a multi-scale cross-K function descriptor to capture the cell arrangement structure, wherein the cell arrangement structure includes one or more of clusters, gaps, intercellular distances, and distribution dynamics between different cell types; Process the cell arrangement through the DINO encoder and ViT-S block to calculate the loss value L o , combined with the identification loss L d The distance L between the multi-scale K function vector a , guiding large language models to optimize the generation of synthetic cell arrangements under multiple prompts.

3. The method according to claim 1, wherein: Further including: The integration into the latent diffusion model is to inject noise into the latent variable x at time steps t=1...T through a Markov chain forward process, use denoising diffusion implicit model sampling, and gradually reduce the noise in the latent variable until the final image is generated.

4. The method according to claim 1, wherein: The method uses a latent diffusion model to merge discrete low-resolution samples into a single high-resolution representation, a spatial context synthesis step, and a random sliding inference strategy to generate high-resolution images.

5. The method according to claim 4, characterized in that: Initialize the large image latent variable Y by sampling from the Gaussian distribution N(0,I), and initialize the first patch from the upper left corner of Y. Then randomly select subsequent patches to complete discrete time step tracking. After denoising all patches, the patches are aggregated through the weight map to form a high-resolution latent representation. The final high-resolution latent representation is back-projected back to the pixel space through the decoder to generate a complete high-resolution pathology image.

6. The method according to claim 2, wherein: The final optimization target L o It's L d and L a A linear combination of .

7. The method according to claim 1, wherein: The discrete low-resolution samples are merged into a single high-resolution representation, which is to sample the high-resolution image using a random sliding strategy. The high-resolution image is decomposed into multiple patches y∈R e , project it into the latent space using encoder E, crop the large instance into multiple patches through multiple projections, and use a weight map to weight the latent vector according to its distance from the tile center, with the weight range in [0.5, 1].

Images