Method for generating cell endoscopy images

Abstract

The present invention discloses a method for generating endoscopic images, and relates to the technical field of medical image processing. The method comprises: obtaining a real image of an endoscopic image and corresponding pathological information, constructing a training data set and a test data set; preprocessing the real image, and performing text analysis on the pathological information; constructing an endoscopic image generation model based on a deep diffusion model, and training the model with a training data set; testing the model with a test data set; applying the final endoscopic image generation model to endoscopic image generation, and generating a new endoscopic image based on the input image and pathological information. The present invention combines pathological information with a diffusion model, and uses different pathological categories of endoscopic images as generation conditions. The generated image retains the key structural features in the medical image, has high medical relevance and structural accuracy, and helps to improve the accuracy in clinical applications.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a method for generating a cell endoscopic image. Background Art

[0002] In the field of medical image analysis, deep learning technology relies on large-scale, diverse, and high-quality data to improve the generalization ability of the model, but it often faces the problem of data imbalance, especially when important pathological samples are scarce. Traditional medical image generation methods mainly include generative adversarial networks (GANs) and variational autoencoders (VAEs). Although these methods have achieved certain results in generating medical images, they are usually single-modal methods, and the generated medical images are not accurate enough in terms of the boundaries, shapes, and features of lesions in the lesion area, and the authenticity and detail accuracy are poor. Summary of the invention

[0003] In order to solve the above technical problems, the present invention provides a method for generating a cell endoscopic image.

[0004] According to one aspect of the present invention, a method for generating a cell endoscopic image is provided, comprising:

[0005] Obtain real images of cell endoscopy and corresponding pathological information, and construct training and test datasets;

[0006] Preprocess the real images of cell endoscopy and perform text analysis on pathological information;

[0007] Based on the processed real images and pathological information, a cell endoscopic image generation model based on the deep diffusion model was constructed, and the model was trained using a training dataset.

[0008] The test data set is used to judge the endoscopic images generated by the trained model, and the qualified ones are the final cell endoscopic image generation model;

[0009] The final endoscopic image generation model is applied to the endoscopic image generation work, and the endoscopic image and pathological information are input. The model generates endoscopic images of corresponding pathological types based on the input images and pathological information.

[0010] Optionally, the obtaining of the real image of the cell endoscopy and the corresponding pathological information includes:

[0011] The patient's endoscopic image provided by the hospital is obtained as a real image, and the pathological diagnosis corresponding to the endoscopic image provided by the doctor is obtained as pathological information, wherein the pathological diagnosis of the endoscopic image includes hyperplastic polyps, sessile serrated lesions, adenomas and invasive carcinomas.

[0012] Optionally, the preprocessing of the real image of the cell endoscopy includes:

[0013] The real images of cell endoscopy were normalized to 512x512 pixels;

[0014] The standardized image is converted into a NumPy array, and RGB channel conversion and normalization are performed. The normalization formula is: ;in, is the normalized image, is the original image, and are the mean and standard deviation of the image data respectively.

[0015] Optionally, the text analysis of the pathological information includes:

[0016] pass tokenize The function marks the pathological information and generates a tensor : ;in, T Indicates pathological information. Pathological information T The tensor generated after labeling;

[0017] The tensor generated Input to the embedding layer of the OpenCLIP text encoding model to generate the embedding vector X , the expression is as follows: ;in, Embedding Indicates embedded processing;

[0018] Embedding vector Perform position encoding to introduce the position information of each tensor in the sequence, where the position encoding is as follows: ; ;in, Indicates the location and the dimension index is an even number The position encoding is calculated using the sine function. Indicates the location and the dimension index is an odd number The position encoding is calculated using the cosine function. is the position in the sequence, is the dimension index, is the dimension size of the embedding;

[0019] Encode the position PE With the embedding vector X Add element by element to generate an embedding vector with position information : ; The embedding vector that will add position information As the input of the Transformer layer, after being processed by multiple attention mechanisms and feedforward networks in the Transformer layer, the processed features are intercepted in the penultimate layer to output a high-dimensional semantic embedding vector context .

[0020] Optionally, the adopting the training data set to train the model includes:

[0021] Use ControlNet to receive the current time step information and encode the time step information into a time embedding vector : ;in, t Indicates the current time step information;

[0022] Time Embedding Vector Then two linear layers and an activation function are passed to generate the final time embedding : ;in, , are the weights of the first and second linear layers, , They are , The corresponding bias, is the activation function;

[0023] Extracting main feature maps from preprocessed intracellular microscopy images : ;in, represents the preprocessed cell endoscopy image, Indicates the use of the convolution module in ControlNet from Extract image features;

[0024] The main feature map Combined with time embedding and the high-dimensional semantic embedding vector obtained through text analysis context Perform further feature extraction and fusion to generate feature maps : ;in, Represents the main feature map , Time Embedding , high-dimensional semantic embedding vector context The three vectors are fused and spliced;

[0025] Generated feature map Further processing through the middle_block module to obtain the feature map , where the middle_block module includes residual blocks and attention blocks, which are responsible for strengthening the dependencies between features, retaining and strengthening temporal and spatial features, and ensuring that the impact on the output in the early stage of training is minimized through the smoothing operation of the zero convolution layer. The expression is as follows: ;in, represents the residual block operation, represents the attention block operation, represents zero convolution operation;

[0026] The real image of the cell endoscope is input into the encoder Autoencode. The image is first represented in the latent space by the encoder method. During this process, the encoder generates feature maps for the mean and standard deviation. Then, the feature maps for the mean and standard deviation are converted into latent vectors for sampling through the quant_conv convolution layer. z The latent space representation of

[0027] The latent space representation is decoded into a generated image through a decoder.

[0028] Optionally, before decoding the latent space representation into a generated image through a decoder, the method further comprises sampling the latent vector z Performing denoising processing, wherein the denoising processing includes:

[0029] For each time step t , generate the corresponding noise image x_noisy t : ;in, is the time step t Corresponding to the noise image, the model will sample the potential vector z Add a time-step-dependent amount of noise to is the time step t The corresponding noise scaling factor, is random noise sampled from a standard normal distribution;

[0030] The feature map With the current noisy image x_noisy , time step t and high-dimensional semantic embedding vectors context Combine and combine into complete conditional information and pass it to the stable diffusion model;

[0031] In the forward propagation process of the Stable Diffusion model, more noise is gradually added to the image as time steps advance. In the backward propagation process, the model will be based on the current noisy image. x_noisy , time stept , context , and the feature map passed from ControlNet , calculate and predict the noise of the current step and apply it to the denoising process, where the denoising process formula is expressed as: ;in, is the time step t- 1 corresponds to the noise image, compared to the time step t The noise image at the time has removed some noise and is gradually approaching the final clear image. is the noise of the model prediction at the current step, Representative feature map , high-dimensional semantic embedding vector context Input of conditions;

[0032] The loss function formula in the denoising process is expressed as follows: ;in, Represents the time step , original image ,noise The expected value of the loss function Used to measure the noise of model predictions The actual noise added By minimizing the gap between them, the model gradually learns to remove noise and thus updates the latent space representation of the image.

[0033] The latent space representation z after denoising will be used to be decoded into the final image by the decoder.

[0034] Optionally, the using the test data set to judge the endoscopic image generated by the trained model includes:

[0035] Input the real endoscopic images and corresponding pathological information in the test data set into the trained endoscopic image generation model to obtain the generated endoscopic verification images;

[0036] The cell endoscopy verification images were compared with the corresponding real images, and quantitative indicators and qualitative evaluation methods were used to evaluate the performance of the cell endoscopy image generation model on the test dataset.

[0037] Optionally, the applying the final cell endoscopic image generation model to the cell endoscopic image generation work comprises:

[0038] The open source Gradio library is used to create a graphical interface, which is used to allow users to interact with the cell endoscopic image generation model through a web interface. The user uploads the cell endoscopic image, inputs pathological information, and adjusts parameters through the graphical interface, and clicks the generate button on the graphical interface to obtain a new cell endoscopic image generated by the model.

[0039] Optionally, after obtaining the new cell endoscopic image generated by the model, the method further comprises:

[0040] An image saving instruction input by a user is received, and in response to the image saving instruction, a generated new endoscopic image is saved into an Excel file, wherein the Excel file includes pathological information, a real endoscopic image, and a new endoscopic image generated based on the pathological information and the real image.

[0041] Beneficial effects of the present invention:

[0042] The present invention proposes a method for generating endoscopic images of cells, which combines pathological information and is based on a deep diffusion model to generate endoscopic images that conform to different pathological categories. By introducing text pathological information as a constraint for image generation, the pathological information is combined with the diffusion model, and different pathological categories of endoscopic images are used as generation conditions. The pathological information text can constrain the generated image and the input image to be the same in lesion type. Through multimodal fusion of text information and image information, a new endoscopic image that is consistent with the lesion category of the original image is generated. The generated new endoscopic image is the same type of lesion as the input image, and differs from the input image in background and some texture details, ensuring that the generated image is both diverse and can retain key structural features in medical images. Therefore, the generated image has high medical relevance and structural accuracy, which helps to improve accuracy in clinical applications.

[0043] In addition, the images generated by the method of the present invention can also serve the task of classifying intestinal cell endoscopic images. Since the data sets of these tasks often face the problem of insufficient labeled data or missing data on rare diseases, the image generation method provided by the present invention can ensure the generation of pseudo images of the same disease category, thereby reducing the difficulty of training the intestinal image lesion classification model based on deep learning and improving the accuracy of the final classification.

[0044] Through the cell endoscopic image generation method based on the deep diffusion model of the present invention, cell endoscopic images that meet clinical needs can be generated more effectively to supplement the data of scarce pathological categories, thereby alleviating the challenges caused by the scarcity of data in certain pathological categories. At the same time, this method provides a balanced and rich data set for subsequent segmentation and classification tasks of cell endoscopic images, significantly improving the accuracy and robustness of downstream tasks.

[0045] Through the cell endoscopic image generation method based on the deep diffusion model of the present invention, doctors can use the deep diffusion model to accurately generate cell endoscopic images of different pathologies through a graphical interface. The generated images can be used for subsequent image quality judgment, which will effectively solve the data imbalance problem of downstream segmentation and classification tasks of cell endoscopic images. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0047] Figure 1 is a flow chart of a method for generating a cell endoscopic image in an embodiment of the present invention;

[0048] Figure 2 is a flow chart of training a cell endoscopic image generation model in an embodiment of the present invention;

[0049] Figure 3 It is a schematic diagram of an interface in which the cell endoscopic image generation method according to an embodiment of the present invention is applied to the cell endoscopic image generation work;

[0050] Figure 4 It is a structural block diagram of a cell endoscopic image generation system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0051] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only embodiments of a part of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work should fall within the scope of protection of the present application. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0052] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0053] The terms "comprises" and "having" and any variations thereof in the specification and claims of this application are intended to cover non-exclusive inclusions. For example, a process, method, product or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products or apparatuses.

[0054] Embodiment 1 of the present invention, referring to Figure 1 , Figure 1 is a flow chart of a method for generating a cell endoscopic image in an embodiment of the present invention, such as Figure 1 As shown, the method comprises the following steps:

[0055] S1, obtain the real images of cell endoscopy and the corresponding pathological information, and construct the training data set and the test data set;

[0056] Endoscopic cell (EC) images can be obtained from medical institutions, clinical research or professional databases. Pathological information usually comes from the diagnosis report of professional pathologists or doctors, including lesion type, lesion nature, pathological grade, location, etc. Each endoscopic cell image and its corresponding pathological information are paired to form the basic unit of the data set. The collected data are split into a training data set and a test data set according to a certain split ratio. The training set is used to train the model and usually accounts for a larger proportion of the total data set (such as 70%-80%). The test set is used to evaluate the performance of the model and usually accounts for a smaller proportion of the total data set (such as 20%-30%).

[0057] Furthermore, in this embodiment, the endoscopic image of the patient provided by the hospital is obtained as the real image, and the pathological diagnosis corresponding to the endoscopic image provided by the doctor is obtained as the pathological information. In this example, the pathological information is the image diagnosis result, including four types: hyperplastic polyps, sessile serrated lesions, adenomas, and invasive carcinomas.

[0058] S2, preprocessing the real images of cell endoscopy and performing text analysis on pathological information;

[0059] Among them, image preprocessing includes operations such as image enhancement, normalization, and denoising; text analysis includes operations such as text cleaning, vectorization, and feature extraction.

[0060] Furthermore, in this implementation, preprocessing the real image of the cell endoscopy includes:

[0061] The real images of cell endoscopy were normalized to 512x512 pixels;

[0062] The standardized image is converted into a NumPy array, and RGB channel conversion and normalization are performed. The normalization formula is: ;in, is the normalized image, is the original image, and are the mean and standard deviation of the image data respectively.

[0063] NumPy Array is a data structure provided by the NumPy library for efficiently storing and manipulating large multidimensional arrays and matrices in Python. RGB channel conversion refers to rearranging or adjusting the data of the three color channels of an image, namely red, green, and blue, in image processing. Using NumPy's slicing and indexing functions, you can easily separate, merge, or reorder the RGB channels.

[0064] By preprocessing the images such as standardization and normalization, the cell endoscopy images can be made consistent with the model input, which helps to accelerate model convergence and improve training stability and performance.

[0065] Furthermore, in this implementation, text analysis of pathological information includes text tokenization and embedding and position encoding, specifically, including:

[0066] pass tokenize The function marks the pathological information and generates a tensor : ;in, T Indicates pathological information. Pathological information T The tensor generated after tokenization; tokenization is the process of breaking text into smaller, more manageable parts, usually words or subwords, using tokenize The function labels the pathological information, and this process generates a tensor, which represents the labeled text data;

[0067] Embed the labeled tensor and convert it into an embedding vector. Embedding is to convert the label into a fixed-size dense vector, usually a floating-point number. These vectors can capture the semantic relationship between the labels and generate a tensor. Input to the embedding layer of the OpenCLIP text encoding model to generate the embedding vector X : ;in, Embedding Indicates embedded processing;

[0068] Embedding vector Perform position encoding to introduce the position information of each tensor in the sequence, where the position encoding is as follows: ; ;in, Indicates the location and the dimension index is an even number The position encoding is calculated using the sine function. Indicates the location and the dimension index is an odd number The position encoding is calculated using the cosine function. is the position in the sequence, is the dimension index, is the size of the embedded dimension; when the dimension index is an even number, the sine function is used to calculate the position encoding, and when the dimension index is an odd number, the cosine function is used to calculate the position encoding; the position encoding PE With the embedding vector X Add element by element to generate an embedding vector with position information : ; The embedding vector that will add position information As the input of the Transformer layer, after being processed by multiple attention mechanisms and feedforward networks in the Transformer layer, the processed features are intercepted in the penultimate layer to output a high-dimensional semantic embedding vector context , this high-dimensional semantic embedding vector context The semantic and location information in the pathology information text is captured.

[0069] S3, based on the processed real images and pathological information, a cell endoscopic image generation model based on the deep diffusion model is constructed, and the model is trained using the training data set;

[0070] The most common generative models in the field of image generation include generative adversarial networks (GANs), variational autoencoders (VAEs), flow-based models, etc. The processed images and corresponding text vectors are used as the input of the model, and appropriate loss functions and optimizers are defined for model training. The model parameters are iteratively updated using the training data set until the predetermined number of training rounds is reached or the loss converges.

[0071] The model constructed in this embodiment is based on the Deep Diffusion Model (DDM). The deep diffusion model simulates the process of data gradually diffusing from random noise to target data. Compared with the traditional generation model, the diffusion model generates samples through a multi-step iterative process, which enables it to generate high-quality and diverse outputs.

[0072] Reference Figure 2 , Figure 2 FIG. 1 is a flow chart of training a cell endoscopic image generation model in an embodiment of the present invention. Figure 2 As shown, in this embodiment, using the training data set to train the model includes:

[0073] ControlNet is a neural network that controls a pre-trained image diffusion model. It uses ControlNet to receive the current time step information and encode the time step information into a time embedding vector. : ;in, t Indicates the current time step information;

[0074] Time Embedding Vector Then two linear layers and an activation function are passed to generate the final time embedding : ;in, , are the weights of the first and second linear layers, , They are , The corresponding bias, is the activation function;

[0075] Extracting main feature maps from preprocessed intracellular microscopy images : ;in, represents the preprocessed cell endoscopy image, Indicates the use of the convolution module in ControlNet from Extract image features;

[0076] The main feature map Combined with time embedding and the high-dimensional semantic embedding vector obtained through text analysis context Perform further feature extraction and fusion to generate feature maps : ;in, Represents the main feature map , Time Embedding , high-dimensional semantic embedding vector context The three vectors are fused and spliced;

[0077] Generated feature map Further processing through the middle_block module to obtain the feature map , where the middle_block module includes residual blocks and attention blocks, which are responsible for strengthening the dependencies between features, retaining and strengthening temporal and spatial features, and ensuring that the impact on the output in the early stage of training is minimized through the smoothing operation of the zero convolution layer, thereby allowing the model to gradually learn how to effectively use these feature information during the training process. This process is expressed as follows: ;in, represents the residual block operation, represents the attention block operation, represents zero convolution operation;

[0078] The real image of the cell endoscope is input into the encoder Autoencode. The image is first represented in the latent space by the encoder method. During this process, the encoder generates feature maps for the mean and standard deviation. Then, the feature maps for the mean and standard deviation are converted into latent vectors for sampling through the quant_conv convolution layer. z The latent space representation of

[0079] The latent space representation is decoded into a generated image through a decoder.

[0080] Furthermore, during the training of the diffusion model, the latent vector z is gradually added with noise at multiple time steps t, and the model will sample the latent vector z A noise amount related to the time step is added to the sample potential vector z Perform denoising, including:

[0081] For each time step t , generate the corresponding noise image x_noisy t : ;in, is the time step t Corresponding to the noise image, the model will sample the potential vector z Add a time-step-dependent amount of noise to is the time step t The corresponding noise scaling factor, is random noise sampled from a standard normal distribution;

[0082] The feature map With the current noisy image x_noisy , time step t and high-dimensional semantic embedding vectors context Combine and combine into complete conditional information and pass it to the stable diffusion model;

[0083] In the forward propagation process of the Stable Diffusion model, more noise is gradually added to the image as time steps advance. In the backward propagation process, the model will be based on the current noisy image. x_noisy , time step t , context , and the feature map passed from ControlNet , calculate and predict the noise of the current step and apply it to the denoising process, where the denoising process can be expressed as: ;in, is the time step t- 1 corresponds to the noise image, compared to the time step t The noise image at the time has removed some noise and is gradually approaching the final clear image. is the noise of the model prediction at the current step, Representative feature map , high-dimensional semantic embedding vector context Input of conditions;

[0084] The loss function formula in the denoising process is expressed as follows: ;in, Represents the time step , original image ,noise The expected value of the loss function Used to measure the noise of model predictions The actual noise added By minimizing the gap between them, the model gradually learns to remove noise and thus updates the latent space representation of the image.

[0085] The latent space representation z after denoising will be decoded into the final image through the decoder to complete the model training.

[0086] S4, using the test data set to judge the endoscopic images generated by the trained model, and the qualified one is the final cell endoscopic image generation model;

[0087] The model is evaluated using a test data set to assess its performance on unknown data, i.e., non-training data. The performance includes image quality, such as clarity and detail retention. The endoscopic images generated by the model are evaluated based on objective quality evaluation indicators, such as PSNR (Peak Signal-to-Noise Ratio) or SSIM (Structural SIMilarity) and subjective expert judgment to determine whether the model is qualified. Qualified models will be used for subsequent application work.

[0088] Furthermore, in this example, the use of the test data set to judge the endoscopic images generated by the trained model includes: inputting the real endoscopic images and corresponding pathological information in the test data set into the trained endoscopic image generation model to obtain the generated endoscopic verification images; comparing the endoscopic verification images with the corresponding real images, and using quantitative indicators and qualitative evaluation methods to evaluate the performance of the endoscopic image generation model on the test data set. Among them, the quantitative aspect uses indicators such as the structural similarity index (SSIM) and the peak signal-to-noise ratio (PSNR) to measure the similarity between the generated images and the real images in terms of details and quality; the qualitative evaluation is performed by medical experts based on whether the generated images retain key pathological features and the supportability of the diagnosis, thereby comprehensively evaluating the generation ability and clinical relevance of the model.

[0089] S5, applying the final cell endoscopic image generation model to the cell endoscopic image generation work, inputting the cell endoscopic image and pathological information, and the model generates a cell endoscopic image corresponding to the pathological type based on the input image and pathological information.

[0090] In this example, an open source Gradio library is used to create a graphical interface, which is used to allow the user to interact with the endoscopic image generation model through a web interface. The user uploads the endoscopic image, inputs pathological information, and adjusts parameters through the graphical interface, and clicks the generate button on the graphical interface to obtain a new endoscopic image generated by the model. After obtaining the new endoscopic image generated by the model, the method further includes: receiving an image saving instruction input by the user, and saving the generated new endoscopic image to an Excel file in response to the image saving instruction, wherein the Excel file includes pathological information, a real image of the endoscopic image, and a new endoscopic image generated based on the pathological information and the real image.

[0091] Reference Figure 3 , Figure 3 FIG. 1 is a schematic diagram of an interface in which the method for generating a cell endoscopic image in an embodiment of the present invention is applied to generating a cell endoscopic image. Figure 3 As shown, this example uses the open source Gradio library to create a graphical interface, allowing doctors to interact with the endoscopic image generation model through a web interface. Doctors upload the real endoscopic images of the patient through the graphical interface, input pathological information, and adjust parameters, and click the Generate button on the graphical interface to obtain new endoscopic images generated by the model. After that, the doctor will judge the quality of the generated image. If the doctor thinks the quality of the generated image is good, he can click the Good button on the graphical interface to save the generated image to an Excel file; if the doctor thinks the quality of the generated image is not good, he can click the Bad button on the graphical interface to not save it.

[0092] The generated new cell endoscopic images integrate the input image features and text features, and can reflect the details of the diseased cell endoscopic images more accurately. They can be used in doctor diagnosis, teaching, scientific research and other fields, providing strong support for medical education and research.

[0093] In addition, the images generated by the method of the present invention can also serve the task of classifying intestinal cell endoscopic images. Since the data sets of these tasks often face the problem of insufficient labeled data or missing data on rare diseases, the image generation method of the present invention can ensure the generation of pseudo images of the same disease category, thereby reducing the difficulty of training the intestinal image lesion classification model based on deep learning and improving the accuracy of the final classification.

[0094] Embodiment 2 of the present invention

[0095] In this embodiment, a cell endoscopy image generation system is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the terms "module" and "unit" can implement a combination of software and / or hardware for a predetermined function. Although the system described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0096] Figure 4 is a structural block diagram of a cell endoscopic image generation system according to an embodiment of the present invention. Figure 4 As shown, the system includes a data acquisition module 100, an image preprocessing module 200, a pathology information analysis module 300, a model building module 400, a model training module 500, a model testing module 600, and a model application module 700, wherein:

[0097] The data acquisition module 100 is used to obtain the real image of the cell endoscope and the corresponding pathological information, and to construct a training data set and a test data set;

[0098] An image preprocessing module 200 is used to preprocess the real image of the cell endoscope;

[0099] Pathology information analysis module 300, used for text analysis of pathology information;

[0100] A model building module 400 is used to build a cell endoscopy image generation model based on a deep diffusion model based on the processed real image and pathological information;

[0101] A model training module 500 is used to train the model using a training data set;

[0102] The model testing module 600 is used to use the test data set to judge the endoscopic images generated by the trained model, and the qualified one is the final cell endoscopic image generation model;

[0103] The model application module 700 is used to apply the final endoscopic image generation model to the endoscopic image generation work, and inputs the endoscopic image and pathological information. The model generates the endoscopic image of the corresponding pathological type according to the input image and pathological information.

[0104] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0105] An embodiment of the present invention further provides a storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0106] Optionally, in this embodiment, the above-mentioned storage medium may include but is not limited to: a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and other media that can store computer programs.

[0107] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0108] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0109] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for generating a cell endoscopic image, characterized in that: include: Obtain real images of cell endoscopy and corresponding pathological information, and construct training and test datasets; Preprocess the real images of cell endoscopy and perform text analysis on pathological information; Based on the processed real images and pathological information, a cell endoscopy image generation model based on the deep diffusion model was constructed, and the model was trained using a training dataset. The test data set is used to judge the endoscopic images generated by the trained model, and the qualified ones are the final cell endoscopic image generation model; The final cell endoscopic image generation model is applied to the cell endoscopic image generation work, and the cell endoscopic image and pathological information are input. The model generates a cell endoscopic image of the corresponding pathological type according to the input image and pathological information; The text analysis of pathological information includes: pass tokenize The function marks the pathological information and generates a tensor : ;in, T Indicates pathological information. Pathological information T The tensor generated after labeling; The tensor generated Input to the embedding layer of the OpenCLIP text encoding model to generate the embedding vector X , the expression is as follows: ;in, Embedding Indicates embedded processing; Embedding vector Perform position encoding to introduce the position information of each tensor in the sequence, where the position encoding is as follows: ; ;in, Indicates the location and the dimension index is an even number The position encoding is calculated using the sine function. Indicates the location and the dimension index is an odd number The position encoding is calculated using the cosine function. is the position in the sequence, is the dimension index, is the dimension size of the embedding; Encode the position PE With the embedding vector X Add element by element to generate an embedding vector with position information : ; The embedding vector that will add position information As the input of the Transformer layer, after being processed by multiple attention mechanisms and feedforward networks in the Transformer layer, the processed features are intercepted in the penultimate layer to output a high-dimensional semantic embedding vector context .

2. The method for generating a cell endoscopic image according to claim 1, characterized in that: The acquisition of the real image of the cell endoscopy and the corresponding pathological information includes: The patient's endoscopic image provided by the hospital is obtained as a real image, and the pathological diagnosis corresponding to the endoscopic image provided by the doctor is obtained as pathological information, wherein the pathological diagnosis of the endoscopic image includes hyperplastic polyps, sessile serrated lesions, adenomas and invasive carcinomas.

3. The method for generating a cell endoscopic image according to claim 1, characterized in that: The preprocessing of the real image of the cell endoscope comprises: The real images of cell endoscopy were normalized to 512x512 pixels; The standardized image is converted into a NumPy array, and RGB channel conversion and normalization are performed. The normalization formula is: ;in, is the normalized image, is the original image, and are the mean and standard deviation of the image data respectively.

4. The method for generating a cell endoscopic image according to claim 1, characterized in that: The adopting of the training data set to train the model comprises: Use ControlNet to receive the current time step information and encode the time step information into a time embedding vector : ;in, t Indicates the current time step information; Time Embedding Vector Then two linear layers and an activation function are passed to generate the final time embedding : ;in, , are the weights of the first and second linear layers, , They are , The corresponding bias, is the activation function; Extracting main feature maps from preprocessed intracellular microscopy images : ;in, represents the preprocessed cell endoscopy image, Indicates the use of the convolution module in ControlNet from Extract image features; The main feature map Combined with time embedding and the high-dimensional semantic embedding vector obtained through text analysis context Perform further feature extraction and fusion to generate feature maps : ;in, Represents the main feature map , Time Embedding , high-dimensional semantic embedding vector context The three vectors are fused and spliced; Generated feature map Further processing through the middle_block module to obtain the feature map , where the middle_block module includes residual blocks and attention blocks, which are responsible for strengthening the dependencies between features, retaining and strengthening temporal and spatial features, and ensuring that the impact on the output in the early stage of training is minimized through the smoothing operation of the zero convolution layer. The expression is as follows: ;in, represents the residual block operation, represents the attention block operation, represents zero convolution operation; The real image of the cell endoscope is input into the encoder Autoencode. The image is first represented in the latent space by the encoder method. During this process, the encoder generates feature maps for the mean and standard deviation. Then, the feature maps for the mean and standard deviation are converted into latent vectors for sampling through the quant_conv convolution layer. z The latent space representation of The latent space representation is decoded into a generated image through a decoder.

5. The method for generating a cell endoscopic image according to claim 4, characterized in that: Before decoding the latent space representation into the generated image through the decoder, the method further includes sampling the latent vector z Performing denoising processing, wherein the denoising processing includes: For each time step t , generate the corresponding noise image x_noisy t : ;in, is the time step t Corresponding to the noise image, the model will sample the potential vector z Add a time-step-dependent amount of noise to is the time step t The corresponding noise scaling factor, is random noise sampled from a standard normal distribution; The feature map With the current noisy image x_noisy , time step t and high-dimensional semantic embedding vectors context Combine and combine into complete conditional information and pass it to the stable diffusion model; In the forward propagation process of the Stable Diffusion model, more noise is gradually added to the image as time steps advance. In the backward propagation process, the model will be based on the current noisy image. x_noisy , time step t , context , and the feature map passed from ControlNet , calculate and predict the noise of the current step and apply it to the denoising process, where the denoising process formula is expressed as: ;in, is the time step t- 1 corresponds to the noise image, compared to the time step t The noise image at the time has removed some noise and is gradually approaching the final clear image. is the noise of the model prediction at the current step, Representative feature map , high-dimensional semantic embedding vector context Input of conditions; The loss function formula in the denoising process is expressed as follows: ;in, Represents the time step , original image ,noise The expected value of the loss function Used to measure the noise of model predictions The actual noise added By minimizing the gap between them, the model gradually learns to remove noise and thus updates the latent space representation of the image. The latent space representation z after denoising will be used to be decoded into the final image by the decoder.

6. The method for generating a cell endoscopic image according to claim 1, characterized in that: The use of the test data set to judge the endoscopic images generated by the trained model includes: Input the real endoscopic images and corresponding pathological information in the test data set into the trained endoscopic image generation model to obtain the generated endoscopic verification images; The cell endoscopy verification images were compared with the corresponding real images, and quantitative indicators and qualitative evaluation methods were used to evaluate the performance of the cell endoscopy image generation model on the test dataset.

7. The method for generating a cell endoscopic image according to claim 1, characterized in that: The method of applying the final cell endoscopic image generation model to the cell endoscopic image generation work includes: The open source Gradio library is used to create a graphical interface, which is used to allow users to interact with the cell endoscopic image generation model through a web interface. The user uploads the cell endoscopic image, inputs pathological information, and adjusts parameters through the graphical interface, and clicks the generate button on the graphical interface to obtain a new cell endoscopic image generated by the model.

8. The method for generating a cell endoscopic image according to claim 7, characterized in that: After obtaining the new cell endoscopic image generated by the model, the method further comprises: An image saving instruction input by a user is received, and in response to the image saving instruction, a generated new endoscopic image is saved into an Excel file, wherein the Excel file includes pathological information, a real endoscopic image, and a new endoscopic image generated based on the pathological information and the real image.

Images