A method and system for rapid conversion of histological microscopic images

Abstract

The application discloses a kind of quick conversion method and system of tissue microscopic image, it is related to image processing technical field, it is applied to computer device, it include: the tissue microscopic image after being obtained using two kinds of preset fluorescent probe to soak and dye in vivo human tissue;Wherein, tissue microscopic image is the image containing first dyed cell nucleus and first dyed cytoplasm;Image conversion is carried out to tissue microscopic image, to obtain the virtual microscopic image containing second dyed cell nucleus, second dyed cytoplasm;Image conversion includes the image conversion based on preset image conversion model or the image conversion based on image pixel and according to preset channel conversion mode carries out.The conversion process of image quick conversion of tissue microscopic image does not need to change the image quality of original tissue microscopic image, obtains the virtual microscopic image of high imaging contrast, high resolution, realizes quick and accurate tissue structure analysis according to virtual microscopic image, it is convenient to intraoperative quick diagnosis and treatment.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a method and system for rapid conversion of histological microscopic images. BACKGROUND

[0002] In order to achieve the goal of accurate edge evaluation in the operation to clearly define the boundary between healthy tissue and diseased tissue, pathological diagnosis is usually used to examine the tissue. As a traditional intraoperative pathology technique, frozen section (FS) is widely used for intraoperative rapid diagnosis. This technique immediately freezes the resected specimen for sectioning, and then observes it under a microscope after hematoxylin-eosin staining (HE). The advantage of frozen section is that it can generate a relatively clear HE-stained image in a short time, helping surgeons make immediate decisions during surgery. However, the frozen section technique has limited application and the details of the prepared sections are not clear enough, often resulting in cell swelling and deformation, which affects morphological recognition.

[0003] In contrast, in recent years, a system and method for imaging the surface of a histological non-sectioned sample (D.D staining method) disclosed in patent publication CN118311029A has provided a new possibility for intraoperative pathology. The D.D staining method uses two fluorescent probes, DAPI (4', 6-diamidino-2-phenylindole) and DiD (1,1'-dioctadecyl-3,3,3',3'-tetramethylindocyanine, 4-perchlorate), to quickly soak and stain the surface cells of fresh tissue, thereby obtaining a D.D-stained image. Similar to traditional HE-stained images, D.D-stained images can also provide histological information. However, the D.D staining method does not require the preparation of sections, thereby avoiding potential tissue damage during the frozen section process and significantly reducing the preparation time. This provides potential advantages for intraoperative rapid diagnosis.

[0004] However, the clinical application of the D.D staining technique still faces the following challenges: HE staining is the gold standard for pathology worldwide, and based on its extensive clinical research and literature support, using HE-stained images for diagnosis can ensure the standardization and consistency of diagnostic results. As an emerging technology, the D.D staining method needs a lot of research and time to establish the same standardized diagnostic system. Pathologists need time to adapt to the new staining modality and learn how to identify pathological features in D.D-stained images, which may affect diagnostic efficiency in the early stage.

[0005] In summary, how to achieve the image information in the conventional HE-stained image from the rapidly obtained D.D-stained image in the intraoperative scenario and reduce the image preparation and generation time is a technical problem to be solved in the field. Summary of the Invention

[0006] In view of this, the purpose of this invention is to provide a method and system for rapid conversion of tissue microscopic images, which can, in intraoperative scenarios, extract image information from rapidly acquired DD-stained images into conventional HE-stained images, reducing image preparation and generation time. The specific solution is as follows:

[0007] In a first aspect, this application discloses a method for rapid conversion of tissue microscopic images, applied to a computer device, comprising:

[0008] Obtain tissue microscopic images after intraoperative human tissue has been immersed and stained using two preset fluorescent probes; wherein, the tissue microscopic images are images containing the nuclei and cytoplasm of the first stained cells;

[0009] The tissue microscopic image is converted to obtain a virtual microscopic image containing the nucleus and cytoplasm of the second stained cells; the image conversion includes image conversion based on a preset image conversion model or image conversion based on image pixels and according to a preset channel conversion method.

[0010] Optionally, before performing image conversion on the tissue micrograph based on the preset image conversion model, the method further includes:

[0011] Paired images constructed using historical tissue microscopic images and corresponding historical HE staining images were used as the data training set;

[0012] The initial image conversion model is trained using the data training set so that the initial image conversion model learns the first mapping relationship of color and texture features between the historical tissue microscopic image and the historical HE staining image in each of the paired images, the second mapping relationship of cell structure and tissue morphology, and the third mapping relationship between the dark field style of the historical tissue microscopic image and the bright field style of the historical HE staining image, so as to obtain the preset image conversion model.

[0013] Optionally, before training the initial image conversion model using the data training set, so that the initial image conversion model learns the first mapping relationship of color and texture features between the historical tissue micrograph and the historical HE staining image in each of the paired images, the second mapping relationship of cell structure and tissue morphology, and the third mapping relationship between the dark field style of the historical tissue micrograph and the bright field style of the historical HE staining image, to obtain the preset image conversion model, the method further includes:

[0014] Construct an initial image transformation model based on a GAN architecture, which includes a generator and a discriminator; wherein the generator includes a first generator and a second generator, and the discriminator includes a first discriminator and a second discriminator.

[0015] Optionally, the initial image conversion model is trained using the data training set so that the initial image conversion model learns the first mapping relationship of color and texture features between the historical tissue microscopic image and the historical HE staining image in each of the paired images, the second mapping relationship of cell structure and tissue morphology, and the third mapping relationship between the dark field style of the historical tissue microscopic image and the bright field style of the historical HE staining image, to obtain a preset image conversion model, including:

[0016] The paired image is input to the first generator so that the first generator can capture the historical tissue color features, historical tissue texture features, historical tissue cell structure, historical tissue morphology and historical tissue brightness information of the historical tissue microscopic image, and perform mapping processing with the corresponding feature information in the corresponding historical HE staining image to obtain the first mapping relationship, the second mapping relationship and the third mapping relationship, and output the generated target HE staining image.

[0017] The historical HE staining image carrying a real image label and the target HE staining image carrying a generated image label are input to the first discriminator so that the first discriminator learns the real image features in each of the historical HE staining images and the generated image features in the target HE staining image, so as to obtain a first target discriminator for predicting whether the target HE staining image output by the first generator is a real image.

[0018] The target HE-stained image is input into the second generator so that the second generator can capture the target tissue color features, target tissue texture features, target tissue cell structure, target tissue morphology and target HE-stained image brightness information of the target HE-stained image, and perform mapping processing with the corresponding feature information in the historical tissue microscopic image to obtain the updated first mapping relationship, second mapping relationship and third mapping relationship, and output the generated target tissue microscopic image.

[0019] The historical tissue microscopic images carrying real image labels and the target tissue microscopic images carrying generated image labels are input to the second discriminator, so that the second discriminator learns the real image features in each of the historical tissue microscopic images and the generated image features in the target tissue microscopic images, so as to obtain a second target discriminator for predicting whether the target tissue microscopic image output by the second generator is a real image.

[0020] Optionally, the rapid conversion method for tissue micrographs further includes:

[0021] Construct an adversarial loss function to constrain adversarial training between the generator and the discriminator;

[0022] Construct a cyclic consistency loss function to constrain the consistency between the target tissue microscopic image and the historical tissue microscopic image output by the generator, and the consistency between the target HE staining image and the historical HE staining image;

[0023] Construct a mapping loss function to constrain the content difference features between the target HE-stained image and the historical tissue micrograph, and the content difference features between the target tissue micrograph and the historical HE-stained image output by the generator;

[0024] The overall model loss function for the initial image conversion model training process is constructed based on the adversarial loss function, the cycle consistency loss function, the mapping loss function, and the corresponding function weight parameters.

[0025] Optionally, the construction of the cyclic consistency loss function for constraining the consistency between the target tissue microscopic image and historical tissue microscopic images output by the generator, and the consistency between the target HE staining image and historical HE staining images, includes:

[0026] Calculate the first brightness contrast factor and the first color-structure contrast factor between the target tissue microscopic image and the historical tissue microscopic image, and the second brightness contrast factor and the second color-structure contrast factor between the target HE-stained image and the historical HE-stained image, respectively;

[0027] The cycle consistency loss function is constructed based on the first brightness contrast factor, the first color-structure contrast factor, the second brightness contrast factor, and the second color-structure contrast factor.

[0028] Optionally, the step of performing image conversion on the tissue microscopic image to obtain a virtual microscopic image containing the nuclei and cytoplasm of the second stained cells includes:

[0029] The tissue microscopic image is separated into RGB channels using a preset image processing tool to obtain the separated first tissue microscopic image, second tissue microscopic image, and third tissue microscopic image.

[0030] The first tissue microscopic image, the second tissue microscopic image, and the third tissue microscopic image are respectively calculated and converted pixel by pixel according to the corresponding RGB channel operation algorithm to obtain the conversion results of each image;

[0031] The image conversion results are combined to obtain a virtual microscopic image containing the nucleus and cytoplasm of the second stained cell.

[0032] Optionally, the tissue micrograph is a dark field image, and the region of the first stained cell nucleus in the tissue micrograph is blue, and the region of the first stained cytoplasm is red.

[0033] Optionally, the virtual microscopic image is a bright-field image, and the region of the second stained cell nucleus in the virtual microscopic image is blue, while the region of the second stained cytoplasm is pink.

[0034] Secondly, this application discloses a rapid conversion system for tissue microscopic images, applied to a computer device, comprising:

[0035] The image acquisition module is used to acquire tissue microscopic images after the human tissue has been immersed and stained using two preset fluorescent probes during surgery; wherein the tissue microscopic image is an image containing the nucleus and cytoplasm of the first stained cell.

[0036] The image conversion module is used to perform image conversion on the tissue microscopic image to obtain a virtual microscopic image containing the nucleus and cytoplasm of the second stained cells; the image conversion includes image conversion based on a preset image conversion model or image conversion based on image pixels and according to a preset channel conversion method.

[0037] Thirdly, this application discloses an electronic device, including:

[0038] Memory, used to store computer programs;

[0039] A processor for executing the computer program to implement the steps of the aforementioned disclosed method for rapid conversion of tissue micrographs.

[0040] Fourthly, this application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the steps of the aforementioned disclosed method for rapidly converting tissue microscopic images.

[0041] As can be seen, this application discloses a rapid conversion method for tissue microscopic images, applied to a computer device, comprising: acquiring a tissue microscopic image after intraoperative human tissue has been soaked and stained using two preset fluorescent probes; wherein the tissue microscopic image is an image containing the nuclei and cytoplasm of first stained cells; performing image conversion on the tissue microscopic image to obtain a virtual microscopic image containing the nuclei and cytoplasm of second stained cells; the image conversion includes image conversion based on a preset image conversion model or image conversion based on image pixels and according to a preset channel conversion method. Therefore, compared to obtaining the corresponding stained image through traditional HE staining, this invention, through the rapid image conversion process of the tissue microscopic image, does not alter the image quality of the original tissue microscopic image, obtaining a virtual microscopic image with high imaging contrast and high resolution. Furthermore, the overall time for acquiring the virtual microscopic image is shorter than that of traditional HE staining images, while the imaging quality of the virtual microscopic image is higher than that of traditional HE staining images. Thus, in intraoperative detection scenarios, rapid and accurate tissue structure analysis can be achieved based on the virtual microscopic image, facilitating rapid intraoperative diagnosis and treatment. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0043] Figure 1 This is a flowchart of a method for rapid conversion of tissue microscopic images disclosed in this application;

[0044] Figure 2 This is a diagram illustrating the model training process of a preset image conversion model disclosed in this application;

[0045] Figure 3(a) shows a global tissue micrograph, Figure 3(b) shows a global virtual micrograph, Figure 3(c) shows a global reference image, Figure 3(d) shows a locally magnified tissue micrograph, Figure 3(e) shows a locally magnified virtual micrograph, and Figure 3(f) shows a locally magnified reference image.

[0046] Figure 4 This is a flowchart of the method for converting tissue microscopic images via RGB channels disclosed in this application;

[0047] Figure 5 This is a schematic diagram of the structure of a rapid conversion system for tissue microscopic images disclosed in this application;

[0048] Figure 6This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0049] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0050] To achieve accurate intraoperative margin assessment and clearly define the boundary between healthy and diseased tissue, pathological diagnosis is typically used to examine the tissue. Frozen sections, as a traditional intraoperative pathology technique, are widely used for rapid intraoperative diagnosis. This technique involves immediately freezing and sectioning the excised specimen, staining it with hematoxylin and eosin, and then observing it under a microscope. The advantage of frozen sections is that they can generate relatively clear HE-stained images in a short time, helping surgeons make immediate decisions during surgery. However, frozen section technology has limited applicability, the resulting sections are not always clear enough, and cell swelling and deformation often occur, affecting morphological identification.

[0051] In contrast, recent years have seen the emergence of a system and method for imaging the surface of histological samples without sections (DD staining), as disclosed in patent publication number CN118311029A, offering new possibilities for intraoperative pathology. DD staining uses two fluorescent probes, DAPI and DiD, to rapidly immerse and stain cells on the surface of fresh tissue, thereby obtaining DD-stained images. Similar to traditional HE staining images, DD-stained images can also provide histological information. However, DD staining does not require section preparation, thus avoiding tissue damage that may occur during frozen sectioning and significantly reducing preparation time. This offers a potential advantage for rapid intraoperative diagnosis.

[0052] Despite this, the clinical application of DD staining technology still faces the following challenges: HE staining is the gold standard in pathology globally, and its extensive clinical research and literature support ensure the standardization and consistency of diagnostic results. As an emerging technology, DD staining requires significant research and time to establish the same standardized diagnostic system. Pathologists need time to adapt to the new staining modality and learn how to identify pathological features in DD-stained images, which may initially affect diagnostic efficiency.

[0053] To this end, the present invention provides a rapid conversion scheme for tissue microscopic images, which enables the acquisition of image information from conventional HE staining images from rapidly acquired DD staining images during surgery, thereby reducing image preparation and generation time.

[0054] Reference Figure 1 As shown, this embodiment of the invention discloses a method for rapid conversion of tissue microscopic images, applied to a computer device, comprising:

[0055] Step S11: Obtain tissue microscopic images after immersion and staining of human tissue during surgery using two preset fluorescent probes; wherein the tissue microscopic images are images containing the nuclei and cytoplasm of the first stained cells.

[0056] In this embodiment, rapid immersion staining (hereinafter referred to as DD staining) of fresh tissue surface cells was performed using two fluorescent probes: DAPI (4',6-diamidinyl-2-phenylindole) and DiD (1,1'-bis(octadecyl-3,3,3',3'-tetramethylindocyanine, 4-perchlorate). DAPI binds to the cell nucleus, while DiD binds to the cell membrane. Under 365 nm laser excitation, DAPI-bound cell nuclei emit blue light, while under 640 nm laser excitation, DiD-bound cell membranes emit red light. A fluorescence microscopy imaging system, based on scattering imaging and depth-of-field extension, performed non-destructive imaging of the stained fresh tissue surface, resulting in a tissue microscopic image. This tissue microscopic image is a dark-field image, with the region of the first stained cell nucleus appearing blue and the region of the first stained cytoplasm appearing red. It should be noted that the total time for preparing and imaging the tissue microscopic image is less than 10 minutes, indicating a short acquisition time.

[0057] Step S12: Perform image conversion on the tissue microscopic image to obtain a virtual microscopic image containing the nucleus and cytoplasm of the second stained cells; the image conversion includes image conversion based on a preset image conversion model or image conversion based on image pixels and according to a preset channel conversion method.

[0058] In this embodiment, the obtained tissue microscopic image is converted. The specific image conversion method can be: to perform image conversion using a preset image conversion model trained specifically for converting tissue microscopic images to virtual microscopic images, or to perform image conversion based on the image pixels in the tissue microscopic image according to a preset channel conversion method.

[0059] In one specific embodiment, if a preset image conversion model is used for image conversion, then the preset image conversion model needs to be trained. Before converting the tissue microscopic image based on the preset image conversion model, the process further includes: using paired images constructed from historical tissue microscopic images and corresponding historical HE-stained images as a data training set; using the data training set to train the initial image conversion model, so that the initial image conversion model learns the first mapping relationship of color and texture features between the historical tissue microscopic images and the historical HE-stained images in each of the paired images, the second mapping relationship of cell structure and tissue morphology, and the third mapping relationship between the dark field style of the historical tissue microscopic images and the bright field style of the historical HE-stained images, to obtain the preset image conversion model. It is understood that, firstly, a data training set is constructed. Specifically, the samples used in this embodiment of the invention are from human colon cancer tissue. DD staining uses fluorescence dark field imaging, using 365nm and 640nm lasers as light sources; HE staining uses bright field white light for illumination. The stained samples are imaged using an Olympus microscope under a 10x objective lens. The process of constructing the dataset includes, but is not limited to: acquiring DD staining images, acquiring HE staining images, and performing necessary image preprocessing. Specifically, freshly excised tissue samples, while kept moist, are rapidly soaked and stained with DAPI and DiD to remove surface cells, and then mounted. A suitable microscope and laser light source system is then set up, and the prepared tissue samples are scanned and imaged at the optimal imaging power of 365nm and 640nm lasers. Finally, a stitching algorithm is used to obtain large-size DD staining images. Freshly excised tissue samples are first fixed in formalin to preserve tissue structure, then dehydrated, paraffin-impregnated, and embedded to form paraffin blocks. Next, the paraffin blocks are sectioned and attached to glass slides, followed by dewaxing and hydration. After these preprocessing steps, the sections are stained with hematoxylin and eosin sequentially. After staining, the tissue samples are mounted. Finally, the mounted samples are observed under a microscope to obtain HE staining images. A total of 414 WSI (Whole Slide Image) images were acquired using the two steps described above. Among them, 214 WSI images were HE-colored, and 200 were DD-colored. To enable the obtained colored images to be used for deep learning model training, the WSI images were cut into 256-pixel * 256-pixel image blocks with a 100-pixel overlap during cutting. The obtained image blocks were then processed to remove excessive background.The final constructed DD-HE unpaired image dataset comprises 11,600 DD-stained image patches and 11,600 HE-stained image patches. 9,000 DD-stained image patches (historical tissue microscopic images) and 2,600 HE-stained image patches (historical HE-stained images) are used as the training set, and 2,600 are used as the validation set. The initial image conversion model (GAN architecture) is then trained using the training set. This allows the initial image conversion model to learn the first mapping relationship between color and texture features of cell nuclei and cytoplasm in historical tissue microscopic images and historical HE-stained images, the second mapping relationship between cell structure and tissue morphology, and the third mapping relationship between the dark field style of historical tissue microscopic images and the bright field style of historical HE-stained images, thus obtaining the preset image conversion model.

[0060] Specifically, before training the initial image conversion model using the data training set to learn the first mapping relationship of color and texture features between the historical tissue micrograph and the historical HE staining image in each of the paired images, the second mapping relationship of cell structure and tissue morphology, and the third mapping relationship between the dark field style of the historical tissue micrograph and the bright field style of the historical HE staining image, to obtain the preset image conversion model, the method further includes: constructing an initial image conversion model based on a GAN architecture containing a generator and a discriminator; wherein the generator includes a first generator and a second generator, and the discriminator includes a first discriminator and a second discriminator. It is understood that the GAN architecture is used to learn the staining conversion from a DD staining image to its equivalent bright field HE staining image. GANs can output clear and realistic images through adversarial training between the generator (G) and the discriminator (D). The generator G aims to deceive the discriminator D by generating fake images that resemble the target image. The discriminator D's goal is to distinguish whether the received image is a generated image or a real image, making it more suitable for performing unpaired DD staining to HE staining conversion tasks.

[0061] In this embodiment, the initial image conversion model is trained using the data training set so that the initial image conversion model learns the first mapping relationship of color and texture features between the historical tissue microscopic images and the historical HE staining images in each of the paired images, the second mapping relationship of cell structure and tissue morphology, and the third mapping relationship between the dark field style of the historical tissue microscopic images and the bright field style of the historical HE staining images, to obtain a preset image conversion model. This includes: inputting the paired images into the first generator to capture the historical tissue color features, historical tissue texture features, historical tissue cell structure, historical tissue morphology, and historical tissue microscopic image brightness information of the historical tissue microscopic images, and mapping these features with the corresponding feature information in the corresponding historical HE staining images to obtain the first, second, and third mapping relationships, and outputting the generated target HE staining image; inputting the historical HE staining image carrying the real image label and the target HE staining image carrying the generated image label into the first discriminator so that the first discriminator learns the first mapping relationship between the dark field style of the historical tissue microscopic images and the bright field style of the historical HE staining images, to obtain a preset image conversion model. The real image features in the historical HE-stained images and the generated image features in the target HE-stained image are used to obtain a first target discriminator for predicting whether the target HE-stained image output by the first generator is a real image. The target HE-stained image is input to the second generator to capture the target tissue color features, target tissue texture features, target tissue cell structure, target tissue morphology, and target HE-stained image brightness information of the target HE-stained image, and map them with the corresponding feature information in the historical tissue microscopic images to obtain updated first, second, and third mapping relationships, and output the generated target tissue microscopic image. The historical tissue microscopic images carrying real image labels and the target tissue microscopic images carrying generated image labels are input to the second discriminator so that the second discriminator learns the real image features in each of the historical tissue microscopic images and the generated image features in the target tissue microscopic image to obtain a second target discriminator for predicting whether the target tissue microscopic image output by the second generator is a real image. It is understood that, referring to Figure 2As shown, this invention provides a specific training process for a preset image conversion model. The initial image conversion model is trained using historical tissue microscopic images (DD-stained images) and historical HE-stained images, both carrying real labels, from a training dataset. Specifically, the initial image conversion model includes a generator, a first generator for converting DD-stained images to HE-stained images, a first discriminator for determining whether the HE images generated by the first generator are real images, a second generator for converting HE-stained images to DD-stained images, and a second discriminator for determining whether the DD images generated by the second generator are real images. Specifically, the DD-stained image distribution is considered the source domain (X domain), the HE-stained image distribution is considered the target domain (Y domain), and the target of CycleGAN is a given edge distribution. Learning conditional mapping and The CycleGAN architecture contains two generators ( and ) and two discriminators ( and The specific training process is as follows: given input (A DD-stained image, i.e., a historical tissue micrograph), x passes through the first generator. Obtain the generated image in the Y domain (can be represented as) ), that is, the target HE-stained image, Then through the second generator Obtain the reconstructed image in the X domain (can be represented as) ), that is, a microscopic image of the target tissue; similarly, for (A single HE-stained image, i.e., a historical HE-stained image, can be processed through a similar procedure) , ), to obtain the generated image in the X domain. Reconstructed image of the Y domain The generated image and the real image are fed into a discriminator in the corresponding domain to train the discriminator's ability to distinguish between real and fake images.

[0062] In this embodiment, an adversarial loss function is constructed to constrain the adversarial training between the generator and the discriminator; a cyclic consistency loss function is constructed to constrain the consistency between the target tissue microscopic image and historical tissue microscopic images output by the generator, and the consistency between the target HE-stained image and historical HE-stained images; a mapping loss function is constructed to constrain the content difference features between the target HE-stained image and historical tissue microscopic images output by the generator, and the content difference features between the target tissue microscopic image and historical HE-stained images; and the overall model loss function for the initial image conversion model training process is constructed based on the adversarial loss function, the cyclic consistency loss function, the mapping loss function, and the corresponding function weight parameters. It can be understood that after the model architecture design is completed, further loss functions are constructed to constrain and optimize the model training; specifically, the adversarial loss function... For adversarial training between the generator and the discriminator, it is defined as follows: ;

[0063] in, Each represents the mathematical expectation of the sample image; This represents the HE image obtained after the real DD image x is processed by the first generator. , Represents a real HE image The DD image obtained after the second generator For the first generator Its goal is to generate a generator that can fool the first discriminator. The image, and the second generator The goal is to generate a generator that can fool the second discriminator. The image. The purpose of the first and second discriminators is to accurately distinguish between real and generated images.

[0064] Cyclic consistency loss function The constraint used to maintain consistency between the generator's output and the original input image is defined as follows: ;

[0065] in, This represents the Manhattan Distance, also known as the L1 distance. This represents the reconstructed image of the input DD-stained image. It is also a reconstructed image of the input HE image. The goal is for the generator to be able to transform the input image from domain X to domain Y, and the generated image to be transformed back to domain X, thus maintaining cyclic consistency with the original input.

[0066] Ontology mapping loss function The generator is encouraged to perform a reverse mapping from the input image to its corresponding target domain image, and then map it back to the input domain. The aim of this process is to preserve as many content features as possible from the original image, ensuring that the generated image is highly similar to the input image. This is defined as follows: ;

[0067] Then, based on the adversarial loss function, the cycle consistency loss function, the mapping loss function, and their respective pre-configured function weight parameters, the total loss function for the model training process is constructed.

[0068] It is important to note that These are the loss functions used in the original CycleGAN. They all compare the differences between two images pixel by pixel using Manhattan distance, but ignore information such as color, texture, and structure, which are important references for pathologists in making diagnoses. Therefore, MS-SSIM (Multiscale Structural Similarity) and L1 hybrid loss are introduced. Traditional cycle consistency loss Enhancements are made to maintain the cyclic invariance of brightness, color, and structure.

[0069] Specifically, the construction of a cyclic consistency loss function to constrain the consistency between the target tissue microscopic image and historical tissue microscopic images output by the generator, and the consistency between the target HE-stained image and historical HE-stained images, includes: calculating a first brightness contrast factor and a first color-structure contrast factor between the target tissue microscopic image and the historical tissue microscopic image, and a second brightness contrast factor and a second color-structure contrast factor between the target HE-stained image and the historical HE-stained image; and constructing the cyclic consistency loss function based on the first brightness contrast factor, the first color-structure contrast factor, the second brightness contrast factor, and the second color-structure contrast factor. It is understood that for MS-SSIM, the brightness contrast factor is first calculated and defined. and color-structure contrast factor The definition is as follows: ; ;

[0070] in, This represents the window corresponding to the two images. They represent Mean pixel intensity express Variance of pixel intensity express covariance of pixel intensity These are two constants to avoid a denominator of 0. It should be noted that, based on the brightness contrast factor and color-structure contrast factor defined above, the first brightness contrast factor and the first color-structure contrast factor characterizing the DD image, and the second brightness contrast factor and the second color-structure contrast factor characterizing the HE image, are obtained respectively.

[0071] MS-SSIM is calculated based on the luminance contrast factor and the color-structure contrast factor, and is defined as follows: ;

[0072] in, Indicates different scales. All are represented as constants. Then, define... and : ; ;

[0073] Where, is a coefficient used to adjust the MS-SSIM loss and cycle consistency loss. The proportion.

[0074] Therefore, the total loss function of the initial image conversion model training process is expressed as follows: ;

[0075] in, These are hyperparameters used to configure the weights of the loss function.

[0076] For training the improved CycleGAN, the settings of this invention are as follows: MS-SSIM loss... , In Total loss function In The initial learning rate was set to 0.0002, and the number of training epochs was 100. The network weights were initialized using a Gaussian distribution with a mean of 0 and a variance of 0.0004. The Adam optimizer was used, where betas = (0.5, 0.999).

[0077] The trained model is validated using a data validation set, and the validated model is used as the preset image conversion model.

[0078] The tissue microscopic image to be converted is input into a preset image conversion model so that the corresponding HE image can be directly converted by the preset image conversion model and output as a virtual microscopic image.

[0079] As shown in Figure 3, Figure 3(a) shows a tissue microscopic image (DD staining image) obtained by immersing and staining human tissue during surgery using the two preset fluorescent probes mentioned in this invention. Figure 3(d) shows the local tissue microscopic image circled in yellow in Figure 3(a), where red cell membranes or cytoplasm and blue cell nuclei are visible against a black background. Figure 3(b) is a bright-field virtual microscopic image (virtual HE staining image) obtained after image conversion of Figure 3(a). Figure 3(e) represents the local virtual microscopic image circled in yellow in Figure 3(b), where HE image information (pink cytoplasm and blue cell nuclei) against a white background, consistent with what researchers typically see, is presented. Figure 3(c) shows the same tissue site treated using traditional techniques (formalin-fixed paraffin sectioning). The reference HE-stained image obtained by paraffin-embedding (FFPE) is shown in Figure 3(f), which represents the local reference HE-stained image highlighted in yellow in Figure 3(c). By comparison, it can be seen that the DD-stained image is similar in detail to the traditional HE-stained image. Especially after generating a virtual HE-stained image using an image conversion model, it can effectively restore the main features of HE staining and has the advantage of high contrast. This indicates that the DD staining method has potential for rapid intraoperative diagnosis, especially when a sectioning step is not required, as image conversion technology can still obtain image information similar to HE staining.

[0080] In another specific implementation, refer to Figure 4 As shown, the tissue microscopic image is subjected to RGB channel separation using a preset image processing tool to obtain a first tissue microscopic image, a second tissue microscopic image, and a third tissue microscopic image. The first, second, and third tissue microscopic images are then calculated and converted pixel-by-pixel according to their respective RGB channel algorithms to obtain conversion results. These conversion results are then combined to obtain a virtual microscopic image containing the nuclei and cytoplasm of the second stained cells. It can be understood that by loading the tissue microscopic image using an image processing tool and performing pixel-by-pixel color separation of the R (Red), G (Green), and B (Blue) channels, the resulting images are obtained pixel-by-pixel. , , If the tissue micrograph is an 8-bit image, the image conversion is performed according to the following formula: ; ; ;

[0081] in, , , These are the channel conversion results for the B, G, and R channels of the converted HE image, respectively. Then, [the following is a description of the process:] , , Image synthesis processing is performed to obtain a converted HE image, which is then output as a virtual microscopic image. If the tissue microscopic image is 16-bit, then 255 in the formula is replaced with 65535. It should be noted that after obtaining the converted HE image, the color can be further optimized by adjusting the white balance to obtain a processed HE image, which is then output as a virtual microscopic image. The virtual microscopic image is a bright-field image, and the region of the second stained cell nucleus in the virtual microscopic image is blue, while the region of the second stained cytoplasm is pink.

[0082] In this way, the image quality (high resolution, high imaging contrast) of DD-stained images can be preserved through the above image conversion. Furthermore, since the preparation and acquisition time of DD-stained images is less than 10 minutes, and the subsequent conversion to virtual microscopic images can be completed within 5 minutes, the overall time for acquiring virtual microscopic images is less than 15 minutes. In intraoperative scenarios, high-quality virtual microscopic images can be provided in a short time, allowing for the presentation of richer pathological information from the virtual microscopic images based on the high contrast characteristics of DD-stained images. In contrast, traditional intraoperative pathology techniques (such as frozen sections) require approximately 20 to 30 minutes of complex procedures for fixing, sectioning, and staining to acquire HE images. Moreover, their applicability is limited, and it is difficult to guarantee staining quality, restricting their application in certain situations.

[0083] As can be seen, this application discloses a rapid conversion method for tissue microscopic images, applied to a computer device, comprising: acquiring a tissue microscopic image after intraoperative human tissue has been soaked and stained using two preset fluorescent probes; wherein the tissue microscopic image is an image containing the nuclei and cytoplasm of first stained cells; performing image conversion on the tissue microscopic image to obtain a virtual microscopic image containing the nuclei and cytoplasm of second stained cells; the image conversion includes image conversion based on a preset image conversion model or image conversion based on image pixels and according to a preset channel conversion method. Therefore, compared to obtaining the corresponding stained image through traditional HE staining, this invention, through the rapid image conversion process of the tissue microscopic image, does not alter the image quality of the original tissue microscopic image, obtaining a virtual microscopic image with high imaging contrast and high resolution. Furthermore, the overall time for acquiring the virtual microscopic image is shorter than that of traditional HE staining images, while the imaging quality of the virtual microscopic image is higher than that of traditional HE staining images. Thus, in intraoperative detection scenarios, rapid and accurate tissue structure analysis can be achieved based on the virtual microscopic image, facilitating rapid intraoperative diagnosis and treatment.

[0084] Reference Figure 5 As shown, the present invention also discloses a rapid conversion system for tissue microscopic images, applied to a computer device, comprising:

[0085] The image acquisition module 11 is used to acquire tissue microscopic images after the human tissue has been soaked and stained with two preset fluorescent probes during surgery; wherein, the tissue microscopic image is an image containing the nucleus and cytoplasm of the first stained cell;

[0086] The image conversion module 12 is used to perform image conversion on the tissue microscopic image to obtain a virtual microscopic image containing the nucleus of the second stained cell and the cytoplasm of the second stained cell; the image conversion includes image conversion based on a preset image conversion model or image conversion based on image pixels and according to a preset channel conversion method.

[0087] As can be seen, this application discloses obtaining tissue microscopic images after intraoperative immersion and staining of human tissue using two preset fluorescent probes; wherein, the tissue microscopic image is an image containing the nuclei and cytoplasm of first stained cells; the tissue microscopic image is converted to obtain a virtual microscopic image containing the nuclei and cytoplasm of second stained cells; the image conversion includes image conversion based on a preset image conversion model or image conversion based on image pixels and according to a preset channel conversion method. Therefore, compared to obtaining the corresponding stained image through traditional HE staining, this invention obtains a high-contrast, high-resolution virtual microscopic image by rapidly converting the tissue microscopic image without altering the image quality of the original tissue microscopic image. Furthermore, the overall time for obtaining the virtual microscopic image is shorter than that of traditional HE staining images, while the imaging quality of the virtual microscopic image is higher than that of traditional HE staining images. Thus, in intraoperative detection scenarios, rapid and accurate tissue structure analysis can be achieved based on the virtual microscopic image, facilitating rapid intraoperative diagnosis and treatment.

[0088] Furthermore, embodiments of this application also disclose an electronic device, Figure 6 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.

[0089] Figure 6 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the rapid conversion method for tissue microscopic images disclosed in any of the foregoing embodiments. Alternatively, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0090] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0091] The processor 21 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 21 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 21 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 21 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 21 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0092] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0093] The operating system 221 manages and controls the various hardware devices and computer programs 222 on the electronic device 20 to enable the processor 21 to perform calculations and processing on the massive amounts of data 223 in the memory 22. It can be Windows Server, Netware, Unix, Linux, etc. The computer program 222, in addition to including a computer program capable of performing the rapid conversion method of tissue microscopic images executed by the electronic device 20 as disclosed in any of the foregoing embodiments, may further include computer programs capable of performing other specific tasks. The data 223 may include data received by the electronic device from external devices, as well as data collected by its own input / output interface 25.

[0094] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned method for rapid conversion of tissue microscopic images. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0095] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0096] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application. The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly in hardware, software modules executed by a processor, or a combination of both. The software module may be located in random access memory (RAM), memory, read-only memory (ROM), electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, removable disks, CD-ROMs (Compact Disc-Read Only Memory), or any other form of storage medium known in the art.

[0097] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0098] The solution provided by the present invention has been described in detail above. Specific examples have been used to illustrate the principle and implementation of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation and application scope based on the idea of ​​the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for rapid conversion of tissue microscopic images, characterized in that, Applied to computer devices, including: Obtain tissue microscopic images after intraoperative human tissue has been immersed and stained using two preset fluorescent probes; wherein, the tissue microscopic images are images containing the nuclei and cytoplasm of the first stained cells; Paired images constructed using historical tissue microscopic images and corresponding historical HE staining images were used as the data training set; An initial image transformation model based on a GAN architecture is constructed, which includes a generator and a discriminator; wherein the generator includes a first generator and a second generator, and the discriminator includes a first discriminator and a second discriminator; The paired image is input to the first generator so that the first generator can capture the historical tissue color features, historical tissue texture features, historical tissue cell structure, historical tissue morphology and historical tissue brightness information of the historical tissue microscopic image, and perform mapping processing with the corresponding feature information in the corresponding historical HE staining image to obtain the first mapping relationship, the second mapping relationship and the third mapping relationship, and output the generated target HE staining image. The historical HE staining image carrying a real image label and the target HE staining image carrying a generated image label are input to the first discriminator so that the first discriminator learns the real image features in each of the historical HE staining images and the generated image features in the target HE staining image, so as to obtain a first target discriminator for predicting whether the target HE staining image output by the first generator is a real image. The target HE-stained image is input into the second generator so that the second generator can capture the target tissue color features, target tissue texture features, target tissue cell structure, target tissue morphology and target HE-stained image brightness information of the target HE-stained image, and perform mapping processing with the corresponding feature information in the historical tissue microscopic image to obtain the updated first mapping relationship, second mapping relationship and third mapping relationship, and output the generated target tissue microscopic image. The historical tissue microscopic image carrying a real image label and the target tissue microscopic image carrying a generated image label are input to the second discriminator so that the second discriminator learns the real image features in each of the historical tissue microscopic images and the generated image features in the target tissue microscopic image, so as to obtain a second target discriminator for predicting whether the target tissue microscopic image output by the second generator is a real image. The tissue microscopic image is converted to obtain a virtual microscopic image containing the nucleus and cytoplasm of the second stained cells; the image conversion includes image conversion based on a preset image conversion model or image conversion based on image pixels and according to a preset channel conversion method.

2. The rapid conversion method for tissue microscopic images according to claim 1, characterized in that, Also includes: Construct an adversarial loss function to constrain adversarial training between the generator and the discriminator; Construct a cyclic consistency loss function to constrain the consistency between the target tissue microscopic image and the historical tissue microscopic image output by the generator, and the consistency between the target HE staining image and the historical HE staining image; Construct a mapping loss function to constrain the content difference features between the target HE-stained image and the historical tissue micrograph, and the content difference features between the target tissue micrograph and the historical HE-stained image output by the generator; The overall model loss function for the initial image conversion model training process is constructed based on the adversarial loss function, the cycle consistency loss function, the mapping loss function, and the corresponding function weight parameters.

3. The rapid conversion method for tissue microscopic images according to claim 2, characterized in that, The cyclic consistency loss function, constructed to constrain the consistency between the target tissue microscopic image and historical tissue microscopic images output by the generator, and the consistency between the target HE-stained image and historical HE-stained images, includes: Calculate the first brightness contrast factor and the first color-structure contrast factor between the target tissue microscopic image and the historical tissue microscopic image, and the second brightness contrast factor and the second color-structure contrast factor between the target HE-stained image and the historical HE-stained image, respectively; The cycle consistency loss function is constructed based on the first brightness contrast factor, the first color-structure contrast factor, the second brightness contrast factor, and the second color-structure contrast factor.

4. The rapid conversion method for tissue microscopic images according to claim 1, characterized in that, The step of image conversion of the tissue microscopic image to obtain a virtual microscopic image containing the nuclei and cytoplasm of the second stained cells includes: The tissue microscopic image is separated into RGB channels using a preset image processing tool to obtain the separated first tissue microscopic image, second tissue microscopic image, and third tissue microscopic image. The first tissue microscopic image, the second tissue microscopic image, and the third tissue microscopic image are respectively calculated and converted pixel by pixel according to the corresponding RGB channel operation algorithm to obtain the conversion results of each image; The image conversion results are combined to obtain a virtual microscopic image containing the nucleus and cytoplasm of the second stained cell.

5. The rapid conversion method for tissue microscopic images according to claim 1, characterized in that, The tissue micrograph is a dark field image, and the region of the first stained cell nucleus in the tissue micrograph is blue, while the region of the first stained cytoplasm is red.

6. The rapid conversion method for tissue microscopic images according to claim 1, characterized in that, The virtual microscopic image is a bright-field image, and the region of the second stained cell nucleus in the virtual microscopic image is blue, while the region of the second stained cytoplasm is pink.

7. A rapid conversion system for tissue microscopic images, characterized in that, Applied to computer devices, including: The image acquisition module is used to acquire tissue microscopic images after the human tissue has been immersed and stained using two preset fluorescent probes during surgery; wherein the tissue microscopic image is an image containing the nucleus and cytoplasm of the first stained cell. The rapid conversion system for tissue microscopic images is further used to construct a paired image set using historical tissue microscopic images and corresponding historical HE-stained images; construct an initial image conversion model based on a GAN architecture, including a generator and a discriminator; wherein the generator includes a first generator and a second generator, and the discriminator includes a first discriminator and a second discriminator; the paired image is input to the first generator so that the first generator can capture the historical tissue color features, historical tissue texture features, historical tissue cell structure, historical tissue morphology, and historical tissue microscopic image brightness information of the historical tissue microscopic image, and perform mapping processing with the corresponding feature information in the corresponding historical HE-stained image to obtain a first mapping relationship, a second mapping relationship, and a third mapping relationship, and output the generated target HE-stained image; the historical HE-stained image carrying the real image label and the target HE-stained image carrying the generated image label are input to the first discriminator so that the first discriminator learns the real image features in each of the historical HE-stained images and the target HE-stained image. The generated image features in the HE staining image are used to obtain a first target discriminator for predicting whether the target HE staining image output by the first generator is a real image. The target HE staining image is input to the second generator so that the second generator can capture the target tissue color features, target tissue texture features, target tissue cell structure, target tissue morphology, and target HE staining image brightness information of the target HE staining image, and perform mapping processing with the corresponding feature information in the historical tissue microscopic images to obtain updated first mapping relationship, second mapping relationship, and third mapping relationship, and output the generated target tissue microscopic image. The historical tissue microscopic images carrying real image labels and the target tissue microscopic images carrying generated image labels are input to the second discriminator so that the second discriminator can learn the real image features in each of the historical tissue microscopic images and the generated image features in the target tissue microscopic image to obtain a second target discriminator for predicting whether the target tissue microscopic image output by the second generator is a real image. The image conversion module is used to perform image conversion on the tissue microscopic image to obtain a virtual microscopic image containing the nucleus and cytoplasm of the second stained cells; the image conversion includes image conversion based on a preset image conversion model or image conversion based on image pixels and according to a preset channel conversion method.

Images