Rapid tissue microscopic image conversion method and system

Abstract

The invention discloses a rapid tissue microscopic image conversion method and system, relates to the technical field of image processing, is applied to a computer device, and comprises the following steps: obtaining a tissue microscopic image obtained by soaking and dyeing intraoperative human tissues by using two preset fluorescent probes; wherein the tissue microscopic image is an image containing a first dyed cell nucleus and a first dyed cytoplasm; performing image conversion on the tissue microscopic image to obtain a virtual microscopic image containing a second dyed cell nucleus and a second dyed cytoplasm; the image conversion comprises image conversion based on a preset image conversion model or image conversion based on image pixels and according to a preset channel conversion mode. According to the method, the image quality of the original tissue microscopic image does not need to be changed in the rapid image conversion process of the tissue microscopic image, the virtual microscopic image with high imaging contrast and high resolution is obtained, rapid and accurate tissue structure analysis is achieved according to the virtual microscopic image, and rapid diagnosis and treatment in an operation are facilitated.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method and system for quickly converting tissue microscopic images. Background Art

[0002] In order to achieve the goal of accurate margin assessment during surgery to clarify the boundary between healthy tissue and diseased tissue, pathological diagnosis is usually used to examine the tissue. As a traditional intraoperative pathological technique, frozen section (FS) is widely used for rapid intraoperative diagnosis. This technology freezes the resected specimen immediately and observes it under a microscope after hematoxylin-eosin staining (HE). The advantage of frozen section is that it can generate clearer HE staining images in a short time, helping surgeons make instant decisions during surgery. However, the scope of application of frozen section technology is limited, the details of the slices produced are not clear enough, and cell swelling and deformation often occur, affecting morphological identification.

[0003] In contrast, in recent years, a system and method (DD staining method) for histological non-section sample surface imaging, as disclosed in patent publication number CN118311029A, has emerged, providing new possibilities for intraoperative pathology. The DD staining method uses two fluorescent probes, DAPI (4',6-diamidino-2-phenylindole) and DiD (1,1'-dioctadecyl-3,3,3', 3'-tetramethylindocyanine, 4-perchlorate), to quickly immerse and stain the surface cells of fresh tissues to obtain DD staining images. Similar to traditional HE staining images, DD staining images can also provide histological information. However, the DD staining method does not require the preparation of sections, thereby avoiding tissue damage that may occur during frozen sectioning and significantly reducing preparation time. This provides a potential advantage for rapid intraoperative diagnosis.

[0004] Nevertheless, the clinical application of DD staining technology still faces the following challenges: HE staining is the gold standard for pathology worldwide. Based on its extensive clinical research and literature support, the use of HE staining image diagnosis can ensure the standardization and consistency of diagnostic results. As an emerging technology, DD staining requires a lot of research and time to establish the same standardized diagnostic system. Pathologists need time to adapt to the new staining modality and learn how to identify pathological features in DD staining images, which may affect diagnostic efficiency in the early stages.

[0005] In summary, how to obtain image information in conventional HE staining images from quickly acquired DD staining images in intraoperative scenarios and reduce image preparation time is a technical problem to be solved in this field. Summary of the invention

[0006] In view of this, the purpose of the present invention is to provide a method and system for rapid conversion of tissue microscopic images, which can achieve the acquisition of image information in conventional HE staining images from rapidly acquired DD staining images in intraoperative scenarios, thereby reducing image preparation and generation time. The specific scheme is as follows:

[0007] In a first aspect, the present application discloses a method for rapid conversion of tissue microscopic images, which is applied to a computer device and comprises:

[0008] Acquire a tissue microscopic image after immersion staining of intraoperative human tissue using two preset fluorescent probes; wherein the tissue microscopic image is an image including a first stained cell nucleus and a first stained cytoplasm;

[0009] The tissue microscopic image is subjected to image conversion to obtain a virtual microscopic image comprising a second stained cell nucleus and a second stained cytoplasm; the image conversion includes image conversion based on a preset image conversion model or image conversion based on image pixels and in accordance with a preset channel conversion method.

[0010] Optionally, before performing image conversion on the tissue microscopic image based on the preset image conversion model, the method further includes:

[0011] Paired images constructed using historical tissue microscopic images and corresponding historical HE staining images were used as data training sets;

[0012] The data training set is used to perform model training on the initial image conversion model, so that the initial image conversion model learns the first mapping relationship between the color features and texture features of the historical tissue microscopic image and the historical HE staining image in each of the paired images, the second mapping relationship between the cell structure and the tissue morphology, and learns the third mapping relationship between the dark field style of the historical tissue microscopic image and the bright field style of the historical HE staining image, so as to obtain a preset image conversion model.

[0013] Optionally, before the initial image conversion model is trained using the data training set so that the initial image conversion model learns a first mapping relationship between color features and texture features between the historical tissue microscopic image and the historical HE staining image in each of the paired images, a second mapping relationship between cell structure and tissue morphology, and learns a third mapping relationship between a dark field style of the historical tissue microscopic image and a bright field style of the historical HE staining image, to obtain a preset image conversion model, the method further includes:

[0014] An initial image conversion model based on a GAN architecture including a generator and a discriminator is constructed; wherein the generator includes a first generator and a second generator, and the discriminator includes a first discriminator and a second discriminator.

[0015] Optionally, the using the data training set to perform model training on the initial image conversion model so that the initial image conversion model learns a first mapping relationship between color features and texture features between the historical tissue microscopic image and the historical HE staining image in each of the paired images, a second mapping relationship between cell structure and tissue morphology, and learns a third mapping relationship between the dark field style of the historical tissue microscopic image and the bright field style of the historical HE staining image, so as to obtain a preset image conversion model, including:

[0016] Input the paired image to the first generator, so that the historical tissue color features, historical tissue texture features, historical tissue cell structure, historical tissue morphology and historical tissue microscopic image brightness information of the historical tissue microscopic image are captured by the first generator, and mapped with the corresponding feature information in the corresponding historical HE staining image to obtain a first mapping relationship, a second mapping relationship and a third mapping relationship, and output the generated target HE staining image;

[0017] Inputting the historical HE-stained images carrying real image labels and the target HE-stained images carrying generated image labels into the first discriminator, so that the first discriminator learns the real image features in each of the historical HE-stained images and the generated image features in the target HE-stained images, so as to obtain a first target discriminator for predicting a discrimination result of whether the target HE-stained image output by the first generator is a real image;

[0018] Inputting the target HE staining image into the second generator, so as to capture the target tissue color features, target tissue texture features, target tissue cell structure, target tissue morphology and target HE staining image brightness information of the target HE staining image through the second generator, and performing mapping processing with the corresponding feature information in the historical tissue microscopic image to obtain updated first mapping relationship, second mapping relationship and third mapping relationship, and outputting the generated target tissue microscopic image;

[0019] The historical tissue microscopic images carrying real image labels and the target tissue microscopic images carrying generated image labels are input into the second discriminator so that the second discriminator learns the real image features in each of the historical tissue microscopic images and the generated image features in the target tissue microscopic images, so as to obtain a second target discriminator for predicting the discrimination result of whether the target tissue microscopic image output by the second generator is a real image.

[0020] Optionally, the method for rapidly converting tissue microscopic images further comprises:

[0021] Constructing an adversarial loss function for constraining adversarial training between the generator and the discriminator;

[0022] Constructing a cycle consistency loss function for constraining the consistency between the target tissue microscopic image and the historical tissue microscopic image output by the generator, and the consistency between the target HE staining image and the historical HE staining image;

[0023] Constructing a mapping loss function for constraining the content difference features between the target HE stained image and the historical tissue microscopic image, and the content difference features between the target tissue microscopic image and the historical HE stained image output by the generator;

[0024] A total model loss function of the initial image conversion model training process is constructed according to the adversarial loss function, the cycle consistency loss function, the mapping loss function and corresponding function weight parameters.

[0025] Optionally, the constructing of a cycle consistency loss function for constraining the consistency between the target tissue microscopic image output by the generator and the historical tissue microscopic image, and the consistency between the target HE staining image and the historical HE staining image, comprises:

[0026] Respectively calculating a first brightness contrast factor and a first color-structure contrast factor between the target tissue microscopic image and the historical tissue microscopic image, and a second brightness contrast factor and a second color-structure contrast factor between the target HE staining image and the historical HE staining image;

[0027] The cycle consistency loss function is constructed according to the first brightness contrast factor, the first color-structure contrast factor, the second brightness contrast factor and the second color-structure contrast factor.

[0028] Optionally, performing image conversion on the tissue microscopic image to obtain a virtual microscopic image containing a second stained cell nucleus and a second stained cytoplasm comprises:

[0029] Separating the RGB channels of the tissue microscopic image by a preset image processing tool to obtain a separated first tissue microscopic image, a second tissue microscopic image, and a third tissue microscopic image;

[0030] Respectively calculating and converting the first tissue microscopic image, the second tissue microscopic image, and the third tissue microscopic image pixel by pixel according to corresponding RGB channel operation algorithms to obtain respective image conversion results;

[0031] The image conversion results are synthesized to obtain a virtual microscopic image including the second stained cell nucleus and the second stained cytoplasm.

[0032] Optionally, the tissue microscopic image is a dark field image, and the color of the first stained cell nucleus region in the tissue microscopic image is blue, and the color of the first stained cytoplasm region is red.

[0033] Optionally, the virtual microscopic image is a bright field image, and the color of the second stained cell nucleus region in the virtual microscopic image is blue, and the color of the second stained cytoplasm region is pink.

[0034] In a second aspect, the present application discloses a system for rapidly converting tissue microscopic images, which is applied to a computer device and comprises:

[0035] An image acquisition module, used to acquire a tissue microscopic image after the intraoperative human tissue is immersed and stained with two preset fluorescent probes; wherein the tissue microscopic image is an image including a first stained cell nucleus and a first stained cytoplasm;

[0036] An image conversion module is used to perform image conversion on the tissue microscopic image to obtain a virtual microscopic image containing a second stained cell nucleus and a second stained cytoplasm; the image conversion includes image conversion based on a preset image conversion model or image conversion based on image pixels and in accordance with a preset channel conversion method.

[0037] In a third aspect, the present application discloses an electronic device, comprising:

[0038] Memory, used to store computer programs;

[0039] A processor is used to execute the computer program to implement the steps of the aforementioned method for rapid conversion of tissue microscopic images.

[0040] In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the steps of the aforementioned method for rapid conversion of tissue microscopic images are implemented.

[0041] It can be seen that the present application discloses a method for rapid conversion of tissue microscopic images, which is applied to a computer device, including: obtaining a tissue microscopic image after immersion staining of intraoperative human tissue using two preset fluorescent probes; wherein the tissue microscopic image is an image containing a first stained nucleus and a first stained cytoplasm; performing image conversion on the tissue microscopic image to obtain a virtual microscopic image containing a second stained nucleus and a second stained cytoplasm; the image conversion includes image conversion based on a preset image conversion model or image conversion based on image pixels and in accordance with a preset channel conversion method. It can be seen that compared with obtaining the corresponding staining image through traditional HE staining, the present invention obtains a virtual microscopic image with high imaging contrast and high resolution through the conversion process of rapid image conversion of the tissue microscopic image without changing the image quality of the original tissue microscopic image, and the overall time for obtaining the virtual microscopic image of the present invention is less than that of the traditional HE staining image, and the imaging quality of the virtual microscopic image is higher than that of the traditional HE staining image. In this way, in the scene of intraoperative detection, rapid and accurate tissue structure analysis can be achieved according to the virtual microscopic image, which is convenient for rapid intraoperative diagnosis and treatment. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0043] Figure 1 A flowchart of a method for rapid conversion of tissue microscopic images disclosed in this application;

[0044] Figure 2 A diagram of a model training process of a preset image conversion model disclosed in this application;

[0045] Figure 3 (a) shows a global tissue microscopic image, Figure 3 (b) shows a global virtual microscopic image, Figure 3 (c) shows a global reference image, Figure 3 (d) shows a locally enlarged tissue microscopic image, Figure 3 (e) shows a locally enlarged virtual microscopic image, and Figure 3 (f) shows a locally enlarged reference image;

[0046] Figure 4 This is a flow chart of the method for converting tissue microscopic images through RGB channels disclosed in this application;

[0047] Figure 5 This is a schematic diagram of the structure of a rapid conversion system for tissue microscopic images disclosed in this application;

[0048] Figure 6This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0050] In order to achieve the goal of accurate margin assessment during surgery to clarify the boundary between healthy tissue and diseased tissue, pathological diagnosis is usually used to examine the tissue. As a traditional intraoperative pathology technique, frozen sections are widely used for rapid intraoperative diagnosis. This technology freezes and slices the resected specimen immediately, and then observes it under a microscope after staining with hematoxylin and eosin. The advantage of frozen sections is that they can generate clearer HE-stained images in a short period of time, helping surgeons make instant decisions during surgery. However, the scope of application of frozen section technology is limited, the details of the slices produced are not clear enough, and cell swelling and deformation often occur, affecting morphological identification.

[0051] In contrast, in recent years, a system and method (DD staining method) for histological non-section sample surface imaging, as disclosed in patent publication number CN118311029A, has emerged, providing new possibilities for intraoperative pathology. The DD staining method uses two fluorescent probes, DAPI and DiD, to quickly immerse and stain the surface cells of fresh tissues to obtain DD staining images. Similar to traditional HE staining images, DD staining images can also provide histological information. However, the DD staining method does not require the preparation of sections, thereby avoiding tissue damage that may occur during the frozen section process and significantly reducing preparation time. This provides a potential advantage for rapid intraoperative diagnosis.

[0052] Nevertheless, the clinical application of DD staining technology still faces the following challenges: HE staining is the gold standard for pathology worldwide. Based on its extensive clinical research and literature support, the use of HE staining image diagnosis can ensure the standardization and consistency of diagnostic results. As an emerging technology, DD staining requires a lot of research and time to establish the same standardized diagnostic system. Pathologists need time to adapt to the new staining modality and learn how to identify pathological features in DD staining images, which may affect diagnostic efficiency in the early stages.

[0053] To this end, the present invention provides a rapid conversion solution for tissue microscopic images, which can achieve the acquisition of image information in conventional HE staining images from rapidly acquired DD staining images in an intraoperative scenario, thereby reducing image preparation and generation time.

[0054] Reference Figure 1 As shown, an embodiment of the present invention discloses a method for quickly converting a tissue microscopic image, which is applied to a computer device and includes:

[0055] Step S11: Acquire a tissue microscopic image after immersion staining of intraoperative human tissue using two preset fluorescent probes; wherein the tissue microscopic image is an image including a first stained cell nucleus and a first stained cytoplasm.

[0056] In this embodiment, the surface cells of fresh tissues are rapidly immersed and stained by two fluorescent probes, DAPI (4',6-diamidino-2-phenylindole) and DiD (1,1'-dioctadecyl-3,3,3',3'-tetramethylindocyanine, 4-perchlorate) (hereinafter referred to as DD staining), wherein DAPI can bind to the cell nucleus and DiD can bind to the cell membrane. Under 365 nm laser excitation, the cell nucleus bound to DAPI emits blue light, and under 640 nm laser excitation, the cell membrane bound to DiD emits red light. The fluorescence microscope imaging system non-destructively images the surface of fresh tissues after immersion based on scattering imaging and depth of field extension functions. A tissue microscopic image is obtained. The tissue microscopic image is a dark field image, and the color of the area of ​​the first stained cell nucleus in the tissue microscopic image is blue, and the color of the area of ​​the first stained cytoplasm is red. It should be noted that the total time of the preparation time and imaging time of the above-mentioned tissue microscopic image is less than 10 minutes, and the time for obtaining the tissue microscopic image is short.

[0057] Step S12: performing image conversion on the tissue microscopic image to obtain a virtual microscopic image including a second stained cell nucleus and a second stained cytoplasm; the image conversion includes image conversion based on a preset image conversion model or image conversion based on image pixels and in accordance with a preset channel conversion method.

[0058] In this embodiment, the tissue microscopic image acquired above is subjected to image conversion. The specific image conversion method may be: performing image conversion using a preset image conversion model trained specifically for converting tissue microscopic images to virtual microscopic images, or performing image conversion based on image pixels in the tissue microscopic image according to a preset channel conversion method.

[0059] In a specific embodiment, if a preset image conversion model is used for image conversion, the preset image conversion model needs to be trained, wherein before the tissue microscopic image is converted based on the preset image conversion model, it also includes: using the paired images constructed by the historical tissue microscopic image and the corresponding historical HE staining image as a data training set; using the data training set to train the initial image conversion model, so that the initial image conversion model learns the first mapping relationship between the color features and texture features of the historical tissue microscopic image and the historical HE staining image in each paired image, the second mapping relationship between the cell structure and the tissue morphology, and learns the third mapping relationship between the dark field style of the historical tissue microscopic image and the bright field style of the historical HE staining image, so as to obtain the preset image conversion model. It can be understood that the data training set is first constructed. Specifically, the sample used in the embodiment of the present invention is from human colon cancer tissue, DD staining uses fluorescent dark field imaging, and uses 365nm and 640nm lasers as light sources; HE staining uses bright field white light for illumination. The stained sample is imaged using an Olympus microscope under a 10x objective lens. The process of constructing the data set includes but is not limited to: obtaining DD staining images, obtaining HE staining images, and performing necessary image preprocessing. Specifically, while keeping the fresh tissue samples removed by surgery moist, the surface cells of the fresh tissues are quickly soaked and stained with DAPI and DiD and sealed. After that, a suitable microscope and laser light source system are set up, and the prepared tissue samples are scanned and imaged at the optimal imaging power of 365nm and 640nm lasers, and finally a large-size DD staining image is obtained by stitching algorithm. The fresh tissue samples removed by surgery are first fixed in formalin to maintain the tissue structure, and then dehydrated, waxed and embedded to form paraffin blocks. Next, the paraffin blocks are sliced ​​and attached to slides, and then the slices are dewaxed and hydrated. After these preprocessing steps, the slices are sequentially stained with hematoxylin and eosin. After staining, the tissue samples are sealed. Finally, the sealed samples are observed under a microscope to obtain HE staining images. A total of 414 WSI images (Whole Slide Image) were collected using the above two steps, of which 214 were HE-stained WSIs and 200 were DD-stained WSIs. In order to enable the obtained stained images to be used for deep learning model training, the above WSI images were cut into image blocks of 256 pixels * 256 pixels, with an overlap of 100 pixels during cutting, and the obtained image blocks were processed to remove the parts with excessive background.The final constructed DD-HE unpaired image dataset includes a total of 11,600 DD-stained image blocks and 11,600 HE-stained image blocks, of which 9,000 DD-stained image blocks (historical tissue microscopic images) and 2,600 HE-stained image blocks (historical HE-stained images) are taken as data training sets, and 2,600 are taken as data validation sets. Then, the data training set is used to train the initial image conversion model (GAN architecture), so that the initial image conversion model learns the first mapping relationship between the color features and texture features of the cell nucleus and cytoplasm in the historical tissue microscopic images and the historical HE-stained images, the second mapping relationship between the cell structure and the tissue morphology, and the third mapping relationship between the dark field style of the historical tissue microscopic images and the bright field style of the historical HE-stained images, so as to obtain the preset image conversion model.

[0060] Specifically, the initial image conversion model is trained using the data training set so that the initial image conversion model learns the first mapping relationship between the color features and texture features of the historical tissue microscopic image and the historical HE staining image in each of the paired images, the second mapping relationship between the cell structure and the tissue morphology, and learns the third mapping relationship between the dark field style of the historical tissue microscopic image and the bright field style of the historical HE staining image to obtain the preset image conversion model, and also includes: constructing an initial image conversion model based on a GAN architecture including a generator and a discriminator; wherein the generator includes a first generator and a second generator, and the discriminator includes a first discriminator and a second discriminator. It is understandable that it should be noted that the GAN architecture is used to learn the staining conversion from a DD staining image to its equivalent bright field HE staining image. GAN can output clear and realistic images through adversarial training of the generator G (Generator) and the discriminator D (Discrimator). The generator G aims to deceive the discriminator D by generating fake images similar to the target image, while the discriminator D aims to distinguish whether the received image is a generated image or a real image, making it more suitable for the unpaired DD dyeing to HE dyeing conversion task.

[0061] In this embodiment, the initial image conversion model is trained using the data training set so that the initial image conversion model learns the first mapping relationship between the color features and texture features of the historical tissue microscopic image and the historical HE staining image in each of the paired images, the second mapping relationship between the cell structure and the tissue morphology, and learns the third mapping relationship between the dark field style of the historical tissue microscopic image and the bright field style of the historical HE staining image to obtain a preset image conversion model, including: inputting the paired image into the first generator so that the historical tissue color features, historical tissue texture features, historical tissue cell structure, historical tissue morphology and brightness information of the historical tissue microscopic image are captured by the first generator, and mapped with the corresponding feature information in the corresponding historical HE staining image to obtain the first mapping relationship, the second mapping relationship and the third mapping relationship, and output the generated target HE staining image; inputting the historical HE staining image carrying the real image label and the target HE staining image carrying the generated image label into the first discriminator so that the first discriminator learns each The real image features in the historical HE staining image and the generated image features in the target HE staining image are used to obtain a first target discriminator for predicting whether the target HE staining image output by the first generator is a real image; the target HE staining image is input to the second generator so that the target tissue color features, target tissue texture features, target tissue cell structure, target tissue morphology and target HE staining image brightness information of the target HE staining image are captured by the second generator, and mapped with the corresponding feature information in the historical tissue microscopic image to obtain an updated first mapping relationship, a second mapping relationship and a third mapping relationship, and output the generated target tissue microscopic image; the historical tissue microscopic image carrying the real image label and the target tissue microscopic image carrying the generated image label are input to the second discriminator so that the second discriminator learns the real image features in each of the historical tissue microscopic images and the generated image features in the target tissue microscopic image to obtain a second target discriminator for predicting whether the target tissue microscopic image output by the second generator is a real image. It can be understood that, with reference to Figure 2As shown, the present invention provides a specific training process for a preset image conversion model, wherein the initial image conversion model is trained by using historical tissue microscopic images (DD stained images) carrying real labels in a data training set and historical HE stained images also carrying real labels as data training sets. Specifically, the initial image conversion model includes a generator, a first generator for converting a DD stained image into an HE stained image, and a first discriminator for determining whether the HE image generated by the first generator is a real image, a second generator for converting the HE stained image into a DD stained image, and a second discriminator for determining whether the DD image generated by the second generator is a real image. Specifically, the distribution of DD stained images is regarded as a source domain (X domain), and the distribution of HE stained images is regarded as a target domain (Y domain). The goal of CycleGAN is to give a marginal distribution , learn the conditional mapping and The CycleGAN architecture consists of two generators ( and ) and two discriminators ( and ). The specific training process is: given input (a DD staining image, i.e. a historical tissue microscopic image), x passes through the first generator Get the generated image in the Y domain (It can be expressed as ), that is, the target HE staining image, Then pass through the generator second generator Get the reconstructed image in the X domain (It can be expressed as ), which is the microscopic image of the target tissue; similarly, for (a HE-stained image, also known as a historical HE-stained image), can be obtained through a similar process ( , ), and obtain the generated image of the X domain and the reconstructed image of the Y domain The images generated by the generator and the real images are sent to the discriminator in the corresponding domain to train the discriminator's ability to distinguish between true and false.

[0062] In this embodiment, an adversarial loss function is constructed to constrain the adversarial training between the generator and the discriminator; a cycle consistency loss function is constructed to constrain the consistency between the target tissue microscopic image output by the generator and the historical tissue microscopic image, and the consistency between the target HE staining image and the historical HE staining image; a mapping loss function is constructed to constrain the content difference features between the target HE staining image output by the generator and the historical tissue microscopic image, and the content difference features between the target tissue microscopic image and the historical HE staining image; the total model loss function of the initial image conversion model training process is constructed according to the adversarial loss function, the cycle consistency loss function, the mapping loss function and the corresponding function weight parameters. It can be understood that after the model architecture design is completed, a loss function is further constructed to constrain and optimize the training of the model. Specifically, the adversarial loss function For adversarial training between the generator and the discriminator, it is defined as follows: ;

[0063] in, Both represent the mathematical expectation of the sample images; represents the HE image obtained after the real DD image x passes through the first generator , Represents a true HE image DD image obtained after passing through the second generator For the first generator , whose goal is to generate a The second generator The goal is to generate a class that can deceive the second discriminator For the first discriminator and the second discriminator, their purpose is to accurately distinguish between real images and generated images.

[0064] Cycle consistency loss function It is used to constrain the consistency between the output of the generator and the original input image, which is defined as follows: ;

[0065] in, Represents Manhattan Distance, also known as L1 distance. represents the reconstructed image of the input DD stained image, It is also the reconstructed image of the input HE image. The purpose is to hope that the generator can transform the input image from domain X to domain Y, and the generated image can be reversely transformed back to domain X again, thus maintaining cycle consistency with the original input.

[0066] Ontology Mapping Loss Function The generator is encouraged to perform a reverse mapping of the input image to its corresponding target domain image, and then map it back to the input domain. The purpose of this process is to preserve the content characteristics of the original image as much as possible, ensuring that the generated image is very similar to the input image. The definition is as follows: ;

[0067] Then, based on the above-mentioned adversarial loss function, cycle consistency loss function, mapping loss function and their corresponding pre-configured function weight parameters, the total loss function of the model training process is constructed.

[0068] It should be noted that It is the loss function used by the original CycleGAN. They all use Manhattan distance to compare the difference between two images pixel by pixel, but ignore the color, texture and structure of the image, which are important references for pathologists to make diagnoses. Therefore, MS-SSIM (Multiscale Structural Similarity) and L1 hybrid loss are introduced. For traditional cycle consistency loss Enhancements are performed to preserve cycle invariance of brightness, color, and structure.

[0069] Specifically, the construction of a cycle consistency loss function for constraining the consistency between the target tissue microscopic image output by the generator and the historical tissue microscopic image, and the consistency between the target HE staining image and the historical HE staining image, includes: respectively calculating the first brightness contrast factor and the first color-structure contrast factor between the target tissue microscopic image and the historical tissue microscopic image, and the second brightness contrast factor and the second color-structure contrast factor between the target HE staining image and the historical HE staining image; constructing the cycle consistency loss function according to the first brightness contrast factor, the first color-structure contrast factor, the second brightness contrast factor, and the second color-structure contrast factor. It can be understood that for MS-SSIM, the brightness contrast factor defined is first calculated. and color-structure contrast factor , defined as follows: ; ;

[0070] in, Represents the windows corresponding to the two images, Respectively The mean pixel intensity, express The variance of pixel intensity, express The covariance of pixel intensities, are two constants to avoid the denominator being 0. It should be noted that, according to the brightness contrast factor and color-structure contrast factor defined above, the first brightness contrast factor and the first color-structure contrast factor representing the DD image and the second brightness contrast factor and the second color-structure contrast factor representing the HE image are obtained respectively.

[0071] MS-SSIM is calculated based on the brightness contrast factor and the color-structure contrast factor, which are defined as follows: ;

[0072] in, Represents different scales, are all expressed as constants. Then, define and : ; ;

[0073] Among them, 𝜆 is a coefficient used to adjust the MS-SSIM loss and cycle consistency loss The proportion occupied.

[0074] Therefore, the total model loss function of the initial image conversion model training process is expressed as follows: ;

[0075] in, is a hyperparameter used to configure the weight of the loss function.

[0076] For the training of the improved CycleGAN, the present invention is set as follows: , In , the total loss function In ; The initial learning rate is set to 0.0002 and the number of training rounds is 100; a Gaussian distribution with a mean of 0 and a variance of 0.0004 is used to initialize the network weights; the Adam optimizer is used with betas=(0.5, 0.999).

[0077] The trained model is validated using the data validation set, and the validated model is used as the preset image conversion model.

[0078] The tissue microscopic image to be converted is input into a preset image conversion model, so that the preset image conversion model can directly convert the corresponding HE image into a virtual microscopic image for output.

[0079] As shown in FIG3 , FIG3 (a) shows a tissue microscopic image (DD staining image) obtained by immersing and staining human tissue during surgery using the two preset fluorescent probes mentioned in the present invention, wherein FIG3 (d) shows a local tissue microscopic image circled in the yellow frame in the tissue microscopic image FIG3 (a), and red cell membrane or cytoplasm and blue cell nucleus can be seen from FIG3 (d) under a black background, while FIG3 (b) is a virtual microscopic image (virtual HE staining image) under bright field obtained after image conversion of FIG3 (a), and FIG3 (e) represents the local virtual microscopic image selected by the yellow frame in FIG3 (b), and from FIG3 (e) it can be seen that the HE image information of pink cytoplasm and blue cell nucleus under a white background that is consistent with what researchers see daily, and FIG3 (c) shows the same tissue site using traditional technology (Formalin-fixed-paraffin section (Formalin-fixed Figure 3 (f) shows the local reference HE staining image selected by the yellow frame in Figure 3 (c). By comparison, it can be seen that the details of the DD staining image are similar to those of the traditional HE staining image, especially after the virtual HE staining image is generated by the image conversion model, it can effectively restore the main features of HE staining and has the advantage of high contrast. This shows that the DD staining method has potential in rapid intraoperative diagnosis, especially when the sectioning step is not required, image information similar to HE staining can still be obtained through image conversion technology.

[0080] In another specific embodiment, referring to Figure 4 As shown, the RGB channels of the tissue microscopic image are separated by a preset image processing tool to obtain the separated first tissue microscopic image, the second tissue microscopic image, and the third tissue microscopic image; the first tissue microscopic image, the second tissue microscopic image, and the third tissue microscopic image are respectively calculated and converted pixel by pixel according to the corresponding RGB channel operation algorithm to obtain each image conversion result; each image conversion result is synthesized to obtain a virtual microscopic image containing a second stained cell nucleus and a second stained cytoplasm. It can be understood that the tissue microscopic image is loaded through the image processing tool, and the color of the three channels of R (Red), G (Green), and B (Blue) is separated pixel by pixel by channel separation. After the separation is completed, the following are obtained respectively. , , If the tissue microscopy image is an eight-bit image, the image conversion is performed according to the following formula: ; ; ;

[0081] in, , , are the channel conversion results of the B, G, and R channels of the converted HE image, and then , , Image synthesis processing is performed to obtain a converted HE image as a virtual microscopic image for output. If the tissue microscopic image is 16 bits, replace 255 in the formula with 65535. It should be noted that after obtaining the converted HE image, the color of the converted HE image can be further optimized by adjusting the white balance to obtain a processed HE image as a virtual microscopic image for output. The virtual microscopic image is a bright field image, and the color of the second stained nucleus area in the virtual microscopic image is blue, and the color of the second stained cytoplasm area is pink.

[0082] In this way, the image quality (high resolution, high imaging contrast) of DD staining images can be preserved through the above-mentioned image conversion. At the same time, since the preparation and acquisition time of DD staining images only takes less than 10 minutes, and the subsequent conversion time to virtual microscopic images can also be completed within 5 minutes, the overall acquisition time of virtual microscopic images is less than 15 minutes. In the intraoperative scenario, high-quality virtual microscopic images can be provided in a short time, so as to present richer pathological information from virtual microscopic images according to the high contrast characteristics of DD staining images. The complicated processes of fixation, slicing, staining, etc. required for obtaining HE images by traditional intraoperative pathology techniques (such as frozen sections) take about 20 to 30 minutes, and the scope of application is limited, and it is difficult to guarantee the staining quality, which limits its application in some cases.

[0083] It can be seen that the present application discloses a method for rapid conversion of tissue microscopic images, which is applied to a computer device, including: obtaining a tissue microscopic image after immersion staining of intraoperative human tissue using two preset fluorescent probes; wherein the tissue microscopic image is an image containing a first stained nucleus and a first stained cytoplasm; performing image conversion on the tissue microscopic image to obtain a virtual microscopic image containing a second stained nucleus and a second stained cytoplasm; the image conversion includes image conversion based on a preset image conversion model or image conversion based on image pixels and in accordance with a preset channel conversion method. It can be seen that compared with obtaining the corresponding staining image through traditional HE staining, the present invention obtains a virtual microscopic image with high imaging contrast and high resolution through the conversion process of rapid image conversion of the tissue microscopic image without changing the image quality of the original tissue microscopic image, and the overall time for obtaining the virtual microscopic image of the present invention is less than that of the traditional HE staining image, and the imaging quality of the virtual microscopic image is higher than that of the traditional HE staining image. In this way, in the scene of intraoperative detection, rapid and accurate tissue structure analysis can be achieved according to the virtual microscopic image, which is convenient for rapid intraoperative diagnosis and treatment.

[0084] Reference Figure 5 As shown, the present invention also discloses a rapid conversion system of tissue microscopic images, which is applied to a computer device and includes:

[0085] The image acquisition module 11 is used to acquire a tissue microscopic image after the intraoperative human tissue is immersed and stained with two preset fluorescent probes; wherein the tissue microscopic image is an image including a first stained cell nucleus and a first stained cytoplasm;

[0086] The image conversion module 12 is used to perform image conversion on the tissue microscopic image to obtain a virtual microscopic image containing a second stained cell nucleus and a second stained cytoplasm; the image conversion includes image conversion based on a preset image conversion model or image conversion based on image pixels and according to a preset channel conversion method.

[0087] It can be seen that the present application discloses a method for obtaining a tissue microscopic image after immersion staining of intraoperative human tissue using two preset fluorescent probes; wherein the tissue microscopic image is an image containing a first stained nucleus and a first stained cytoplasm; the tissue microscopic image is image converted to obtain a virtual microscopic image containing a second stained nucleus and a second stained cytoplasm; the image conversion includes image conversion based on a preset image conversion model or image conversion based on image pixels and in accordance with a preset channel conversion method. It can be seen that compared with obtaining the corresponding staining image through traditional HE staining, the present invention obtains a virtual microscopic image with high imaging contrast and high resolution by performing a conversion process of rapid image conversion on the tissue microscopic image without changing the image quality of the original tissue microscopic image, and the overall time for obtaining the virtual microscopic image of the present invention is less than that of the traditional HE staining image, and the imaging quality of the virtual microscopic image is higher than that of the traditional HE staining image. In this way, in the scene of intraoperative detection, rapid and accurate tissue structure analysis can be achieved based on the virtual microscopic image, which is convenient for rapid intraoperative diagnosis and treatment.

[0088] Furthermore, the present application also discloses an electronic device. Figure 6 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the diagram cannot be regarded as any limitation on the scope of use of the present application.

[0089] Figure 6 The present invention provides a schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the method for rapid conversion of tissue microscopic images disclosed in any of the aforementioned embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0090] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0091] Among them, the processor 21 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 21 can be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 21 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 21 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 21 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0092] In addition, the memory 22, as a carrier for storing resources, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0093] The operating system 221 is used to manage and control the hardware devices and computer programs 222 on the electronic device 20, so as to realize the operation and processing of the massive data 223 in the memory 22 by the processor 21, which can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program that can be used to complete the method for quickly converting tissue microscopic images performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program that can be used to complete other specific tasks. In addition to data received by the electronic device and transmitted from an external device, the data 223 can also include data collected by its own input and output interface 25.

[0094] Furthermore, the present application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the aforementioned method for rapidly converting tissue microscopic images is implemented. The specific steps of the method may refer to the corresponding contents disclosed in the aforementioned embodiments, and will not be described in detail here.

[0095] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0096] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application. The steps of the method or algorithm described in conjunction with the embodiments disclosed herein can be implemented directly with hardware, a software module executed by a processor, or a combination of the two. The software module can be placed in a random access memory RAM (Random Access Memory), memory, read-only memory ROM (Read Only Memory), electrically programmable EPROM (Electrically Programmable Read Only Memory), electrically erasable programmable EEPROM (ElectricErasable Programmable Read Only Memory), register, hard disk, removable disk, CD-ROM (CompactDisc-Read Only Memory), or any other form of storage medium known in the technical field.

[0097] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0098] The scheme provided by the present invention is introduced in detail above. Specific examples are used in this article to illustrate the principle and implementation mode of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation mode and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A method for rapid conversion of tissue microscopic images, characterized in that: Applicable to computer devices, including: Acquire a tissue microscopic image after immersion staining of intraoperative human tissue using two preset fluorescent probes; wherein the tissue microscopic image is an image including a first stained cell nucleus and a first stained cytoplasm; The tissue microscopic image is subjected to image conversion to obtain a virtual microscopic image comprising a second stained cell nucleus and a second stained cytoplasm; the image conversion includes image conversion based on a preset image conversion model or image conversion based on image pixels and in accordance with a preset channel conversion method.

2. The method for rapid conversion of tissue microscopic images according to claim 1, characterized in that: Before performing image conversion on the tissue microscopic image based on the preset image conversion model, the method further includes: Paired images constructed using historical tissue microscopic images and corresponding historical HE staining images were used as data training sets; The data training set is used to perform model training on the initial image conversion model, so that the initial image conversion model learns the first mapping relationship between the color features and texture features of the historical tissue microscopic image and the historical HE staining image in each of the paired images, the second mapping relationship between the cell structure and the tissue morphology, and learns the third mapping relationship between the dark field style of the historical tissue microscopic image and the bright field style of the historical HE staining image, so as to obtain a preset image conversion model.

3. The method for rapid conversion of tissue microscopic images according to claim 2, characterized in that: The method further comprises: performing model training on the initial image conversion model by using the data training set so that the initial image conversion model learns a first mapping relationship between color features and texture features between the historical tissue microscopic image and the historical HE staining image in each of the paired images, a second mapping relationship between cell structure and tissue morphology, and learns a third mapping relationship between the dark field style of the historical tissue microscopic image and the bright field style of the historical HE staining image to obtain the preset image conversion model. An initial image conversion model based on a GAN architecture including a generator and a discriminator is constructed; wherein the generator includes a first generator and a second generator, and the discriminator includes a first discriminator and a second discriminator.

4. The method for rapid conversion of tissue microscopic images according to claim 3, characterized in that: The method of using the data training set to perform model training on the initial image conversion model so that the initial image conversion model learns a first mapping relationship between color features and texture features between the historical tissue microscopic image and the historical HE staining image in each of the paired images, a second mapping relationship between cell structure and tissue morphology, and learns a third mapping relationship between the dark field style of the historical tissue microscopic image and the bright field style of the historical HE staining image to obtain a preset image conversion model, including: Input the paired image to the first generator, so that the historical tissue color features, historical tissue texture features, historical tissue cell structure, historical tissue morphology and historical tissue microscopic image brightness information of the historical tissue microscopic image are captured by the first generator, and mapped with the corresponding feature information in the corresponding historical HE staining image to obtain a first mapping relationship, a second mapping relationship and a third mapping relationship, and output the generated target HE staining image; Inputting the historical HE-stained images carrying real image labels and the target HE-stained images carrying generated image labels into the first discriminator, so that the first discriminator learns the real image features in each of the historical HE-stained images and the generated image features in the target HE-stained images, so as to obtain a first target discriminator for predicting a discrimination result of whether the target HE-stained image output by the first generator is a real image; Inputting the target HE staining image into the second generator, so as to capture the target tissue color features, target tissue texture features, target tissue cell structure, target tissue morphology and target HE staining image brightness information of the target HE staining image through the second generator, and performing mapping processing with the corresponding feature information in the historical tissue microscopic image to obtain updated first mapping relationship, second mapping relationship and third mapping relationship, and outputting the generated target tissue microscopic image; The historical tissue microscopic images carrying real image labels and the target tissue microscopic images carrying generated image labels are input into the second discriminator so that the second discriminator learns the real image features in each of the historical tissue microscopic images and the generated image features in the target tissue microscopic images, so as to obtain a second target discriminator for predicting the discrimination result of whether the target tissue microscopic image output by the second generator is a real image.

5. The method for rapid conversion of tissue microscopic images according to claim 3, characterized in that: Also includes: Constructing an adversarial loss function for constraining adversarial training between the generator and the discriminator; Constructing a cycle consistency loss function for constraining the consistency between the target tissue microscopic image and the historical tissue microscopic image output by the generator, and the consistency between the target HE staining image and the historical HE staining image; Constructing a mapping loss function for constraining the content difference features between the target HE stained image and the historical tissue microscopic image, and the content difference features between the target tissue microscopic image and the historical HE stained image output by the generator; A total model loss function of the initial image conversion model training process is constructed according to the adversarial loss function, the cycle consistency loss function, the mapping loss function and corresponding function weight parameters.

6. The method for rapid conversion of tissue microscopic images according to claim 5, characterized in that: The constructing of a cycle consistency loss function for constraining the consistency between the target tissue microscopic image output by the generator and the historical tissue microscopic image, and the consistency between the target HE staining image and the historical HE staining image, comprises: Respectively calculating a first brightness contrast factor and a first color-structure contrast factor between the target tissue microscopic image and the historical tissue microscopic image, and a second brightness contrast factor and a second color-structure contrast factor between the target HE staining image and the historical HE staining image; The cycle consistency loss function is constructed according to the first brightness contrast factor, the first color-structure contrast factor, the second brightness contrast factor and the second color-structure contrast factor.

7. The method for rapid conversion of tissue microscopic images according to claim 1, characterized in that: The image conversion of the tissue microscopic image to obtain a virtual microscopic image containing a second stained cell nucleus and a second stained cytoplasm comprises: Separating the RGB channels of the tissue microscopic image by a preset image processing tool to obtain a separated first tissue microscopic image, a second tissue microscopic image, and a third tissue microscopic image; Respectively calculating and converting the first tissue microscopic image, the second tissue microscopic image, and the third tissue microscopic image pixel by pixel according to corresponding RGB channel operation algorithms to obtain respective image conversion results; The image conversion results are synthesized to obtain a virtual microscopic image including the second stained cell nucleus and the second stained cytoplasm.

8. The method for rapid conversion of tissue microscopic images according to claim 1, characterized in that: The tissue microscopic image is a dark field image, and the color of the first stained cell nucleus region in the tissue microscopic image is blue, and the color of the first stained cytoplasm region is red.

9. The method for rapid conversion of tissue microscopic images according to claim 1, characterized in that: The virtual microscopic image is a bright field image, and the color of the second stained cell nucleus region in the virtual microscopic image is blue, and the color of the second stained cytoplasm region is pink.

10. A rapid conversion system for tissue microscopic images, characterized in that: Applicable to computer devices, including: An image acquisition module, used to acquire a tissue microscopic image after the intraoperative human tissue is immersed and stained with two preset fluorescent probes; wherein the tissue microscopic image is an image including a first stained cell nucleus and a first stained cytoplasm; An image conversion module is used to perform image conversion on the tissue microscopic image to obtain a virtual microscopic image containing a second stained cell nucleus and a second stained cytoplasm; the image conversion includes image conversion based on a preset image conversion model or image conversion based on image pixels and in accordance with a preset channel conversion method.

Images