Training method, virtual staining method, and virtual staining system

Abstract

The present invention provides a training method, a virtual staining method, and a virtual staining system. The training method comprises: obtaining a Raman image of a cell sample and a corresponding ground truth stained image of the cell sample; on the basis of the Raman image and the ground truth stained image, training a cycle generative adversarial network to cause the value of a loss function of a generator network in the cycle generative adversarial network to converge; and on the basis of the generator network of the trained cycle generative adversarial network in which the value of the loss function has converged, obtaining a target generator network model, wherein the target generator network model generates a target virtual stained image on the basis of a target Raman image of a target cell sample, and the target virtual stained image is used for enhanced display of cellular structures in the target cell sample; and wherein the loss function of the generator network comprises a cellular structure similarity constraint term, and the cellular structure similarity constraint term constrains the consistency of cellular structures in an input image and an output image of the generator network on the basis of a structural similarity (SSIM) function.

Description

Training method, virtual dyeing method and virtual dyeing system

[0001] This application claims priority to Chinese Patent Application No. 202410050288.7 filed on January 12, 2024. The contents of the above-mentioned Chinese patent application disclosure are hereby incorporated by reference in their entirety as a part of this application. Technical Field

[0002] The present invention relates to the field of image processing, and more particularly, to a training method for obtaining a target generation network model and a virtual staining method and a virtual staining system for enhancing the display of cell structures. Background Art

[0003] In the medical field, cytopathology is a key branch of pathology and a crucial component of clinical pathology. In cytopathology, staining cell specimens (also known as samples) allows images of the stained samples to reveal details about the cell structure within the specimens. Clinicians and researchers can use the cellular structure revealed in these stained images to make diagnoses and decisions.

[0004] In early technologies, artificial chemical staining methods were often used to generate stained images of cell specimens. Specifically, the artificial chemical staining method first fixed the cell specimen, then stained it with dyes such as hematoxylin, eosin, Giemsa, and various other reagents. Finally, after operations such as transparency and sealing, the stained image of the cell specimen could be observed under a bright field microscope. However, artificial chemical staining methods for cell specimens are usually complex and time-consuming. The staining process is difficult to control consistently in terms of factors such as reagents, operations, processing environment, and specimen scanners. This can lead to inconsistent staining results and interfere with further analysis of the staining results.

[0005] To overcome these technical issues, virtual staining techniques have been gradually developed, which use computers to process specimen images to achieve staining. Virtual staining is a technique that uses a specific algorithm to process digital images of specimens, either labeled with fluorescent markers or without them, to impart pseudocolor to the specimen images. This technique aims to enhance the contrast of cellular structures and achieve staining results that are as close as possible to those achieved with common dyes such as hematoxylin and eosin (H&E). Virtual staining can significantly shorten staining time, reduce labor costs, and minimize damage to specimens, thus addressing the shortcomings of traditional chemical staining methods for cytopathology.

[0006] Virtual staining mainly includes virtual staining methods based on exogenous fluorescent labeling microscopy and virtual staining methods based on label-free microscopy. The exogenous dyes introduced by virtual staining methods based on exogenous fluorescent labeling microscopy will have irreversible effects on the physical and chemical properties of the specimen, interfering with other specimen analyses. Furthermore, the autofluorescence effect of the specimen and other environmental factors can also affect the virtual staining results. Therefore, compared with virtual staining methods based on exogenous fluorescent labeling microscopy, virtual staining methods based on label-free microscopy are more desirable.

[0007] Virtual staining methods based on label-free microscopy primarily utilize generative adversarial networks (GANs) to virtually stain the generated images, generating virtual stained images that doctors or researchers can use to make judgments and decisions. However, these methods still face two challenges.

[0008] On the one hand, the currently commonly used imaging technology for virtual staining methods based on label-free microscopic imaging cannot provide clear and accurate cell structure information. As a result, the method of generating adversarial network virtual staining based on images generated by currently commonly used imaging technology is difficult to accurately achieve cell-level staining and cannot accurately provide cell structure information.

[0009] On the other hand, the generative adversarial network virtual staining method currently used in virtual staining methods based on label-free microscopy lacks the ability to recognize cell structures and cannot guarantee the consistency of cell structures in input and output images (for example, it is easy to mistake the cell nucleus for cytoplasm). As a result, the generated virtual staining images cannot accurately provide information about cell structures.

[0010] Therefore, there is a need for an improved virtual staining method that can accurately provide information on cell structure. Summary of the Invention

[0011] To address the above issues, the present invention provides a training method for obtaining a target generative network model, a virtual staining method for enhancing the display of cell structure, and a virtual staining system. This training method uses Raman images and real stained images of cell samples for training, and constrains the loss function in the recurrent generative adversarial network to ensure cell structure consistency. This allows the target generative network model obtained based on the trained recurrent generative adversarial network to generate virtual stained images that are more consistent with the real stained images, improving the accuracy of virtual cell staining and achieving enhanced display of cell structure in cell samples.

[0012] According to one aspect of the present invention, a training method for obtaining a target generative network model is provided, comprising: obtaining a Raman image of a cell sample and a corresponding real staining image of the cell sample; training a cyclic generative adversarial network based on the Raman image and the real staining image to converge the value of the loss function of the generative network in the cyclic generative adversarial network; and obtaining a target generative network model based on the generative network in the trained cyclic generative adversarial network with the converged loss function value, wherein the target generative network model generates a target virtual staining image based on the target Raman image of the target cell sample, and the target virtual staining image is used to enhance the display of the cell structure in the target cell sample; wherein the generative network comprises a first generative network and a second generative network, and the loss function comprises a first loss function for the first generative network and a second loss function for the second generative network, and the first loss function and the second loss function comprise a cell structure similarity constraint term, and the cell structure constraint term constrains the consistency of the cell structure in the input image and the output image of the generative network based on the structural similarity SSIM function, wherein the cell structure similarity constraint term of the first loss function comprises The cell structure similarity constraint term of the second loss function includes Wherein, a is a Raman image in the Raman image domain A consisting of one or more Raman images, G A→B (a) is to input a into the first generation network G A→B The generated intermediate virtual stained image, SSIM(G A→B (a), a) is the structural similarity value between the intermediate virtual stained image and the Raman image, is the expected value of the structural similarity loss value corresponding to each Raman image in the Raman image domain A; and b is a real stained image in the real stained image domain B consisting of one or more real stained images, G B→A (b) Input b into the second generation network G B→A The generated intermediate Raman image, SSIM(G B→A (b),b) is the structural similarity value between the intermediate Raman image and the true dyeing image, is the expected value of the structural similarity loss value corresponding to each real stained image in the real stained image domain B.

[0013] According to some embodiments of the present invention, training a cyclic generative adversarial network based on the Raman image and the real stained image so that the value of the loss function of the generative network in the cyclic generative adversarial network converges includes: inputting the Raman image into the first generative network to generate an intermediate virtual stained image, inputting the intermediate virtual stained image into the first discriminant network in the cyclic generative adversarial network to determine the probability that the intermediate virtual stained image is judged as the real stained image, and inputting the real stained image into the second generative network to generate an intermediate Raman image, inputting the intermediate Raman image into the second discriminant network in the cyclic generative adversarial network to determine the probability that the intermediate Raman image is judged as the Raman image, and adjusting the parameters of the cyclic generative adversarial network based on the probabilities determined by the first discriminant network and the second discriminant network so that the values ​​of the first loss function and the second loss function converge respectively.

[0014] According to some embodiments of the present invention, adjusting the parameters of the cyclic generative adversarial network based on the probabilities determined by the first discriminant network and the second discriminant network so that the values ​​of the first loss function and the second loss function converge respectively includes: determining the values ​​of the first loss function and the second loss function based on the probabilities determined by the first discriminant network and the second discriminant network; adjusting the parameters of the cyclic generative adversarial network based on the values ​​of the first loss function and the second loss function so that the values ​​of the first loss function and the second loss function converge respectively.

[0015] According to some embodiments of the present invention, the expression of the first loss function is: L G (G A→B )=L adv (G A→B )+γL cycle +pL SSIM1

[0016] L G (G A→B ) is the first loss function, L adv (G A→B ) is the adversarial loss constraint term of the first loss function, L cycle is the cycle consistency loss constraint term of the first loss function, L SSIM1 is the cell structure similarity constraint term of the first loss function; wherein, D B is the first discriminant network, D B (G A→B (a)) is the probability that the intermediate virtual stained image is judged to be the real stained image, is the expected value of the adversarial loss value corresponding to each Raman image in the Raman image domain A; wherein, G B→A (G A→B (a)) Inputting the intermediate virtual stained image into the second generation network G B→A The reconstructed Raman image generated, is the expected value of the mean absolute error between each Raman image in the Raman image domain A and the corresponding reconstructed Raman image, G A→B (G B→A (a)) is to input the intermediate Raman image into the first generation network G A→B The generated reconstructed true stained image, is the expected value of the mean absolute error between each real stained image in the real stained image domain B and the corresponding reconstructed real stained image; wherein, γ=10, p=2.

[0017] According to some embodiments of the present invention, the expression of the second loss function is: L G (G B→A )=L adv (G B→A )+γL cycle +pL SSIM2

[0018] Among them, L G (G B→A ) is the second loss function, L adv (G B→A ) is the adversarial loss constraint term of the second loss function, L cycle is the cycle consistency loss constraint term of the second loss function, L SSIM2 is the cell structure similarity constraint term of the second generative network; wherein, D A is the second discriminant network, D A (G B→A (b)) is the probability that the intermediate Raman image is judged to be the Raman image, is the expected value of the adversarial loss value corresponding to each real stained image in the real stained image domain B.

[0019] According to some embodiments of the present invention, the loss function further includes a congruent mapping loss constraint term, and the expression of the congruent mapping loss constraint term is:

[0020] Among them, G B→A (a)) is to input a into the second generation network G B→AThe generated congruent mapping Raman image, is the expected value of the mean absolute error between each Raman image in the Raman image domain A and the corresponding congruent mapping Raman image, G A→B (b)) is to input b into the first generation network G A→B The generated congruent mapping true stained image, is the expected value of the mean absolute error between each true stained image in the true stained image domain B and the corresponding congruent mapping true stained image; wherein the first loss function can also be expressed as: L G (G A→B )=L adv (G A→B )+γL cycle +λL idt +pL SSIM1 , the second loss function can also be expressed as: L G (G B→A )=L adv (G B→A )+γL cycle +λL idt +pL SSIM2 , where λ=1.

[0021] According to some embodiments of the present invention, training a cyclic generative adversarial network based on the Raman image and the real stained image so that the value of the loss function of the generative network in the cyclic generative adversarial network converges includes: iteratively training the cyclic generative adversarial network based on multiple Raman images and multiple corresponding real stained images until the value of the loss function of the generative network in the cyclic generative adversarial network converges.

[0022] According to some embodiments of the present invention, obtaining a target generative network model based on the generative network in a trained cyclic generative adversarial network with a converged loss function value includes: in response to the values ​​of the first loss function and the second loss function respectively converging, using the parameters of the first generative network in the trained cyclic generative adversarial network with a converged loss function value to obtain the target generative network model.

[0023] According to some embodiments of the present invention, the training method further includes: performing image registration on the Raman image and the corresponding true stained image so that the Raman image and the true stained image are aligned at the pixel level; inputting the registered Raman image and the corresponding registered true stained image into the trained cyclic generative adversarial network with a converged loss function value to verify the trained cyclic generative adversarial network with a converged loss function value.

[0024] According to some embodiments of the present invention, verifying the trained cyclic generative adversarial network with a converged loss function value includes: determining whether the value of the loss function of the generative network in the trained cyclic generative adversarial network with a converged loss function value is overfitting; when it is determined that the value of the loss function of the generative network in the trained cyclic generative adversarial network with a converged loss function value is not overfitting, obtaining a target generative network model based on the generative network in the trained cyclic generative adversarial network with a converged loss function value; and when it is determined that the value of the loss function of the generative network in the trained cyclic generative adversarial network with a converged loss function value is overfitting, adjusting the parameters of the trained cyclic generative adversarial network with a converged loss function value and training the cyclic generative adversarial network with the adjusted parameters based on the Raman image and the real stained image.

[0025] According to some embodiments of the present invention, obtaining a Raman image of a cell sample and a corresponding true staining image of the cell sample includes: performing Raman scattering microscopy imaging on the cell sample to obtain the Raman image of the cell sample; and performing chemical staining and bright-field microscopy imaging on the cell sample for which the Raman image has been generated to obtain the corresponding true staining image of the cell sample.

[0026] According to some embodiments of the present invention, performing Raman scattering microscopic imaging on the cell sample to obtain the Raman image of the cell sample includes: irradiating the cell sample with a first excitation light and a second excitation light; in response to irradiating the cell sample with the first excitation light and the second excitation light, receiving reflected light having at least one Raman shift characteristic peak from the cell sample to obtain the Raman image of the cell sample.

[0027] According to some embodiments of the present invention, the first excitation light and the second excitation light each have a predetermined frequency, and in response to the first excitation light and the second excitation light irradiating the cell sample, receiving reflected light having at least one Raman shift characteristic peak from the cell sample to obtain the Raman image of the cell sample includes: in response to the first excitation light and the second excitation light having a predetermined frequency irradiating the cell sample, receiving reflected light having only one Raman shift characteristic peak from the cell sample to obtain the Raman image of the cell sample.

[0028] According to some embodiments of the present invention, the first excitation light and the second excitation light are pulsed lasers, and the frequency difference between the first excitation light and the second excitation light is 2800-3100 cm -1 , repetition frequency is greater than 50MHz, and pulse width is 100fs-20ps.

[0029] According to some embodiments of the present invention, the first excitation light includes pump light, the second excitation light includes Stokes light, and in response to the first excitation light and the second excitation light irradiating the cell sample, receiving reflected light having at least one Raman shift characteristic peak from the cell sample to obtain the Raman image of the cell sample includes: in response to the pump light and the Stokes light irradiating the cell sample, receiving reflected light having at least one Raman shift characteristic peak from the cell sample to obtain a coherent Raman scattering imaging image of the cell sample, the coherent Raman scattering imaging image includes a stimulated Raman scattering imaging image, a stimulated Raman photothermal imaging image, or a coherent anti-Stokes Raman scattering imaging image.

[0030] According to some embodiments of the present invention, performing Raman scattering microscopy on the cell sample to obtain the Raman image of the cell sample includes: performing Raman scattering microscopy on each part of the cell sample in a local field of view to obtain a local Raman image of the cell sample; splicing the local Raman images of each part of the cell sample to generate the Raman image of the cell sample; and chemically staining and bright-field microscopy on the cell sample for which the Raman image has been generated to obtain the corresponding true staining image of the cell sample includes: chemically staining the cell sample for which the Raman image has been generated, and performing bright-field microscopy on each part of the chemically stained cell sample in a local field of view to obtain a local true staining image of the cell sample, and splicing the local true staining images of each part of the cell sample to generate the true staining image of the cell sample.

[0031] According to another aspect of the present invention, a virtual staining method for enhancing the display of cell structure is also provided, comprising: obtaining a target Raman image of a target cell sample; inputting the target Raman image into a target generation network model according to any one of the above-mentioned training methods to obtain a target virtual staining image of the target cell sample, wherein the target virtual staining image is used to enhance the display of the cell structure in the target cell sample.

[0032] According to some embodiments of the present invention, obtaining a target Raman image of a target cell sample includes: performing Raman scattering microscopy imaging on the target cell sample to obtain the target Raman image of the target cell sample.

[0033] According to some embodiments of the present invention, performing Raman scattering microscopic imaging on the target cell sample to obtain the target Raman image of the target cell sample includes: irradiating the target cell sample with a first excitation light and a second excitation light; in response to irradiating the target cell sample with the first excitation light and the second excitation light, receiving reflected light having at least one Raman shift characteristic peak from the target cell sample to obtain the target Raman image of the target cell sample.

[0034] According to some embodiments of the present invention, the first excitation light and the second excitation light each have a predetermined frequency, and in response to the first excitation light and the second excitation light irradiating the target cell sample, receiving reflected light having at least one Raman shift characteristic peak from the target cell sample to obtain the target Raman image of the target cell sample includes: in response to the first excitation light and the second excitation light having a predetermined frequency irradiating the target cell sample, receiving reflected light having only one Raman shift characteristic peak from the target cell sample to obtain the target Raman image of the target cell sample.

[0035] According to some embodiments of the present invention, the first excitation light and the second excitation light are pulsed lasers, and the frequency difference between the first excitation light and the second excitation light is 2800-3100 cm -1 , repetition frequency is greater than 50MHz, and pulse width is 100fs-20ps.

[0036] According to some embodiments of the present invention, the first excitation light includes pump light, the second excitation light includes Stokes light, and in response to the first excitation light and the second excitation light irradiating the target cell sample, reflected light having at least one Raman shift characteristic peak is received from the target cell sample to obtain the target Raman image of the target cell sample; in response to the pump light and the Stokes light irradiating the target cell sample, reflected light having at least one Raman shift characteristic peak is received from the target cell sample to obtain a coherent Raman scattering imaging image of the cell sample, the coherent Raman scattering imaging image includes a stimulated Raman scattering imaging image, a stimulated Raman photothermal imaging image, or a coherent anti-Stokes Raman scattering imaging image.

[0037] According to some embodiments of the present invention, performing Raman scattering microscopy imaging on the target cell sample to obtain the target Raman image of the target cell sample includes: performing Raman scattering microscopy imaging on each local part of the target cell sample in a local field of view to obtain a local target Raman image of the target cell sample; and splicing the local target Raman images of each local part of the target cell sample to obtain the target Raman image of the target cell sample.

[0038] According to another aspect of the present invention, a virtual staining system is also provided, comprising: an image acquisition component configured to obtain a target Raman image of a target cell sample; and an image processing component configured to input the target Raman image into a target generation network model according to any one of the above-mentioned training methods to obtain a target virtual staining image of the target cell sample, wherein the target virtual staining image is used to enhance the display of the cell structure in the target cell sample.

[0039] According to some embodiments of the present invention, the image acquisition component is configured to perform Raman scattering microscopic imaging on the target cell sample to obtain the target Raman image of the target cell sample.

[0040] According to some embodiments of the present invention, the virtual staining system further includes: a laser source configured to generate a first excitation light and a second excitation light for irradiating a target cell sample; and the image acquisition component is further configured to irradiate the target cell sample in response to the first excitation light and the second excitation light, and receive reflected light having at least one Raman shift characteristic peak from the target cell sample to obtain the target Raman image of the target cell sample.

[0041] According to some embodiments of the present invention, wherein the first excitation light and the second excitation light each have a predetermined frequency, the image acquisition component is further configured to: in response to the first excitation light and the second excitation light having the predetermined frequency irradiating the target cell sample, receive reflected light having only one Raman shift characteristic peak from the target cell sample to obtain the target Raman image of the target cell sample.

[0042] According to some embodiments of the present invention, the virtual staining system further includes: an optical path component, comprising a two-dimensional galvanometer assembly and a first filter configured to guide the first excitation light and the second excitation light to the target cell sample; a sample carrying component configured to carry the target cell sample to receive irradiation by the first excitation light and the second excitation light; and an objective lens component configured to receive the reflected light from the target cell sample and transmit the reflected light to the image acquisition component.

[0043] According to some embodiments of the present invention, the virtual staining system further includes: an automatic focusing component, wherein the automatic focusing component includes a focus detection unit, a second filter, and a movable component for moving the objective lens component; the focus detection unit is configured to: generate a third excitation light, the third excitation light is irradiated onto the sample-carrying component through the second filter; detect the detection reflected light reflected from the sample-carrying component and returned to the focus detection unit; and control the movable component according to the detection result so that the objective lens component receives the reflected light from the target cell sample.

[0044] According to some embodiments of the present invention, the first excitation light and the second excitation light are pulsed lasers, and the frequency difference between the first excitation light and the second excitation light is 2800-3100 cm -1 , repetition frequency is greater than 50MHz, and pulse width is 100fs-20ps.

[0045] According to some embodiments of the present invention, the first excitation light includes pump light, the second excitation light includes Stokes light, and the image acquisition component is further configured to: in response to the pump light and the Stokes light irradiating the cell sample, receive reflected light having at least one Raman shift characteristic peak from the cell sample to obtain a coherent Raman scattering imaging image of the cell sample, wherein the coherent Raman scattering imaging image includes a stimulated Raman scattering imaging image, a stimulated Raman photothermal imaging image, or a coherent anti-Stokes Raman scattering imaging image.

[0046] According to some embodiments of the present invention, the image acquisition component is configured to: perform Raman scattering microscopic imaging of various parts of the target cell sample under a local field of view to obtain a local target Raman image of the target cell sample; and stitch the local target Raman images of various parts of the target cell sample to obtain the target Raman image of the target cell sample.

[0047] Embodiments of the present invention provide a training method for obtaining a target generation network model, a virtual staining method for enhancing the display of cell structure, and a virtual staining system.

[0048] Therefore, according to the training method for obtaining a target generation network model in an embodiment of the present invention, Raman imaging technology is used to obtain Raman images that can more accurately describe the cell structure of a cell sample; at the same time, in the training process using the Raman images and real stained images of the cell samples, a structural similarity function is used to constrain the loss function in the recurrent generative adversarial network for cell structure consistency, so that the target generation network model obtained based on the trained recurrent generative adversarial network can generate a virtual stained image that is more consistent with the real stained image, avoiding problems such as the virtual staining process reversing the staining of the cell nucleus and cytoplasm, thereby improving the accuracy of virtual cell staining and achieving enhanced display of cell structure.

[0049] In this way, according to the virtual staining method and virtual staining system for enhancing the display of cell structure according to the embodiments of the present invention, the target cell sample is virtually stained by the target generation network model obtained by the above-mentioned training method, and a virtual staining image that accurately enhances the display of the target cell sample structure can be obtained, thereby realizing the expansion of virtual staining imaging from the tissue level to the cellular level, so that the virtual staining image has significant consistency with the real staining image, so that clinicians or researchers can analyze the cell structure of the target cell sample based on the virtual staining image to make accurate judgments and decisions. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are merely some exemplary embodiments of the present invention, and those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0051] FIG1 shows an example diagram of a virtual staining method based on ultraviolet surface excitation microscopy in the prior art;

[0052] FIG2 shows example diagrams of Raman images and true stained images obtained based on different methods including the training method according to an embodiment of the present invention;

[0053] FIG3 shows a flow chart of a training method for obtaining a target generation network model according to some embodiments of the present invention;

[0054] FIG4 shows an architecture diagram of a recurrent generative adversarial network according to some embodiments of the present invention;

[0055] FIG5 shows a flow chart of training a recurrent generative adversarial network according to some embodiments of the present invention;

[0056] FIG6 shows a schematic diagram of the network structure of a generator network and a discriminator network according to some embodiments of the present invention;

[0057] FIG7 shows a schematic diagram of stitching and registering a Raman image and a true stained image according to some embodiments of the present invention;

[0058] FIG8 shows a schematic diagram of evaluating a target generation network model according to some embodiments of the present invention;

[0059] FIG9 shows a flow chart of a virtual staining method for enhancing display of cell structures according to some embodiments of the present invention;

[0060] FIG10 shows a block diagram of a virtual coloring system according to some embodiments of the present invention;

[0061] FIG11 shows an example structural diagram of a virtual coloring system 1000 according to some embodiments of the present invention;

[0062] FIG12 shows a schematic diagram of a microscopic imaging system for Raman images according to some embodiments of the present invention;

[0063] FIG13 shows the spontaneous Raman scattering spectra of pure lipid samples and pure protein samples in the carbon-hydrogen bond vibration region according to some embodiments of the present invention, that is, a graph of the spontaneous Raman scattering intensity under different frequency differences between the pump light and the Stokes light;

[0064] FIG14 shows another example structural diagram of a virtual coloring system 1000 according to some embodiments of the present invention;

[0065] FIG15 shows a block diagram of an electronic device 1500 according to some embodiments of the present invention. DETAILED DESCRIPTION

[0066] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings of the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the described embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0067] Unless otherwise defined, the technical or scientific terms used in the present invention should have the usual meanings understood by people with ordinary skills in the field to which the present invention belongs. The "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly. In order to keep the following description of the embodiments of the present invention clear and concise, the present invention omits detailed descriptions of some known functions and known components.

[0068] Flowcharts are used in the present invention to illustrate the steps of the method according to embodiments of the present invention. It should be understood that the preceding or following steps do not necessarily need to be performed in exact order. Instead, the various steps may be performed in reverse order or simultaneously. Furthermore, other operations may be added to these processes, or one or more steps may be removed from these processes.

[0069] In the description and drawings of the present invention, elements are described in singular or plural form, depending on the embodiment. However, the singular and plural forms are appropriately selected for the situations presented merely for convenience of explanation and are not intended to limit the present invention thereto. Therefore, a singular form may include a plural form, and a plural form may also include a singular form, unless the context clearly indicates otherwise.

[0070] As mentioned above, virtual staining mainly includes virtual staining methods based on exogenous fluorescent labeling microscopy and virtual staining methods based on label-free microscopy.

[0071] Among these, virtual staining methods based on microscopic imaging of exogenous fluorescent markers provide basic chemical composition information by capturing the contrast of fluorescence signal intensities at different locations within the specimen during imaging. As shown in Figure 1, microscopy with ultraviolet surface excitation (MUSE) is a representative method of this type. It enables virtual H&E staining of tissues without the need for fixation, embedding, or sectioning, reducing the traditional overnight process to a 2-3 minute fluorescence staining and a few seconds of image processing.

[0072] Specifically, virtual staining methods based on exogenous fluorescent labeling imaging first stain the specimen with fluorescent dyes (such as acridine orange, DAPI, etc.) to emphasize key features in virtual staining, such as cell nuclei within the specimen. The specimen is then imaged and converted into a stained image based on the grayscale value of each pixel in the result. For example, in MUSE, after soaking the tissue with Hoechst 33342 and Rhodamine B or other dyes, a UV LED is used to excite the fluorescent markers on the surface layer and image it. The colors in the image are then unmixed, adjusted, and remixed to make the virtual staining result closer to H&E staining.

[0073] However, the exogenous dyes introduced by the virtual staining method based on exogenous fluorescent labeling microscopy will have irreversible effects on the physical and chemical properties of the specimen, which can easily interfere with other analyses of the specimen. In addition, the autofluorescence effect of the specimen and other environmental factors will also affect the virtual staining results. In addition, the virtual staining algorithm based on exogenous fluorescent labeling microscopy is usually based on experience or deduction, and it is difficult to ensure the authenticity of the visual effect of the staining results. Therefore, compared with the virtual staining method based on exogenous fluorescent labeling microscopy, it is more desirable to use a virtual staining method based on label-free microscopy to obtain more realistic virtual staining images.

[0074] Among label-free microscopy-based virtual staining methods, currently used imaging methods primarily include autofluorescence imaging (AFI), photoacoustic imaging (PAI), and quantitative phase imaging (QPI). Generative adversarial networks (GANs) are then used to virtually stain the generated images. Due to the lack of exogenous labels to provide contrast, these methods rely on other methods to provide information for virtual staining. For example, relevant information can be obtained based on imaging parameters and image processing methods. Deep learning techniques enable trained deep neural networks to automatically identify high-dimensional features in the input data that can be used to address specific tasks. GANs are a model architecture commonly used for virtual staining in histology, and their significant contribution is to achieve staining results that are highly similar to the actual stained images. GAN-based virtual staining techniques can be used for virtual staining of autofluorescence, photoacoustic, and quantitative phase images. They can be used to identify cell specimens in various tissues, including thyroid and liver, and can be used to virtually stain with staining methods similar to those of various dyes, such as H&E and Masson.

[0075] However, there are two problems with the virtual staining method based on label-free microscopy, which result in the virtual staining method based on label-free microscopy being unable to provide information about cell structure as accurately as real staining images.

[0076] On the one hand, autofluorescence imaging has poor molecular specificity and is highly dependent on components such as NADH and FAD, making it difficult to distinguish cell structures when imaging cells. The spatial resolution of photoacoustic imaging is only at the micron level, which cannot clearly image cells and their internal structures. Quantitative phase imaging also cannot provide sufficient contrast for cell structures. Therefore, the current common imaging technology used for label-free virtual staining cannot accurately identify cell structures, which in turn makes it difficult for generative adversarial network (GAN) virtual staining to achieve cellular-level staining and provide accurate cellular-level information. As a result, clinicians and researchers cannot make accurate diagnoses and decisions based on virtual staining images obtained based on such methods.

[0077] On the other hand, there are two main ways to train GANs: one is supervised training, in which the training dataset contains pixel-level matched unlabeled microscopic images and corresponding stained images, resulting in high accuracy of virtual staining results, and the loss function can clearly represent the training progress. However, the preparation of datasets for supervised training is difficult, and it is difficult to obtain a large number of pixel-level matched unlabeled microscopic images and corresponding stained images; the other is unsupervised training, which usually uses a cycle generative adversarial network (CycleGAN). The dataset only needs two image domains from the model input and output, without the need for exact matching, which significantly reduces the difficulty of dataset preparation. However, since in this case it is impossible to directly evaluate the accuracy of the staining results by referencing the real image (ground truth, GT) during training, existing technologies have found it difficult to achieve virtual staining at the cellular structure level using unsupervised training cycle generative adversarial networks.

[0078] Specifically, during the research and development process, the inventors of the present invention discovered through a large number of experimental studies that the virtual staining process of the virtual staining image generated by the generative network trained by the traditional method does not maintain the consistency of the cell structure in the input image and the output image. The machine learning algorithm easily mistakenly takes the cell nucleus as the cytoplasm and stains the cell nucleus with the color corresponding to the cytoplasm. Correspondingly, the cytoplasm is also stained with the color corresponding to the cell nucleus. That is, the virtual staining process has the problem of reversing the staining of the cell nucleus and cytoplasm.

[0079] For example, FIG2 shows examples of Raman images and true staining images obtained based on different methods, including the training method of an embodiment of the present invention. As shown in FIG2 , the first row in FIG2 corresponds to a Raman image of a cell sample (the cell sample may be exfoliated cells from peritoneal lavage fluid). Specifically, the left half a in the first row is the Raman image, and the right half b and c are magnified images of two regions in the left Raman image. The second row corresponds to the true staining image of the cell sample (the true staining image may be obtained by staining the cell sample using a hematoxylin-eosin staining method, for example). The third row corresponds to a first virtual staining image of the cell sample, which is generated using a generative network using conventional techniques. By comparison, it can be seen that in the true staining image in the second row, the cells indicated by the arrows have a light color on the left (corresponding to the cytoplasm) and a dark color on the right (corresponding to the nucleus). However, in the first virtual staining image in the third row, the nucleus and cytoplasm are reversed. In FIG2 , the cells indicated by the arrows in the first virtual staining image have a dark color on the left (corresponding to the nucleus) and a light color on the right (corresponding to the cytoplasm), which is exactly the opposite of the corresponding positions in the true staining image.

[0080] In order to solve the above technical problems, the present invention provides a training method for obtaining a target generation network model, a virtual staining method for enhancing the display of cell structure, and a virtual staining system.

[0081] The following will illustrate the training method for obtaining a target generation network model, the virtual staining method for enhancing the display of cell structure, and the virtual staining system provided by the present application through specific embodiments.

[0082] First embodiment

[0083] The training method for obtaining the target generation network model provided by the present invention will be described in detail below with reference to the accompanying drawings.

[0084] Figure 3 shows a flow chart of a training method for obtaining a target generation network model according to some embodiments of the present invention. As shown in Figure 3, first, in step S310, a Raman image of a cell sample and a corresponding true staining image of the cell sample are obtained.

[0085] In one example, cell samples may include cancer cells, immune cells, lymphocytes, mesothelial cells, epithelial cells, blood cells, granulocytes, and the like. Cell samples may be collected from organs or tissues of animals or plants. Cell samples may be cell samples requiring cell structure analysis in practical application scenarios, such as clinical application scenarios, research and development scenarios, and the applicable scenarios of the embodiments of the present invention are not specifically limited herein.

[0086] Since the chemical composition of cell samples changes after chemical staining, affecting the results of coherent Raman scattering microscopy, it is necessary to first perform Raman scattering microscopy imaging on unstained cell samples before chemically staining the cell samples to ensure the accuracy of the Raman image of the cell samples.

[0087] According to one embodiment of the present invention, Raman scattering microscopy can be performed on a cell sample to obtain a (coherent) Raman image of the cell sample. The cell sample for which the Raman image has been generated is then chemically stained and subjected to bright-field microscopy to obtain a corresponding true stained image of the cell sample.

[0088] In one example, chemical staining and brightfield microscopy can be specific methods used in the artificial chemical staining method described above. Those skilled in the art will appreciate how to obtain a true staining image using the artificial chemical staining method, and therefore, this will not be described in detail herein. In one example, the true staining image can be a true H&E staining image.

[0089] In one example, a real stained image of a cell sample obtained by an artificial chemical staining method is used to input a recurrent generative adversarial network together with a Raman image of the cell sample to serve as a training set for the recurrent generative adversarial network. It should be noted that in this article, the Raman image of the cell sample and the corresponding real stained image of the cell sample are not intended to refer to only a single image, but can include one or more images or image sets, thereby enabling iterative training of the recurrent generative adversarial network.

[0090] Coherent Raman scattering microscopy is a technology that uses spatially resolved spectral information to generate images. The images obtained by coherent Raman scattering microscopy can fully reflect the chemical information of cell structures. In other words, coherent Raman scattering microscopy has obvious advantages when considering spatial resolution, molecular specificity and other aspects. It can utilize the Raman shift characteristic peaks of lipids and proteins that are widely present in various cells, and can therefore be widely used for cell imaging. In addition, coherent Raman imaging of cell samples has molecular specificity at the chemical bond level, and solves the defects of weak spontaneous Raman scattering signals and slow imaging speeds. Therefore, compared with other imaging methods such as API, the use of Raman imaging can more accurately image cell structures, so that accurate virtual staining can be achieved at the cell structure level based on neural networks.

[0091] In this document, the term coherent Raman scattering microscopy is used interchangeably with coherent Raman imaging, Raman imaging, and the like.

[0092] It should be noted that because the cell sample must first be Raman imaged and then artificially stained, and artificial staining takes time, the cells in the obtained Raman image and the actual stained image may be offset, resulting in the images not being strictly aligned at the pixel level. However, according to the principles of recurrent generative adversarial networks, such images can also be used to train the model. By randomly cropping the same number of fixed-size images from each of the two types of images, a training set for the recurrent generative network model can be obtained.

[0093] After obtaining the Raman image of the cell sample and the corresponding real staining image of the cell sample, in step S320, a cyclic generative adversarial network can be trained based on the Raman image and the real staining image to converge the value of the loss function of the generative network in the cyclic generative adversarial network.

[0094] During training, the recurrent generative adversarial network (GAN) uses Raman images and real stained images as training samples. By pitting the generative and discriminative networks within the GAN against each other, the GAN can fully learn how to generate virtual stained images based on Raman images that closely resemble the real stained images. Once the GAN is trained, the target generative network can generate realistic virtual stained images based on the input Raman images. These virtual stained images, similar to the real stained images, can be used to enhance the display of cellular structures in target cell samples.

[0095] Specifically, Figure 4 shows an architecture diagram of a cyclic generative adversarial network according to some embodiments of the present invention. As shown in Figure 4, according to one embodiment of the present disclosure, the generative network may include a first generative network G A→B and the second generation network G B→A The discriminant network may include a first discriminant network D B and the second discriminant network D A A represents the Raman image domain A consisting of one or more Raman images, and B represents the real dyeing image domain B consisting of one or more real dyeing images.

[0096] The first generation network G A→B It is used to convert the input Raman image into an intermediate virtual stained image and output the intermediate virtual stained image. B Identify the input to the first discriminant network D B Is the image a real colored image or the first generated network G A→B The output is the intermediate virtual stained image, and the probability that the input image is the real stained image is output. Similarly, the second generation network G B→A With the second discriminant network D ASimilarly, the two discriminant networks supervise the two generative networks respectively, driving the cyclic generative adversarial network to continuously improve the similarity between the generated image (i.e., the intermediate virtual stained image or the intermediate Raman image) and the image used for discrimination (i.e., the real stained image or the original Raman image), making it increasingly difficult for the discriminant network to distinguish the input image.

[0097] When training a neural network model, a loss function is typically defined, and the loss function value is obtained from the output of the neural network model. Generally speaking, the larger the loss function value, the worse the current performance of the neural network model, and the greater the degree of penalty for the neural network model. Subsequently, the backpropagation algorithm updates the neural network model parameters in a direction that reduces the loss value, allowing the neural network model to produce better results. Furthermore, the greater the penalty, the greater the magnitude of the change in the neural network model parameters. Therefore, for adversarial recurrent generative adversarial networks, the loss function can be converged by optimizing the network parameters, so that the trained recurrent generative adversarial network can be used for virtual coloring.

[0098] According to one embodiment of the present disclosure, the loss function may include a first loss function for the first generative network and a second loss function for the second generative network. The loss function may also include loss functions for the discriminant network respectively. Considering that when the loss function of the generative network converges, the loss function of the discriminant network will also converge, and the target generative network model (i.e., a model that can be used to virtually stain the Raman image of the cell sample to be identified) is generated based on the generative network part in the cyclic adversarial generative network, it is preferable to consider the loss function of the generative network without considering the loss function of the discriminant network.

[0099] Figure 5 shows a training flowchart for a recurrent generative adversarial network according to some embodiments of the present invention. As shown in Figure 5, training the recurrent generative adversarial network based on Raman images and real stained images to converge the loss function of the generative network in the recurrent generative adversarial network can be specifically achieved by the following steps.

[0100] In step S510, the Raman image can be input into the first generation network G A→B To generate an intermediate virtual stained image,

[0101] In step S520, the intermediate virtual dyed image can be input into the first discriminant network D in the cyclic generative adversarial network. B To determine the probability that the intermediate virtual stained image is judged as a real stained image, and

[0102] In step S530, the real dyed image can be input into the second generation network G B→A To generate an intermediate Raman image,

[0103] In step S540, the intermediate Raman image can be input into the second discriminant network D in the cyclic generative adversarial network. A To determine the probability that the intermediate Raman image is judged as a Raman image,

[0104] In step S550, based on the first discriminant network D B and the second discriminant network D A The determined probability adjusts the parameters of the cyclic generation adversarial network so that the values ​​of the first loss function and the second loss function converge. According to one embodiment of the present invention, the first discriminant network D B and the second discriminant network D A The determined probabilities determine values ​​of a first loss function and a second loss function; and parameters of a cyclic generative adversarial network are adjusted based on the values ​​of the first loss function and the second loss function so that the values ​​of the first loss function and the second loss function converge.

[0105] As previously mentioned, the Raman image of the cell sample may include one or more Raman images, while the corresponding true staining image of the cell sample may include one or more corresponding true staining images. Therefore, according to one embodiment of the present disclosure, a recurrent generative adversarial network may be iteratively trained based on the multiple Raman images and the multiple corresponding true staining images until the loss function of the generator network in the recurrent generative adversarial network converges.

[0106] In an example of the present invention, the above steps S510-S550 may be repeatedly performed until the values ​​of the first loss function and the second loss function converge.

[0107] Figure 6 shows a schematic diagram of the network structure of the generator network and the discriminator network according to some embodiments of the present invention. As shown in Figure 6, the cyclic generative adversarial network can be built using, for example, PyTorch 1.7.0, the training graphics card can be used, for example, NVIDIA Tesla P100-16G, the optimizer can be used, for example, Adam (with β values ​​of 0.5 and 0.999), and the learning time can be set to 10 -4 , and is reduced to 10 in the last 1 / 3 of the training phase -5 Regarding the model architecture of the cyclic generative adversarial network, the generative network (the first generative network and the second generative network) uses ResNet, and the discriminative network (the first discriminative network and the second discriminative network) uses 70×70PatchGAN.

[0108] As shown in Figure 6, residual block is the residual block, BN is batch normalization, Averaging is averaging, ReLU is the linear rectification function, Conv is the convolution layer, S is the convolution stride, and ConvTranspose is the deconvolution layer, where: ResidualBlock(x)=x+Conv{Conv{x}}

[0109] Among them, Conv{} is a convolution with a step size of 1, batch normalization and ReLU are performed in sequence, and the convolution kernel size is 3 except for the generator input and output convolution which is 7.

[0110] In one example, considering the huge amount of computation required in the training process, the training process of the target generation network in an embodiment of the present invention can be performed on a server instead of a terminal. After the server completes the training, the target generation network is sent to the terminal for use by the terminal.

[0111] In order to control the consistency of the cell area and its internal nucleus and cytoplasm structure before and after virtual staining, improve the accuracy at the cellular level, and avoid problems such as the virtual staining process reversing the staining of the nucleus and cytoplasm, according to some embodiments of the present invention, the loss function may include a cell structure similarity constraint term. The first loss function and the second loss function may include a cell structure similarity constraint term, which may be based on a structural similarity (SSIM) function to constrain the consistency of the cell structure in the input image and the output image of the generated network. In some examples, the cell structure constraint term may also be based on a function such as a neural network perceptual loss or a morphological algorithm to constrain the consistency of the cell structure in the input image and the output image of the generated network.

[0112] In one example, the first loss function of the first generation network may include at least a first cell structure constraint term, and the first cell structure constraint term is used to constrain the consistency of the cell structure in the input image and the output image of the first generation network; and the loss function of the second generation network may include at least a second cell structure constraint term, and the second cell structure constraint term is used to constrain the consistency of the cell structure in the input image and the output image of the second generation network.

[0113] The cell structure constraint expression of the first loss function can be shown as Formula 1 or Formula 2:

[0114] In one example, the cell structure constraint term of the first loss function also includes the first generation network G A→BWith the second generation network G B→A However, in the calculation, only the first half can be considered, that is, only the first generation network G A→B This is due to the first generation network G A→B When the loss function is calculated, only the first generation network G A→B The loss of the relevant part is effective, and the loss of the second generation network G B→A The relevant part is due to the first generation network G A→B The parameters are irrelevant and do not affect the first generation network G A→B Similarly, the cell structure constraint term of the second loss function can also be simplified, that is, only considering the second generation network G B→A Relevant parts.

[0115] Therefore, according to one embodiment of the present invention, the cell structure similarity constraint term L of the first loss function is SSIM1 Can include The cell structure similarity constraint term L of the second loss function SSIM2 Can include

[0116] Where a is a Raman image in the Raman image domain A consisting of one or more Raman images, G A→B (a) is to input a into the first generation network G A→B The generated intermediate virtual stained image, SSIM(G A→B (a), a) is the structural similarity value between the intermediate virtual stained image and the Raman image, is the expected value of the structural similarity loss value corresponding to each Raman image in the Raman image domain A; and b is a real stained image in the real stained image domain B consisting of one or more real stained images, G B→A (b) Input b into the second generation network G B→A The generated intermediate Raman image, SSIM(G B→A (b),b) is the structural similarity value between the intermediate Raman image and the true dyeing image, is the expected value of the structural similarity loss value corresponding to each real stained image in the real stained image domain B.

[0117] In addition to the cell structure constraint, for the cycle generative adversarial network, the loss function (including the first loss function and the second loss function) can also include constraints such as adversarial loss and cycle consistency loss.

[0118] Adversarial loss Ladv This is a loss function constraint that is defined for both the generator and the discriminator, and that contradicts each other, placing them in a competitive relationship. For the generator, the more confidently the discriminator identifies its output as false, that is, the closer the discriminator's output is to 0, the greater the penalty it receives. Simultaneously, for the discriminator, on the one hand, the more confidently it identifies the generator's output as true, that is, the closer its own output is to 1, the greater the penalty it receives. On the other hand, the more confidently it identifies a real image as false, that is, the closer its own output is to 0, the greater the penalty it receives.

[0119] The formula for the adversarial loss constraint term can be, for example, a least squares GAN (LSGAN), as shown in the following formulas 3 to 6. A→B The expression of the adversarial loss constraint term, Formula 4 is the second generation network G B→A The expression of the adversarial loss constraint term, Formula 5 is the second discriminant network D B The expression of the adversarial loss constraint term, Formula 6 is the first discriminant network D A The expression of the adversarial loss constraint term is:

[0120] Wherein, a is a Raman image in a Raman image domain A consisting of one or more Raman images, and b is a real dyeing image in a real dyeing image domain B consisting of one or more real dyeing images. dataA 、P dataB are the probability distributions of the two types of images reflected by the training set (the real staining image set of cell samples and the Raman image set of cell samples). E is the expected value, such as The probability distribution of Raman images of cell samples follows P dataA The expected value of a specific function (constraint term) when G A→B With D B Correspondingly, G B→A With D A Correspondingly, L D is the total loss of the discriminant network.

[0121] Cycle consistency loss L cycle Only the generative network is defined, and its design idea comes from the following: when a sentence is translated from language A to language B, and then translated from language B back to language A, the result should be the same as the original sentence. Similarly, the Raman image of the cell sample is transformed through two generative networks in sequence, and the reconstructed Raman image should also be the same as the original Raman image of the original cell sample. The greater the difference between the reconstructed Raman image and the original Raman image, the better the first discriminant network D AThe purpose of cycle consistency loss is that any generative network should not lose its key information when processing the image, otherwise it will not be able to reconstruct the original image.

[0122] Since the Cycle Generative Adversarial Network has no labels during training, the cycle consistency loss is an additional constraint on the network. The following formula 7 is the expression of the cycle consistency loss constraint term, where ||·||1 is the L1 loss (mean absolute value loss):

[0123] The loss value is obtained in the above way, so that the training of the recurrent generative adversarial network model only requires an unpaired dataset. That is, for one of the Raman images, there is no need for a real stained image that corresponds to its content at the pixel level.

[0124] Based on the various constraints described above, according to one embodiment of the present invention, the expression of the first loss function can be shown as Formula 8: L G (G A→B )=L adv (G A→B )+γL cycle +pL SSIM1 Formula 8

[0125] L G (G A→B ) is the first loss function, L adv (G A→B ) is the adversarial loss constraint term of the first loss function, L cycle is the cycle consistency loss constraint term of the first loss function, L SSIM1 is the cell structure similarity constraint term of the first loss function;

[0126] in, D B is the first discriminant network, D B (G A→B (a)) is the probability that the intermediate virtual stained image is judged as the real stained image, is the expected value of the adversarial loss value corresponding to each Raman image in the Raman image domain A;

[0127] in,

[0128] G B→A (G A→B (a)) Input the intermediate virtual dyed image into the second generation network G B→A The reconstructed Raman image generated, is the expected value of the mean absolute error between each Raman image in the Raman image domain A and the corresponding reconstructed Raman image, G A→B(G B→A (a)) Input the intermediate Raman image into the first generation network G A→B The generated reconstructed true stained image, is the expected value of the mean absolute error between each true stained image in the true stained image domain B and the corresponding reconstructed true stained image; where γ = 10 and p = 2.

[0129] According to one embodiment of the present invention, the expression of the second loss function can be shown as Formula 9: L G (G B→A )=L adv (G B→A )+γL cycle +pL SSIM2 Formula 9

[0130] Among them, L G (G B→A ) is the second loss function, L adv (G B→A ) is the adversarial loss constraint term of the second loss function, L cycle is the cycle consistency loss constraint term of the second loss function, L SSIM2 is the cell structure similarity constraint of the second generation network; D A is the second discriminant network, D A (G B→A (b)) is the probability that the intermediate Raman image is judged as a Raman image, is the expected value of the adversarial loss value corresponding to each real colored image in the real colored image domain B.

[0131] To further strengthen the constraints on cyclic generative adversarial learning, according to one embodiment of the present invention, the loss function also includes a congruent mapping loss constraint term to ensure that when a class X image is input to a generator that converts a class Y image into a class X image, the result should be the same as the original image.

[0132] The expression of the congruent mapping loss constraint term is shown in Formula 10:

[0133] Among them, G B→A (a)) is to input a into the second generation network G B→A The generated congruent mapping Raman image, is the expected value of the mean absolute error between each Raman image in the Raman image domain A and the corresponding congruent mapping Raman image, G A→B (b)) is to input b into the first generation network G A→B The generated congruent mapping true stained image, is the expected value of the mean absolute error between each true stained image in the true stained image domain B and the corresponding congruent mapping true stained image.

[0134] Therefore, the first loss function can also be expressed as: L G (G A→B )=L adv (G A→B )+γL cycle +λL idt +pL SSIM1 , the second loss function can also be expressed as: L G (G B→A )=L adv (G B→A )+γL cycle +λL idt +pL SSIM2 , where λ=1.

[0135] After the loss function of the cyclic generative adversarial network converges, in step S330, a target generative network model can be obtained based on the generative network in the trained cyclic generative adversarial network with the converged loss function value, wherein the target generative network model can generate a target virtual stained image based on the target Raman image of the target cell sample, and the target virtual stained image is used to enhance the display of the cell structure in the target cell sample.

[0136] Specifically, according to one embodiment of the present invention, in response to the values ​​of the first loss function and the second loss function respectively converging, the parameters of the first generative network in the trained cyclic generative adversarial network with the converged loss function values ​​can be used to obtain the target generative network model.

[0137] In one example, the target cell sample may be a cell sample to be stained rather than a cell sample for training, that is, only a Raman image is obtained for the target cell sample rather than a true staining image.

[0138] In another example, both a Raman image and a real staining image of the target cell sample can be obtained for the target cell sample to test whether the virtual staining image generated by the target generation network model is consistent with the real staining image, that is, whether it meets the requirements.

[0139] In one example, the target generation network model can be a separate model or a part of another model. For example, the other model can also include an input and output part so that the model can directly receive the Raman image of the target cell sample and output a virtual stained image of the target cell sample.

[0140] Refer to Figure 2 again. As shown in Figure 2, the fourth row in Figure 2 corresponds to the second virtual staining image of the cell sample, and the second virtual staining image is generated by the target generation network model obtained by the embodiment of the present invention. It can be seen from Figure 2 that the cells pointed by the arrows in the second virtual staining image of the fourth row have a light color on the left (corresponding to the cytoplasm) and a dark color on the right (corresponding to the nucleus), which is consistent with the real staining image of the second row. The staining effect is highly close to the real staining image, and the staining accuracy of the cell structure is high, which has higher accuracy than the existing virtual staining method.

[0141] The above describes in detail the training method for obtaining the target generation network model in conjunction with Figures 2 to 6. As can be seen from the above detailed description, the present invention can obtain Raman images that can more accurately describe the cell structure of cell samples by using Raman imaging technology; at the same time, in the process of training using Raman images and real stained images of cell samples, the loss function in the recurrent generative adversarial network is constrained by the structural similarity function to ensure the consistency of the cell structure, so that the target generation network model obtained based on the trained recurrent generative adversarial network can generate virtual stained images that are more consistent with the real stained images, avoiding problems such as the virtual staining process reversing the staining of the cell nucleus and cytoplasm, improving the accuracy of virtual cell staining, and achieving enhanced display of cell structure.

[0142] In this way, by virtually staining the target cell sample through the target generation network model obtained by the training method, a virtual staining image that accurately enhances the display of the target cell sample structure can be obtained, thereby expanding the virtual staining imaging from the tissue level to the cellular level, making the virtual staining image significantly consistent with the real staining image, so that clinicians or researchers can analyze the cell structure of the cell sample based on the virtual staining image to make accurate judgments and decisions.

[0143] Second embodiment

[0144] On the basis of the first embodiment, in order to obtain a Raman image of a cell sample, according to one embodiment of the present invention, multicolor imaging can be used, that is, multiple sets of laser wavelength parameters are used to respectively excite different molecular vibration modes in the sample, and then the concentrations of different chemical components at each point in the image are calculated.

[0145] For example, stimulated Raman histology (SRH) acquires two images per field of view by adjusting the pump light wavelength. These images are then transformed into images similar to those stained with H&E staining using a specific algorithm. Additionally, there are two-color methods (acquiring two images per field of view) for examining the distribution of proteins and lipids in a sample, and three-color methods (acquiring three images per field of view) for examining the distribution of proteins, lipids, and DNA.

[0146] In one example, a cell sample can be illuminated using a first excitation light and a second excitation light. In response to the illumination of the cell sample with the first excitation light and the second excitation light, reflected light having at least one Raman shift characteristic peak can be received from the cell sample to obtain a Raman image of the cell sample. It should be noted that the first and second in the first excitation light and the second excitation light do not represent any order, quantity, or importance, but are merely used to distinguish different components.

[0147] In one example, coherent Raman imaging uses two laser beams (pump light and Stokes light) with different frequencies for imaging, mainly including stimulated Raman scattering (SRS), stimulated Raman photothermal (SRP) and coherent anti-Stokes Raman scattering (CARS). According to one embodiment of the present invention, the first excitation light may include pump light, and the second excitation light may include Stokes light. Conversely, the first excitation light may also be Stokes light, and the second excitation light may be pump light. In response to the pump light and Stokes light irradiating the cell sample, reflected light having at least one Raman shift characteristic peak can be received from the cell sample to obtain a coherent Raman scattering imaging image of the cell sample. The coherent Raman scattering imaging image may include a stimulated Raman scattering imaging image, a stimulated Raman photothermal imaging image, or a coherent anti-Stokes Raman scattering imaging image.

[0148] According to one embodiment of the present invention, the first excitation light and the second excitation light can be pulsed lasers, and the frequency difference between the first excitation light and the second excitation light can be 2800-3100 cm -1 , the repetition frequency can be greater than 50MHz, and the pulse width can be 100fs-20ps.

[0149] In one example, cell samples can also be imaged using other types of coherent Raman scattering microscopy, such as stimulated Raman scattering (SRS) microscopy. The specific implementation of Raman imaging is not limited herein. In one example, in addition to coherent Raman scattering microscopy, surface-enhanced Raman scattering imaging, single-walled carbon nanotube Raman imaging, etc. can also be used as an alternative.

[0150] In one example, a coherent Raman scattering microscopic imaging device may be used to perform coherent Raman scattering microscopic imaging on a target cell sample, and a computer device may acquire the coherent Raman image.

[0151] In the case where coherent Raman scattering imaging uses multiple Raman shift characteristic peaks to image the sample separately, the laser used is required to be tunable. However, lasers with tuning functions are usually very expensive and unstable. Therefore, according to another embodiment of the present invention, a laser with a predetermined frequency can be used, that is, a laser that does not require a tuning function. Through such a method, the cost of the imaging device can be reduced and the stability of the imaging device can be improved. In addition, by using a laser with a predetermined frequency, the laser does not need a tuning function, thereby miniaturizing the imaging device. Therefore, by setting the excitation light to a predetermined frequency, the imaging device can be made more suitable for clinical applications.

[0152] In one example, the first excitation light and the second excitation light may each have a predetermined frequency. In response to the first excitation light and the second excitation light having the predetermined frequencies irradiating the cell sample, reflected light having only one Raman shift characteristic peak may be received from the cell sample to obtain a Raman image of the cell sample.

[0153] Based on the disclosure of the second embodiment above, the present invention can utilize a laser with a predetermined frequency, thereby enabling Raman imaging of cell samples using a single characteristic peak. This eliminates the need for laser tuning, which, on the one hand, allows for miniaturization of the imaging device, and on the other hand, reduces costs and improves stability, making the imaging device used in the present invention more suitable for clinical applications.

[0154] Third embodiment

[0155] FIG7 shows a schematic diagram of stitching and registering a Raman image and a true stained image according to some embodiments of the present invention.

[0156] As shown in Figure 7, after obtaining a cell sample, the cell sample can be imaged using Raman imaging technology to obtain a Raman image of the cell sample. Then, the cell sample is chemically stained to obtain a true staining image of the cell sample.

[0157] According to one embodiment of the present invention, in the process of obtaining a Raman image and a true staining image of a cell sample, in order to obtain a Raman image with higher resolution and / or reduce the difficulty of imaging, Raman scattering microscopy imaging can be performed on various parts of the cell sample in a local field of view to obtain a local Raman image of the cell sample; and then the local Raman images of the various parts of the cell sample are spliced ​​to generate a Raman image of the cell sample.

[0158] Similarly, the cell sample for which a Raman image has been generated can be chemically stained, and bright-field microscopic imaging can be performed on each part of the chemically stained cell sample under a local field of view to obtain a local true staining image of the cell sample, and then the local true staining images of each part of the cell sample can be spliced ​​to generate a true staining image of the cell sample.

[0159] In this way, clearer imaging at the cell structure level can be achieved, so that subsequent virtual staining can be further accurately performed based on the imaged images, and the difficulty of imaging caused by the performance of the imaging equipment (for example, the imaging equipment is not accurate enough to capture a whole clear image at one time) can be reduced.

[0160] In addition, according to one embodiment of the present invention, after the recurrent generative adversarial network is trained to converge the loss function or during the training process, the Raman image and the corresponding real stained image aligned at the pixel level can be used to verify the recurrent generative adversarial network to ensure that the virtual stained image obtained by the generated target generation network model is accurate.

[0161] Specifically, verification using the registered Raman images and the corresponding true stained images can ensure that the loss function does not overfit. After verifying that the loss function does not overfit and converges using the registered Raman images and the corresponding true stained images, training the cyclic GAN can be terminated and a target GAN model can be generated based on the cyclic GAN.

[0162] In one example, the actual staining image of the cell sample can be used as a reference to register each cell in the Raman image of the cell sample, obtaining two types of paired images with complete fields of view, so that the same cell is ultimately in the same position in the registered Raman image of the cell sample and the registered actual staining image of the cell sample.

[0163] According to some embodiments of the present invention, image registration can be performed on a Raman image and a corresponding true stained image so that the Raman image and the true stained image are aligned at the pixel level; the registered Raman image and the corresponding registered true stained image are input into a trained cyclic generative adversarial network with a converged loss function value to verify the trained cyclic generative adversarial network with a converged loss function value.

[0164] According to some embodiments of the present invention, verifying a trained cyclic generative adversarial network with a converged loss function value can be specifically achieved in the following manner: determining whether the value of the loss function of the generative network in the trained cyclic generative adversarial network with a converged loss function value is overfitting; when it is determined that the value of the loss function of the generative network in the trained cyclic generative adversarial network with a converged loss function value is not overfitting, obtaining a target generative network model based on the generative network in the trained cyclic generative adversarial network with a converged loss function value; and when it is determined that the value of the loss function of the generative network in the trained cyclic generative adversarial network with a converged loss function value is overfitting, adjusting the parameters of the trained cyclic generative adversarial network with the converged loss function value and training the cyclic generative adversarial network with the adjusted parameters based on the Raman image and the real stained image.

[0165] In one example, after obtaining the unpaired and paired data sets, the Raman image may be subjected to noise reduction and contrast enhancement, and the true stained image may be subjected to preprocessing such as stain normalization to improve image quality.

[0166] In addition, the registered Raman images and the real dyed images can also be used to verify the beneficial effects of the training method of the present invention. The beneficial effects of the training method implemented by the present invention will be further illustrated below using two sets of experimental data.

[0167] (1) The target generation network model was evaluated on the test set. The test set contains approximately 600 registered monochromatic Raman images of cell samples and 600 real stained images of cell samples. The images were randomly cropped from 5 of the 32 samples in the dataset, with each sample containing an average of approximately 200 cells.

[0168] In the test results, the SSIM of the virtual stained image and the real stained image reached 0.881±0.015, and the Dice coefficient of the cell nucleus area reached 0.804±0.137, indicating that the target generation network can convert the label-free monochrome Raman image into a virtual stained image and achieve accurate mapping of the cell structure.

[0169] (2) Evaluation of the target generation network by pathologists. The dataset contains 32 samples from 7 positive patients and 4 negative patients. For example, the pathologists can evaluate the target generation network model by referring to the evaluation diagram of the target generation network model shown in Figure 8.

[0170] When counting cells at the cell level, pathologists were able to distinguish normal cells from cancer cells in virtual stained images with high accuracy. The accuracy was 99.4%, the specificity was 99.5%, and the sensitivity was 99.7%. This compared to the Cohen's Kappa coefficient for cell classification in real stained images, which was 0.782±0.284. This indicates that the cell classifications made by doctors using virtual and real stained images were highly consistent. When counting samples at the sample level, the accuracy was 93.8%, the sensitivity was 100%, and the specificity was 90.9%.

[0171] Based on the content disclosed in the third embodiment above, the Raman images of the cell sample and the corresponding real stained images can be divided into a training set, a validation set, and a test set.

[0172] Among them, for the training set, by stitching the images, clearer imaging at the cell structure level can be achieved, so that subsequent virtual staining based on the imaged images can be further accurately performed, and the imaging difficulty caused by the performance of the imaging equipment (for example, the imaging equipment is not accurate enough to capture a whole clear image at one time) can be reduced.

[0173] For the validation set, by using Raman images and corresponding real stained images registered at the pixel level to verify the recurrent generative adversarial network, it can be ensured that the loss function will not be overfitted, so that the generated target generation network model can generate virtual stained images consistent with the real stained images, ensuring accurate enhanced display of cell structures.

[0174] For the test set, the registered Raman images and the corresponding real staining images were tested, which showed that the target generation network model obtained based on the training method of the present invention can accurately enhance the virtual staining image displayed for the target cell sample structure, and realize the expansion of virtual staining imaging from the tissue level to the cellular level, so that the virtual staining image and the real staining image have significant consistency, so that clinicians or researchers can analyze the cell structure of the target cell sample based on the virtual staining image to make accurate judgments and decisions.

[0175] Fourth embodiment

[0176] The virtual staining method for enhancing the display of cell structure provided by the present invention will be described in detail below with reference to the accompanying drawings.

[0177] FIG9 illustrates a flow chart of a virtual staining method for enhancing the display of cell structures according to some embodiments of the present invention. As shown in FIG9 , the virtual staining method for enhancing the display of cell structures of the present invention can first obtain a target Raman image of a target cell sample at step S910, and then input the target Raman image into a target generation network model generated according to the aforementioned training method at step S920 to obtain a target virtual staining image of the target cell sample. The target virtual staining image is used to enhance the display of the cell structure in the target cell sample.

[0178] According to one embodiment of the present invention, Raman scattering microscopy imaging can be performed on a target cell sample to obtain a target Raman image of the target cell sample. Specifically, according to one embodiment of the present invention, the target cell sample can be irradiated with a first excitation light and a second excitation light; in response to the irradiation of the target cell sample with the first excitation light and the second excitation light, reflected light having at least one Raman shift characteristic peak can be received from the target cell sample to obtain a target Raman image of the target cell sample.

[0179] According to another embodiment of the present invention, the first excitation light and the second excitation light may each have a predetermined frequency. In this case, in response to irradiating the target cell sample with the first excitation light and the second excitation light having the predetermined frequencies, reflected light having only one Raman shift characteristic peak may be received from the target cell sample to obtain a target Raman image of the target cell sample.

[0180] In one example, the first excitation light and the second excitation light can be pulsed lasers, and the frequency difference between the first excitation light and the second excitation light is 2800-3100 cm -1 , repetition frequency is greater than 50MHz, and pulse width is 100fs-20ps.

[0181] According to another embodiment of the present invention, the first excitation light may include pump light, and the second excitation light may include Stokes light. In this case, in response to irradiating the target cell sample with the pump light and the Stokes light, reflected light having at least one Raman shift characteristic peak may be received from the target cell sample to obtain a coherent Raman scattering imaging image of the cell sample. The coherent Raman scattering imaging image includes a stimulated Raman scattering imaging image, a stimulated Raman photothermal imaging image, or a coherent anti-Stokes Raman scattering imaging image.

[0182] According to one embodiment of the present invention, Raman scattering microscopic imaging can also be performed on various parts of the target cell sample in a local field of view to obtain a local target Raman image of the target cell sample; then, the local target Raman images of various parts of the target cell sample can be spliced ​​to obtain a target Raman image of the target cell sample.

[0183] For some specific details of the virtual staining method for enhancing the display of cell structure disclosed in the fourth embodiment of the present invention, reference can also be made to the training method for obtaining the target generation network model described in the first to third embodiments, so the same content will not be repeated here.

[0184] Based on the content disclosed in the fourth embodiment of the present invention, the virtual staining method of the present invention can be used to virtually stain the target cell sample to obtain a virtual staining image that accurately enhances the display of the target cell sample structure, thereby expanding the virtual staining imaging from the tissue level to the cellular level, so that the virtual staining image has significant consistency with the real staining image, so that clinicians or researchers can analyze the cellular structure of the target cell sample based on the virtual staining image to make accurate judgments and decisions.

[0185] In one example, a computer device can be used to execute the training method and virtual coloring method provided in embodiments of the present invention. The computer device can be a terminal or a server. Terminals can include, but are not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, portable wearable devices, and medical electronic devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart car devices, etc. Portable wearable devices can include smart watches, smart bracelets, head-mounted devices, etc. The server can be implemented as a standalone server or a server cluster consisting of multiple servers.

[0186] Fifth embodiment

[0187] The virtual coloring system provided by the present invention will be described in detail below with reference to the accompanying drawings.

[0188] FIG10 illustrates a block diagram of a virtual staining system according to some embodiments of the present invention. As shown in FIG10 , virtual staining system 1000 may include an image acquisition component 1010 and an image processing component 1020. Image acquisition component 1010 may be configured to obtain a target Raman image of a target cell sample; and image processing component 1020 may be configured to input the target Raman image into a target generation network model generated according to the aforementioned training method to obtain a target virtual staining image of the target cell sample. The target virtual staining image is used to enhance the display of cellular structures in the target cell sample.

[0189] According to one embodiment of the present invention, the image acquisition component 1010 may be configured to perform Raman scattering microscopic imaging on the target cell sample to obtain a target Raman image of the target cell sample.

[0190] According to one embodiment of the present invention, the virtual coloring system 1000 may further include:

[0191] a laser source configured to generate a first excitation light and a second excitation light for irradiating a target cell sample; and

[0192] The image acquisition component 1010 can also be configured to irradiate the target cell sample in response to the first excitation light and the second excitation light, receive reflected light having at least one Raman shift characteristic peak from the target cell sample, and obtain a target Raman image of the target cell sample.

[0193] According to one embodiment of the present invention, the first excitation light and the second excitation light may each have a predetermined frequency, and the image acquisition component 1010 may also be configured as follows:

[0194] In response to irradiating a target cell sample with first excitation light and second excitation light having a predetermined frequency, reflected light having only one Raman shift characteristic peak is received from the target cell sample to obtain a target Raman image of the target cell sample.

[0195] According to one embodiment of the present invention, the virtual coloring system 1000 may further include:

[0196] an optical path component, the optical path component including a two-dimensional galvanometer assembly and a first filter configured to guide the first excitation light and the second excitation light to the target cell sample;

[0197] a sample carrying component configured to carry a target cell sample to be irradiated by the first excitation light and the second excitation light; and

[0198] The objective lens component is configured to receive reflected light from the target cell sample and transmit the reflected light to the image acquisition component 1010 .

[0199] FIG11 shows an exemplary structural diagram of a virtual staining system 1000 according to some embodiments of the present invention. As shown in FIG11 , the virtual staining system 1000 includes a laser emitting device 1110 , a sample holding component 1120 , an image acquisition component 1010 , and an image processing component 1020 .

[0200] The laser emitting device 1110 includes a laser source 1111 , a two-dimensional galvanometer assembly 1112 and a first filter 1113 on an optical path component, and an objective lens component 1114 .

[0201] The laser source 1111 is used to generate a first excitation light and a second excitation light, which are output collinearly.

[0202] This design simplifies the optical path structure between the laser source 1111 and the objective lens component 1114, avoiding the need to split and adjust the wavelength of the single wavelength excitation light beam output by the laser source 1111, thereby improving the compactness of the device, reducing the volume, and facilitating commercial development. The laser source 1111 can be constructed to output a first excitation light and a second excitation light of fixed wavelengths. In some embodiments, as described above, the laser source 1111 can also be a tunable laser source, so that the wavelength of the first excitation light (and / or the second excitation light) can be selected within a certain range, while the wavelength of the second excitation light (and / or the first excitation light) is fixed. In the case where the laser source 1111 is a tunable laser source, the laser source 1111 can be integrated with a control circuit for controlling the laser source 1111 to output a specific form of laser light, or the laser source 1111 can communicate with the image processing component 1020 or other computer control device via a cable 1141 and be controlled by the image processing component 1020 to output a laser light of a specific wavelength.

[0203] According to one embodiment of the present invention, FIG12 shows a schematic diagram of a microscopic imaging system for Raman images according to some embodiments of the present invention. As shown in FIG12 , EOM is an electro-optical modulator, DM is a dichroic mirror, M is a reflective silver mirror, OBJ is an objective lens, CON is a condenser lens, FL is a filter, and PD is a photodiode.

[0204] The laser (picoEmerald™ S, Applied Physics & Electronics) has a repetition rate of 80 MHz and a pulse width of 2 ps. The emitted pump light wavelength can be tuned in the range of 700-960 nm. In one example, the pump light wavelength during imaging is 796.8 nm, corresponding to the Raman shift characteristic peak of 2850 cm -1The Stokes light wavelength was fixed at 1031 nm, and the two overlapped spatially and temporally. The Stokes light was modulated to approximately 20 MHz by an electro-optical modulator. The collinear pump and Stokes light beams were coupled to a two-dimensional galvanometer scanner (GVS012-2D, Thorlab) and then input into an inverted microscope (IX73, Olympus). Both beams were focused onto the sample via a 60X water-immersion objective (LUMPlanFL N, numerical aperture 1.0, Olympus), inducing specific molecular resonances and generating stimulated Raman loss and gain. The beams were then collected by another water-immersion objective of the same model. The Stokes light was removed by a low-pass filter (ET980SP, Chroma) and detected by a 10 mm × 10 mm silicon photodiode (S3994-01, Hamamatsu) reverse-biased at 48 V DC. After the signal was extracted by a lock-in amplifier (HF2LI, Zurich Instruments), the analog output representing the SRS signal entered a data acquisition card (PCIE-6363, National Instruments) and was input into a computer to display the SRS image of the sample on LabVIEW 2018 software.

[0205] According to one embodiment of the present invention, Figure 13 shows the spontaneous Raman scattering spectra of pure lipid samples (using triolein (TO) as an example) and pure protein samples (using bovine serum albumin (BSA) as an example) in the carbon-hydrogen bond vibration region (2800-3100 cm⁻¹) according to some embodiments of the present invention. This graph depicts the spontaneous Raman scattering intensity at different frequency differences between pump light and Stokes light. As shown in Figure 13, lipids and proteins generate strong Raman signals in this range. Therefore, the Raman images obtained in this range can reflect the differences in chemical concentrations within different cellular structures within the target cell sample. It should be noted that while stimulated Raman scattering spectra are relatively close to spontaneous Raman scattering spectra, coherent anti-Stokes Raman scattering spectra differ significantly from spontaneous Raman scattering spectra. Therefore, the correct Raman spectrum should be selected for reference based on the specific coherent Raman scattering microscopy imaging method.

[0206] In some embodiments, the laser source 1111 may be a laser source with a fixed output wavelength, thereby obtaining a coherent Raman image of the sample under a single Raman shift characteristic peak. For example, in some embodiments, the 2850 cm-1 wavelength corresponding to the 796.8 nm pump light and the 1031 nm Stokes light in stimulated Raman scattering microscopy imaging is -1 In the Raman shift channel, as shown in FIG11 , the Raman signal of lipids is relatively strong, which can be used to reflect the spatial distribution of lipids in the target cell sample. In some embodiments, the 2850 cm-1 corresponding to the 780 nm pump light and the 1003 nm Stokes light under coherent anti-Stokes Raman scattering microscopy imaging is -1The Raman shift channel can also reflect the lipid component characteristics in the target cell sample 1180 .

[0207] In some embodiments, the laser source 1111 may also be a tunable laser source, thereby obtaining a coherent Raman image of the sample under multiple Raman shift characteristic peaks. For example, in some embodiments, the 2850 cm-1 corresponding to the 796.8 nm and 791.8 nm pump light and the 1031 nm Stokes light under stimulated Raman scattering microscopy imaging is -1 and 2930cm -1 Raman shift channel, as shown in Figure 11, the Raman signal of protein in the latter is stronger, which can reflect the characteristics of the protein component in the target cell sample in addition to the lipid component characteristics in the target cell sample.

[0208] Of course, the specific wavelength of the laser output by the laser source 1111 is not limited to the above, and the laser source 1111 can be constructed to output laser light having other wavelength ranges, which depends on the specific components in the target cell sample that are desired to be detected. It should be understood that the components that are desired to be detected are not limited to the lipids or proteins discussed above.

[0209] In some embodiments, the control circuit integrated in the image processing component 1020 or the laser source 1111 may be further configured to control the laser source 1111 to output a pulsed laser. According to one embodiment of the present invention, the first excitation light and the second excitation light may be pulsed lasers, and the frequency difference between the first excitation light and the second excitation light is 2800-3100 cm -1 , a repetition frequency greater than 50 MHz, and a pulse width of 100 fs-20 ps. According to one embodiment of the present invention, the first excitation light may include pump light, the second excitation light may include Stokes light, and the image acquisition component 1010 is further configured to irradiate the cell sample in response to the pump light and the Stokes light, receive reflected light having at least one Raman shift characteristic peak from the cell sample, and obtain a coherent Raman scattering imaging image of the cell sample. The coherent Raman scattering imaging image includes a stimulated Raman scattering imaging image, a stimulated Raman photothermal imaging image, or a coherent anti-Stokes Raman scattering imaging image. The frequency and pulse width of the pulsed laser are not limited to this. The laser source 1111 can be controlled to output pulsed lasers of other frequencies and pulse widths.

[0210] In some embodiments, the sample holding component 1120 may be a mechanical translation stage for holding the target cell sample 1180 .

[0211] In some embodiments, the image acquisition component 1010 may be a photomultiplier tube or a photodiode.

[0212] The first and second excitation lights enter the two-dimensional galvanometer assembly 1112, which adjusts the optical paths of the first and second excitation lights. The first and second excitation lights exiting the two-dimensional galvanometer assembly 1112 sequentially pass through the first filter 1113 and the objective lens assembly 1114. The first and second excitation lights pass through the first filter 1113, and the objective lens assembly 1114 focuses the first and second excitation lights onto the sample carrier 1120. The sample on the sample carrier 1120 generates signal light under the influence of the first and second excitation lights. After the signal light passes through the objective lens 1114, the first filter 1113 reflects the signal light to the image acquisition assembly 1010. The image acquisition assembly 1010 generates a Raman image based on the signal light and outputs the Raman image to the image processing assembly 1020 via the cable 1141. The Raman image is then displayed on the display 1142 via the cable 1141.

[0213] According to one embodiment of the present invention, the virtual system may further include an automatic focusing component, wherein the automatic focusing component may include a focus detection unit, a second filter and a movable component for moving the objective lens component; the focus detection unit may be configured to: generate a third excitation light, the third excitation light is irradiated onto the sample-carrying component through the second filter; detect the detection reflected light reflected by the sample-carrying component and returned to the focus detection unit; and control the movable component according to the detection result so that the objective lens component receives the reflected light from the target cell sample.

[0214] Figure 14 shows another example structural diagram of the virtual staining system 1000 according to some embodiments of the present invention. Specifically, as shown in Figure 14, in order to accurately focus the first excitation light and the second excitation light to the target cell sample 1180 at the sample supporting part 1120 through the objective lens 1114, the laser emitting device 1110 also includes an automatic focusing mechanism, and the automatic focusing mechanism includes a focus detection unit 1422, a second filter 1421 and a moving component. The second filter 1421 is respectively arranged to correspond to the first filter 1113 and the objective lens component 1114. After the first excitation light and the second excitation light pass through the first filter 1113, they are reflected to the objective lens 1114 through the second filter 1421. After the signal light passes through the objective lens 1114, it is reflected to the first filter 1113 through the second filter 1421. The third excitation light generated by the focus detection unit 1422 passes through the second filter 1421 and is then focused parallel or collinearly with the first and second excitation lights, respectively, by the objective lens 1114 onto the sample support 1120, generating reflected light on the sample support 1120. The reflected light then returns to the focus detection unit 1422 along the original path of the third excitation light, where it is detected. The objective lens 1114 is mounted on a movable assembly, which moves the objective lens 1114 based on the detection results of the focus detection unit 1422 to adjust the distance between the objective lens 1114 and the sample support 1120.

[0215] In one example of the present invention, the image processing component 1020 may include processing Raman images and real stained images during the model training phase to obtain a model dataset and train the model to obtain a target generation network model. The implementation of each sub-function required to achieve this function has been described above and will not be repeated here.

[0216] In the model prediction stage after the model training is completed, the Raman image of the cell sample is input into the target generation network model to obtain a virtual stained image output by the target generation network model. The virtual stained image can be used to enhance the display of the cell structure in the target cell sample.

[0217] The functional modules in the image processing component 1020 can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the corresponding operations of the above modules.

[0218] According to one embodiment of the present invention, the image acquisition component 1010 can also be configured to perform Raman scattering microscopic imaging of various parts of the target cell sample under a local field of view to obtain a local target Raman image of the target cell sample; and splice the local target Raman images of various parts of the target cell sample to obtain a target Raman image of the target cell sample.

[0219] For some specific details of the virtual coloring system 1000 described in the fifth embodiment, reference can be made to the contents disclosed in the first to fourth embodiments, and therefore the same contents will not be repeated here.

[0220] Based on the content disclosed in the fifth embodiment of the present invention, the virtual staining system of the present invention can be used to virtually stain the target cell sample to obtain a virtual staining image that accurately enhances the display of the target cell sample structure, thereby expanding the virtual staining imaging from the tissue level to the cellular level, so that the virtual staining image has significant consistency with the real staining image, so that clinicians or researchers can analyze the cellular structure of the target cell sample based on the virtual staining image to make accurate judgments and decisions.

[0221] FIG15 shows a block diagram of an electronic device 1500 according to some embodiments of the present invention.

[0222] 15 , electronic device 1500 may include a processor 1501 and a memory 1502. Processor 1501 and memory 1502 may be connected via a bus 1503. Electronic device 1500 may be any type of portable device (e.g., a smart camera, a smartphone, a tablet computer, etc.) or any type of fixed device (e.g., a desktop computer, a server, etc.).

[0223] The processor 1501 can perform various actions and processes according to the program stored in the memory 1502. Specifically, the processor 1501 can be an integrated circuit chip with signal processing capabilities. The above-mentioned processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The various methods, steps, and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor, etc., and can be an X86 architecture or an ARM architecture.

[0224] Memory 1502 stores computer-executable instructions that, when executed by processor 1501, implement the training method for obtaining a target generation network model and / or the virtual staining method for enhancing display of cell structure. Memory 1502 may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. Volatile memory may be a random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct memory bus random access memory (DR RAM). It should be noted that memory of the methods described herein is intended to comprise, but not be limited to, these and any other suitable types of memory.

[0225] Furthermore, the method for determining video compatibility according to the present invention may be recorded in a computer-readable recording medium. Specifically, the present invention may provide a computer-readable recording medium storing computer-executable instructions that, when executed by a processor, cause the processor to execute the method for determining video compatibility as described above.

[0226] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the present invention are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions.

[0227] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architectures, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or part of the code, and the module, program segment, or part of the code contains at least one executable instruction for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or can be implemented using a combination of dedicated hardware and computer instructions.

[0228] In general, various example embodiments of the present invention may be implemented in hardware or dedicated circuitry, software, firmware, logic, or any combination thereof. Certain aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device. When aspects of embodiments of the present invention are illustrated or described as block diagrams, flow charts, or using some other graphical representation, it will be understood that the blocks, devices, systems, techniques, or methods described herein may be implemented, as non-limiting examples, in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or a controller or other computing device, or some combination thereof.

[0229] Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. It should also be understood that terms such as those defined in common dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology and should not be interpreted in an idealized or highly formal sense, unless expressly defined as such herein.

[0230] The above is an illustration of the present invention and should not be considered as limiting thereof. Although several exemplary embodiments of the present invention have been described, it will be readily understood by those skilled in the art that many modifications may be made to the exemplary embodiments without departing from the novel teachings and advantages of the present invention. Therefore, all such modifications are intended to be included within the scope of the present invention as defined by the claims. It should be understood that the above is an illustration of the present invention and should not be considered as being limited to the specific embodiments disclosed, and modifications to the disclosed embodiments and other embodiments are intended to be included within the scope of the appended claims. The present invention is defined by the claims and their equivalents.

Claims

1. A training method for obtaining a target generation network model, comprising: Obtaining a Raman image of a cell sample and a corresponding true staining image of the cell sample; Training a cycle generative adversarial network based on the Raman image and the true staining image so that the value of the loss function of the generative network in the cycle generative adversarial network converges; And Obtaining a target generation network model based on the generative network in the trained cycle generative adversarial network with a converged loss function value, wherein the target generation network model generates a target virtual staining image based on a target Raman image of a target cell sample, and the target virtual staining image is used to enhance the display of cell structures in the target cell sample; Wherein, the generative network includes a first generative network and a second generative network, the loss function includes a first loss function for the first generative network and a second loss function for the second generative network, and the first loss function and the second loss function include a cell structure similarity constraint term, and the cell structure constraint term constrains the consistency of the cell structures in the input image and the output image of the generative network based on the structural similarity SSIM function, Among them, the structural similarity constraint term of the first loss function includes The structural similarity constraint term of the second loss function includes Among them, a is a Raman image belonging to a Raman image domain A composed of one or more of the said Raman images, G A→B (a) is an intermediate virtual staining image generated by inputting a into the first generation network G A→B The structural similarity value between the intermediate virtual staining image and the Raman image is SSIM(G A→B (a), a). is the expected value of the structural similarity loss value corresponding to each Raman image in the Raman image domain A; and b is a real staining image belonging to the real staining image domain B composed of one or more of the real staining images, G B→A (b) is the intermediate Raman image generated by inputting b into the second generation network G B→A The generated intermediate Raman image, SSIM(G B→A (b), b) is the structural similarity value between the intermediate Raman image and the real staining image, Is the expected value of the structural similarity loss value corresponding to each true staining image in the true staining image domain B.

2. The training method according to claim 1, wherein, Training a cycle generative adversarial network based on the Raman image and the true staining image so that the value of the loss function of the generative network in the cycle generative adversarial network converges includes: Inputting the Raman image into the first generative network to generate an intermediate virtual staining image, Inputting the intermediate virtual staining image into the first discriminative network in the cycle generative adversarial network to determine the probability that the intermediate virtual staining image is judged to be the true staining image, and Inputting the true staining image into the second generative network to generate an intermediate Raman image, Inputting the intermediate Raman image into the second discriminative network in the cycle generative adversarial network to determine the probability that the intermediate Raman image is judged to be the Raman image, Adjusting the parameters of the cycle generative adversarial network based on the probabilities determined by the first discriminative network and the second discriminative network so that the values of the first loss function and the second loss function converge respectively.

3. The training method according to claim 2, wherein Adjusting the parameters of the cycle generative adversarial network based on the probabilities determined by the first discriminative network and the second discriminative network so that the values of the first loss function and the second loss function converge respectively includes: Determining the values of the first loss function and the second loss function based on the probabilities determined by the first discriminative network and the second discriminative network; Adjusting the parameters of the cycle generative adversarial network based on the values of the first loss function and the second loss function so that the values of the first loss function and the second loss function converge respectively.

4. The training method according to claim 3, wherein The expression of the first loss function is: L G (G A→B ) = L adv (G A→B ) + γL cycle + pL SSIM1 L G (G A→B ) is the first loss function, L adv (G A→B ) is the adversarial loss constraint term of the first loss function, L cycle is the cycle consistency loss constraint term of the first loss function, L SSIM1 is the structural similarity constraint term of the cells for the first loss function; Among them, D B is the first discrimination network, -D B (G A→B (a)) is the probability that the intermediate virtual stained image is judged as the real stained image Is the expected value of the adversarial loss value corresponding to each Raman image in the Raman image domain A; Among them, G B→A (G A→B (a)) is to input the intermediate virtual staining image into the second generation network G B→A The generated reconstructed Raman image, is the expected value of the mean absolute error value between each Raman image and the corresponding reconstructed Raman image in the Raman image domain A, G A→B (G B→A (a)) is the reconstructed true staining image generated by inputting the intermediate Raman image into the first generation network G A→B ​ Is the expected value of the mean absolute error value between each true staining image in the true staining image domain B and the corresponding reconstructed true staining image; Wherein, γ = 10, p = 2.

5. The training method according to claim 4, wherein, The expression of the second loss function is: L G (G B→A ) = L adv (G B→A ) + γL cycle + pL SSIM2 where, L G (G B→A ) is the second loss function, L adv (G B→A ) is the adversarial loss constraint term of the second loss function, L cycle is the cycle consistency loss constraint term of the second loss function, L SSIM2 is the structural similarity constraint term of the cells of the second generation network; Among them, D A is the second discrimination network, D A (G B→A (b)) is the probability that the intermediate Raman image is judged as the Raman image is the expected value of the adversarial loss value corresponding to each real staining image in the real staining image domain B.

6. The training method according to claim 5, wherein, The loss function further includes a congruent mapping loss constraint term, and the expression of the congruent mapping loss constraint term is: Among them, G B→A (a)) is the congruent mapping Raman image generated by inputting a into the second generation network G B→A The generated congruent mapping Raman image, is the expected value of the mean absolute error between each Raman image in the Raman image domain A and the corresponding congruent mapped Raman image, G A→B (b)) is the congruent mapped true staining image generated by inputting b into the first generation network G A→B ​ is the expected value of the mean absolute error value between each real staining image in the real staining image domain B and the corresponding congruent mapping real staining image; wherein, the first loss function can also be expressed as: L G (G A→B ) = L adv (G A→B ) + γL cycle + λL idt + pL SSIM1 , the second loss function can also be expressed as: L G (G B→A ) = L adv (G B→A ) + γL cycle + λL idt + pL SSIM2 , where λ = 1.

7. The training method according to claim 1, wherein, Training the cycle generative adversarial network based on the Raman image and the real staining image so that the value of the loss function of the generative network in the cycle generative adversarial network converges includes: Iteratively training the cycle generative adversarial network based on multiple Raman images and multiple corresponding real staining images until the value of the loss function of the generative network in the cycle generative adversarial network converges.

8. The training method according to claim 2, wherein, Obtaining the target generative network model based on the generative network in the trained cycle generative adversarial network with a converged loss function value includes: In response to the values of the first loss function and the second loss function each converging, using the parameters of the first generative network in the trained cycle generative adversarial network with a converged loss function value to obtain the target generative network model.

9. The training method according to any one of claims 1-8, further comprising: Performing image registration on the Raman image and the corresponding real staining image so that the Raman image and the real staining image are pixel-level aligned; Inputting the registered Raman image and the corresponding registered real staining image into the trained cycle generative adversarial network with a converged loss function value to verify the trained cycle generative adversarial network with a converged loss function value.

10. The training method according to claim 9, wherein, Verifying the trained cycle generative adversarial network with a converged loss function value includes: Determining whether the value of the loss function of the generative network in the trained cycle generative adversarial network with a converged loss function value is overfitting; When it is determined that the value of the loss function of the generative network in the trained cycle generative adversarial network with a converged loss function value is not overfitting, obtaining the target generative network model based on the generative network in the trained cycle generative adversarial network with a converged loss function value; and When it is determined that the value of the loss function of the generative network in the trained cycle generative adversarial network with a converged loss function value is overfitting, adjusting the parameters of the trained cycle generative adversarial network with a converged loss function value and training the cycle generative adversarial network with the adjusted parameters based on the Raman image and the real staining image.

11. The training method according to any one of claims 1-8, wherein, Obtaining the Raman image of the cell sample and the corresponding real staining image of the cell sample includes: Performing Raman scattering microscopy imaging on the cell sample to obtain the Raman image of the cell sample; Performing chemical staining and bright-field microscopy imaging on the cell sample for which the Raman image has been generated to obtain the corresponding real staining image of the cell sample.

12. The training method according to claim 11, wherein, Performing Raman scattering microscopy imaging on the cell sample to obtain the Raman image of the cell sample includes: Irradiating the cell sample with a first excitation light and a second excitation light; In response to irradiation of the cell sample with the first excitation light and the second excitation light, receiving reflected light having at least one Raman shift characteristic peak from the cell sample to obtain the Raman image of the cell sample.

13. The training method according to claim 12, wherein, The first excitation light and the second excitation light each have a predetermined frequency, and In response to irradiation of the cell sample with the first excitation light and the second excitation light, receiving reflected light having at least one Raman shift characteristic peak from the cell sample to obtain the Raman image of the cell sample includes: In response to irradiation of the cell sample with the first excitation light and the second excitation light having a predetermined frequency, receiving reflected light having only one Raman shift characteristic peak from the cell sample to obtain the Raman image of the cell sample.

14. The training method according to claim 12, wherein, The first excitation light and the second excitation light are pulsed lasers, and the frequency difference between the first excitation light and the second excitation light is 2800 - 3100 cm -1 , the repetition frequency is greater than 50 MHz, and the pulse width is 100 fs - 20 ps.

15. The training method according to claim 14, wherein The first excitation light includes pump light, the second excitation light includes Stokes light, and in response to irradiation of the cell sample with the first excitation light and the second excitation light, receiving reflected light having at least one Raman shift characteristic peak from the cell sample to obtain the Raman image of the cell sample includes: In response to irradiation of the cell sample with the pump light and the Stokes light, receiving reflected light having at least one Raman shift characteristic peak from the cell sample to obtain a coherent Raman scattering imaging image of the cell sample, the coherent Raman scattering imaging image including a stimulated Raman scattering imaging image, a stimulated Raman photothermal imaging image, or a coherent anti-Stokes Raman scattering imaging image.

16. The training method according to claim 11, wherein Performing Raman scattering microscopy on the cell sample to obtain the Raman image of the cell sample includes: Performing Raman scattering microscopy on each part of the cell sample under a local field of view to obtain a local Raman image of the cell sample; Stitching the local Raman images of each part of the cell sample to generate the Raman image of the cell sample; and Performing chemical staining and bright-field microscopy on the cell sample on which the Raman image has been generated to obtain the corresponding true staining image of the cell sample includes: Performing chemical staining on the cell sample on which the Raman image has been generated, and performing bright-field microscopy on each part of the chemically stained cell sample under a local field of view to obtain a local true staining image of the cell sample, Stitching the local true staining images of each part of the cell sample to generate the true staining image of the cell sample.

17. A virtual staining method for enhancing the display of cell structures, comprising: Obtaining a target Raman image of a target cell sample; Inputting the target Raman image into the target generation network model according to any one of claims 1-16 to obtain a target virtual staining image of the target cell sample, the target virtual staining image being used to enhance the display of cell structures in the target cell sample.

18. The virtual dyeing method according to claim 17, wherein, Obtaining a target Raman image of a target cell sample includes: Performing Raman scattering microscopy on the target cell sample to obtain the target Raman image of the target cell sample.

19. The virtual dyeing method according to claim 18, wherein, Performing Raman scattering microscopy imaging on the target cell sample to obtain the target Raman image of the target cell sample includes: Irradiating the target cell sample with a first excitation light and a second excitation light; In response to the irradiation of the target cell sample with the first excitation light and the second excitation light, receiving reflected light having at least one Raman shift characteristic peak from the target cell sample to obtain the target Raman image of the target cell sample.

20. The virtual dyeing method according to claim 19, wherein, The first excitation light and the second excitation light each have a predetermined frequency, and In response to the irradiation of the target cell sample with the first excitation light and the second excitation light, receiving reflected light having at least one Raman shift characteristic peak from the target cell sample to obtain the target Raman image of the target cell sample includes: In response to the irradiation of the target cell sample with the first excitation light and the second excitation light having a predetermined frequency, receiving reflected light having only one Raman shift characteristic peak from the target cell sample to obtain the target Raman image of the target cell sample.

21. The virtual staining method according to claim 19, wherein, The first excitation light and the second excitation light are pulsed lasers, and the frequency difference between the first excitation light and the second excitation light is 2800 - 3100 cm -1 , the repetition frequency is greater than 50 MHz, and the pulse width is 100 fs - 20 ps.

22. The virtual dyeing method according to claim 21, wherein, The first excitation light includes pump light, the second excitation light includes Stokes light, and in response to the irradiation of the target cell sample with the first excitation light and the second excitation light, receiving reflected light having at least one Raman shift characteristic peak from the target cell sample to obtain the target Raman image of the target cell sample: In response to the irradiation of the target cell sample with the pump light and the Stokes light, receiving reflected light having at least one Raman shift characteristic peak from the target cell sample to obtain a coherent Raman scattering imaging image of the cell sample, the coherent Raman scattering imaging image including a stimulated Raman scattering imaging image, a stimulated Raman photothermal imaging image, or a coherent anti-Stokes Raman scattering imaging image.

23. The virtual staining method according to claim 18, wherein Performing Raman scattering microscopy imaging on the target cell sample to obtain the target Raman image of the target cell sample includes: Performing Raman scattering microscopy imaging on each local part of the target cell sample under a local field of view to obtain a local target Raman image of the target cell sample; Stitching the local target Raman images of each local part of the target cell sample to obtain the target Raman image of the target cell sample.

24. A virtual staining system, comprising: An image acquisition component configured to obtain a target Raman image of a target cell sample; And An image processing component configured to input the target Raman image into the target generation network model according to any one of claims 1-16 to obtain a target virtual staining image of the target cell sample, the target virtual staining image being used to enhance the display of cell structures in the target cell sample.

25. The virtual dyeing system according to claim 24, wherein, The image acquisition component is configured to perform Raman scattering microscopy imaging on the target cell sample to obtain the target Raman image of the target cell sample.

26. The virtual staining system according to claim 25, wherein The virtual staining system further comprises: A laser source configured to generate a first excitation light and a second excitation light for irradiating a target cell sample; and The image acquisition component is further configured to receive, in response to the first excitation light and the second excitation light irradiating the target cell sample, reflected light having at least one Raman shift characteristic peak from the target cell sample, so as to obtain the target Raman image of the target cell sample.

27. The virtual staining system according to claim 26, wherein, The first excitation light and the second excitation light each have a predetermined frequency, and the image acquisition component is further configured to: Receive, in response to the first excitation light and the second excitation light having the predetermined frequency irradiating the target cell sample, reflected light having only one Raman shift characteristic peak from the target cell sample, so as to obtain the target Raman image of the target cell sample.

28. The virtual staining system according to claim 26, further comprising: An optical path component, the optical path component including a two-dimensional galvanometer assembly and a first filter configured to guide the first excitation light and the second excitation light to the target cell sample; A sample carrying component configured to carry the target cell sample to receive the irradiation of the first excitation light and the second excitation light; And An objective lens component configured to receive the reflected light from the target cell sample and transmit the reflected light to the image acquisition component.

29. The virtual staining system according to claim 26, wherein, The first excitation light and the second excitation light are pulsed lasers, and the frequency difference between the first excitation light and the second excitation light is 2800 - 3100 cm -1 , the repetition frequency is greater than 50 MHz, and the pulse width is 100 fs - 20 ps.

30. The virtual staining system according to claim 29, wherein, The first excitation light includes pump light, the second excitation light includes Stokes light, and the image acquisition component is further configured to: Receive, in response to the pump light and the Stokes light irradiating the cell sample, reflected light having at least one Raman shift characteristic peak from the cell sample, so as to obtain a coherent Raman scattering imaging image of the cell sample, the coherent Raman scattering imaging image including a stimulated Raman scattering imaging image, a stimulated Raman optothermal imaging image, or a coherent anti-Stokes Raman scattering imaging image.

31. The virtual dyeing method according to claim 25, wherein, The image acquisition component is configured to: Perform Raman scattering microscopy imaging on each part of the target cell sample under a local field of view to obtain a local target Raman image of the target cell sample; Stitch the local target Raman images of each part of the target cell sample to obtain the target Raman image of the target cell sample.

Images