Training method, virtual staining method and virtual staining system

Abstract

The present invention provides a training method, a virtual staining method, and a virtual staining system. The training method includes obtaining a Raman image of a cell sample and a corresponding true staining image of the cell sample; training a cycle generative adversarial network based on the Raman image and the true staining image so that the value of the loss function of the generative network in the cycle generative adversarial network converges; and obtaining a target generative network model based on the generative network in the trained cycle generative adversarial network with a converged value of the loss function, wherein the target generative network model generates a target virtual staining image based on a target Raman image of a target cell sample, and the target virtual staining image is used to enhance the display of cell structures in the target cell sample; wherein the loss function of the generative network includes a cell structure similarity constraint term, and the cell structure constraint term constrains the consistency of cell structures in the input image and the output image of the generative network based on the structural similarity SSIM function.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and more specifically, to a training method for obtaining a target generation network model, a virtual staining method for enhancing the display of cell structures, and a virtual staining system. Background Art

[0002] In the medical field, cytopathology is an important branch of pathology and an important part of clinical pathology. In cytopathology, by staining cell specimens (also called samples), the cell structure and other conditions in the cell specimens can be reflected based on the images of the stained cell samples. Clinicians and researchers can make diagnoses and decisions based on the cell structure reflected by the stained images of cell specimens.

[0003] In the early technology, an artificial chemical staining method was usually used to generate stained images of cell specimens. Specifically, in the artificial chemical staining method, the cell specimen is first fixed, then stained with dyes such as hematoxylin, eosin, Giemsa, and other various reagents, and finally, after operations such as clearing and mounting, the stained image of the cell specimen can be observed under the bright field of a microscope. However, the artificial chemical staining method for cell specimens usually has complex steps and long time consumption, and its staining process is difficult to control consistently in terms of factors such as reagents, operations, processing environments, and specimen scanners. Therefore, it will lead to inconsistent staining results and interfere with the further analysis of the staining results.

[0004] To overcome the above technical problems, a virtual staining technology for realizing staining by computer processing of specimen images has been gradually developed. The virtual staining technology is a technology that gives a pseudo-color to a specimen image by performing specific algorithm processing on a digital image collected after the specimen is labeled with fluorescence or without labeling. The purpose of this technology is to enhance the contrast of cell structures and can achieve results as close as possible to the staining results of common dyes such as hematoxylin and eosin (H&E). The virtual staining technology can significantly shorten the staining time, reduce labor costs, and reduce the damage to specimens, making up for the defects of traditional chemical staining methods in cytopathology.

[0005] Virtual staining mainly includes a virtual staining method based on exogenous fluorescence-labeled microscopy imaging and a virtual staining method based on label-free microscopy imaging. Among them, the exogenous dyes introduced in the virtual staining method based on exogenous fluorescence-labeled microscopy imaging will have an irreversible impact on the physical and chemical properties of the specimen, interfering with other analyses of the specimen, and the autofluorescence effect of the specimen and other environmental factors will also affect the virtual staining results. Therefore, compared with the virtual staining method based on exogenous fluorescence-labeled microscopy imaging, it is more desirable to use the virtual staining method based on label-free microscopy imaging.

[0006] The virtual staining method based on label-free microscopy mainly uses a generative adversarial network (GAN) to perform virtual staining on the generated images to generate virtual staining images for doctors or researchers to make judgments and decisions. However, there are still two problems with the virtual staining method based on label-free microscopy.

[0007] On the one hand, the currently commonly used imaging techniques in the virtual staining method based on label-free microscopy cannot provide clear and accurate cell structure information. Therefore, it is difficult to accurately achieve cell-level staining by using the images generated by the currently commonly used imaging techniques for GAN virtual staining, and it is impossible to accurately provide the information of cell structure.

[0008] On the other hand, the currently adopted GAN virtual staining method in the virtual staining method based on label-free microscopy lacks the ability to recognize cell structures and cannot ensure the consistency of cell structures in the input image and the output image (for example, it is easy to misidentify the cell nucleus as the cytoplasm). Therefore, the generated virtual staining images cannot accurately provide the information of cell structure.

[0009] Therefore, an improved virtual staining method that can accurately provide the information of cell structure is needed. Summary of the Invention

[0010] In view of the above problems, the present invention provides a training method for obtaining a target generation network model, a virtual staining method for enhancing the display of cell structures, and a virtual staining system. The training method is trained by using the Raman images and real staining images of cell samples, and the loss function in the cyclic generative adversarial network is constrained for cell structure consistency, so that the target generation network model obtained based on the trained cyclic generative adversarial network can generate virtual staining images more consistent with the real staining images, improving the accuracy of cell virtual staining and achieving enhanced display of cell structures in cell samples.

[0011] According to one aspect of the present invention, there is provided a training method for obtaining a target generation network model, including: obtaining a Raman image of a cell sample and a corresponding true staining image of the cell sample; training a cycle generative adversarial network based on the Raman image and the true staining image so that the value of the loss function of the generative network in the cycle generative adversarial network converges; and obtaining a target generation network model based on the generative network in the trained cycle generative adversarial network with a converged loss function value, wherein the target generation network model generates a target virtual staining image based on a target Raman image of a target cell sample, and the target virtual staining image is used to enhance the display of cell structures in the target cell sample; wherein, the generative network includes a first generative network and a second generative network, the loss function includes a first loss function for the first generative network and a second loss function for the second generative network, and the first loss function and the second loss function include a cell structure similarity constraint term, and the cell structure constraint term constrains the consistency of the cell structures in the input image and the output image of the generative network based on the structural similarity SSIM function, wherein, the cell structure similarity constraint term of the first loss function includes The cell structure similarity constraint term of the second loss function includes wherein, a is a Raman image belonging to a Raman image domain A composed of one or more of the Raman images, and G A→B (a) is an intermediate virtual staining image generated by inputting a into the first generative network G A→B , and SSIM(G A→B (a), a) is the structural similarity value between the intermediate virtual staining image and the Raman image, is the expected value of the structural similarity loss value corresponding to each Raman image in the Raman image domain A; and b is a true staining image belonging to a true staining image domain B composed of one or more of the true staining images, and G B→A (b) is an intermediate Raman image generated by inputting b into the second generative network G B→A , and SSIM(G B→A (b), b) is the structural similarity value between the intermediate Raman image and the true staining image, is the expected value of the structural similarity loss value corresponding to each true staining image in the true staining image domain B.

[0012] According to some embodiments of the present invention, training a cycle generative adversarial network based on the Raman image and the real staining image to converge the values of the loss functions of the generative network in the cycle generative adversarial network includes: inputting the Raman image into the first generative network to generate an intermediate virtual staining image, inputting the intermediate virtual staining image into the first discriminative network in the cycle generative adversarial network to determine the probability that the intermediate virtual staining image is judged as the real staining image, and inputting the real staining image into the second generative network to generate an intermediate Raman image, inputting the intermediate Raman image into the second discriminative network in the cycle generative adversarial network to determine the probability that the intermediate Raman image is judged as the Raman image, and adjusting the parameters of the cycle generative adversarial network based on the probabilities determined by the first discriminative network and the second discriminative network so that the values of the first loss function and the second loss function converge respectively.

[0013] According to some embodiments of the present invention, adjusting the parameters of the cycle generative adversarial network based on the probabilities determined by the first discriminative network and the second discriminative network so that the values of the first loss function and the second loss function converge respectively includes: determining the values of the first loss function and the second loss function based on the probabilities determined by the first discriminative network and the second discriminative network; adjusting the parameters of the cycle generative adversarial network based on the values of the first loss function and the second loss function so that the values of the first loss function and the second loss function converge respectively.

[0014] According to some embodiments of the present invention, the expression of the first loss function is:

[0015] L G (G A→B )=L adv (G A→B )+γL cycle +pL SSIM1

[0016] L G (G A→B ) is the first loss function, L adv (G A→B ) is the adversarial loss constraint term of the first loss function, L cycle is the cycle consistency loss constraint term of the first loss function, L SSIM1 is the structural similarity constraint term of the first loss function for cells; where D B is the first discriminative network, D B (G A→B(a)) is the probability that the intermediate virtual staining image is judged as the real staining image, is the expected value of the adversarial loss value corresponding to each Raman image in the Raman image domain A; wherein, G B→A (G A→B (a)) is the reconstructed Raman image generated by inputting the intermediate virtual staining image into the second generation network G B→A and is the expected value of the mean absolute error value between each Raman image in the Raman image domain A and the corresponding reconstructed Raman image, G A→B (G B→A (a)) is the reconstructed real staining image generated by inputting the intermediate Raman image into the first generation network G A→B and is the expected value of the mean absolute error value between each real staining image in the real staining image domain B and the corresponding reconstructed real staining image; wherein, γ = 10, p = 2.

[0017] According to some embodiments of the present invention, wherein, the expression of the second loss function is:

[0018] L G (G B→A ) = L adv (G B→A ) + γL cycle + pL SSIM2

[0019] wherein, L G (G B→A ) is the second loss function, L adv (G B→A ) is the adversarial loss constraint term of the second loss function, L cycle is the cycle consistency loss constraint term of the second loss function, L SSIM2 is the structural similarity constraint term of the second generation network; wherein, D A is the second discriminant network, D A (G B→A (b)) is the probability that the intermediate Raman image is judged as the Raman image, is the expected value of the adversarial loss value corresponding to each real staining image in the real staining image domain B.

[0020] According to some embodiments of the present invention, wherein, the loss function further includes a congruent mapping loss constraint term, and the expression of the congruent mapping loss constraint term is:

[0021]

[0022] Among them, G B→A (a)) is the congruent mapping Raman image generated by inputting a into the second generation network G B→A The generated congruent mapping Raman image, is the expected value of the mean absolute error value between each Raman image in the Raman image domain A and the corresponding congruent mapping Raman image, G A→B (b)) is the congruent mapping real staining image generated by inputting b into the first generation network G A→B The generated congruent mapping real staining image, is the expected value of the mean absolute error value between each real staining image in the real staining image domain B and the corresponding congruent mapping real staining image; among them, the first loss function can also be expressed as: L G (G A→B ) = L adc (G A→B ) + γL cycle + λL idt + pL SSIM1 , the second loss function can also be expressed as: L G (G B→A ) = L adv (G B→A ) + γL cycle + λL idt + pL SSIM2 , where λ = 1.

[0023] According to some embodiments of the present invention, wherein training the cycle generative adversarial network based on the Raman image and the real staining image so that the value of the loss function of the generative network in the cycle generative adversarial network converges includes: iteratively training the cycle generative adversarial network based on a plurality of Raman images and a plurality of corresponding real staining images until the value of the loss function of the generative network in the cycle generative adversarial network converges.

[0024] According to some embodiments of the present invention, wherein obtaining the target generative network model based on the generative network in the trained cycle generative adversarial network with a converged loss function value includes: in response to the values of the first loss function and the second loss function each converging, using the parameters of the first generative network in the trained cycle generative adversarial network with a converged loss function value to obtain the target generative network model.

[0025] According to some embodiments of the present invention, the training method further includes: performing image registration on the Raman image and the corresponding true staining image so that the Raman image and the true staining image are aligned at the pixel level; inputting the registered Raman image and the corresponding registered true staining image into the trained cyclic generative adversarial network with a converged loss function value to verify the trained cyclic generative adversarial network with the converged loss function value.

[0026] According to some embodiments of the present invention, wherein verifying the trained cyclic generative adversarial network with a converged loss function value includes: determining whether the loss function value of the generative network in the trained cyclic generative adversarial network with a converged loss function value is overfitting; when it is determined that the loss function value of the generative network in the trained cyclic generative adversarial network with a converged loss function value is not overfitting, obtaining a target generative network model based on the generative network in the trained cyclic generative adversarial network with a converged loss function value; and when it is determined that the loss function value of the generative network in the trained cyclic generative adversarial network with a converged loss function value is overfitting, adjusting the parameters of the trained cyclic generative adversarial network with a converged loss function value and training the cyclic generative adversarial network with the adjusted parameters based on the Raman image and the true staining image.

[0027] According to some embodiments of the present invention, wherein obtaining the Raman image of the cell sample and the corresponding true staining image of the cell sample includes: performing Raman scattering microscopy imaging on the cell sample to obtain the Raman image of the cell sample; performing chemical staining and bright-field microscopy imaging on the cell sample for which the Raman image has been generated to obtain the corresponding true staining image of the cell sample.

[0028] According to some embodiments of the present invention, wherein performing Raman scattering microscopy imaging on the cell sample to obtain the Raman image of the cell sample includes: irradiating the cell sample with a first excitation light and a second excitation light; in response to the irradiation of the cell sample with the first excitation light and the second excitation light, receiving reflected light having at least one Raman shift characteristic peak from the cell sample to obtain the Raman image of the cell sample.

[0029] According to some embodiments of the present invention, the first excitation light and the second excitation light each have a predetermined frequency, and in response to the first excitation light and the second excitation light irradiating the cell sample, receiving reflected light having at least one Raman shift characteristic peak from the cell sample to obtain the Raman image of the cell sample includes: in response to the first excitation light and the second excitation light having a predetermined frequency irradiating the cell sample, receiving reflected light having only one Raman shift characteristic peak from the cell sample to obtain the Raman image of the cell sample.

[0030] According to some embodiments of the present invention, the first excitation light and the second excitation light are pulsed lasers, and the frequency difference between the first excitation light and the second excitation light is 2800 - 3100 cm -1 , the repetition frequency is greater than 50 MHz, and the pulse width is 100 fs - 20 ps.

[0031] According to some embodiments of the present invention, the first excitation light includes pump light, the second excitation light includes Stokes light, and in response to the first excitation light and the second excitation light irradiating the cell sample, receiving reflected light having at least one Raman shift characteristic peak from the cell sample to obtain the Raman image of the cell sample includes: in response to the pump light and the Stokes light irradiating the cell sample, receiving reflected light having at least one Raman shift characteristic peak from the cell sample to obtain the coherent Raman scattering imaging image of the cell sample, and the coherent Raman scattering imaging image includes stimulated Raman scattering imaging image, stimulated Raman optothermal imaging image or coherent anti-Stokes Raman scattering imaging image.

[0032] According to some embodiments of the present invention, performing Raman scattering microscopy on the cell sample to obtain the Raman image of the cell sample includes: performing Raman scattering microscopy on each part of the cell sample under a local field of view to obtain the local Raman image of the cell sample; stitching the local Raman images of each part of the cell sample to generate the Raman image of the cell sample; and performing chemical staining and bright-field microscopy on the cell sample on which the Raman image has been generated to obtain the corresponding true staining image of the cell sample includes: performing chemical staining on the cell sample on which the Raman image has been generated, and performing bright-field microscopy on each part of the chemically stained cell sample under a local field of view to obtain the local true staining image of the cell sample, and stitching the local true staining images of each part of the cell sample to generate the true staining image of the cell sample.

[0033] According to another aspect of the present invention, there is also provided a virtual staining method for enhancing the display of cell structures, including: obtaining a target Raman image of a target cell sample; inputting the target Raman image into a target generation network model according to any one of the above training methods to obtain a target virtual staining image of the target cell sample, where the target virtual staining image is used to enhance the display of cell structures in the target cell sample.

[0034] According to some embodiments of the present invention, wherein obtaining a target Raman image of a target cell sample includes: performing Raman scattering microscopy imaging on the target cell sample to obtain the target Raman image of the target cell sample.

[0035] According to some embodiments of the present invention, wherein performing Raman scattering microscopy imaging on the target cell sample to obtain the target Raman image of the target cell sample includes: irradiating the target cell sample with a first excitation light and a second excitation light; in response to the irradiation of the target cell sample with the first excitation light and the second excitation light, receiving reflected light having at least one Raman shift characteristic peak from the target cell sample to obtain the target Raman image of the target cell sample.

[0036] According to some embodiments of the present invention, wherein the first excitation light and the second excitation light each have a predetermined frequency, and in response to the irradiation of the target cell sample with the first excitation light and the second excitation light, receiving reflected light having at least one Raman shift characteristic peak from the target cell sample to obtain the target Raman image of the target cell sample includes: in response to the irradiation of the target cell sample with the first excitation light and the second excitation light having a predetermined frequency, receiving reflected light having only one Raman shift characteristic peak from the target cell sample to obtain the target Raman image of the target cell sample.

[0037] According to some embodiments of the present invention, wherein the first excitation light and the second excitation light are pulsed lasers, and the frequency difference between the first excitation light and the second excitation light is 2800 - 3100 cm -1 , the repetition frequency is greater than 50 MHz, and the pulse width is 100 fs - 20 ps.

[0038] According to some embodiments of the present invention, the first excitation light includes pump light, the second excitation light includes Stokes light, and in response to the first excitation light and the second excitation light irradiating the target cell sample, reflected light having at least one Raman shift characteristic peak is received from the target cell sample to obtain the target Raman image of the target cell sample: in response to the pump light and the Stokes light irradiating the target cell sample, reflected light having at least one Raman shift characteristic peak is received from the target cell sample to obtain a coherent Raman scattering imaging image of the cell sample, and the coherent Raman scattering imaging image includes a stimulated Raman scattering imaging image, a stimulated Raman photothermal imaging image, or a coherent anti-Stokes Raman scattering imaging image.

[0039] According to some embodiments of the present invention, performing Raman scattering microscopy imaging on the target cell sample to obtain the target Raman image of the target cell sample includes: performing Raman scattering microscopy imaging on each part of the target cell sample under a local field of view to obtain a local target Raman image of the target cell sample; stitching the local target Raman images of each part of the target cell sample to obtain the target Raman image of the target cell sample.

[0040] According to another aspect of the present invention, there is also provided a virtual staining system, including: an image acquisition component configured to obtain a target Raman image of a target cell sample; and an image processing component configured to input the target Raman image into the target generation network model according to any one of the above training methods to obtain a target virtual staining image of the target cell sample, and the target virtual staining image is used to enhance the display of cell structures in the target cell sample.

[0041] According to some embodiments of the present invention, the image acquisition component is configured to perform Raman scattering microscopy imaging on the target cell sample to obtain the target Raman image of the target cell sample.

[0042] According to some embodiments of the present invention, the virtual staining system further includes: a laser source configured to generate a first excitation light and a second excitation light for irradiating the target cell sample; and the image acquisition component is further configured to, in response to the first excitation light and the second excitation light irradiating the target cell sample, receive reflected light having at least one Raman shift characteristic peak from the target cell sample to obtain the target Raman image of the target cell sample.

[0043] According to some embodiments of the present invention, the first excitation light and the second excitation light each have a predetermined frequency, and the image acquisition component is further configured to: in response to the first excitation light and the second excitation light having the predetermined frequency irradiating the target cell sample, receive reflected light having only one Raman shift characteristic peak from the target cell sample, so as to obtain the target Raman image of the target cell sample.

[0044] According to some embodiments of the present invention, the virtual staining system further includes: an optical path component, the optical path component includes a two-dimensional galvanometer assembly and a first filter configured to guide the first excitation light and the second excitation light to the target cell sample; a sample carrier component configured to carry the target cell sample to receive the irradiation of the first excitation light and the second excitation light; and an objective lens component configured to receive the reflected light from the target cell sample and transmit the reflected light to the image acquisition component.

[0045] According to some embodiments of the present invention, the virtual staining system further includes: an autofocus component, wherein the autofocus component includes a focus detection unit, a second filter, and a moving component for moving the objective lens component; the focus detection unit is configured to: generate a third excitation light, the third excitation light irradiates the sample carrier component through the second filter; detect the detected reflected light reflected back to the focus detection unit by the sample carrier component; and control the moving component according to the detection result so that the objective lens component receives the reflected light from the target cell sample.

[0046] According to some embodiments of the present invention, the first excitation light and the second excitation light are pulsed lasers, and the frequency difference between the first excitation light and the second excitation light is 2800 - 3100 cm -1 , the repetition frequency is greater than 50 MHz, and the pulse width is 100 fs - 20 ps.

[0047] According to some embodiments of the present invention, the first excitation light includes pump light, the second excitation light includes Stokes light, and the image acquisition component is further configured to: in response to the pump light and the Stokes light irradiating the cell sample, receive reflected light having at least one Raman shift characteristic peak from the cell sample, so as to obtain a coherent Raman scattering imaging image of the cell sample, and the coherent Raman scattering imaging image includes a stimulated Raman scattering imaging image, a stimulated Raman optothermal imaging image, or a coherent anti-Stokes Raman scattering imaging image.

[0048] According to some embodiments of the present invention, the image acquisition component is configured to: perform Raman scattering microscopy imaging on each part of the target cell sample under a local field of view to obtain a local target Raman image of the target cell sample; and splice the local target Raman images of each part of the target cell sample to obtain the target Raman image of the target cell sample.

[0049] Embodiments of the present invention provide a training method for obtaining a target generation network model, a virtual staining method for enhancing the display of cell structures, and a virtual staining system.

[0050] Therefore, in the training method for obtaining a target generation network model according to the embodiments of the present invention, a Raman image that can more accurately describe the cell structure of a cell sample is obtained by using Raman imaging technology; at the same time, during the training process using the Raman image and the real staining image of the cell sample, the loss function in the cycle generative adversarial network is constrained for cell structure consistency through a structural similarity function, so that the target generation network model obtained based on the trained cycle generative adversarial network can generate a virtual staining image that is more consistent with the real staining image, avoiding problems such as the inversion of nuclear and cytoplasmic staining during the virtual staining process, improving the accuracy of cell virtual staining, and achieving enhanced display of cell structures.

[0051] In this way, in the virtual staining method and virtual staining system for enhancing the display of cell structures according to the embodiments of the present invention, the target cell sample is virtually stained by the target generation network model obtained by the above training method, and a virtual staining image that accurately enhances the display of the structure of the target cell sample can be obtained, realizing the extension of virtual staining imaging from the tissue level to the cell level, making the virtual staining image have significant consistency with the real staining image, so that clinicians or researchers can analyze the cell structure of the target cell sample based on the virtual staining image to make accurate judgments and decisions. Description of the Drawings

[0052] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some exemplary embodiments of the present invention, and those of ordinary skill in the art can obtain other drawings without creative efforts based on these drawings.

[0053] Figure 1 Shows an example diagram of a virtual staining method based on ultraviolet surface excitation microscopy imaging in the prior art;

[0054] Figure 2Shows an example diagram of a Raman image and a true staining image obtained by different methods based on the training method including the embodiments of the present invention;

[0055] Figure 3 Shows a flowchart of a training method for obtaining a target generation network model according to some embodiments of the present invention;

[0056] Figure 4 Shows an architecture diagram of a cyclic generative adversarial network according to some embodiments of the present invention;

[0057] Figure 5 Shows a training flowchart of a cyclic generative adversarial network according to some embodiments of the present invention;

[0058] Figure 6 Shows a schematic diagram of the network structures of a generation network and a discriminator network according to some embodiments of the present invention;

[0059] Figure 7 Shows a schematic diagram of splicing and registration processing of a Raman image and a true staining image according to some embodiments of the present invention;

[0060] Figure 8 Shows an evaluation schematic diagram of a target generation network model according to some embodiments of the present invention;

[0061] Figure 9 Shows a flowchart of a virtual staining method for enhancing the display of cell structures according to some embodiments of the present invention;

[0062] Figure 10 Shows a block diagram of a virtual staining system according to some embodiments of the present invention;

[0063] Figure 11 Shows an example structural diagram of a virtual staining system 1000 according to some embodiments of the present invention;

[0064] Figure 12 Shows a schematic diagram of a microscopic imaging system of a Raman image according to some embodiments of the present invention;

[0065] Figure 13 Shows a graph of the spontaneous Raman scattering spectra of lipid pure samples and protein pure samples in the carbon-hydrogen bond vibration region, that is, the curve of the spontaneous Raman scattering intensity at different frequency differences between the pump light and the Stokes light, according to some embodiments of the present invention;

[0066] Figure 14 Shows another example structural diagram of a virtual staining system 1000 according to some embodiments of the present invention;

[0067] Figure 15The structural diagram of the electronic device 1500 according to some embodiments of the present invention is shown. Detailed implementation manners

[0068] In order to make the objectives, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the described embodiments of the present invention without creative efforts shall fall within the scope of protection of the present invention.

[0069] Unless otherwise defined, the technical terms or scientific terms used in the present invention shall have the ordinary meanings understood by those of ordinary skill in the art to which the present invention pertains. The "first", "second" and similar terms used in the present invention do not denote any order, quantity or importance, but are only used to distinguish different components. The terms such as "including" or "comprising" mean that the elements or items appearing before this term cover the elements or items listed after this term and their equivalents, without excluding other elements or items. The terms such as "connected" or "coupled" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The terms such as "upper", "lower", "left" and "right" are only used to represent relative positional relationships. When the absolute position of the object being described changes, the relative positional relationship may also change accordingly. To keep the following description of the embodiments of the present invention clear and concise, some details of known functions and known components are omitted in the present invention.

[0070] Flowcharts are used in the present invention to illustrate the steps of the methods according to the embodiments of the present invention. It should be understood that the steps before or after do not necessarily need to be carried out precisely in sequence. On the contrary, they can be carried out in reverse order or various steps can be processed simultaneously. At the same time, other operations can also be added to these processes, or one or several steps can be removed from these processes.

[0071] In the description and drawings of the present invention, elements are described in singular or plural forms according to the embodiments. However, the singular and plural forms are appropriately selected for the presented cases only for the convenience of explanation and are not intended to limit the present invention thereto. Therefore, the singular form may include the plural form, and the plural form may also include the singular form, unless the context clearly indicates otherwise.

[0072] As mentioned above, virtual staining mainly includes virtual staining methods based on exogenous fluorescence-labeled microscopy imaging and virtual staining methods based on label-free microscopy imaging.

[0073] Among them, the virtual staining method based on exogenous fluorescence-labeled microscopy provides basic chemical composition information for virtual staining by collecting the contrast of fluorescence signal intensities at different positions of the specimen during imaging. For example, Figure 1 as shown, microscopy with ultraviolet surface excitation (MUSE) is representative of such methods. It can achieve virtual H&E staining of tissues without fixation, embedding, and sectioning, converting the traditional overnight processing into 2-3 minutes of fluorescence staining and a few seconds of image processing.

[0074] Specifically, the virtual staining method based on exogenous fluorescence-labeled imaging first stains the specimen with fluorescent dyes (such as acridine orange, DAPI, etc.) to emphasize key features in virtual staining such as cell nuclei inside the specimen. Subsequently, the specimen is imaged, and the result is converted into a stained image according to the gray values of each pixel. For example, in MUSE, after soaking the tissue with Hoechst 33342 and Rhodamine B or other various dyes, the fluorescence label on its surface layer is excited by an ultraviolet LED and imaged. Subsequently, the colors in the image are unmixed, adjusted, and remixed to make the virtual staining result closer to H&E staining.

[0075] However, the exogenous dyes introduced by the virtual staining method based on exogenous fluorescence-labeled microscopy will have an irreversible impact on the physical and chemical properties of the specimen, easily interfering with other analyses of the specimen. Moreover, the autofluorescence effect of the specimen and other environmental factors will also affect the virtual staining result. In addition, the virtual staining algorithms based on exogenous fluorescence-labeled microscopy usually come from experience or derivation, and it is difficult to ensure the authenticity of the staining result in terms of visual effect. Therefore, compared with the virtual staining method based on exogenous fluorescence-labeled microscopy, it is more desirable to adopt a virtual staining method based on label-free microscopy to obtain a more realistic virtual staining image.

[0076] In the virtual staining method based on label-free microscopy imaging, the currently used imaging methods mainly include autofluorescence imaging (AFI), photoacoustic imaging (PAI), quantitative phase imaging (QPI), etc., and then the generative adversarial network (GAN) is used to perform virtual staining on the generated images. Due to the lack of exogenous labels to provide contrast, such methods need to obtain relevant information for virtual staining through other methods. For example, relevant information can be obtained based on imaging parameters and image processing methods. Deep learning technology enables the trained deep neural network to automatically find high-dimensional features in the input data that can be used to process specific tasks. The generative adversarial network is a model architecture currently commonly used for virtual staining in histology, and its outstanding contribution is to make the staining results highly similar to real stained images. The virtual staining technology based on GAN can be used for virtual staining of autofluorescence images, photoacoustic images, and quantitative phase images, can be used to identify cell specimens in various tissues including the thyroid, liver, etc., and can perform virtual staining in a staining manner similar to various dyes such as H&E and Masson.

[0077] However, there are two problems with the virtual staining method based on label-free microscopy imaging, which result in the inability of the virtual staining method based on label-free microscopy imaging to provide information about cell structure as accurately as real stained images.

[0078] On the one hand, the molecular specificity of autofluorescence imaging is poor, highly dependent on components such as NADH and FAD, and it is difficult to distinguish cell structures when imaging cells; the spatial resolution of photoacoustic imaging is only at the micron level and cannot clearly image cells and their internal structures; quantitative phase imaging also cannot provide sufficient contrast for cell structures. Therefore, the currently common imaging techniques for label-free virtual staining cannot accurately identify cell structures, which in turn leads to difficulties in virtual staining by the generative adversarial network (GAN) at the cell level and inability to accurately provide cell-level information. As a result, clinicians and scientific researchers cannot make accurate diagnoses and decisions based on the virtual staining images obtained by such methods.

[0079] On the other hand, there are mainly two ways to train GANs: one is supervised training, in which the training dataset contains unlabeled microscopic images and corresponding stained images with pixel-level matching, so that the virtual staining result has a high accuracy, and the loss function can clearly represent the training progress. However, it is difficult to prepare the dataset for supervised training, and it is very difficult to obtain a large number of unlabeled microscopic images and corresponding stained images with pixel-level matching; the other is unsupervised training, usually using Cycle Generative Adversarial Network (CycleGAN). The dataset only needs to come from two image domains of the model input and output without precise matching, which significantly reduces the difficulty of dataset preparation. However, since the accuracy of the staining result cannot be directly evaluated by referring to the real image (ground truth, GT) during training in this case, it is difficult for the prior art to achieve virtual staining at the cellular structure level by using the unsupervised training CycleGAN.

[0080] Specifically, during the R & D process, the inventors of the present invention found through a large number of experimental studies that for the virtual staining image generated by the generative network trained by the traditional method, the consistency of the cellular structure in the input image and the output image is not maintained during the virtual staining process. The machine learning algorithm is prone to misidentifying the cell nucleus as the cytoplasm and staining the cell nucleus with the color corresponding to the cytoplasm. Correspondingly, the cytoplasm will also be stained with the color corresponding to the cell nucleus, that is, there is a problem of reversing the staining of the cell nucleus and the cytoplasm during the virtual staining process.

[0081] Exemplarily, Figure 2 shows an example diagram of Raman images and real stained images obtained by different methods including the training method of the embodiments of the present invention. As Figure 2 shown, Figure 2 the first row in corresponds to the Raman image of the cell sample (the cell sample can be exfoliated cells in peritoneal lavage fluid). Specifically, in the left half a of the first row is the Raman image, and the right halves b and c are enlarged views of two regions in the left Raman image. The second row corresponds to the real stained image of the cell sample (the real stained image can be obtained, for example, by staining the cell sample with hematoxylin-eosin staining method). The third row corresponds to the first virtual staining image of the cell sample, and the first virtual staining image is generated by the generative network of the traditional technology. By comparison, it can be seen that in the real stained image in the second row, the color on the left side of the cell pointed by the arrow is light (corresponding to the cytoplasm), and the color on the right side is dark (corresponding to the cell nucleus). However, in the first virtual staining image in the third row, the staining of the cell nucleus and the cytoplasm is reversed, Figure 2 in the first virtual staining image in, the color on the left side of the cell pointed by the arrow is dark (corresponding to the cell nucleus), and the color on the right side is light (corresponding to the cytoplasm), which is exactly opposite to the corresponding position in the real stained image.

[0082] To solve the above technical problems, the present invention provides a training method for obtaining a target generation network model, a virtual staining method for enhancing the display of cell structures, and a virtual staining system.

[0083] The following will illustrate through specific embodiments the training method for obtaining a target generation network model, the virtual staining method for enhancing the display of cell structures, and the virtual staining system provided by the present application.

[0084] First Embodiment

[0085] The following will refer to the accompanying drawings to elaborate on the training method for obtaining a target generation network model provided by the above present invention.

[0086] Figure 3 The flowchart of the training method for obtaining a target generation network model according to some embodiments of the present invention is shown. As Figure 3 shown, first, in step S310, a Raman image of a cell sample and a corresponding true staining image of the cell sample are obtained.

[0087] In one example, the cell sample may include cancer cells, immune cells, lymphocytes, mesothelial cells, epithelial cells, blood cells, granulocytes, etc. The cell sample can be collected from organs or tissues in animals and plants. The cell sample can be a cell sample that needs to be analyzed for cell structure in an actual application scenario, such as a clinical application scenario, a research and development scenario, etc. There is no specific limitation on the scenarios to which the embodiments of the present invention can be applied herein.

[0088] Since the chemical composition of the cell sample will change after chemical staining, which affects the results of coherent Raman scattering microscopy imaging. Therefore, it is necessary to first perform Raman scattering microscopy imaging on the unstained sample cell sample, and then perform chemical staining on the cell sample, so as to ensure the imaging accuracy of the Raman image of the cell sample.

[0089] According to an embodiment of the present invention, Raman scattering microscopy imaging can be performed on the cell sample to obtain a (coherent) Raman image of the cell sample. Then, the cell sample with the generated Raman image is chemically stained and bright-field microscopy imaging is performed to obtain a corresponding true staining image of the cell sample.

[0090] In one example, chemical staining and bright-field microscopy imaging can be, for example, the specific means adopted by the artificial chemical staining method as described above. Those skilled in the art know clearly how to obtain a true staining image through the artificial chemical staining method, so it will not be elaborated herein. In one example, the true staining image can be an H&E true staining image.

[0091] In one example, the true staining image of a cell sample obtained by an artificial chemical staining method is used, together with the Raman image of the cell sample, as the training set of the cycle generative adversarial network. It should be noted that in this article, the Raman image of the cell sample and the corresponding true staining image of the cell sample are not intended to refer only to a single image, but may include one or more images or an image set, so as to enable iterative training of the cycle generative adversarial network.

[0092] Coherent Raman scattering microscopy is a technique that generates images using spatially resolved spectral information. The images obtained by coherent Raman scattering microscopy can fully reflect the chemical information of cell structures. In other words, coherent Raman scattering microscopy has obvious advantages in terms of comprehensive consideration of spatial resolution, molecular specificity, etc. It can utilize the Raman shift characteristic peaks of lipids and proteins widely present in various cells, and thus can be widely used in cell imaging. In addition, the coherent Raman imaging of cell samples has molecular specificity at the chemical bond level and solves the defects of weak spontaneous Raman scattering signals and slow imaging speed. Therefore, compared with other imaging methods such as API, the Raman imaging method can be used to image cell structures more accurately, facilitating subsequent accurate virtual staining at the cell structure level based on neural networks.

[0093] In this article, the term "coherent Raman scattering microscopy" can be used interchangeably with terms such as "coherent Raman imaging" and "Raman imaging".

[0094] It should be noted that since it is necessary to first perform Raman imaging on the cell sample and then perform artificial staining, and artificial staining takes some time, the cells in the obtained Raman image and the true staining image may be displaced, resulting in the images not being strictly registered at the pixel level. However, according to the principle of the cycle generative adversarial network, such images can also be used to train the model. By randomly cropping the same number of images of a fixed size from the two types of images respectively, the training set of the cycle generative network model can be obtained.

[0095] After obtaining the Raman image of the cell sample and the corresponding true staining image of the cell sample, in step S320, the cycle generative adversarial network can be trained based on the Raman image and the true staining image to make the value of the loss function of the generative network in the cycle generative adversarial network converge.

[0096] During the training process of the cycle generative adversarial network, Raman images and real staining images can be used as training samples. Through the confrontation between the generative network and the discriminative network included in the cycle generative adversarial network, the generative network can fully learn how to generate virtual staining images that are infinitely close to real staining images based on Raman images. In this way, after the cycle generative adversarial network is trained, the target generative network can generate realistic virtual staining images according to the input Raman images. These virtual staining images are similar to real staining images and can be used to enhance the display of cell structures in the target cell samples.

[0097] Specifically, Figure 4 shows an architecture diagram of a cycle generative adversarial network according to some embodiments of the present invention. As Figure 4 shown, according to an embodiment of the present disclosure, the generative network may include a first generative network G A→B and a second generative network G B→A . The discriminative network may include a first discriminative network D B and a second discriminative network D A . A represents the Raman image domain A composed of one or more Raman images, and B represents the real staining image domain B composed of one or more real staining images.

[0098] The first generative network G A→B is used to convert the input Raman image into an intermediate virtual staining image and output the intermediate virtual staining image. The first discriminative network D B identifies whether the image input to the first discriminative network D B is a real staining image or the intermediate virtual staining image output by the first generative network G A→B , and outputs the probability that the input image is a real staining image. Similarly, the second generative network G B→A is similar to the second discriminative network D A . The two discriminative networks play a supervisory role for the two generative networks respectively, so as to drive the cycle generative adversarial network to continuously improve the similarity between the generated (i.e., intermediate virtual staining images or intermediate Raman images) images and the images used for discrimination (i.e., real staining images or original Raman images), making it increasingly difficult for the discriminative network to distinguish the input images.

[0099] In the training of a neural network model, it is usually necessary to define a loss function, and the value of the loss function is obtained from the output of the neural network model. Generally speaking, the larger the value of the loss function, the worse the current performance of the neural network model, and the greater the degree of punishment for the neural network model. Subsequently, through the backpropagation algorithm, the parameters of the neural network model are updated in the direction of reducing the loss value, so that the neural network model can give better results. In addition, the greater the degree of punishment, the greater the change amplitude of the neural network model parameters. Therefore, for the adversarial cycle generative adversarial network, the loss function can be converged by optimizing the network parameters, so that the trained cycle generative adversarial network can be used for virtual staining.

[0100] According to an embodiment of the present disclosure, the loss function may include a first loss function for the first generative network and a second loss function for the second generative network. The loss function may also include loss functions for the discriminative network respectively. Considering that when the loss function of the generative network converges, the loss function of the discriminative network will also converge, and the target generative network model (i.e., the model that can be used for virtual staining of the Raman image of the cell sample to be recognized) is generated based on the generative network part in the cycle adversarial generative network, so preferably the loss function of the generative network can be considered without considering the loss function of the discriminative network.

[0101] Figure 5 The training flow chart of the cycle generative adversarial network according to some embodiments of the present invention is shown. As Figure 5 shown, training the cycle generative adversarial network based on the Raman image and the real staining image to converge the value of the loss function of the generative network in the cycle generative adversarial network can be specifically achieved through the following steps.

[0102] In step S510, the Raman image can be input into the first generative network G A→B to generate an intermediate virtual staining image.

[0103] In step S520, the intermediate virtual staining image can be input into the first discriminative network D in the cycle generative adversarial network B to determine the probability that the intermediate virtual staining image is judged as a real staining image, and

[0104] In step S530, the real staining image can be input into the second generative network G B→A to generate an intermediate Raman image.

[0105] In step S540, the intermediate Raman image can be input into the second discriminative network D in the cycle generative adversarial network A to determine the probability that the intermediate Raman image is judged as a Raman image.

[0106] In step S550, based on the first discriminative network DB and the second discriminant network D A to determine the parameters of the probability-adjusted cycle generative adversarial network to converge the values of the first loss function and the second loss function respectively. According to an embodiment of the present invention, based on the first discriminant network D B and the second discriminant network D A to determine the values of the first loss function and the second loss function; based on the values of the first loss function and the second loss function, adjust the parameters of the cycle generative adversarial network to converge the values of the first loss function and the second loss function respectively.

[0107] As described above, the Raman image of the cell sample may include one or more Raman images, while the corresponding true staining image of the cell sample may include one or more corresponding true staining images. Therefore, according to an embodiment of the present disclosure, the cycle generative adversarial network can be iteratively trained based on multiple Raman images and multiple corresponding true staining images until the value of the loss function of the generative network in the cycle generative adversarial network converges.

[0108] In an example of the present invention, the above steps S510-S550 can be repeatedly executed until the values of the first loss function and the second loss function converge respectively.

[0109] Figure 6 shows a schematic diagram of the network structure of the generative network and the discriminant network according to some embodiments of the present invention. As Figure 6 shown, the cycle generative adversarial network can be constructed using, for example, PyTorch 1.7.0, the training graphics card uses, for example, NVIDIA Tesla P100-16G, the optimizer uses, for example, Adam (β values are 0.5 and 0.999), and the learning rate can be set to 10 -4 , and is reduced to 10 -5 in the last 1 / 3 of the training phase. Regarding the model architecture of the cycle generative adversarial network, the generative network (the first generative network and the second generative network) adopts ResNet, and the discriminant network (the first discriminant network and the second discriminant network) adopts a 70×70 PatchGAN.

[0110] Further as Figure 6 shown, residual block is a residual block, BN is batch normalization, Averaging is to find the average, ReLU is a rectified linear unit, Conv is a convolutional layer, S is the convolutional stride, ConvTranspose is a transposed convolutional layer, where:

[0111]

[0112] ResidualBlock(x) = x + Conv{Conv{x}}

[0113] Wherein, Conv{} performs convolution with a stride of 1, batch normalization, and ReLU in sequence. The kernel size is 3 except for the convolution of the input and output of the generator, which is 7.

[0114] In one example, considering the huge computational amount in the training process, the training process of the target generation network in the embodiments of the present invention can be executed on the server rather than on the terminal. After the training is completed on the server, the target generation network is sent to the terminal for the terminal to use.

[0115] In order to control the consistency of the cell region and its internal nuclear and cytoplasmic structures before and after virtual staining, improve the accuracy at the cell level, and avoid problems such as the inversion of staining of the nucleus and cytoplasm during the virtual staining process, according to some embodiments of the present invention, the loss function may include a cell structure similarity constraint term. The first loss function and the second loss function may include a cell structure similarity constraint term, and this cell structure constraint term may be based on the structural similarity (SSIM) function to constrain the consistency of the cell structures in the input image and the output image of the generation network. In some examples, the cell structure constraint term may also be based on functions such as neural network perceptual loss or morphological algorithms to constrain the consistency of the cell structures in the input image and the output image of the generation network.

[0116] In one example, the first loss function of the first generation network may include at least a first cell structure constraint term, and the first cell structure constraint term is used to constrain the consistency of the cell structures in the input image and the output image of the first generation network; and the loss function of the second generation network may include at least a second cell structure constraint term, and the second cell structure constraint term is used to constrain the consistency of the cell structures in the input image and the output image of the second generation network.

[0117] The cell structure constraint expression of the first loss function may be as shown in Formula 1 or Formula 2:

[0118]

[0119]

[0120] In one example, although the cell structure constraint term of the first loss function simultaneously includes parts of the first generation network G A→B and the second generation network G B→A , in actual calculation, only the first half part can be considered, that is, only the part including the first generation network G A→B is considered. This is because for the first generation network GA→B When calculating the gradient of the loss function, only the loss of the part related to the first generation network G A→B is effective, while the part related to the second generation network G B→A is not related to the parameters of the first generation network G A→B and does not affect the parameter update of the first generation network G A→B Similarly, for the cell structure constraint term of the second loss function, simplified calculation can also be applied, that is, only the part related to the second generation network G B→A is considered.

[0121] Therefore, according to an embodiment of the present invention, the cell structure similarity constraint term L of the first loss function SSIM1 may include the cell structure similarity constraint term L of the second loss function SSIM2 may include

[0122] where a is a Raman image belonging to the Raman image domain A composed of one or more Raman images, and G A→B (a) is the intermediate virtual stained image generated by inputting a into the first generation network G A→B , and SSIM(G A→B (a), a) is the structural similarity value between the intermediate virtual stained image and the Raman image, is the expected value of the structural similarity loss value corresponding to each Raman image in the Raman image domain A; and b is a real stained image belonging to the real stained image domain B composed of one or more real stained images, and G B→A (b) is the intermediate Raman image generated by inputting b into the second generation network G B→A , and SSIM(G B→A (b), b) is the structural similarity value between the intermediate Raman image and the real stained image, is the expected value of the structural similarity loss value corresponding to each real stained image in the real stained image domain B.

[0123] In addition to the cell structure constraint term, for the cyclic generative adversarial network, the loss function (including the first loss function and the second loss function) may further include constraint terms such as adversarial loss and cycle consistency loss.

[0124] The adversarial loss L advIt is a loss function constraint term that defines both the generation network and the discriminator network, and is mutually contradictory, making the two form a competitive relationship. For the generation network, the more the output of the generation network is surely recognized as false by the discriminator network, that is, the output of the discriminator network is close to 0, the greater the penalty it receives; at the same time, for the discriminator network, on the one hand, the more surely it recognizes the output of the generation network as true, that is, its own output is close to 1, the greater the penalty it receives, and on the other hand, the more surely it recognizes the real image as false, that is, its own output is close to 0, the greater the penalty it also receives.

[0125] The formula for the adversarial loss constraint term can, for example, adopt the least squares GAN (LSGAN), specifically see the following Formula 3 - Formula 6. Among them, Formula 3 is the adversarial loss constraint term expression of the first generation network G A→B ; Formula 4 is the adversarial loss constraint term expression of the second generation network G B→A ; Formula 5 is the adversarial loss constraint term expression of the second discriminator network D B ; Formula 6 is the adversarial loss constraint term expression of the first discriminator network D A :

[0126]

[0127]

[0128]

[0129]

[0130] Among them, a is a Raman image belonging to the Raman image domain A composed of one or more Raman images, and b is a real stained image belonging to the real stained image domain B composed of one or more real stained images. P dataA , P data are respectively the probability distributions of the two types of images reflected by the training set (the set of real stained images of cell samples and the set of Raman images of cell samples). E is the expected value. For example, is the expected value of a specific function (constraint term) when the probability distribution of the Raman image of the cell sample follows P dataA . Among them, G A→B corresponds to D B , G B→A corresponds to D A , and L D is the total loss of the discriminator network.

[0131] The cycle consistency loss L cycleIt is only defined for the generation network, and the design idea comes from: when a sentence is translated from language A to language B and then back from language B to language A, the resulting sentence should be the same as the original sentence. Similarly, when the Raman image of a cell sample is successively transformed through two generation networks, the resulting reconstructed Raman image should also be the same as the original Raman image of the original cell sample. The greater the difference between the resulting reconstructed Raman image and the original Raman image, the greater the penalty on the first discriminant network D A will receive. The purpose of the cycle consistency loss is that any generation network should not lose its key information when processing images, otherwise the original image cannot be reconstructed.

[0132] Since the cycle generative adversarial network has no labels during training, the cycle consistency loss is an additional constraint on the network. The following formula 7 is the expression of the cycle consistency loss constraint term, where ||·|| 1 is the L1 loss (mean absolute value loss):

[0133]

[0134] By obtaining the loss value in the above manner, the training of the model of the cycle generative adversarial network only requires an unpaired data set, that is, for one Raman image, there is no need to have a real staining image corresponding to its content at the pixel level.

[0135] Based on the various constraint terms described above, according to an embodiment of the present invention, the expression of the first loss function can be as shown in formula 8:

[0136] L G (G A→B ) = L adv (G A→B ) + γL cycle + pL SSIM1 Formula 8

[0137] L G (G A→B ) is the first loss function, L adv (G A→B ) is the adversarial loss constraint term of the first loss function, L cycle is the cycle consistency loss constraint term of the first loss function, L SSIM1 is the structural similarity constraint term of the first loss function for cells;

[0138] Among them, D B is the first discriminant network, D B (G A→B (a)) is the probability that the intermediate virtual staining image is judged as a real staining image, is the expected value of the adversarial loss value corresponding to each Raman image in the Raman image domain A;

[0139] Among them,

[0140] G B→A (G A→B (a)) is the reconstructed Raman image generated by inputting the intermediate virtual staining image into the second generation network G B→A ; is the expected value of the mean absolute error value between each Raman image in the Raman image domain A and the corresponding reconstructed Raman image, G A→B (G B→A (a)) is the reconstructed real staining image generated by inputting the intermediate Raman image into the first generation network G A→B ; is the expected value of the mean absolute error value between each real staining image in the real staining image domain B and the corresponding reconstructed real staining image; among them, γ = 10, p = 2.

[0141] According to an embodiment of the present invention, the expression of the second loss function can be as shown in Formula 9:

[0142] L G (G B→A ) = L adv (G B→A ) + γL cycle + pL SSI Formula 9 Among them, L G (G B→A ) is the second loss function, L adv (G B→A ) is the adversarial loss constraint term of the second loss function, L cycle is the cycle consistency loss constraint term of the second loss function, L SSIM2 is the structural similarity constraint term of the second generation network; among them, D A is the second discriminant network, D A (G B→A (b)) is the probability that the intermediate Raman image is judged to be a Raman image, is the expected value of the adversarial loss value corresponding to each real staining image in the real staining image domain B.

[0143] In order to further enhance the constraint on the cyclic generative adversarial network, according to an embodiment of the present invention, the loss function further includes a congruent mapping loss constraint term to achieve that when an X-class image is input into the generator that converts a Y-class image into an X-class image, the obtained result should be the same as the original image.

[0144] The possible expression of the congruent mapping loss constraint term is shown in Formula 10:

[0145]

[0146] Wherein, G B→A (a)) is the congruent mapping Raman image generated by inputting a into the second generation network G B→A The expected value of the mean absolute error value between each Raman image in the Raman image domain A and the corresponding congruent mapping Raman image, G is the expected value of the mean absolute error value between each Raman image in the Raman image domain A and the corresponding congruent mapping Raman image, G A→B (b)) is the congruent mapping true staining image generated by inputting b into the first generation network G A→B The expected value of the mean absolute error value between each true staining image in the true staining image domain B and the corresponding congruent mapping true staining image. is the expected value of the mean absolute error value between each true staining image in the true staining image domain B and the corresponding congruent mapping true staining image.

[0147] Therefore, the first loss function can also be expressed as: L G (G A→B ) = L adv (G A→B ) + γL cycle + λL idt + pL SS , the second loss function can also be expressed as: L G (G B→A ) = L adv (G B→A ) + γL cycle + λL idt + pL SS , where λ = 1.

[0148] After the loss function of the cyclic generative adversarial network converges, in step S330, a target generative network model can be obtained based on the generative network in the trained cyclic generative adversarial network with the converged value of the loss function, where the target generative network model can generate a target virtual staining image based on the target Raman image of the target cell sample, and the target virtual staining image is used to enhance the display of the cell structure in the target cell sample.

[0149] Specifically, according to an embodiment of the present invention, in response to the values of the first loss function and the second loss function converging respectively, the parameters of the first generative network in the trained cyclic generative adversarial network with the converged value of the loss function can be used to obtain the target generative network model.

[0150] In one example, the target cell sample can be a cell sample to be stained rather than a cell sample for training. That is, only the Raman image is obtained for the target cell sample without obtaining the true staining image.

[0151] In another example, it is also possible to obtain both the Raman image and the true staining image of the target cell sample for the target cell sample, for testing whether the virtual staining image generated by the target generation network model is consistent with the true staining image, that is, whether it meets the requirements.

[0152] In one example, the target generation network model can be a separate model or a part of other models. For example, other models can also include an input-output part so that the model can directly receive the Raman image of the target cell sample and output the virtual staining image of the target cell sample.

[0153] Refer again to Figure 2 . As Figure 2 shown, Figure 2 the second virtual staining image corresponding to the cell sample in the fourth row, and the second virtual staining image is generated by the target generation network model obtained through the embodiments of the present invention. Through Figure 2 it can be seen that the color on the left side of the cell pointed by the arrow in the second virtual staining image of the fourth row is light (corresponding to the cytoplasm), and the color on the right side is dark (corresponding to the nucleus), which is consistent with the true staining image in the second row. Its staining effect is highly close to the true staining image, and the staining accuracy of the cell structure is high, having higher accuracy compared with the existing virtual staining methods.

[0154] The above has described in detail the training method for obtaining the target generation network model in combination with Figures 2 - 6 . From the content described in detail above, the present invention can obtain a Raman image that can more accurately describe the cell structure of the cell sample by using Raman imaging technology; at the same time, during the training process using the Raman image and the true staining image of the cell sample, the loss function in the cycle generative adversarial network is constrained for cell structure consistency through a structural similarity function, so that the target generation network model obtained based on the trained cycle generative adversarial network can generate a virtual staining image that is more consistent with the true staining image, avoiding problems such as reversing the staining of the nucleus and cytoplasm during the virtual staining process, improving the accuracy of cell virtual staining, and achieving enhanced display of the cell structure.

[0155] In this way, by using the target generation network model obtained by this training method to perform virtual staining on the target cell sample, a virtual staining image with accurate enhanced display of the structure of the target cell sample can be obtained, realizing the extension of virtual staining imaging from the tissue level to the cell level, making the virtual staining image have significant consistency with the true staining image, so that clinicians or researchers can analyze the cell structure of the cell sample based on the virtual staining image to make accurate judgments and decisions.

[0156] Second Embodiment

[0157] Based on the first embodiment, in order to obtain the Raman image of the cell sample, according to an embodiment of the present invention, multi-color imaging can be used, that is, multiple sets of laser wavelength parameters are used to respectively excite different molecular vibration modes in the sample, and then the concentrations of different chemical components at each point of the image are obtained through calculation.

[0158] For example, stimulated Raman histology (SRH) obtains 2 images for each field of view by adjusting the pump light wavelength, and is transformed into a result close to the H&E stained image through a specific algorithm. In addition, there is also the Two-color method for detecting the distribution of proteins and lipids in the sample (obtaining 2 images for each field of view), or the Three-color method for detecting the distribution of proteins, lipids and DNA (obtaining 3 images for each field of view).

[0159] In one example, the cell sample can be irradiated with a first excitation light and a second excitation light; in response to the irradiation of the cell sample with the first excitation light and the second excitation light, reflected light having at least one Raman shift characteristic peak can be received from the cell sample to obtain the Raman image of the cell sample. It should be noted that the first and second in the above first excitation light and second excitation light do not represent any order, quantity or importance, but are only used to distinguish different components.

[0160] In one example, coherent Raman imaging uses two lasers with different frequencies (pump light and Stokes light) for imaging, mainly including stimulated Raman scattering (SRS), stimulated Raman photothermal (SRP) and coherent anti-Stokes Raman scattering (CARS). According to an embodiment of the present invention, the first excitation light may include pump light, and the second excitation light may include Stokes light. Conversely, the first excitation light may also be Stokes light, and the second excitation light may be pump light. In response to the irradiation of the cell sample with the pump light and the Stokes light, reflected light having at least one Raman shift characteristic peak can be received from the cell sample to obtain the coherent Raman scattering imaging image of the cell sample. The coherent Raman scattering imaging image may include a stimulated Raman scattering imaging image, a stimulated Raman photothermal imaging image or a coherent anti-Stokes Raman scattering imaging image.

[0161] According to an embodiment of the present invention, the first excitation light and the second excitation light may be pulsed lasers, and the frequency difference between the first excitation light and the second excitation light may be 2800 - 3100 cm -1 , and the repetition frequency may be greater than 50 MHz, and the pulse width may be 100 fs - 20 ps.

[0162] In one example, other types of coherent Raman scattering microscopy imaging methods can also be used to image cell samples, such as stimulated Raman scattering (SRS) microscopy imaging. The specific implementation method of Raman imaging is not limited herein. In one example, in addition to coherent Raman scattering microscopy imaging, surface-enhanced Raman scattering imaging, single-walled carbon nanotube Raman imaging, etc. can alternatively be used.

[0163] In one example, a coherent Raman scattering microscopy device can be used to perform coherent Raman scattering microscopy imaging on a target cell sample, and a computer device acquires the coherent Raman image.

[0164] In the case where coherent Raman scattering imaging uses multiple Raman shift characteristic peaks to image a sample respectively, the laser used must be tunable. However, lasers with tuning functions are usually very costly and unstable. Therefore, according to another embodiment of the present invention, a laser with a predetermined frequency can be used, that is, a laser that does not require a tuning function. By such a method, the cost of the imaging device can be reduced and the stability of the imaging device can be improved. In addition, by using a laser with a predetermined frequency, the laser can be without a tuning function, thereby miniaturizing the imaging device. Therefore, by setting the excitation light to a predetermined frequency, the imaging device can be made more suitable for clinical applications.

[0165] In one example, the first excitation light and the second excitation light can each have a predetermined frequency. In response to the first excitation light and the second excitation light with a predetermined frequency irradiating the cell sample, reflected light with only one Raman shift characteristic peak can be received from the cell sample to obtain a Raman image of the cell sample.

[0166] Based on the content disclosed in the above second embodiment, the present invention can use a laser with a predetermined frequency, so that Raman imaging of a cell sample can be performed using a single characteristic peak. In this way, the laser can be without a tuning function. On the one hand, the imaging device can be miniaturized, and on the other hand, the cost can be reduced and the stability can be improved, so that the imaging device used in the present invention is more suitable for clinical applications.

[0167] Third Embodiment

[0168] Figure 7 Shows a schematic diagram of stitching and registration processing of a Raman image and a true staining image according to some embodiments of the present invention.

[0169] Such as Figure 7As shown, after obtaining a cell sample, Raman imaging technology can be used to image the cell sample to obtain a Raman image of the cell sample. Then, the cell sample is chemically stained to obtain a true stained image of the cell sample.

[0170] According to an embodiment of the present invention, in the process of obtaining the Raman image and the true stained image of the cell sample, in order to obtain a Raman image with higher resolution and / or reduce the difficulty of imaging, Raman scattering microscopy imaging can be performed on each part of the cell sample under a local field of view to obtain a local Raman image of the cell sample; then, the local Raman images of each part of the cell sample are stitched together to generate a Raman image of the cell sample.

[0171] Similarly, the cell sample with the generated Raman image can be chemically stained, and bright-field microscopy imaging is performed on each part of the chemically stained cell sample under a local field of view to obtain a local true stained image of the cell sample, and then the local true stained images of each part of the cell sample are stitched together to generate a true stained image of the cell sample.

[0172] In this way, clearer imaging at the cell structure level can be achieved, so as to accurately perform virtual staining based on the subsequent imaging-based images, and the imaging difficulty caused by the performance of the imaging device (for example, the imaging device has low precision and cannot capture a clear whole image at one time) can also be reduced.

[0173] In addition, according to an embodiment of the present invention, after training the cycle generative adversarial network to make the loss function converge or during the training process, the Raman image registered at the pixel level and the corresponding true stained image can be used to verify the cycle generative adversarial network to ensure that the virtual stained image obtained through the generated target generative network model is accurate.

[0174] Specifically, by using the registered Raman image and the corresponding true stained image for verification, it can be ensured that the loss function will not overfit. After verifying that the loss function has not overfitted and has converged using the registered Raman image and the corresponding true stained image, the training of the cycle generative adversarial network can be ended and the target generative network model can be generated based on this cycle generative adversarial network.

[0175] In one example, the true stained image of the cell sample can be used as a reference to register each cell in the Raman image of the cell sample to obtain two types of images with a complete field of view and paired, so that the same cell in the registered Raman image of the cell sample and the registered true stained image of the cell sample is in the same position.

[0176] According to some embodiments of the present invention, image registration can be performed on the Raman image and the corresponding real staining image so that the Raman image and the real staining image are aligned at the pixel level; the registered Raman image and the corresponding registered real staining image are input into a trained cyclic generative adversarial network with a converged loss function value to verify the trained cyclic generative adversarial network with a converged loss function value.

[0177] According to some embodiments of the present invention, verifying the trained cyclic generative adversarial network with a converged loss function value can be specifically implemented in the following manner: determining whether the loss function value of the generative network in the trained cyclic generative adversarial network with a converged loss function value is overfitting; when it is determined that the loss function value of the generative network in the trained cyclic generative adversarial network with a converged loss function value is not overfitting, obtaining a target generative network model based on the generative network in the trained cyclic generative adversarial network with a converged loss function value; and when it is determined that the loss function value of the generative network in the trained cyclic generative adversarial network with a converged loss function value is overfitting, adjusting the parameters of the trained cyclic generative adversarial network with a converged loss function value and training the cyclic generative adversarial network with the adjusted parameters based on the Raman image and the real staining image.

[0178] In one example, after obtaining the unpaired and paired datasets, noise reduction and contrast enhancement processing can also be performed on the Raman image, and preprocessing such as stain normalization can be performed on the real staining image to improve the image quality.

[0179] In addition, the registered Raman image and the real staining image can also be used to verify the beneficial effects of the training method of the present invention. The beneficial effects of the training method implemented by the present invention will be further described below through two sets of experimental data.

[0180] (1) Evaluate the target generative network model on the test set. The test set contains approximately 600 monochromatic Raman images of registered cell samples and real staining images of cell samples each, which are obtained by randomly cropping 5 samples out of 32 samples in the dataset, and each sample contains approximately 200 cells on average.

[0181] In the test results, the SSIM between the virtual staining image and the real staining image reaches 0.881±0.015, and the Dice coefficient of the nucleus region reaches 0.804±0.137, indicating that the target generative network can convert the unlabeled monochromatic Raman image into a virtual staining image and achieve accurate mapping of cell structures.

[0182] (2) The target generation network is evaluated by a pathologist. The dataset contains a total of 32 samples from 7 positive patients and 4 negative patients. Exemplarily, the way for the pathologist to evaluate the target generation network model can refer to Figure 8 the evaluation schematic diagram of the target generation network model shown.

[0183] When counting by cell, the pathologist can distinguish normal cells and cancer cells in the virtual stained image with high accuracy. The accuracy rate is 99.4%, the specificity is 99.5%, the sensitivity is 99.7%, and the Cohen's Kappa coefficient for cell classification under the real stained image is 0.782 ± 0.284. This indicates that there is a significant consistency in the cell classification by the doctor through the virtual stained image and the real stained image. When counting by sample, the accuracy rate is 93.8%, the sensitivity is 100%, and the specificity is 90.9%.

[0184] Based on the content disclosed in the above third embodiment, the Raman image of the cell sample and the corresponding real stained image can be divided into a training set, a validation set, and a test set.

[0185] Among them, for the training set, by stitching the images, clearer imaging of the cell structure level can be achieved, so as to accurately perform virtual staining on the subsequent imaging-based images, and it can also reduce the imaging difficulty caused by the performance of the imaging device (for example, the imaging device has low precision and cannot capture a clear whole image at one time).

[0186] For the validation set, by using the Raman image registered at the pixel level and the corresponding real stained image to verify the cycle generative adversarial network, it can be ensured that the loss function will not overfit, so that the generated target generation network model can generate a virtual stained image consistent with the real stained image, ensuring accurate enhanced display of the cell structure.

[0187] For the test set, testing with the registered Raman image and the corresponding real stained image shows that the target generation network model obtained by the training method based on the present invention can accurately enhance the display of the virtual stained image of the target cell sample structure, realizing the expansion of virtual staining imaging from the tissue level to the cell level, making the virtual stained image and the real stained image have significant consistency, so that clinicians or researchers can analyze the cell structure of the target cell sample based on the virtual stained image to make accurate judgments and decisions.

[0188] Fourth Embodiment

[0189] The virtual staining method for enhancing the display of cell structure provided by the above-mentioned present invention will be described in detail below with reference to the accompanying drawings.

[0190] Figure 9 The flowchart of a virtual staining method for enhancing the display of cell structures according to some embodiments of the present invention is shown. As Figure 9 shown, the virtual staining method for enhancing the display of cell structures of the present invention may first obtain a target Raman image of a target cell sample in step S910, and then in step S920, the target Raman image may be input into a target generation network model generated according to the foregoing training method to obtain a target virtual staining image of the target cell sample, and the target virtual staining image is used to enhance the display of cell structures in the target cell sample.

[0191] According to an embodiment of the present invention, Raman scattering microscopy imaging may be performed on the target cell sample to obtain a target Raman image of the target cell sample. Specifically, according to an embodiment of the present invention, a first excitation light and a second excitation light may be used to irradiate the target cell sample; in response to the first excitation light and the second excitation light irradiating the target cell sample, reflected light having at least one Raman shift characteristic peak may be received from the target cell sample to obtain a target Raman image of the target cell sample.

[0192] According to another embodiment of the present invention, the first excitation light and the second excitation light may each have a predetermined frequency. In this case, in response to the first excitation light and the second excitation light having a predetermined frequency irradiating the target cell sample, reflected light having only one Raman shift characteristic peak may be received from the target cell sample to obtain a target Raman image of the target cell sample.

[0193] In one example, the first excitation light and the second excitation light may be pulsed lasers, and the frequency difference between the first excitation light and the second excitation light is 2800 - 3100 cm -1 , the repetition frequency is greater than 50 MHz, and the pulse width is 100 fs - 20 ps.

[0194] According to still another embodiment of the present invention, the first excitation light may include pump light, and the second excitation light may include Stokes light. In this case, in response to the pump light and the Stokes light irradiating the target cell sample, reflected light having at least one Raman shift characteristic peak may be received from the target cell sample to obtain a coherent Raman scattering imaging image of the cell sample, and the coherent Raman scattering imaging image includes a stimulated Raman scattering imaging image, a stimulated Raman photothermal imaging image, or a coherent anti-Stokes Raman scattering imaging image.

[0195] According to an embodiment of the present invention, Raman scattering microscopy imaging may also be performed on each local part of the target cell sample under a local field of view to obtain a local target Raman image of the target cell sample; then, the local target Raman images of each local part of the target cell sample may be stitched together to obtain a target Raman image of the target cell sample.

[0196] For some specific details of the virtual staining method for enhancing the display of cell structures disclosed in the fourth embodiment of the present invention, reference may also be made to the training methods for obtaining the target generation network model described in the first to third embodiments. Therefore, the same content will not be elaborated here.

[0197] Based on the content disclosed in the fourth embodiment of the present invention, the virtual staining method of the present invention can be used to perform virtual staining on a target cell sample to obtain a virtual staining image that accurately enhances the display of the structure of the target cell sample, realizing the extension of virtual staining imaging from the tissue level to the cell level, making the virtual staining image have significant consistency with the real staining image, so that clinicians or researchers can analyze the cell structure of the target cell sample based on the virtual staining image to make accurate judgments and decisions.

[0198] In one example, a computer device can be used to execute the training method and the virtual staining method provided in the embodiments of the present invention. The computer device can be a terminal or a server. Among them, the terminal can include, but is not limited to, various personal computers, laptop computers, smart phones, tablet computers, Internet of Things devices, portable wearable devices, and medical electronic devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc.; the portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc.; the server can be implemented by an independent server or a server cluster composed of multiple servers.

[0199] Fifth Embodiment

[0200] The virtual staining system provided by the present invention will be described in detail below with reference to the accompanying drawings.

[0201] Figure 10 A block diagram of a virtual staining system according to some embodiments of the present invention is shown. As Figure 10 shown, the virtual staining system 1000 may include an image acquisition component 1010 and an image processing component 1020. Among them, the image acquisition component 1010 may be configured to obtain a target Raman image of a target cell sample; and the image processing component 1020 may be configured to input the target Raman image into a target generation network model generated according to the foregoing training method to obtain a target virtual staining image of the target cell sample, and the target virtual staining image is used to enhance the display of the cell structure in the target cell sample.

[0202] According to an embodiment of the present invention, the image acquisition component 1010 may be configured to perform Raman scattering microscopy imaging on a target cell sample to obtain a target Raman image of the target cell sample.

[0203] According to an embodiment of the present invention, the virtual staining system 1000 may further include:

[0204] A laser source that can be configured to generate a first excitation light and a second excitation light for irradiating a target cell sample; and

[0205] The image acquisition component 1010 can also be configured to receive, in response to the first excitation light and the second excitation light irradiating the target cell sample, reflected light having at least one Raman shift characteristic peak from the target cell sample, so as to obtain a target Raman image of the target cell sample.

[0206] According to an embodiment of the present invention, the first excitation light and the second excitation light can each have a predetermined frequency, and the image acquisition component 1010 can also be configured to:

[0207] In response to the first excitation light and the second excitation light having a predetermined frequency irradiating the target cell sample, receive reflected light having only one Raman shift characteristic peak from the target cell sample, so as to obtain a target Raman image of the target cell sample.

[0208] According to an embodiment of the present invention, the virtual staining system 1000 may further include

[0209] An optical path component, the optical path component includes a two-dimensional galvanometer assembly and a first filter configured to guide the first excitation light and the second excitation light to the target cell sample;

[0210] A sample carrier component configured to carry the target cell sample to receive the irradiation of the first excitation light and the second excitation light; and

[0211] An objective lens component configured to receive the reflected light from the target cell sample and transmit the reflected light to the image acquisition component 1010.

[0212] Figure 11 Shows an exemplary structural diagram of a virtual staining system 1000 according to some embodiments of the present invention. As Figure 11 shown, the virtual staining system 1000 includes a laser emission device 1110, a sample carrier component 1120, an image acquisition component 1010, and an image processing component 1020.

[0213] The laser emission device 1110 includes a laser source 1111, a two-dimensional galvanometer assembly 1112 and a first filter 1113 on the optical path component, and an objective lens component 1114.

[0214] The laser source 1111 is used to generate the first excitation light and the second excitation light, and the two are output collinearly.

[0215] This design simplifies the optical path structure between the laser source 1111 and the objective lens component 1114, avoiding the need to split and adjust the wavelength of the single-wavelength excitation beam output by the laser source 1111, thereby improving the compactness of the device, reducing the volume, and facilitating commercial development. The laser source 1111 can be configured to output a first excitation light and a second excitation light with fixed wavelengths. In some embodiments, as described above, the laser source 1111 can also be a tunable laser source, such that the wavelength of the first excitation light (and / or the second excitation light) can be selected within a certain range, while the wavelength of the second excitation light (and / or the first excitation light) is fixed. In the case where the laser source 1111 is a tunable laser source, the laser source 1111 can integrate a control circuit for controlling the laser source 1111 to output a specific form of laser, or the laser source 1111 can communicate with the image processing component 1020 or other computing control devices through a cable 1141 and be controlled by the image processing component 1020 to output a laser with a specific wavelength.

[0216] According to an embodiment of the present invention, Figure 12 FIG. shows a schematic diagram of a microscopic imaging system for Raman images according to some embodiments of the present invention. As Figure 12 shown, where EOM is an electro-optic modulator, DM is a dichroic mirror, M is a silver mirror, OBJ is an objective lens, CON is a condenser, FL is a filter, and PD is a photodiode.

[0217] The repetition frequency of the laser (picoEmeraldTM S, Applied Physics & Electronics) can be 80 MHz, the pulse width can be 2 ps, and the wavelength of the emitted pump light can be tuned within the range of 700 - 960 nm. In one example, the wavelength of the pump light is 796.8 nm during imaging, corresponding to the Raman shift characteristic peak 2850 cm -1, the wavelength of the Stokes light is fixed at 1031 nm. The two lights overlap both spatially and temporally, and the Stokes light is modulated by an electro-optic modulator to about 20 MHz. The collinear pump and Stokes lights are coupled to a two-dimensional galvanometer scanning mirror (GVS012-2D, Thorlab), and then input into an inverted microscope (IX73, Olympus). The two lights are focused on the sample by a 60X water immersion objective lens (LUMPlanFL N, numerical aperture 1.0, Olympus), which triggers the resonance of specific molecules in the sample, generates stimulated Raman loss and gain, and then is collected by another water immersion objective lens of the same model. The Stokes light therein is removed by a low-pass filter (ET980SP, Chroma), and then received by a silicon photodiode (S3994-01, Hamamatsu) with a size of 10 mm × 10 mm and a 48 V DC reverse bias voltage. After the signal is extracted by a lock-in amplifier (HF2LI, Zurich Instruments), the analog output representing the SRS signal enters a data acquisition card (PCIE-6363, National Instruments) and is input into a computer to display the SRS image of the sample on the LabVIEW 2018 software.

[0218] According to an embodiment of the present invention, Figure 13 shows the spontaneous Raman scattering spectra of pure lipid samples (taking triolein (TO) as an example) and pure protein samples (taking bovine serum albumin (BSA) as an example) in the carbon-hydrogen bond vibration region (2800 - 3100 cm-1) according to some embodiments of the present invention, that is, the curve graph of the spontaneous Raman scattering intensity at different frequency differences between the pump light and the Stokes light. As Figure 13 shown, lipids and proteins can generate strong Raman signals in this interval. Therefore, the Raman images obtained in this interval can reflect the concentration differences of chemical substances in different structures of the cells contained in the target cell sample. It should be noted that the stimulated Raman scattering spectrum is relatively close to the spontaneous Raman scattering spectrum, but the coherent anti-Stokes Raman scattering spectrum is quite different from the spontaneous Raman scattering spectrum. The correct Raman spectrum should be selected as a reference according to the specific coherent Raman scattering microscopy method.

[0219] In some embodiments, the laser source 1111 can be a laser source with a fixed output wavelength, so as to obtain the coherent Raman image of the sample under a single Raman shift characteristic peak. For example, in some embodiments, the 2850 cm corresponding to the 796.8 nm pump light and the 1031 nm Stokes light under stimulated Raman scattering microscopy -1 in the Raman shift channel, as Figure 11As shown, the Raman signal of lipids is strong and can be used to reflect the spatial distribution of lipids in the target cell sample. In some embodiments, the 2850 cm -1 Raman shift channel corresponding to the 780 nm pump light and 1003 nm Stokes light under coherent anti-Stokes Raman scattering microscopy can also reflect the lipid component characteristics in the target cell sample 1180.

[0220] In some embodiments, the laser source 1111 can also be a tunable laser source, so as to obtain the sample coherent Raman images under multiple Raman shift characteristic peaks. For example, in some embodiments, the 2850 cm -1 and 2930 cm -1 Raman shift channels corresponding to the 796.8 nm and 791.8 nm pump lights and 1031 nm Stokes light respectively under stimulated Raman scattering microscopy, as Figure 11 shown, in the latter, the Raman signal of proteins is strong, and in addition to reflecting the lipid component characteristics in the target cell sample, it can also reflect the protein component characteristics in the target cell sample.

[0221] Of course, the specific wavelength of the laser output by the laser source 1111 is not limited to the above, and the laser source 1111 can be configured to output lasers within other wavelength ranges, which depends on the specific components to be detected in the target cell sample. It should be understood that the components to be detected are not limited to lipids or proteins discussed above.

[0222] In some embodiments, the control circuit integrated in the image processing component 1020 or the laser source 1111 can be further configured to control the laser source 1111 to output pulsed laser. According to an embodiment of the present invention, the first excitation light and the second excitation light can be pulsed lasers, the frequency difference between the first excitation light and the second excitation light is 2800 - 3100 cm -1 , the repetition frequency is greater than 50 MHz, and the pulse width is 100 fs - 20 ps. According to an embodiment of the present invention, the first excitation light can include pump light, the second excitation light can include Stokes light, and the image acquisition component 1010 is also configured to receive the reflected light with at least one Raman shift characteristic peak from the cell sample in response to the pump light and Stokes light irradiating the cell sample, so as to obtain the coherent Raman scattering imaging image of the cell sample, and the coherent Raman scattering imaging image includes stimulated Raman scattering imaging image, stimulated Raman photothermal imaging image or coherent anti-Stokes Raman scattering imaging image. The frequency and pulse width of the pulsed laser are not limited to this, and the laser source 1111 can be controlled to output pulsed lasers with other frequencies and pulse widths.

[0223] In some embodiments, the sample carrier component 1120 can be a mechanical displacement stage for carrying the target cell sample 1180.

[0224] In some embodiments, the image acquisition component 1010 may be a photomultiplier tube or a photodiode.

[0225] The first excitation light and the second excitation light are incident on the two-dimensional galvanometer assembly 1112, and the two-dimensional galvanometer assembly 1112 adjusts the optical paths of the first excitation light and the second excitation light. The first excitation light and the second excitation light leaving the two-dimensional galvanometer assembly 1112 sequentially pass through the first filter 1113 and the objective lens component 1114. The first excitation light and the second excitation light transmit through the first filter 1113, and the objective lens component 14 focuses the first excitation light and the second excitation light onto the sample carrier component 1120; the sample on the sample carrier device 1120 generates signal light under the action of the first excitation light and the second excitation light. After the signal light passes through the objective lens 1114, the first filter 1113 reflects the signal light to the image acquisition component 1010. The image acquisition component 1010 generates a Raman image according to the signal light and outputs the Raman image to the image processing component 1020 through the cable 1141, and then displays it on the display 1142 through the cable 1141.

[0226] According to an embodiment of the present invention, the virtual system may further include an autofocus component. Among them, the autofocus component may include a focus detection unit, a second filter, and a moving component for moving the objective lens component; the focus detection unit may be configured to: generate a third excitation light, and the third excitation light is irradiated on the sample carrier component through the second filter; detect the detection reflected light reflected back to the focus detection unit by the sample carrier component; and control the moving component according to the detection result so that the objective lens component receives the reflected light from the target cell sample.

[0227] Figure 14 Another exemplary structural diagram of the virtual staining system 1000 according to some embodiments of the present invention is shown. Specifically, as Figure 14As shown, in order to accurately focus the first excitation light and the second excitation light onto the target cell sample 1180 at the sample carrier member 1120 through the objective lens 1114, the laser emission device 1110 further includes an autofocus mechanism. The autofocus mechanism includes a focus detection unit 1422, a second filter 1421, and a moving component. The second filter 1421 is correspondingly arranged with the first filter 1113 and the objective lens component 1114 respectively. After passing through the first filter 1113, the first excitation light and the second excitation light are reflected by the second filter 1421 to the objective lens 1114. The signal light passes through the objective lens 1114 and then is reflected by the second filter 1421 to the first filter 1113. After the third excitation light generated by the focus detection unit 1422 passes through the second filter 1421, it is parallel or collinear with the first excitation light and the second excitation light respectively, and is focused onto the sample carrier member 1120 through the objective lens 1114, generating reflected light on the sample carrier member 1120. The reflected light returns to the focus detection unit 1422 along the original path of the third excitation light, and the focus detection unit 1422 detects the reflected light. The objective lens 1114 is installed on the moving component, and the moving component moves the objective lens 1114 according to the detection result of the focus detection unit 1422 to adjust the distance between the objective lens 1114 and the sample carrier member 1120.

[0228] In an example of the present invention, the functions of the image processing component 1020 may include: in the model training stage, processing the Raman image and the real staining image to obtain the data set of the model and training the model to obtain the target generation network model. The implementation methods of each sub-function required to implement this function have been described above and will not be elaborated here;

[0229] In the model prediction stage after the model training is completed, the Raman image of the cell sample is input into the target generation network model, and a virtual staining image output by the target generation network model is obtained. This virtual staining image can be used to enhance the display of the cell structure in the target cell sample.

[0230] The functional modules in the image processing component 1020 can be embedded in the processor in the computer device in hardware form or independent of it, or stored in the memory in the computer device in software form, so that the processor can call and execute the operations corresponding to each above module.

[0231] According to an embodiment of the present invention, the image acquisition component 1010 can also be configured to perform Raman scattering microscopy imaging on each part of the target cell sample under a local field of view to obtain the local target Raman image of each part of the target cell sample; stitching the local target Raman images of each part of the target cell sample to obtain the target Raman image of the target cell sample.

[0232] Some specific details of the virtual staining system 1000 described in the fifth embodiment can also refer to the content disclosed in the first to fourth embodiments. Therefore, the same content will not be repeated here.

[0233] Based on the content disclosed in the fifth embodiment of the present invention, the virtual staining system of the present invention can be used to perform virtual staining on a target cell sample to obtain a virtual staining image that accurately enhances the display of the structure of the target cell sample, realizing the extension of virtual staining imaging from the tissue level to the cell level, making the virtual staining image have significant consistency with the real staining image, so that clinicians or researchers can analyze the cell structure of the target cell sample based on the virtual staining image to make accurate judgments and decisions.

[0234] Figure 15 The structural diagram of an electronic device 1500 according to some embodiments of the present invention is shown.

[0235] See Figure 15 , the electronic device 1500 may include a processor 1501 and a memory 1502. The processor 1501 and the memory 1502 can both be connected through a bus 1503. The electronic device 1500 can be any type of portable device (such as a smart camera, a smartphone, a tablet computer, etc.) or any type of fixed device (such as a desktop computer, a server, etc.).

[0236] The processor 1501 can perform various actions and processes according to the programs stored in the memory 1502. Specifically, the processor 1501 can be an integrated circuit chip with signal processing capabilities. The above-mentioned processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. It can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc., and can be of the X86 architecture or the ARM architecture.

[0237] The memory 1502 stores computer-executable instructions that, when executed by the processor 1501, implement the above-described method for obtaining a target generation network model and / or the method for enhancing virtual staining of cell structures. The memory 1502 can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. The non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which serves as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus random access memory (DR RAM). It should be noted that the memories of the methods described herein are intended to include but are not limited to these and any other suitable types of memories.

[0238] In addition, the method for determining the matchability of a video according to the present invention can be recorded on a computer-readable recording medium. Specifically, according to the present invention, there can be provided a computer-readable recording medium storing computer-executable instructions that, when executed by a processor, can cause the processor to execute the method for determining the matchability of a video as described above.

[0239] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present invention are all information and data that have been authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data need to comply with the relevant laws, regulations, and standards of relevant countries and regions.

[0240] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, a program segment, or a part of code that contains at least one executable instruction for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than that marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, as well as combinations of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system for performing the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0241] In general, the various exemplary embodiments of the present invention can be implemented in hardware or dedicated circuits, software, firmware, logic, or any combination thereof. Some aspects can be implemented in hardware, while other aspects can be implemented in firmware or software that can be executed by a controller, a microprocessor, or other computing devices. When aspects of the embodiments of the present invention are illustrated or described as block diagrams, flowcharts, or using some other graphical representation, it will be understood that the blocks, devices, systems, techniques, or methods described herein can be implemented as non-limiting examples in hardware, software, firmware, dedicated circuits or logic, general hardware or a controller or other computing devices, or some combination thereof.

[0242] Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. It should also be understood that terms such as those defined in a commonly used dictionary should be interpreted as having a meaning consistent with their meaning in the context of the relevant art, and should not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0243] The above is an explanation of the present invention and should not be considered a limitation thereof. Although several exemplary embodiments of the present invention have been described, those skilled in the art will readily understand that many modifications can be made to the exemplary embodiments without departing from the novel teachings and advantages of the present invention. Therefore, all such modifications are intended to be included within the scope of the present invention as defined by the claims. It should be understood that the above is an explanation of the present invention and should not be considered limited to the specific embodiments disclosed, and modifications to the disclosed embodiments and other embodiments are intended to be included within the scope of the appended claims. The present invention is defined by the claims and their equivalents.

Claims

1. A training method for obtaining a target generation network model, comprising: Obtain a Raman image of a cell sample and a corresponding true staining image of the cell sample, wherein the Raman image is based on a frequency difference of 2800-3100 cm -1 obtained by irradiating with a first excitation light and a second excitation light; Training a cyclic generative adversarial network based on the Raman image and the true stained image so that a value of a loss function of a generating network in the cyclic generative adversarial network converges; as well as Obtaining a target generation network model based on the generation network in the trained cyclic generative adversarial network with a converged loss function value, wherein the target generation network model generates a target virtual staining image based on a target Raman image of a target cell sample, and the target virtual staining image is used to enhance the display of a cell structure in the target cell sample; The generation network includes a first generation network and a second generation network, the loss function includes a first loss function for the first generation network and a second loss function for the second generation network, the first loss function and the second loss function include a cell structure similarity constraint term and an adversarial loss constraint term, and the cell structure similarity constraint term constrains the consistency of the cell structure in the input image and the output image of the generation network based on the structural similarity SSIM function, Wherein, training a recurrent generative adversarial network based on the Raman image and the real stained image so that the value of the loss function of the generative network in the recurrent generative adversarial network converges includes: Inputting the Raman image into the first generation network to generate an intermediate virtual staining image, Inputting the intermediate virtual dyed image into the first discriminant network in the recurrent generative adversarial network to determine the probability that the intermediate virtual dyed image is judged as the real dyed image, and Inputting the true dyeing image into the second generation network to generate an intermediate Raman image, Inputting the intermediate Raman image into a second discriminant network in the recurrent generative adversarial network to determine the probability that the intermediate Raman image is judged to be the Raman image, adjusting the parameters of the cyclic generative adversarial network based on the probabilities determined by the first discriminant network and the second discriminant network so that the values ​​of the first loss function and the second loss function converge respectively, Among them, the cell structure similarity constraint term of the first loss function includes The cell structure similarity constraint term of the second loss function includes Wherein, a is a Raman image in the Raman image domain A consisting of one or more Raman images, G A→B (a) is to input a into the first generation network G A→B The generated intermediate virtual stained image, SSIM (G A→B (a), a) is the structural similarity value between the intermediate virtual staining image and the Raman image, is the expected value of the structural similarity loss value corresponding to each Raman image in the Raman image domain A; and b is a real stained image in the real stained image domain B consisting of one or more real stained images, G B→A (b) is to input b into the second generation network G B→A The generated intermediate Raman image, SSIM (G B→A (b), b) is the structural similarity value between the intermediate Raman image and the real dyeing image. is the expected value of the structural similarity loss value corresponding to each real stained image in the real stained image domain B, Among them, the adversarial loss constraint term L of the first loss function adv (G A→B )equal D B is the first discriminant network, D B (G A→B (a)) is the probability that the intermediate virtual stained image is judged as the real stained image, is the expected value of the adversarial loss value corresponding to each Raman image in the Raman image domain A, Among them, the adversarial loss constraint term L of the second loss function adv (G B→A )equal D A is the second discriminant network, D A (G B→A (b)) is the probability that the intermediate Raman image is judged to be the Raman image, is the expected value of the adversarial loss value corresponding to each real stained image in the real stained image domain B.

2. The training method according to claim 1, wherein: Adjusting the parameters of the cyclic generative adversarial network based on the probabilities determined by the first discriminant network and the second discriminant network so that the values ​​of the first loss function and the second loss function converge respectively includes: Determining values ​​of the first loss function and the second loss function based on the probabilities determined by the first discriminant network and the second discriminant network; The parameters of the cyclic generative adversarial network are adjusted based on the values ​​of the first loss function and the second loss function so that the values ​​of the first loss function and the second loss function converge respectively.

3. The training method according to claim 2, wherein: The expression of the first loss function is: L G (G A→B )=L adv (G A→B )+γL cycle +pL SSIM1 L G (G A→B ) is the first loss function, L adv (G A→B ) is the adversarial loss constraint term of the first loss function, L cycle is the cycle consistency loss constraint term of the first loss function, L SSIM1 is the cell structure similarity constraint term of the first loss function; in, G B→A (G A→B (a)) is to input the intermediate virtual stained image into the second generation network G B→A The reconstructed Raman image generated is is the expected value of the mean absolute error between each Raman image in the Raman image domain A and the corresponding reconstructed Raman image, G A→B (G B→A (a)) is to input the intermediate Raman image into the first generation network G A→B The generated reconstructed true stained image, is the expected value of the mean absolute error between each real stained image in the real stained image domain B and the corresponding reconstructed real stained image; Among them, γ=10, p=2.

4. The training method according to claim 3, wherein: The expression of the second loss function is: L G (G B→A )=L adv (G B→A )+γL cycle +pL SSIM2 Among them, L G (G B→A ) is the second loss function, L adv (G B→A ) is the adversarial loss constraint term of the second loss function, L cycle is the cycle consistency loss constraint term of the second loss function, L SSIM2 The cell structure similarity constraint term of the second generated network is defined.

5. The training method according to claim 4, wherein: The loss function also includes a congruent mapping loss constraint term, and the expression of the congruent mapping loss constraint term is: Among them, G B→A (a)) is to input a into the second generation network G B→A The generated congruent mapping Raman image, is the expected value of the mean absolute error between each Raman image in the Raman image domain A and the corresponding congruent mapping Raman image, G A→B (b)) is to input b into the first generation network G A→B The generated congruent mapping true stained image, is the expected value of the mean absolute error between each real stained image in the real stained image domain B and the corresponding congruent mapped real stained image; Among them, the first loss function is: L G (G A→B )=L adv (G A→B )+γL cycle +λL idt +pL SSIM1 , The second loss function is: L G (G B→A )=L adv (G B→A )+γL cycle +λL idt +pL SSIM2 , Among them, λ=1.

6. The training method according to claim 1, wherein: Training a cyclic generative adversarial network based on the Raman image and the real stained image so that the value of the loss function of the generative network in the cyclic generative adversarial network converges includes: The cyclic generative adversarial network is iteratively trained based on multiple Raman images and multiple corresponding true stained images until the value of the loss function of the generative network in the cyclic generative adversarial network converges.

7. The training method according to claim 1, wherein: Obtaining a target generative network model based on the generative network in the trained cyclic generative adversarial network with a converged loss function value includes: In response to the values ​​of the first loss function and the second loss function each converging, the target generative network model is obtained using the parameters of the first generative network in the trained cyclic generative adversarial network having the converged loss function values.

8. The training method according to any one of claims 1 to 7, further comprising: Performing image registration on the Raman image and the corresponding true staining image so that the Raman image is aligned with the true staining image at a pixel level; The registered Raman image and the corresponding registered true stained image are input into the trained recurrent generative adversarial network with a converged loss function value to verify the trained recurrent generative adversarial network with a converged loss function value.

9. The training method according to claim 8, wherein: Verifying the trained recurrent generative adversarial network having a converged value of the loss function comprises: Determining whether a value of a loss function of a generative network in the trained cyclic generative adversarial network having a converged value of the loss function is overfitted; When it is determined that the value of the loss function of the generative network in the trained cyclic generative adversarial network with the converged loss function value is not overfitted, obtaining a target generative network model based on the generative network in the trained cyclic generative adversarial network with the converged loss function value; and When it is determined that the value of the loss function of the generative network in the trained recurrent generative adversarial network with a converged loss function value is overfitted, the parameters of the trained recurrent generative adversarial network with a converged loss function value are adjusted and the recurrent generative adversarial network with the adjusted parameters is trained based on the Raman image and the true stained image.

10. The training method according to any one of claims 1 to 7, wherein: Obtaining a Raman image of a cell sample and a corresponding true staining image of the cell sample includes: Performing Raman scattering microscopic imaging on the cell sample to obtain the Raman image of the cell sample; The cell sample for which the Raman image has been generated is subjected to chemical staining and bright field microscopic imaging to obtain the corresponding real staining image of the cell sample.

11. The training method according to claim 10, wherein: Performing Raman scattering microscopic imaging on the cell sample to obtain the Raman image of the cell sample includes: irradiating the cell sample with the first excitation light and the second excitation light; In response to the first excitation light and the second excitation light irradiating the cell sample, a reflected light having at least one Raman shift characteristic peak is received from the cell sample to obtain the Raman image of the cell sample.

12. The training method according to claim 11, wherein: The first excitation light and the second excitation light each have a predetermined frequency, and In response to the first excitation light and the second excitation light irradiating the cell sample, receiving reflected light having at least one Raman shift characteristic peak from the cell sample to obtain the Raman image of the cell sample comprises: In response to the first excitation light and the second excitation light having a predetermined frequency irradiating the cell sample, reflected light having only one Raman shift characteristic peak is received from the cell sample to obtain the Raman image of the cell sample.

13. The training method according to claim 11, wherein: The first excitation light and the second excitation light are pulsed lasers with a repetition frequency greater than 50 MHz and a pulse width of 100 fs-20 ps.

14. The training method according to claim 13, wherein: The first excitation light includes pump light, the second excitation light includes Stokes light, and in response to irradiating the cell sample with the first excitation light and the second excitation light, receiving reflected light having at least one Raman shift characteristic peak from the cell sample to obtain the Raman image of the cell sample includes: In response to the pump light and the Stokes light irradiating the cell sample, a reflected light having at least one Raman shift characteristic peak is received from the cell sample to obtain a coherent Raman scattering imaging image of the cell sample, wherein the coherent Raman scattering imaging image includes a stimulated Raman scattering imaging image, a stimulated Raman photothermal imaging image, or a coherent anti-Stokes Raman scattering imaging image.

15. The training method according to claim 10, wherein: Performing Raman scattering microscopic imaging on the cell sample to obtain the Raman image of the cell sample includes: Performing Raman scattering microscopic imaging on each local part of the cell sample in a local field of view to obtain a local Raman image of the cell sample; splicing the local Raman images of various parts of the cell sample to generate the Raman image of the cell sample; and Obtaining the corresponding real staining image of the cell sample by chemical staining and bright field microscopic imaging of the cell sample for which the Raman image has been generated comprises: The cell sample for which the Raman image has been generated is chemically stained, and each part of the chemically stained cell sample is subjected to bright field microscopic imaging in a local field of view to obtain a local true staining image of the cell sample. The local true staining images of various parts of the cell sample are stitched together to generate the true staining image of the cell sample.

16. A virtual staining method for enhancing the display of cell structure, comprising: Obtaining a target Raman image of a target cell sample; The target Raman image is input into the target generation network model according to any one of claims 1 to 15 to obtain a target virtual staining image of the target cell sample, wherein the target virtual staining image is used to enhance the display of the cell structure in the target cell sample.

17. The virtual coloring method according to claim 16, wherein: Obtaining a targeted Raman image of a target cell sample includes: The target cell sample is subjected to Raman scattering microscopic imaging to obtain the target Raman image of the target cell sample.

18. The virtual coloring method according to claim 17, wherein: Performing Raman scattering microscopic imaging on the target cell sample to obtain the target Raman image of the target cell sample includes: irradiating the target cell sample using a first excitation light and a second excitation light; In response to the first excitation light and the second excitation light irradiating the target cell sample, a reflected light having at least one Raman shift characteristic peak is received from the target cell sample to obtain the target Raman image of the target cell sample.

19. The virtual coloring method according to claim 18, wherein: The first excitation light and the second excitation light each have a predetermined frequency, and In response to irradiating the target cell sample with the first excitation light and the second excitation light, receiving reflected light having at least one Raman shift characteristic peak from the target cell sample to obtain the target Raman image of the target cell sample comprises: In response to the first excitation light and the second excitation light having a predetermined frequency irradiating the target cell sample, reflected light having only one Raman shift characteristic peak is received from the target cell sample to obtain the target Raman image of the target cell sample.

20. The virtual coloring method according to claim 18, wherein: The first excitation light includes pump light, the second excitation light includes Stokes light, and in response to the first excitation light and the second excitation light irradiating the target cell sample, a reflected light having at least one Raman shift characteristic peak is received from the target cell sample to obtain the target Raman image of the target cell sample: In response to the pump light and the Stokes light irradiating the target cell sample, a reflected light having at least one Raman shift characteristic peak is received from the target cell sample to obtain a coherent Raman scattering imaging image of the cell sample, wherein the coherent Raman scattering imaging image includes a stimulated Raman scattering imaging image, a stimulated Raman photothermal imaging image, or a coherent anti-Stokes Raman scattering imaging image.

21. The virtual coloring method according to claim 17, wherein: Performing Raman scattering microscopic imaging on the target cell sample to obtain the target Raman image of the target cell sample includes: Performing Raman scattering microscopic imaging on each local part of the target cell sample in a local field of view to obtain a local target Raman image of the target cell sample; The local target Raman images of various parts of the target cell sample are stitched together to obtain the target Raman image of the target cell sample.

22. A virtual dyeing system, comprising: An image acquisition component configured to obtain a target Raman image of a target cell sample; as well as An image processing component is configured to input the target Raman image into a target generation network model according to any one of claims 1-15 to obtain a target virtual staining image of the target cell sample, wherein the target virtual staining image is used to enhance the display of the cell structure in the target cell sample.

23. The virtual coloring system according to claim 22, wherein: The image acquisition component is configured to perform Raman scattering microscopic imaging on the target cell sample to obtain the target Raman image of the target cell sample.

24. The virtual coloring system according to claim 23, wherein: The virtual dyeing system also includes: a laser source configured to generate a first excitation light and a second excitation light for irradiating a target cell sample; and The image acquisition component is also configured to irradiate the target cell sample in response to the first excitation light and the second excitation light, receive reflected light having at least one Raman shift characteristic peak from the target cell sample, and obtain the target Raman image of the target cell sample.

25. The virtual coloring system according to claim 24, wherein: The first excitation light and the second excitation light each have a predetermined frequency, and the image acquisition component is further configured as follows: In response to the first excitation light and the second excitation light having a predetermined frequency irradiating the target cell sample, reflected light having only one Raman shift characteristic peak is received from the target cell sample to obtain the target Raman image of the target cell sample.

26. The virtual coloring system according to claim 24, further comprising: An optical path component, the optical path component comprising a two-dimensional galvanometer component and a first filter configured to guide the first excitation light and the second excitation light to the target cell sample; a sample carrying component, configured to carry the target cell sample to be irradiated by the first excitation light and the second excitation light; as well as The objective lens component is configured to receive the reflected light from the target cell sample and transmit the reflected light to the image acquisition component.

27. The virtual coloring system according to claim 24, wherein: The first excitation light and the second excitation light are pulsed lasers, and the frequency difference between the first excitation light and the second excitation light is 2800-3100 cm -1 , repetition frequency is greater than 50MHz, and pulse width is 100fs-20ps.

28. The virtual coloring system according to claim 27, wherein: The first excitation light includes pump light, the second excitation light includes Stokes light, and the image acquisition component is further configured as: In response to the pump light and the Stokes light irradiating the cell sample, a reflected light having at least one Raman shift characteristic peak is received from the cell sample to obtain a coherent Raman scattering imaging image of the cell sample, wherein the coherent Raman scattering imaging image includes a stimulated Raman scattering imaging image, a stimulated Raman photothermal imaging image, or a coherent anti-Stokes Raman scattering imaging image.

29. The virtual coloring system according to claim 23, wherein: The image acquisition component is configured as follows: Performing Raman scattering microscopic imaging on each local part of the target cell sample in a local field of view to obtain a local target Raman image of the target cell sample; The local target Raman images of various parts of the target cell sample are stitched together to obtain the target Raman image of the target cell sample.

Images