Training method of virtual staining model and generation method of bright field virtual staining image

Abstract

The present application relates to a method for training a virtual staining model and a method for generating brightfield virtual staining images; the training method for the virtual staining model includes: obtaining training data; performing a first registration to obtain a first training data set; performing a second registration to obtain a second training data set; using a convolutional neural network model to build a virtual staining model; sending the first training data set to the virtual staining model for training; after the loss function of the virtual staining model converges, using the second training data set to train the virtual staining model. The method proposed in the present application automatically generates dynamic magnification, multi-scale, and multi-type brightfield virtual staining images from darkfield images of unstained tissue sections obtained by immunofluorescence or other imaging techniques through image registration, combined with traditional image processing and deep learning image generation; the generated images are consistent with the real staining image area, providing support for pathologists to make rapid and accurate diagnoses.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method for training a virtual staining model and a method for generating a brightfield virtual staining image. Background Art

[0002] In routine pathology diagnosis, pathologists must obtain tissue from the patient's lesioned area through puncture or surgery. Professional technicians then perform the necessary procedures, including fixation, sampling, dehydration, transparency, waxing, embedding, sectioning, slicing, printing, staining, and sealing, before the pathologist can make a diagnosis. This process is time-consuming and labor-intensive, and some materials used in the preparation process are potentially harmful, posing a health hazard to the technicians. Furthermore, the staining of the sections can be affected by various factors, including the overall preparation process, the technician's proficiency, and the dye used. This can result in the sections not staining to the desired standard, impacting the pathologist's final diagnosis and, to a certain extent, the patient's treatment.

[0003] In the related art, when preparing tissue pathology sections, corresponding immunohistochemical sections are stained according to different needs. This process is time-consuming and labor-intensive, and each step in the staining process will affect the final image. Summary of the Invention

[0004] In order to at least overcome the problems existing in the related art to a certain extent, the present application provides a training method for a virtual staining model and a method for generating a bright field virtual staining image.

[0005] According to a first aspect of an embodiment of the present application, a method for training a virtual dyeing model is provided, comprising:

[0006] Acquire training data; the training data includes a dark field scan image and a plurality of corresponding bright field scan images;

[0007] Performing a first registration on the dark field scan image and the corresponding multiple bright field scan images to obtain a first training data set; performing a second registration on the dark field scan image and the corresponding multiple bright field scan images to obtain a second training data set; wherein the accuracy of the second registration is greater than the accuracy of the first registration;

[0008] A convolutional neural network model was used to build a virtual dyeing model;

[0009] Sending the first training data set to the virtual dyeing model for training;

[0010] After the loss function of the virtual coloring model converges, the virtual coloring model is trained using a second training data set.

[0011] Furthermore, the dark field scan image and the corresponding multiple bright field scan images are first registered, including:

[0012] Mark the points on the characteristic areas or corners of the dark field scan image;

[0013] Find the corresponding feature area or corner marking point on the bright field scan image;

[0014] The dark field scan images and bright field scan images are paired according to the marked points.

[0015] Furthermore, after performing the first registration, the method further includes:

[0016] After pairing, the dark field scan image and the bright field scan image were sampled separately using a non-overlap cropping method;

[0017] Sampling is performed at multiple magnifications.

[0018] Furthermore, the virtual dyeing model is constructed using a convolutional neural network model, including:

[0019] A generative adversarial convolutional neural network model was used to build a virtual dyeing model;

[0020] Configure the model's generator and divide its output layer into multiple branches, with each branch corresponding to a bright field image.

[0021] Configure the discriminator of the model so that the number of branches in the input and output layers of the discriminator is the same as the number of branches in the output layer of the generator.

[0022] Furthermore, the first training data set is fed into the virtual dyeing model for training, including:

[0023] The dark field scan image is normalized and then fed into the generator, which outputs the generated image after convolution, pooling, and nonlinear layers. Each branch of the output layer corresponds to a bright field scan image.

[0024] The discriminator performs binary classification on the output results, and multiple branches of the output layer respectively output the binary classification results of multiple bright field scan images.

[0025] Furthermore, during the training process using the first training data set, the objective function is:

[0026] L GAN (G,D Y ,X,Y)=E y-Pdata(y) [logD y ]+E x-Pdata(x) [log(1-D Y (G(x)))];

[0027] Among them, the model mapping relationship G represents the generator that converts X to Y, that is, G(x) = y; X represents the image domain of the dark field scan image, and Y represents the image domain of the bright field virtual staining image generated by the model; D Y is the discriminator; x is the dark field scan image, y is the bright field virtual staining image generated by the model; x-Pdata(x) and y-Pdata(y) represent the distribution of the data; E is the mathematical expectation.

[0028] Furthermore, the cycle consistency loss of the virtual coloring model is defined as:

[0029] L cyc (F,G,X,Y)=E x-Pdata(x) [||F(G(x)-x)||]+E x-Pdata(x) [log(1-D Y (G(x)))];

[0030] The model mapping relationship F represents the generator from Y to X, that is, F(y)=x.

[0031] Furthermore, during the training process using the second training data set, the objective function is:

[0032] L cGAN (G,D)=E x,y [logD(x,y)]+E x,z [log1-D(x,G(x,z))];

[0033] L L1 (G)=E x,y,z [||yG(x,z)||1].

[0034] According to a second aspect of an embodiment of the present application, a method for generating a brightfield virtual stained image is provided, comprising:

[0035] Acquire dark field scanning images;

[0036] Inputting the dark field scanned image into a virtual staining model for processing to generate a corresponding bright field virtual staining image;

[0037] The virtual staining model is obtained by the training method of the virtual staining model described in any one of the above embodiments.

[0038] According to a third aspect of an embodiment of the present application, a training device for a virtual dyeing model is provided, comprising:

[0039] An acquisition module is used to acquire training data; the training data includes a dark field scanning image and a plurality of corresponding bright field scanning images;

[0040] a registration module configured to perform a first registration on the darkfield scan image and the corresponding multiple brightfield scan images to obtain a first training data set; and perform a second registration on the darkfield scan image and the corresponding multiple brightfield scan images to obtain a second training data set; wherein the accuracy of the second registration is greater than the accuracy of the first registration;

[0041] A model building module is used to build a virtual dyeing model using a convolutional neural network model;

[0042] A first training module is used to send a first training data set to the virtual dyeing model for training;

[0043] The second training module is used to train the virtual dyeing model using a second training data set after the loss function of the virtual dyeing model converges.

[0044] The technical solutions provided by the embodiments of this application have the following beneficial effects:

[0045] The method proposed in this application uses image registration, combined with traditional image processing and deep learning image generation, to automatically generate dynamic magnification, multi-scale, and multi-type bright-field virtual stained images from dark-field images of unstained tissue sections obtained by immunofluorescence or other imaging techniques; the generated images are consistent with the actual stained image areas, providing support for pathologists to make rapid and accurate diagnoses.

[0046] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0048] Figure 1 The figure is a flowchart of a method for training a virtual dyeing model according to an exemplary embodiment.

[0049] Figure 2 It is a schematic diagram of the overall process of a model training method according to an exemplary embodiment.

[0050] FIG3 is a schematic diagram showing a first low-magnification tissue registration according to an exemplary embodiment.

[0051] Fig. 4 is a schematic diagram showing the correspondence between a dark field image and multiple bright fields according to an exemplary embodiment.

[0052] Figure 5 It is a schematic diagram of a training process according to an exemplary embodiment.

[0053] Figure 6 is a schematic diagram of a generator network structure according to an exemplary embodiment.

[0054] Figure 7 FIG. 4 is a schematic diagram of a discriminator network structure according to an exemplary embodiment.

[0055] Figure 8 It is a block diagram of a training device for a virtual dyeing model according to an exemplary embodiment. DETAILED DESCRIPTION

[0056] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of methods and apparatus consistent with certain aspects of the present application, as detailed in the appended claims.

[0057] To further describe the technical solution of the present application, we first briefly introduce the current state of the art in this field.

[0058] When preparing tissue pathology slides, immunohistochemical staining is performed according to specific requirements. Brightfield images are images of scanned tissue specimens that have undergone specific, multi-type staining. Using light, the images are generated against a bright background based on the light absorption of various points within the specimen. Darkfield images are images of unstained biological specimens or other transparent samples that absorb very little light, resulting in poor contrast and difficult to observe.

[0059] Current technology requires staining sections, a process that is affected by various factors, including the overall preparation process, the technician's proficiency, and the dye used. This can result in sections not staining to the desired standard, impacting the pathologist's final diagnosis and, to a certain extent, the patient's treatment. Therefore, the development of an efficient, non-invasive, repeatable, and highly consistent staining system is urgently needed.

[0060] Figure 1 1 is a flowchart of a method for training a virtual dyeing model according to an exemplary embodiment. The method may include the following steps:

[0061] Step S1, obtaining training data; the training data includes a dark field scanning image and a plurality of corresponding bright field scanning images;

[0062] Step S2: performing a first registration on the dark field scan image and the corresponding multiple bright field scan images to obtain a first training data set; performing a second registration on the dark field scan image and the corresponding multiple bright field scan images to obtain a second training data set; wherein the accuracy of the second registration is greater than the accuracy of the first registration;

[0063] Step S3, using a convolutional neural network model to build a virtual dyeing model;

[0064] Step S4, sending the first training data set to the virtual dyeing model for training;

[0065] Step S5: After the loss function of the virtual dyeing model converges, the virtual dyeing model is trained using a second training data set.

[0066] The method proposed in this application uses image registration, combined with traditional image processing and deep learning image generation, to automatically generate dynamic magnification, multi-scale, and multi-type bright-field virtual stained images from dark-field images of unstained tissue sections obtained by immunofluorescence or other imaging techniques; the generated images are consistent with the actual stained image areas, providing support for pathologists to make rapid and accurate diagnoses.

[0067] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0068] The present invention proposes a virtual imaging technology that can convert dark field to bright field, which only requires rough registration first and then supplemented with a small amount of refined registration data, and can input multiple virtual staining outputs.

[0069] First, unstained tissue sections are scanned using a darkfield microscope or digital scanner to obtain darkfield images. The scanned tissue is then stained for specific, multiple types of staining and scanned using a microscope or digital scanner to obtain multiple brightfield images. The brightfield and darkfield images are roughly registered to ensure overall image alignment. The darkfield image is cropped to a fixed size, such as 256×256 or 512×12, at any scan magnification, and at any low magnification. Corresponding images of the same size are cropped from the corresponding locations on the brightfield image. A generative adversarial convolutional neural network model is trained that takes darkfield image patches as input and outputs multiple brightfield-stained image patches, with the output images being the same size as the brightfield images. After the model loss function converges, the model is fine-tuned using the finely registered darkfield and brightfield data to achieve virtual staining.

[0070] like Figure 2 In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention are further described in detail below with reference to the accompanying drawings.

[0071] 1. Obtain a low-magnification image. Since the maximum magnification of a common scanned image is 40X, the image dimensions may exceed 100,000 in length and width. Therefore, to reduce computational complexity, when performing the first registration, directly obtain a 2X image of the scanned image.

[0072] 2. First tissue registration: This involves rough registration of tissues at low magnification, as shown in Figure 3. The images acquired in step 1 are the 2X images, consisting of a darkfield image and a corresponding HE brightfield image. A keypoint feature-based image registration method is employed.

[0073] In some embodiments, step S2 performs a first registration of the dark field scan image and the corresponding multiple bright field scan images, including: marking points on the characteristic areas or corners on the dark field scan image; finding the corresponding characteristic areas or corners marked points on the bright field scan image; and pairing the dark field scan image and the bright field scan image in pairs according to the marked points.

[0074] Using a self-developed annotation tool, we used the darkfield image as a template, marking areas or corners with distinct features on the darkfield image. We then found corresponding areas or corners on the brightfield image, pairing them up and annotating at least four groups of these points to complete the initial registration of the low-magnification images. As shown in Figure 4, one darkfield image can correspond to multiple brightfield images.

[0075] 3. Preprocessing of the first registration data. The first registration data completed in step 2 is sampled at multiple magnifications, such as 40X, 20X, 10X, and 5X. The data is paired and cropped to 256×256 and 512×512 sizes, using a non-overlapping cropping method.

[0076] In some embodiments, after the first registration is performed, the method further includes: sampling the dark field scan image and the bright field scan image respectively by non-overlap cropping after pairing; and performing sampling at multiple magnifications respectively.

[0077] 4. Input and output settings of the first registration data model generator. For a dark field image, corresponding bright field images of multiple magnifications are obtained at multiple magnifications. When outputting, multiple bright field images can be generated at one time, such as Figure 5 As shown. That is, in the input layer of the model, the dark field image is normalized to between [-1, 1] and then input into the generator. After a large number of convolutional pooling and nonlinear layers, the generated image is output. The detailed structure of the generator is as follows Figure 6 As shown, each branch of the output layer corresponds to a bright field image, and each image has three RGB channels.

[0078] In some embodiments, step S4 sends the first training data set into the virtual staining model for training, including: normalizing the dark field scan image and sending it into the generator, outputting the generated image after convolution pooling and nonlinear layer, and each branch of the output layer corresponds to a bright field scan image; the discriminator performs binary classification on the output result, and multiple branches of the output layer respectively output the binary classification results of multiple bright field scan images.

[0079] 5. First registration data model discriminator input and output settings, the discriminator detailed structure is as follows Figure 7 As shown in the figure, since the generator generates multiple brightfield virtual stained images, the number of branches in the discriminator's input and output layers should correspond to the number of generated brightfield images. For example, if the generator generates four brightfield images, the discriminator's output layer should contain four binary classification outputs.

[0080] In some embodiments, step S3 uses a convolutional neural network model to build a virtual staining model, including: using a generative adversarial convolutional neural network model to build a virtual staining model; configuring the generator of the model, configuring the output layer of the generator into multiple branches, each branch corresponding to a bright field image; configuring the discriminator of the model, and the number of branches of the input layer and output layer of the discriminator is the same as the number of branches of the output layer of the generator.

[0081] 6. During the training phase, the model is first trained on a large amount of first registration data, and then fine-tuned on the fine registration data.

[0082] 7. Model training uses the generative adversarial convolutional neural network model for training. Generator (refer to Figure 6 ) and the discriminator (refer to Figure 7 ) models are all implemented using pytorch, and their network structures are shown in the figure.

[0083] For the generation network x to y and its discriminant network, the objective function in the first registration data training stage is:

[0084] L GAN (G,D Y ,X,Y)=E y-Pdata(y) [logD y ]+E x-Pdata(x) [log(1-D Y (G(x)))]

[0085] The purpose of this formula is to make the generated network generate an image Y that is as similar as possible to the real Y. Similarly, the objective function of the generated network yx and its discriminant network can be constructed.

[0086] Because the training data is not fully aligned, the mapping G can completely map all x to y, thus invalidating the loss. Therefore, a cycle consistency loss is proposed here, requiring that the two mappings F and G can transform y and x back respectively, that is, F(G(x)) = x and G(F(y)) = y. The cycle consistency loss is defined as follows:

[0087] L cyc (F,G,X,Y)

[0088] =E x-Pdata(x) [||F(G(x)-x)||]+E x-Pdata(x) [log(1-D Y (G(x)))]

[0089] After the model pre-training is completed, the model is fine-tuned using a smaller amount of fine-aligned data. The loss function is as follows:

[0090] L cGAN (G,D)=E x,y [logD(x,y)]+E x,z [log 1-D(x,G(x,z))]

[0091] L L1 (G)=E x,y,z [||yG(x,z)||1]

[0092] Where X represents the dark field unstained image domain, Y represents the bright field virtual stained image domain, x represents the dark field unstained image, and y represents the bright field stained image. x-Pdata(x) and y-Pdata(y) represent the distribution of the data. The model mapping relationship G and F are the generators for converting X to Y and Y to X, respectively, and the corresponding discriminator is D X , D Y , E is the expectation.

[0093] 8. When the model is trained on the first registration data, it is sufficient to train for about 20 epochs based on the data of the registration data, so that the model can initially complete the staining of the dark field image.

[0094] An epoch represents the complete forward computation and backpropagation process after all data is fed into the network. Iteratively training all data once is not sufficient; multiple iterations are required to achieve fit and convergence. As the number of epochs increases, the number of weight update iterations increases, and the model progresses from an initial underfitting state to an optimized fit and eventually overfitting. The specific number of epochs depends on the diversity of the dataset; the greater the diversity, the more epochs are required.

[0095] 9. When the model is trained on the precisely registered data, the parameters of the generator except for the last five modules (each module includes convolution, pooling and nonlinear layers) are solidified, and only the parameters of the last five modules are trained; similarly, the discriminator only trains the parameters of the last three modules.

[0096] 10. The training adopts mini-batch stochastic gradient descent and Adam optimizer. The number of dark field images input in each batch is 1, the initial learning rate is 0.001, and the momentum parameters β1 = 0.5 and β2 = 0.999.

[0097] 11. During the inference phase of the model, the input dark-field image also needs to be normalized to [-1, 1]. The indexes of the output multiple bright-field images correspond one-to-one to the positions during training.

[0098] The present invention proposes a virtual imaging technology for converting darkfield to brightfield, requiring only a first registration plus a small amount of registration data, and a single input with multiple virtual staining outputs. This technology primarily implements a virtual staining imaging technology that takes one type of darkfield input image and outputs multiple types of brightfield images at multiple scales. This effectively avoids the need for large amounts of pixel-level image registration data, simplifying the processing of input images. Any darkfield image can be input as long as it can reflect detailed information such as the structure of the tissue, and the output is a staining image of multiple types and sizes. Virtual staining images can be consistent with real images, meaning that there is no substantial difference between the two when performing medical diagnosis.

[0099] The key points of the present invention are: 1. By inputting a dark field image, multiple bright field images can be obtained. In the training stage, it is only necessary to prepare multiple types of bright field data and the corresponding dark field data, so that in actual use, one input and multiple types of output can be obtained. 2. The first registration data is pre-trained, and the fine registration data is used to fine-tune the model. Since the cost of obtaining and labeling refined matching data is relatively high, the first registration data is used for pre-training, and the refined registration data is used to fine-tune the model for training. This can ensure that the data cost is reduced without losing accuracy. 3. The model is suitable for automatic adaptation of dynamic magnifications, that is, in the training stage, the data magnifications of the first registration and fine registration range from 1 to 40 times. A large amount of training data allows the model to automatically adapt to the magnification without the need for interpolation or requiring a fixed magnification.

[0100] The present application adopts the above technical solution, which has the following beneficial effects: 1. One-to-many, that is, one input dark field image can obtain multiple types of bright field images. It is only necessary to prepare multiple types of bright field images corresponding to one dark field image in the training stage, so that during training, this effect can be achieved while realizing multi-task learning without having to build another model. 2. The first registration data is pre-trained, and the fine registration data is then used to fine-tune the model, which can greatly reduce the data cost while ensuring the accuracy of the model. Because in the coarse matching data, most of the image information of the input dark field image and the corresponding multiple bright field images exists in their respective images, but is not completely aligned. After fine-tuning the fine registration data, a higher accuracy can be achieved. 3. Dynamic adaptation of magnification. The training data contains matching data of various magnifications, which allows the model to automatically adapt to the magnification, thereby improving the adaptability of the model, avoiding the situation where a model can only adapt to one magnification, and improving the conversion efficiency of dark field to bright field images.

[0101] The present application also provides a method for generating a brightfield virtual staining image, comprising the following steps: obtaining a darkfield scanned image; and processing the darkfield scanned image into a virtual staining model to generate a corresponding brightfield virtual staining image. The virtual staining model is obtained using the virtual staining model training method described in any of the aforementioned embodiments, which will not be further described in this embodiment.

[0102] Figure 8 1 is a block diagram of a training device for a virtual dyeing model according to an exemplary embodiment. The device includes:

[0103] An acquisition module is used to acquire training data; the training data includes a dark field scanning image and a plurality of corresponding bright field scanning images;

[0104] a registration module configured to perform a first registration on the darkfield scan image and the corresponding multiple brightfield scan images to obtain a first training data set; and perform a second registration on the darkfield scan image and the corresponding multiple brightfield scan images to obtain a second training data set; wherein the accuracy of the second registration is greater than the accuracy of the first registration;

[0105] A model building module is used to build a virtual dyeing model using a convolutional neural network model;

[0106] A first training module is used to send a first training data set to the virtual dyeing model for training;

[0107] The second training module is used to train the virtual dyeing model using a second training data set after the loss function of the virtual dyeing model converges.

[0108] Regarding the apparatus in the above-described embodiment, the specific steps for executing the operations of each module have been described in detail in the embodiments of the method and will not be elaborated on here. The modules in the above-described virtual staining model training apparatus can be implemented in whole or in part through software, hardware, or a combination thereof. The modules can be embedded in or independent of a processor in a computer device in hardware form, or can be stored in a memory in a computer device in software form, so that the processor can call and execute the operations corresponding to the modules.

[0109] It can be understood that the same or similar parts of the above embodiments can be referenced to each other, and the contents not described in detail in some embodiments can refer to the same or similar contents in other embodiments.

[0110] It should be noted that, in the description of this application, the terms "first", "second", etc. are used for descriptive purposes only and should not be understood as indicating or implying relative importance. In addition, in the description of this application, unless otherwise specified, the meaning of "plurality" refers to at least two.

[0111] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present application belong.

[0112] It should be understood that various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0113] Those skilled in the art will understand that all or part of the steps in the method of the above embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0114] In addition, the functional units in the various embodiments of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into a module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0115] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0116] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present application. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0117] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A training method for a virtual dyeing model, characterized in that: include: Acquire training data; the training data includes a dark field scan image and a plurality of corresponding bright field scan images; Performing a first registration on the dark field scan image and the corresponding multiple bright field scan images to obtain a first training data set; Performing a second registration on the dark field scan image and the corresponding multiple bright field scan images to obtain a second training data set; wherein the accuracy of the second registration is greater than the accuracy of the first registration; A virtual staining model is constructed using a convolutional neural network model, including: constructing the virtual staining model using a generative adversarial convolutional neural network model; configuring a generator of the model, configuring an output layer of the generator into multiple branches, each branch corresponding to a bright field image; configuring a discriminator of the model, wherein the number of branches in the input layer and output layer of the discriminator is the same as the number of branches in the output layer of the generator; The first training data set is fed into the virtual staining model for training, including: normalizing the dark field scan image and feeding it into the generator, outputting the generated image after convolution pooling and nonlinear layers, with each branch of the output layer corresponding to a bright field scan image; the discriminator performs binary classification on the output result, and multiple branches of the output layer respectively output binary classification results of multiple bright field scan images; wherein, during the training process using the first training data set, the objective function is: L GAN (G,D Y ,X,Y)=E y-Pdata(y) [logD y ]+E x-Pdata(x) [log(1-D Y (G(x)))]; Among them, the model mapping relationship G represents the generator that converts X to Y, that is, G(x) = y; X represents the image domain of the dark field scan image, and Y represents the image domain of the bright field virtual staining image generated by the model; D Y is the discriminator; x is the dark field scan image, y is the bright field virtual staining image generated by the model; x-Pdata(x) and y-Pdata(y) represent the distribution of the data; E is the mathematical expectation; After the loss function of the virtual dyeing model converges, training the virtual dyeing model using a second training data set; Among them, the cycle consistency loss of the virtual coloring model is defined as: L cyc (F,G,X,Y)=E x-Pdata(x) [||F(G(x)-x)||] +E x-Pdata(x) [log(1-D Y (G(x)))]; The model mapping relationship F represents the generator from Y to X, that is, F(y) = x; During the training process using the second training data set, the objective function is: L cGAN (G,D)=E x,y [logD(x,y)]+E x,z [log1-D(x,G(x,z))]; L L1 (G)=E x,y,z [||y-G(x,z)||1]。 2. The method according to claim 1, characterized in that Performing a first registration of the dark field scan image and the corresponding multiple bright field scan images includes: Mark the points on the characteristic areas or corners of the dark field scan image; Find the corresponding feature area or corner marking point on the bright field scan image; The dark field scan images and bright field scan images are paired according to the marked points.

3. The method according to claim 2, characterized in that After the first registration, the following steps are also included: After pairing, the dark field scan image and the bright field scan image were sampled separately using a non-overlap cropping method; Sampling is performed at multiple magnifications.

4. A method for generating a bright field virtual staining image, characterized in that: include: Acquire dark field scanning images; Inputting the dark field scanned image into a virtual staining model for processing to generate a corresponding bright field virtual staining image; Wherein, the virtual staining model is obtained by the training method of the virtual staining model according to any one of claims 1 to 3.

5. A training device for a virtual dyeing model, characterized in that: include: An acquisition module is used to acquire training data; the training data includes a dark field scanning image and a plurality of corresponding bright field scanning images; a registration module, configured to perform a first registration on the dark field scan image and the corresponding plurality of bright field scan images to obtain a first training data set; Performing a second registration on the dark field scan image and the corresponding multiple bright field scan images to obtain a second training data set; wherein the accuracy of the second registration is greater than the accuracy of the first registration; A model building module is used to build a virtual staining model using a convolutional neural network model; specifically, it is used to build a virtual staining model using a generative adversarial convolutional neural network model; configure the model's generator, configure the generator's output layer into multiple branches, each branch corresponding to a bright field image; configure the model's discriminator, and configure the number of branches in the discriminator's input layer and output layer to be the same as the number of branches in the generator's output layer; The first training module is used to feed the first training data set into the virtual staining model for training; specifically, it is used to feed the normalized dark field scan image into the generator, and output the generated image after convolution pooling and nonlinear layer, and each branch of the output layer corresponds to a bright field scan image; the discriminator performs binary classification on the output result, and multiple branches of the output layer respectively output the binary classification results of multiple bright field scan images; wherein, during the training process using the first training data set, the objective function is: L GAN (G,D Y ,X,Y)=E y-Pdata(y) [logD y ]+E x-Pdata(x) [log(1-D Y (G(x)))]; Among them, the model mapping relationship G represents the generator that converts X to Y, that is, G(x) = y; X represents the image domain of the dark field scan image, and Y represents the image domain of the bright field virtual staining image generated by the model; D Y is the discriminator; x is the dark field scan image, y is the bright field virtual staining image generated by the model; x-Pdata(x) and y-Pdata(y) represent the distribution of the data; E is the mathematical expectation; The second training module is configured to train the virtual coloring model using a second training data set after the loss function of the virtual coloring model converges; wherein the cycle consistency loss of the virtual coloring model is defined as: L cyc (F,G,X,Y)=E x-Pdata(x) [||F(G(x)-x)||] +E x-Pdata(x) [log(1-D Y (G(x)))]; The model mapping relationship F represents the generator from Y to X, that is, F(y) = x; During the training process using the second training data set, the objective function is: L cGAN (G,D)=E x,y [logD(x,y)]+E x,z [log1-D(x,G(x,z))]; L l1 (G)=E x,y,z [||y-G(x,z)||1]。

Images