Image segmentation model training method and device for dyed image, equipment and medium

Abstract

The invention discloses an image segmentation model training method and device of a stained image, equipment and a medium, and relates to the technical field of computers, and the method comprises the steps: obtaining a plurality of pairs of HE stained images and mIHC stained images of the same tissue slice, carrying out the pixel-by-pixel image registration processing of each pair of stained images, and obtaining a plurality of HE stained images and mIHC stained images; obtaining target mask information of the HE dyeing image according to an image registration result, and constructing an image training set based on the HE dyeing image and the corresponding target mask information; training a U-Net network structure by using the image training set so that each convolutional layer of the U-Net network extracts each slice image feature of the HE dyeing image and constructs a feature map, and converting the feature map into initial probability prediction results of different cell class groups; and according to the initial probability prediction result and the target mask information, performing back propagation updating on network parameters of the U-Net network to update the U-Net network, and obtaining the updated U-Net network as a preset image segmentation model.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an image segmentation model training method, device, equipment and medium for dyed images. Background Art

[0002] H&E staining (Hematoxylin and eosin stains) and IHC staining (Immunohistochemistry) are widely used in pathology to observe the morphological characteristics of tissues and cells. In the prior art, it is necessary to extract pathological information from tissue staining sections and segment various cell types (tumor cells and immune cells such as T, B and macrophages) through a semantic segmentation model. In order to train a semantic segmentation model, it is necessary to first obtain a data set, which is mainly divided into two parts: (1) H&E stained sections; (2) corresponding labels of different cell types. The prior art mainly relies on traditional manual annotation or staining adjacent slice layers into H&E sections and IHC sections respectively and then aligning them, and obtaining the annotation information of various cell types through the information of corresponding IHC sections. For traditional manual annotation, the main problems are that it is too time-consuming and labor-intensive, and it is impossible to obtain accurate pixel-level annotation. For the method of aligning adjacent slices, since adjacent slices are not completely consistent, the accuracy of alignment is greatly reduced, and accurate pixel-level annotation cannot be obtained.

[0003] In summary, how to obtain accurate HE image annotation information to train an accurate image segmentation model based on the HE image and the annotation information is a technical problem to be solved in this field. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a method, device, equipment and medium for training an image segmentation model of a dyed image, which can obtain accurate HE image annotation information to train an accurate image segmentation model based on the HE image and the annotation information. The specific scheme is as follows:

[0005] In a first aspect, the present application discloses a method for training an image segmentation model of a stained image, comprising:

[0006] Acquire several pairs of HE staining images and mIHC staining images of the same tissue section, and perform pixel-by-pixel image registration processing of the staining modality on each pair of the HE staining images and the mIHC staining images, so as to obtain target mask information of the HE staining images according to the image registration results, and construct an image training set based on the HE staining images and the corresponding target mask information; wherein the target mask information is label information that annotates different cell population types and cell population image positions in the staining image;

[0007] Using the image training set to train the U-Net network structure, so that each convolution layer of the U-Net network extracts the image features of each slice of the HE-stained image to obtain a corresponding feature map, and converts the feature map into initial probability prediction results of different cell populations;

[0008] The network parameters of the U-Net network are back-propagated and updated according to the initial probability prediction result and the label information in the corresponding target mask information to update the U-Net network and obtain the updated U-Net network as the preset image segmentation model.

[0009] Optionally, obtain HE staining images and mIHC staining images of the same tissue section, including:

[0010] After HE staining is performed on a tissue section, the HE-stained tissue section is scanned by a scanning machine to obtain a HE-stained image of the tissue section;

[0011] The HE-stained tissue section is decolorized and mIHC-stained, and then scanned by the scanning machine to obtain an mIHC-stained image of the same tissue section.

[0012] Optionally, performing pixel-by-pixel image registration processing of the staining modality on each pair of the HE staining image and the mIHC staining image to obtain target mask information of the HE staining image according to the image registration result includes:

[0013] Performing structural registration processing on the HE staining image and the mIHC staining image using a preset multi-cell group annotation registration algorithm to obtain a registered HE staining image;

[0014] The visualization information of the mIHC staining image is used as mask information of the image area corresponding to the HE staining image after registration to obtain the target mask information.

[0015] Optionally, the using the image training set to train the U-Net network structure so that each convolution layer of the U-Net network extracts the image features of each slice of the HE-stained image to obtain a corresponding feature map, and converts the feature map into initial probability prediction results of different cell populations, including:

[0016] Constructing a U-Net network structure including a contraction path and an expansion path; wherein the contraction path is provided with a plurality of convolution blocks consisting of two convolution layers and a ReLU activation function and a maximum pooling layer for downsampling; and the expansion path is provided with a deconvolution layer for upsampling and an output layer including a convolution layer and a softmax activation function;

[0017] Inputting the HE stained image into the U-Net network structure, extracting image features of each slice of the image through the convolution layer of the contraction path of the U-Net network structure to obtain a corresponding initial feature map, performing nonlinear feature processing on the initial feature map using the ReLU activation function to obtain a process feature map, and downsampling the process feature map through the maximum pooling layer to obtain a target feature map;

[0018] Inputting the target feature map into the extension path of the U-Net network structure to perform upsampling processing on the target feature map through the deconvolution layer, so as to gradually restore the feature map size of the target feature map to the size of the HE stained image, so as to obtain a restored target feature map;

[0019] The restored target feature map is input into the output layer, so as to convert the restored target feature map into probability predictions for different cell groups through the output layer, thereby obtaining an initial probability prediction result of semantic segmentation.

[0020] Optionally, back-propagating and updating the network parameters of the U-Net network according to the initial probability prediction result and the label information in the corresponding target mask information to update the U-Net network to obtain the updated U-Net network as the preset image segmentation model includes:

[0021] Using a preset loss function to calculate the loss function of the initial probability prediction result and the label information in the corresponding target mask information to obtain a corresponding loss function value;

[0022] The back propagation algorithm and the loss function value are used to calculate the gradient information of different network structures from the output layer in sequence, so as to update the U-Net network according to the gradient information, and obtain the updated U-Net network as the preset image segmentation model.

[0023] Optionally, extracting image features of each slice of the image through the convolution layer of the contraction path of the U-Net network structure includes:

[0024] The shape features, size features, and texture features of cells in the image are extracted through the convolutional layer of the contraction path of the U-Net network structure to obtain the image features of each slice.

[0025] Optionally, after obtaining the updated U-Net network as the preset image segmentation model, the method further includes:

[0026] When there is an image segmentation request for a HE-stained image, determining the HE-stained image to be segmented based on the image segmentation request;

[0027] The HE-stained image to be segmented is input into a preset image segmentation model, so as to perform tumor cell group labeling processing, T lymphocyte group labeling processing, B lymphocyte group labeling processing, and macrophage group labeling processing on the HE-stained image to be segmented through the preset image segmentation model, so as to segment the image area where each cell group is located, and output the corresponding tumor cell group category information and corresponding image area position information, T lymphocyte group category information and corresponding image area position information, B lymphocyte group category information and corresponding image area position information, and macrophage group category information and corresponding image area position information.

[0028] In a second aspect, the present application discloses an image segmentation model training device for stained images, comprising:

[0029] A training set acquisition module, used to acquire several pairs of HE staining images and mIHC staining images of the same tissue section, and perform pixel-by-pixel image registration processing of the staining modality on each pair of the HE staining images and the mIHC staining images, so as to acquire target mask information of the HE staining images according to the image registration results, so as to construct an image training set based on the HE staining images and the corresponding target mask information; wherein the target mask information is label information that annotates different cell population types and cell population image positions in the staining image;

[0030] A model training module, used to train the U-Net network structure using the image training set, so that each convolution layer of the U-Net network extracts the image features of each slice of the HE-stained image to obtain a corresponding feature map, and converts the feature map into an initial probability prediction result of different cell populations;

[0031] A model updating module is used to back-propagate and update the network parameters of the U-Net network according to the initial probability prediction result and the label information in the corresponding target mask information, so as to update the U-Net network and obtain the updated U-Net network as the preset image segmentation model.

[0032] In a third aspect, the present application discloses an electronic device, comprising:

[0033] Memory, used to store computer programs;

[0034] A processor is used to execute the computer program to implement the steps of the image segmentation model training method for stained images disclosed above.

[0035] In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the steps of the image segmentation model training method for stained images disclosed above are implemented.

[0036] It can be seen that the present application discloses an image segmentation model training method for stained images, including: obtaining several pairs of HE stained images and mIHC stained images of the same tissue slice, and performing pixel-by-pixel image registration processing of the staining modality on each pair of the HE stained images and the mIHC stained images, so as to obtain the target mask information of the HE stained image according to the image registration result, so as to construct an image training set based on the HE stained image and the corresponding target mask information; wherein the target mask information is label information that marks different cell population types and cell population image positions in the stained image; using the image training set to train the U-Net network structure, so that each convolution layer of the U-Net network extracts the image features of each slice of the HE stained image to obtain a corresponding feature map, and converts the feature map into an initial probability prediction result of different cell populations; back-propagating and updating the network parameters of the U-Net network according to the initial probability prediction result and the label information in the corresponding target mask information to update the U-Net network, and obtain the updated U-Net network as a preset image segmentation model. It can be seen that by obtaining the HE staining image and mIHC staining image of the same tissue slice and performing pixel-by-pixel image registration processing, the information of the two staining images is fully utilized. The mIHC staining image can provide more detailed labeling information about different cell populations. After being registered with the HE staining image, the target mask information of the HE staining image can be accurately obtained according to the registration result. This information accurately marks the different cell population types and cell population image positions in the staining image. Based on such accurate annotation information training model, the model can more accurately identify and segment different cell populations, thereby improving the accuracy of cell population segmentation. Moreover, using the U-Net network structure for training, each convolution layer can effectively extract the image features of each slice of the HE staining image. The contraction path of U-Net gradually extracts features of different scales through multiple convolution and downsampling operations, which can effectively capture everything from the detailed features of local cells to the more macroscopic distribution features of cell populations. The expansion path can fuse these features and restore them to the original image resolution, so that the model can make full use of image features for cell population segmentation, further improving the segmentation accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0038] Figure 1 A flow chart of an image segmentation model training method for a stained image disclosed in the present application;

[0039] Figure 2 A flow chart of an image segmentation method for a stained image disclosed in this application;

[0040] Figure 3 This is a schematic diagram of the structure of an image segmentation model training device for stained images disclosed in this application;

[0041] Figure 4 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0043] H&E staining and IHC staining are widely used in pathology to observe the morphological characteristics of tissues and cells. In the prior art, it is necessary to extract pathology-related information from tissue staining sections and segment various cell types (cell groups such as T, B, and macrophages in tumor cells and immune cells) through a semantic segmentation model. In order to train a semantic segmentation model, it is necessary to first obtain a data set, which is mainly divided into two parts: (1) H&E stained sections; (2) corresponding labels of different cell types. The prior art mainly relies on traditional manual annotation or staining adjacent slice layers into H&E sections and IHC sections respectively and then aligning them, and obtaining the annotation information of various cell types through the information of corresponding IHC sections. For traditional manual annotation, the main problems are that it is too time-consuming and labor-intensive, and it is impossible to obtain accurate pixel-level annotation. For the method of aligning adjacent slices, since adjacent slices are not completely consistent, the accuracy of alignment will be greatly reduced, and accurate pixel-level annotation cannot be obtained.

[0044] To this end, the present invention provides an image segmentation model training scheme for stained images, which can obtain accurate HE image annotation information. Training an accurate image segmentation model based on HE images and annotation information is a technical problem to be solved in the field.

[0045] Reference Figure 1 As shown, the embodiment of the present application discloses a method for training an image segmentation model of a stained image, comprising:

[0046] Step S11: obtaining several pairs of HE staining images and mIHC staining images of the same tissue section, and performing pixel-by-pixel image registration processing of the staining modality on each pair of the HE staining images and the mIHC staining images, so as to obtain target mask information of the HE staining images according to the image registration results, so as to construct an image training set based on the HE staining images and the corresponding target mask information; wherein the target mask information is label information that annotates different cell population types and cell population image positions in the staining image.

[0047] In this embodiment, after HE staining a tissue section, the HE-stained tissue section is scanned by a film scanning machine to obtain a HE-stained image of the tissue section; the HE-stained tissue section is decolorized and mIHC-stained, and scanned by the film scanning machine to obtain an mIHC-stained image of the same tissue section. It can be understood that a large number of HE-stained images are collected as part of the training images of the image training set. These images should cover a variety of tissue sample conditions, including different cell distribution, morphology and other characteristics, to ensure that the model can learn a rich variety of sample patterns, such as images of tissue sections from different patients and different lesions after HE staining. Then the same tissue section of the above-mentioned partial training image is decolorized and mIHC-stained to obtain an mIHC-stained image of the same tissue section. Specifically, when performing mIHC staining, since it can mark different cell types or cell markers, annotation information about different cell groups, that is, label information, can be obtained at the same time in this process. The annotation information marks the specific locations of different cell groups (such as tumor cells, T cells, B cells, macrophages, etc.) in each slice, and the value of each pixel in the annotation information represents the cell type to which it belongs. These annotation information are also characterized, such as converted into a specific vector representation, so that they can be better matched and associated with image features in the future.

[0048] In this embodiment, the HE staining image and the mIHC staining image are structurally registered using a preset multi-cell group annotation registration algorithm to obtain a registered HE staining image; the visualization information of the mIHC staining image is used as the mask information of the image area corresponding to the registered HE staining image to obtain the target mask information. It can be understood that pixel-by-pixel image registration is performed on each pair of HE staining images and mIHC staining images, and initial matching points are found in the HE staining image and the mIHC staining image. These matching points can be specific positions of cells (such as the center position of a cell group, the position of a single cell with obvious features, etc.). For example, the approximate center position of the tumor cell group is found in the HE staining image, and then the center position of the corresponding marked tumor cell group is found in the mIHC staining image as the initial matching point. Based on the determined initial matching point, the transformation parameters between the two images are calculated to achieve registration. Since the HE staining image and the mIHC staining image of the same tissue section are used, the image transformation only includes translation and rotation during the registration process.

[0049] For translation transformation, the displacement of the two images in the horizontal and vertical directions is calculated so that the matching points can be aligned in space. For example, if the center position of a cell group in the HE-stained image differs from the center position of the corresponding cell group in the mIHC-stained image by a certain number of pixels in the horizontal direction, the horizontal translation parameter can be obtained by calculating this difference.

[0050] For rotation transformation, the angle that the two images need to be rotated is calculated so that the orientation of the images can match. For example, by analyzing the characteristics such as the arrangement direction of cells, the angle that needs to be rotated between the two images is determined to achieve the alignment of structures such as cell groups in the direction.

[0051] In order to obtain a more accurate registration effect, the calculated transformation parameters need to be optimized. Optimization algorithms such as the least squares method and the gradient descent method can be used to adjust the transformation parameters with the goal of minimizing the difference between the two images (such as the distance based on feature points, the difference in cell group annotations, etc.). Through continuous iterative optimization, the two images can achieve the best matching state at the pixel level. At this time, the output registered image is the registered HE-stained image.

[0052] After completing the pixel-level registration of the HE-stained image and the mIHC-stained image, the existing cell group annotation information in the mIHC-stained image (different cell types or cell markers marked during the mIHC staining process) is used to correspond these annotation information to the registered HE-stained image. Because the two images have been registered at the pixel level, the cell group annotations in the mIHC-stained image can be accurately transferred to the HE-stained image.

[0053] A mask is generated based on the cell group annotation information transferred to the HE-stained image. Each pixel in the mask corresponds to a position in the HE-stained image, and its value represents the cell type to which the position belongs. For example, if a pixel belongs to a tumor cell type in the transferred annotation, the value of the pixel in the mask is set to a specific value representing the tumor cell. In this way, a mask with the position and category information of different cell groups in the HE-stained image is obtained, which can be used for subsequent image segmentation and other related operations.

[0054] Step S12: Use the image training set to train the U-Net network structure so that each convolution layer of the U-Net network extracts the image features of each slice of the HE stained image to obtain a corresponding feature map, and converts the feature map into initial probability prediction results of different cell populations.

[0055] In this embodiment, a U-Net network structure including a contraction path and an expansion path is constructed; wherein, a plurality of convolution blocks consisting of two convolution layers and a ReLU activation function and a maximum pooling layer for downsampling are set in the contraction path; a deconvolution layer for upsampling and an output layer including a convolution layer and a softmax activation function are set in the expansion path. It can be understood that the U-Net network structure is constructed, including its contraction path (Contracting Path) and expansion path (Expanding Path). In the contraction path, a plurality of convolution blocks consisting of two 3×3 convolution layers and a ReLU activation function, and a 2×2 maximum pooling layer are set for downsampling; in the expansion path, a transposed convolution (deconvolution) layer is set for upsampling, and a cropping and splicing operation with the contraction path feature map is performed at an appropriate position, and finally the output layer is set to a 1×1 convolution layer and a softmax activation function. In medical image segmentation, U-Net learns the feature representation of different cells in the input image (such as HE staining image) through the convolution layer and downsampling operation of the contraction path. For example, for tumor cells, the network will learn features such as their irregular shape and high nuclear-to-cytoplasmic ratio; for immune cells, it will learn features such as their relatively small size and specific staining pattern. These feature representations are automatically learned by the network, and there is no need to manually design feature extractors. The design of the contraction path and the expansion path enables U-Net to fuse multi-scale information. In the contraction path, the network learns global and abstract features; in the expansion path, by splicing the feature maps of the contraction path, the network can combine these global features with local detail features. For example, when segmenting cells, the network can not only use global features such as the overall shape of the cell to determine the cell type, but also use detail features such as the nuclear structure inside the cell to improve the accuracy of segmentation.

[0056] In this embodiment, the HE stained image is input into the U-Net network structure, so as to extract the image features of each slice of the image through the convolution layer of the contraction path of the U-Net network structure to obtain the corresponding initial feature map, and the nonlinear feature processing is performed on the initial feature map using the ReLU activation function to obtain the process feature map, and the process feature map is downsampled through the maximum pooling layer to obtain the target feature map; the target feature map is input into the expansion path of the U-Net network structure, so as to upsample the target feature map through the deconvolution layer, so as to gradually restore the feature map size of the target feature map to the size of the HE stained image, so as to obtain the restored target feature map; the restored target feature map is input into the output layer, so as to convert the restored target feature map into a probability prediction of different cell groups through the output layer, so as to obtain the initial probability prediction result of semantic segmentation. It can be understood that the HE stained image in the training set is then input into the U-Net network structure, and then in the U-Net network structure, the input HE stained image first passes through the contraction path. The shape features, size features, and texture features of the cells in the image are extracted through the convolution layer of the contraction path of the U-Net network structure to obtain the features of each slice image. In this way, after each convolution layer, the ReLU activation function is used to increase the nonlinear expression ability. As the downsampling proceeds, the receptive field of the network can be expanded, so that the network can learn more macroscopic features, while also reducing the amount of calculation and the number of parameters. In this way, the network gradually extracts more macroscopic features, and the number of feature channels gradually increases. Then, through the expansion path, upsampling is performed by transposed convolution, and the size of the feature map is gradually restored to a size close to that of the input image, and the feature map of the contraction path is spliced ​​in the process to fuse multi-scale information, so that the network can use both global features and detailed features for segmentation. Finally, through the 1×1 convolution layer and softmax activation function of the output layer, the feature map generated by the expansion path is converted into a probability prediction of different cell groups, and the preliminary results of semantic segmentation, that is, the initial probability prediction results, are obtained. It should be noted that the output layer of U-Net is a 1×1 convolution layer, which converts the target feature map generated by the expansion path into the required segmentation result. For example, if you want to segment different cell clusters in an image, the number of channels in the output layer is the same as the number of cell clusters, and each channel represents the probability distribution of a cell cluster. Through a softmax activation function, these probability distributions can be converted into the probability that each pixel belongs to a different cell cluster, thereby obtaining the final initial probability prediction result.

[0057] Step S13: back-propagating and updating the network parameters of the U-Net network according to the initial probability prediction result and the label information in the corresponding target mask information to update the U-Net network and obtain the updated U-Net network as a preset image segmentation model.

[0058] In this embodiment, a preset loss function is used to calculate the loss function of the initial probability prediction result and the label information in the corresponding target mask information to obtain the corresponding loss function value; the gradient information of different network structures is calculated from the output layer in turn using the back propagation algorithm and the loss function value, so as to update the U-Net network according to the gradient information, and obtain the updated U-Net network as the preset image segmentation model. It can be understood that the initial probability prediction result obtained by forward propagation is compared with the corresponding mask information (i.e., the real annotation information) to calculate the loss function. Commonly used preset loss functions such as cross entropy loss functions can measure the degree of difference between the predicted result and the real result. For example, if the probability that a pixel in the predicted result is predicted to be a tumor cell is inconsistent with the cell type (assuming it is a T cell) to which the pixel actually belongs in the mask, then the value of the loss function will reflect this difference. According to the calculated loss function, the gradient of each neuron is calculated in turn from the output layer to the input layer using the back propagation algorithm. These gradients represent the degree of influence of each parameter (such as the weight and bias of the convolutional layer) on the loss function. Then, the parameters of the network are updated based on these gradients. Optimization algorithms such as stochastic gradient descent and its variants (such as Adagrad (Adaptive Gradient Algorithm), Adadelta (Adaptive Delta), Adam (Adaptive Moment Estimation), etc.) are usually used to adjust the parameters so that the network can output a segmentation result that is closer to the actual mask information when the same or similar HE-stained image is input next time. Repeat the above process of inputting data, forward propagation, calculating loss function, back propagation and parameter updating for the HE-stained images of the entire training data set, and perform multiple rounds of iterative training. In each round of training, the network adjusts its own parameters according to the new gradient information and continuously optimizes the segmentation ability of different cell groups. After a certain number of rounds of training, the trained model needs to be evaluated. You can set aside a part of the validation data set (this data set also consists of HE-stained images and corresponding mask information, but does not participate in the training process), input the HE-stained images of the validation data set into the trained model, obtain the predicted segmentation results, and then compare them with the corresponding mask information, and measure the performance of the model by calculating evaluation indicators such as accuracy, recall, and F1 value. If the model performance does not meet expectations, you can continue training and adjust the training parameters (such as learning rate, training rounds, etc.) until a satisfactory trained semantic segmentation model is obtained as the preset image segmentation model.

[0059] The untrained semantic segmentation model (U-Net network) first learns the visual features of the HE-stained slices in the dataset. In the contraction path, these features are extracted through a series of 3×3 convolutional layers. For example, for cells in HE-stained slices, the model learns the shape, size, texture and other features of the cells. The ReLU activation function after each convolutional layer increases the nonlinear expression ability of the network, enabling it to learn more complex cell feature patterns. At the same time, the model learns the mask information corresponding to the HE-stained slices that annotates different cell types. These masks contain the location and category information of each cell in the image, which is the goal of model learning. For example, the mask clearly defines the specific locations of different cell groups such as tumor cells, T cells, B cells and macrophages in the HE-stained image.

[0060] U-Net is an end-to-end neural network. From the input original image to the output segmentation result, all parameters are automatically learned through training data. During the training process, the network parameters are adjusted according to the loss function (such as the cross entropy loss function, which is used to measure the difference between the segmentation result and the true label) through the back-propagation algorithm. In this way, the network can continuously optimize its parameters to adapt to different medical image segmentation tasks, such as accurately segmenting different cell groups such as tumor cells, T cells, B cells and macrophages in HE-stained images. Therefore, the preset image segmentation model obtained after training is a model that learns the association between the image features of each slice in the HE-stained image and the corresponding mIHC-stained image and the corresponding mask information; the mask information includes the position information and category information of each cell group in the HE-stained image.

[0061] It can be seen that the present application discloses an image segmentation model training method for stained images, including: obtaining several pairs of HE stained images and mIHC stained images of the same tissue slice, and performing pixel-by-pixel image registration processing of the staining modality on each pair of the HE stained images and the mIHC stained images, so as to obtain the target mask information of the HE stained image according to the image registration result, so as to construct an image training set based on the HE stained image and the corresponding target mask information; wherein the target mask information is label information that marks different cell population types and cell population image positions in the stained image; using the image training set to train the U-Net network structure, so that each convolution layer of the U-Net network extracts the image features of each slice of the HE stained image to obtain a corresponding feature map, and converts the feature map into an initial probability prediction result of different cell populations; back-propagating and updating the network parameters of the U-Net network according to the initial probability prediction result and the label information in the corresponding target mask information to update the U-Net network, and obtain the updated U-Net network as a preset image segmentation model. It can be seen that by obtaining the HE staining image and mIHC staining image of the same tissue slice and performing pixel-by-pixel image registration processing, the information of the two staining images is fully utilized. The mIHC staining image can provide more detailed labeling information about different cell populations. After being registered with the HE staining image, the target mask information of the HE staining image can be accurately obtained according to the registration result. This information accurately marks the different cell population types and cell population image positions in the staining image. Based on such accurate annotation information training model, the model can more accurately identify and segment different cell populations, thereby improving the accuracy of cell population segmentation. Moreover, using the U-Net network structure for training, each convolution layer can effectively extract the image features of each slice of the HE staining image. The contraction path of U-Net gradually extracts features of different scales through multiple convolution and downsampling operations, which can effectively capture everything from the detailed features of local cells to the more macroscopic distribution features of cell populations. The expansion path can fuse these features and restore them to the original image resolution, so that the model can make full use of image features for cell population segmentation, further improving the segmentation accuracy.

[0062] Reference Figure 2 As shown, the present invention further specifically discloses an image segmentation method for a stained image, comprising:

[0063] Step S21: when there is an image segmentation request for a HE-stained image, determining a HE-stained image to be segmented based on the image segmentation request.

[0064] In this embodiment, when there is a request for image segmentation of a HE-stained image, the HE-stained image involved in the request is used as the HE-stained image to be segmented.

[0065] Step S22: Input the HE-stained image to be segmented into a preset image segmentation model, so as to perform tumor cell group labeling processing, T lymphocyte group labeling processing, B lymphocyte group labeling processing, and macrophage group labeling processing on the HE-stained image to be segmented through the preset image segmentation model, so as to segment the image area where each cell group is located, and output the corresponding tumor cell group category information and corresponding image area position information, T lymphocyte group category information and corresponding image area position information, B lymphocyte group category information and corresponding image area position information, and macrophage group category information and corresponding image area position information.

[0066] In this embodiment, the model identifies the characteristics of tumor cell clusters in the input HE-stained image to be segmented based on the learned feature patterns and associations. These features may include the unique nuclear morphology of tumor cells (such as irregular shape, large nuclear-cytoplasmic ratio, etc.), cytoplasmic staining characteristics, etc. Through a series of neural network operations, the model annotates the areas in the image that meet the characteristics of tumor cell clusters and determines which pixel areas belong to tumor cell clusters. Then, these areas marked as tumor cell clusters are segmented from the entire image, and their specific location range in the image is clarified, thereby achieving the segmentation of the image area where the tumor cell cluster is located.

[0067] T lymphocyte group annotation processing and region segmentation:

[0068] For the T lymphocyte population, the model also uses the knowledge it has learned to identify its characteristics. T lymphocytes usually have relatively small cell size, specific nuclear morphology, and specific staining patterns under HE staining.

[0069] These features are used to identify the T lymphocyte clusters in the image, and the pixel regions belonging to the T lymphocyte clusters are accurately labeled. Subsequently, these labeled regions are segmented to determine their location information in the image, thus completing the segmentation of the image region where the T lymphocyte clusters are located.

[0070] B lymphocyte group annotation processing and region segmentation:

[0071] B lymphocytes also have unique features in HE-stained images, such as different nucleus shapes and cytoplasmic staining. The model learns these features and accurately finds the pixel areas corresponding to B lymphocytes in the input image. After labeling, these areas are segmented from the image, their specific locations in the image are clarified, and the area segmentation of the image where the B lymphocytes are located is achieved.

[0072] Macrophage group annotation processing and regional segmentation:

[0073] Macrophages have some obvious characteristics, such as large cell volume, irregular cell shape, and specific staining performance under HE staining. The model identifies macrophages in the HE-stained image to be segmented based on these characteristics. Then it performs annotation processing, divides the pixel areas belonging to the macrophages, and determines the positions of these areas in the image, thus achieving the segmentation of the image area where the macrophages are located.

[0074] After completing the labeling and region segmentation of tumor cell groups, T lymphocyte groups, B lymphocyte groups, and macrophage groups, the preset image segmentation model will output the following corresponding information:

[0075] Tumor cell population information output:

[0076] Output the category information of the tumor cell cluster, clearly identify these segmented areas as belonging to the tumor cell cluster. At the same time, output the corresponding image area position information, detailing the specific position range of the tumor cell cluster in the image, such as giving the position of each pixel area belonging to the tumor cell cluster in the form of coordinates or relative position description.

[0077] T lymphocyte population information output:

[0078] Output the category information of T lymphocyte groups, indicating that these segmented areas are T lymphocyte groups. And output the location information of the corresponding image area, accurately present the specific location of the T lymphocyte group in the image, which is convenient for subsequent analysis of T lymphocyte distribution and other conditions.

[0079] B lymphocyte population information output:

[0080] Output the category information of B lymphocyte groups and the corresponding image area location information, clearly define the location range of B lymphocyte groups in the image, and provide data support for studying the distribution of B lymphocytes in tissues.

[0081] Macrophage population information output:

[0082] Finally, the category information of the macrophage population and the corresponding image area position information are output, so that the distribution position of the macrophage population in the HE-stained image to be segmented can be accurately understood, which is helpful for further analyzing the role of macrophages in related pathological processes.

[0083] The HE-stained image to be segmented is input into a preset image segmentation model for processing. The model is obtained by learning the relationship between the image features of each slice in the HE-stained image and the corresponding mIHC-stained image and the corresponding mask information. This means that the model has fully learned the features of different stained images and accurate cell group annotation information (mask information), and can accurately identify the characteristic patterns of each cell group in the HE-stained image, thereby achieving high-precision segmentation of the image area where each cell group is located.

[0084] It can be seen that by inputting the HE-stained slice image to be segmented into the preset image segmentation model for processing, the model is obtained by learning the relationship between the features of each slice image in the HE-stained image and the corresponding mIHC-stained image and the corresponding mask information. This means that the model has fully learned the features of different staining images and accurate cell group annotation information, and can accurately identify the characteristic patterns of each cell group in the HE-stained image, thereby achieving high-precision segmentation of the image area where each cell group is located.

[0085] Reference Figure 3 As shown, the present invention also discloses a device for training a stained image segmentation model, comprising:

[0086] The training set acquisition module 11 is used to acquire several pairs of HE staining images and mIHC staining images of the same tissue section, and perform pixel-by-pixel image registration processing of the staining modality on each pair of the HE staining images and the mIHC staining images, so as to acquire target mask information of the HE staining images according to the image registration results, so as to construct an image training set based on the HE staining images and the corresponding target mask information; wherein the target mask information is label information that annotates different cell population types and cell population image positions in the staining image;

[0087] A model training module 12 is used to train the U-Net network structure using the image training set so that each convolution layer of the U-Net network extracts the image features of each slice of the HE-stained image to obtain a corresponding feature map, and converts the feature map into initial probability prediction results of different cell populations;

[0088] The model updating module 13 is used to back-propagate and update the network parameters of the U-Net network according to the initial probability prediction result and the label information in the corresponding target mask information, so as to update the U-Net network and obtain the updated U-Net network as the preset image segmentation model.

[0089] It can be seen that the present application discloses obtaining several pairs of HE staining images and mIHC staining images of the same tissue slice, and performing pixel-by-pixel image registration processing of the staining modality on each pair of the HE staining images and the mIHC staining images, so as to obtain the target mask information of the HE staining image according to the image registration result, so as to construct an image training set based on the HE staining image and the corresponding target mask information; wherein the target mask information is label information that marks different cell population types and cell population image positions in the staining image; the U-Net network structure is trained using the image training set so that each convolutional layer of the U-Net network extracts the image features of each slice of the HE staining image to obtain a corresponding feature map, and converts the feature map into an initial probability prediction result of different cell populations; the network parameters of the U-Net network are back-propagated and updated according to the initial probability prediction result and the label information in the corresponding target mask information to update the U-Net network, and the updated U-Net network is obtained as a preset image segmentation model. It can be seen that by obtaining the HE staining image and mIHC staining image of the same tissue slice and performing pixel-by-pixel image registration processing, the information of the two staining images is fully utilized. The mIHC staining image can provide more detailed labeling information about different cell populations. After being registered with the HE staining image, the target mask information of the HE staining image can be accurately obtained according to the registration result. This information accurately marks the different cell population types and cell population image positions in the staining image. Based on such accurate annotation information training model, the model can more accurately identify and segment different cell populations, thereby improving the accuracy of cell population segmentation. Moreover, using the U-Net network structure for training, each convolution layer can effectively extract the image features of each slice of the HE staining image. The contraction path of U-Net gradually extracts features of different scales through multiple convolution and downsampling operations, which can effectively capture everything from the detailed features of local cells to the more macroscopic distribution features of cell populations. The expansion path can fuse these features and restore them to the original image resolution, so that the model can make full use of image features for cell population segmentation, further improving the segmentation accuracy.

[0090] Furthermore, the present application also discloses an electronic device. Figure 4 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the diagram cannot be regarded as any limitation on the scope of use of the present application.

[0091] Figure 4A schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the image segmentation model training method for stained images disclosed in any of the aforementioned embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0092] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0093] Among them, the processor 21 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 21 can be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 21 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 21 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 21 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0094] In addition, the memory 22, as a carrier for storing resources, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0095] Among them, the operating system 221 is used to manage and control the hardware devices and computer programs 222 on the electronic device 20, so as to realize the operation and processing of the processor 21 on the massive data 223 in the memory 22, which can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program that can be used to complete the image segmentation model training method of the dyed image performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program that can be used to complete other specific tasks. In addition to data received by the electronic device and transmitted from an external device, the data 223 can also include data collected by its own input and output interface 25, etc.

[0096] Furthermore, the present application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the image segmentation model training method for the stained image disclosed above is implemented. The specific steps of the method can refer to the corresponding contents disclosed in the above embodiments, and will not be repeated here.

[0097] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0098] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application. The steps of the method or algorithm described in conjunction with the embodiments disclosed herein can be implemented directly with hardware, a software module executed by a processor, or a combination of the two. The software module can be placed in a random access memory RAM (Random Access Memory), memory, read-only memory ROM (Read Only Memory), electrically programmable EPROM (Electrically Programmable Read Only Memory), electrically erasable programmable EEPROM (ElectricErasable Programmable Read Only Memory), register, hard disk, removable disk, CD-ROM (CompactDisc-Read Only Memory), or any other form of storage medium known in the technical field.

[0099] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0100] The scheme provided by the present invention is introduced in detail above. Specific examples are used in this article to illustrate the principle and implementation mode of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation mode and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A method for training an image segmentation model for a stained image, characterized in that: include: Acquire several pairs of HE staining images and mIHC staining images of the same tissue section, and perform pixel-by-pixel image registration processing of the staining modality on each pair of the HE staining images and the mIHC staining images, so as to obtain target mask information of the HE staining images according to the image registration results, and construct an image training set based on the HE staining images and the corresponding target mask information; wherein the target mask information is label information that annotates different cell population types and cell population image positions in the staining image; Using the image training set to train the U-Net network structure, so that each convolution layer of the U-Net network extracts the image features of each slice of the HE-stained image to obtain a corresponding feature map, and converts the feature map into initial probability prediction results of different cell populations; The network parameters of the U-Net network are back-propagated and updated according to the initial probability prediction result and the label information in the corresponding target mask information to update the U-Net network and obtain the updated U-Net network as the preset image segmentation model.

2. The image segmentation model training method for stained images according to claim 1, characterized in that: Acquire HE staining images and mIHC staining images of the same tissue section, including: After HE staining is performed on a tissue section, the HE-stained tissue section is scanned by a scanning machine to obtain a HE-stained image of the tissue section; The HE-stained tissue section is decolorized and mIHC-stained, and then scanned by the scanning machine to obtain an mIHC-stained image of the same tissue section.

3. The image segmentation model training method for stained images according to claim 2, characterized in that: The step of performing pixel-by-pixel image registration processing of the staining modality on each pair of the HE staining image and the mIHC staining image to obtain target mask information of the HE staining image according to the image registration result includes: Performing structural registration processing on the HE staining image and the mIHC staining image using a preset multi-cell group annotation registration algorithm to obtain a registered HE staining image; The visualization information of the mIHC staining image is used as mask information of the image area corresponding to the HE staining image after registration to obtain the target mask information.

4. The image segmentation model training method for stained images according to claim 1, characterized in that: The method of using the image training set to train the U-Net network structure so that each convolution layer of the U-Net network extracts the image features of each slice of the HE-stained image to obtain a corresponding feature map, and converting the feature map into initial probability prediction results of different cell populations includes: Constructing a U-Net network structure including a contraction path and an expansion path; wherein the contraction path is provided with a plurality of convolution blocks consisting of two convolution layers and a ReLU activation function and a maximum pooling layer for downsampling; and the expansion path is provided with a deconvolution layer for upsampling and an output layer including a convolution layer and a softmax activation function; Inputting the HE stained image into the U-Net network structure, extracting image features of each slice of the image through the convolution layer of the contraction path of the U-Net network structure to obtain a corresponding initial feature map, performing nonlinear feature processing on the initial feature map using the ReLU activation function to obtain a process feature map, and downsampling the process feature map through the maximum pooling layer to obtain a target feature map; Inputting the target feature map into the extension path of the U-Net network structure to perform upsampling processing on the target feature map through the deconvolution layer, so as to gradually restore the feature map size of the target feature map to the size of the HE stained image, so as to obtain a restored target feature map; The restored target feature map is input into the output layer, so as to convert the restored target feature map into probability predictions for different cell groups through the output layer, thereby obtaining an initial probability prediction result of semantic segmentation.

5. The image segmentation model training method for stained images according to claim 4, characterized in that: The method of back-propagating and updating the network parameters of the U-Net network according to the initial probability prediction result and the label information in the corresponding target mask information to update the U-Net network and obtain the updated U-Net network as the preset image segmentation model includes: Using a preset loss function to calculate the loss function of the initial probability prediction result and the label information in the corresponding target mask information to obtain a corresponding loss function value; The back propagation algorithm and the loss function value are used to calculate the gradient information of different network structures from the output layer in sequence, so as to update the U-Net network according to the gradient information, and obtain the updated U-Net network as the preset image segmentation model.

6. The image segmentation model training method for stained images according to claim 5, characterized in that: The extracting image features of each slice of the image through the convolution layer of the contraction path of the U-Net network structure includes: The shape features, size features, and texture features of cells in the image are extracted through the convolutional layer of the contraction path of the U-Net network structure to obtain the features of each slice image.

7. The image segmentation model training method for stained images according to claim 5, characterized in that: After obtaining the updated U-Net network as the preset image segmentation model, the method further includes: When there is an image segmentation request for a HE-stained image, determining the HE-stained image to be segmented based on the image segmentation request; The HE-stained image to be segmented is input into a preset image segmentation model, so as to perform tumor cell group labeling processing, T lymphocyte group labeling processing, B lymphocyte group labeling processing, and macrophage group labeling processing on the HE-stained image to be segmented through the preset image segmentation model, so as to segment the image area where each cell group is located, and output the corresponding tumor cell group category information and corresponding image area position information, T lymphocyte group category information and corresponding image area position information, B lymphocyte group category information and corresponding image area position information, and macrophage group category information and corresponding image area position information.

8. An image segmentation model training device for stained images, characterized in that: include: A training set acquisition module, used to acquire several pairs of HE staining images and mIHC staining images of the same tissue section, and perform pixel-by-pixel image registration processing of the staining modality on each pair of the HE staining images and the mIHC staining images, so as to acquire target mask information of the HE staining images according to the image registration results, so as to construct an image training set based on the HE staining images and the corresponding target mask information; wherein the target mask information is label information that annotates different cell population types and cell population image positions in the staining image; A model training module, used to train the U-Net network structure using the image training set, so that each convolution layer of the U-Net network extracts the image features of each slice of the HE-stained image to obtain a corresponding feature map, and converts the feature map into an initial probability prediction result of different cell populations; A model updating module is used to back-propagate and update the network parameters of the U-Net network according to the initial probability prediction result and the label information in the corresponding target mask information, so as to update the U-Net network and obtain the updated U-Net network as the preset image segmentation model.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor is used to execute the computer program to implement the steps of the image segmentation model training method for stained images as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Used to store computer programs; wherein, when the computer program is executed by a processor, the steps of the image segmentation model training method for stained images as described in any one of claims 1 to 7 are implemented.

Images