Image segmentation model training method and device for stained images, equipment and medium

Abstract

The application discloses a method and device for training an image segmentation model of a stained image, equipment and a medium, relates to the technical field of computers, and comprises the following steps: obtaining a plurality of pairs of HE stained images and mIHC stained images of the same tissue section, performing pixel-by-pixel image registration processing on each pair of stained images, obtaining target mask information of the HE stained images according to the image registration result, and constructing an image training set based on the HE stained images and the corresponding target mask information; training a U-Net network structure by using the image training set, so that each convolutional layer of the U-Net network extracts slice image features of the HE stained images and constructs a feature map, the feature map is converted into initial probability prediction results of different cell groups, and the network parameters of the U-Net network are updated through backward propagation according to the initial probability prediction results and the target mask information, so that the U-Net network is updated, and the updated U-Net network is used as a preset image segmentation model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a staining image image segmentation model training method and device, equipment and medium. BACKGROUND

[0002] H&E staining (Hematoxylin and eosin stains) and IHC staining (Immunohistochemistry) are widely used in pathology to observe the morphological features of tissues and cells. In the prior art, information related to pathology needs to be extracted from tissue stained sections, and various cell types (tumor cells and T, B and macrophage cell groups in immune cells) are segmented out by a semantic segmentation model. In order to train a semantic segmentation model, a data set needs to be obtained, which mainly includes two parts: (1) H&E stained sections; (2) corresponding labels of different cell types. The prior art mainly relies on traditional manual annotation or registration of adjacent sections after staining adjacent sections into H&E sections and IHC sections to obtain annotation information of various cell types by corresponding IHC sections. For traditional manual annotation, the main problem is that it is too time-consuming and labor-intensive, and accurate pixel-level annotation cannot be obtained. For the method of registering adjacent sections, since adjacent sections are not completely consistent, the accuracy of registration is greatly reduced, so accurate pixel-level annotation cannot be obtained.

[0003] In summary, how to obtain accurate HE image annotation information to train an accurate image segmentation model based on HE images and annotation information is a technical problem to be solved in the art. SUMMARY

[0004] Therefore, the purpose of the present application is to provide a staining image image segmentation model training method and device, equipment and medium, which can obtain accurate HE image annotation information to train an accurate image segmentation model based on HE images and annotation information. The specific scheme is as follows:

[0005] In a first aspect, the present application discloses a staining image image segmentation model training method, comprising:

[0006] Obtain a plurality of pairs of HE staining images and mIHC staining images of the same tissue section, and perform pixel-by-pixel image registration processing on each pair of HE staining images and mIHC staining images to obtain target mask information of the HE staining images according to the image registration result, and construct an image training set based on the HE staining images and the corresponding target mask information; wherein the target mask information is label information marking different cell group types and cell group image positions in the staining image.

[0007] training the U-Net network structure by using the image training set, so that each convolutional layer of the U-Net network extracts slice image features of the HE staining image to obtain a corresponding feature map, and converts the feature map into an initial probability prediction result of different cell groups;

[0008] updating the network parameters of the U-Net network according to the initial probability prediction result and the label information in the corresponding target mask information, to update the U-Net network, and obtaining an updated U-Net network as a preset image segmentation model.

[0009] Optionally, the HE staining image and the mIHC staining image of the same tissue slice are obtained, including:

[0010] After HE staining of a tissue slice, the HE stained tissue slice is scanned by a scanning machine to obtain the HE staining image of the tissue slice;

[0011] The HE stained tissue slice is subjected to decolorization treatment and mIHC staining, and is scanned by the scanning machine to obtain the mIHC staining image of the same tissue slice.

[0012] Optionally, the pixel-by-pixel image registration processing of each pair of the HE staining image and the mIHC staining image in the staining modalities is performed to obtain target mask information of the HE staining image according to the image registration result, including:

[0013] The HE staining image and the mIHC staining image are subjected to structural registration processing by using a preset multi-cell group annotation registration algorithm to obtain a registered HE staining image;

[0014] The visual information of the mIHC staining image is used as mask information of a corresponding image region of the registered HE staining image to obtain the target mask information.

[0015] Optionally, the U-Net network structure is trained by using the image training set, so that each convolutional layer of the U-Net network extracts slice image features of the HE staining image to obtain a corresponding feature map, and converts the feature map into an initial probability prediction result of different cell groups, including:

[0016] A U-Net network structure including a contraction path and an expansion path is constructed; wherein a plurality of convolutional blocks composed of two convolutional layers and a ReLU activation function and a maximum pooling layer for down sampling are arranged in the contraction path; a deconvolutional layer for up sampling and an output layer including a convolutional layer and a softmax activation function are arranged in the expansion path.

[0017] inputting the HE staining image into the U-Net network structure, extracting slice image features of the image through the convolutional layers of the contraction path of the U-Net network structure to obtain corresponding initial feature maps, performing nonlinear feature processing on the initial feature maps by using the ReLU activation function to obtain process feature maps, and performing down-sampling processing on the process feature maps through the max-pooling layer to obtain a target feature map;

[0018] inputting the target feature map into the expansion path of the U-Net network structure, performing up-sampling processing on the target feature map through the deconvolutional layer to gradually restore the feature map size of the target feature map to the size of the HE staining image to obtain a restored target feature map;

[0019] inputting the restored target feature map into the output layer to convert the restored target feature map into a probability prediction of different cell groups through the output layer to obtain an initial probability prediction result of semantic segmentation.

[0020] Optionally, the network parameters of the U-Net network are updated by back propagation according to the initial probability prediction result and the label information in the corresponding target mask information to update the U-Net network to obtain an updated U-Net network as a preset image segmentation model, including:

[0021] the initial probability prediction result and the label information in the corresponding target mask information are calculated by using a preset loss function to obtain a corresponding loss function value;

[0022] the gradient information of different network structures is calculated from the output layer by using a back propagation algorithm and the loss function value, and the U-Net network is updated according to the gradient information to obtain an updated U-Net network as a preset image segmentation model.

[0023] Optionally, the convolutional layers of the contraction path of the U-Net network structure extract slice image features of the image, including:

[0024] the shape features, size features and texture features of the cells in the image are extracted through the convolutional layers of the contraction path of the U-Net network structure to obtain the slice image features.

[0025] Optionally, after the updated U-Net network is obtained as a preset image segmentation model, the method further includes:

[0026] when there is an image segmentation request of the HE staining image, determining a HE staining image to be segmented based on the image segmentation request.

[0027] input the HE staining image to be segmented into a preset image segmentation model, perform tumor cell class group annotation processing, T lymphocyte class group annotation processing, B lymphocyte class group annotation processing, and macrophage class group annotation processing on the HE staining image to be segmented by the preset image segmentation model, segment the image region where each cell class group is located, and output corresponding tumor cell class group category information and corresponding image region position information, T lymphocyte class group category information and corresponding image region position information, B lymphocyte class group category information and corresponding image region position information, and macrophage class group category information and corresponding image region position information.

[0028] In a second aspect, the present application discloses a staining image image segmentation model training device, comprising:

[0029] The training set acquisition module is configured to acquire a plurality of pairs of HE staining images and mIHC staining images of the same tissue section, perform pixel-by-pixel image registration processing of the staining modalities on each pair of the HE staining image and the mIHC staining image, acquire target mask information of the HE staining image according to the image registration result, and construct an image training set based on the HE staining image and the corresponding target mask information; wherein the target mask information is label information marking different cell group types and cell group image positions in the staining image.

[0030] The model training module is configured to train a U-Net network structure using the image training set, so that each convolutional layer of the U-Net network extracts features of each slice image of the HE staining image to obtain a corresponding feature map, and converts the feature map into an initial probability prediction result of different cell groups.

[0031] The model updating module is configured to perform backpropagation updating on network parameters of the U-Net network according to the initial probability prediction result and the label information in the corresponding target mask information, to update the U-Net network, and obtain an updated U-Net network as a preset image segmentation model.

[0032] In a third aspect, the present application discloses an electronic device, comprising:

[0033] The memory is configured to save a computer program.

[0034] The processor is configured to execute the computer program to implement the steps of the staining image image segmentation model training method disclosed above.

[0035] In a fourth aspect, the present application discloses a computer readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the steps of the above disclosed image segmentation model training method for stained images.

[0036] It can be seen that the present application discloses an image segmentation model training method for stained images, comprising: obtaining a plurality of pairs of HE stained images and mIHC stained images of the same tissue section, and performing pixel-by-pixel image registration processing on each pair of the HE stained image and the mIHC stained image in the same tissue section to obtain target mask information of the HE stained image according to the image registration result, and constructing an image training set based on the HE stained image and the corresponding target mask information; wherein the target mask information is label information marking different cell population types and cell population image positions in the stained image; training a U-Net network structure using the image training set, so that each convolutional layer of the U-Net network extracts slice image features of the HE stained image to obtain corresponding feature maps, and converts the feature maps into initial probability prediction results of different cell populations; and updating network parameters of the U-Net network according to the initial probability prediction results and the label information in the corresponding target mask information, to update the U-Net network to obtain an updated U-Net network as a preset image segmentation model. As can be seen, by obtaining the HE stained image and the mIHC stained image of the same tissue section and performing pixel-by-pixel image registration processing, the information of the two types of stained images is fully utilized. The mIHC stained image can provide more detailed label information about different cell populations. After registration with the HE stained image, the target mask information of the HE stained image can be accurately obtained according to the registration result, which accurately marks different cell population types and cell population image positions in the stained image. Training the model based on such accurate label information enables the model to more accurately identify and segment different cell populations, improving the accuracy of cell population segmentation. Moreover, the U-Net network structure is used for training, and each convolutional layer thereof can effectively extract slice image features of the HE stained image. The contraction path of the U-Net gradually extracts features of different scales through multiple convolution and down-sampling operations, effectively capturing features from local cell details to more macroscopic cell population distribution features. The expansion path can fuse these features and restore them to the original image resolution, so that the model can fully utilize image features for cell population segmentation, further improving segmentation accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description only constitute a part of the embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of the provided drawings.

[0038] Figure 1 A flow chart of a training method of an image segmentation model of a stained image disclosed in the present application;

[0039] Figure 2 A flow chart of an image segmentation method of a stained image disclosed in the present application;

[0040] Figure 3 A schematic structural view of a training device of an image segmentation model of a stained image disclosed in the present application;

[0041] Figure 4 A structural view of an electronic device disclosed in the present application. DETAILED DESCRIPTION

[0042] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments only constitute a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0043] H&E staining and IHC staining are widely used in pathology for observing morphological features of tissues and cells. In the prior art, information related to pathology needs to be extracted from tissue stained sections, and various cell types (tumor cells and T, B and macrophage cell groups in immune cells) are segmented out by a semantic segmentation model. In order to train a semantic segmentation model, a data set needs to be obtained, which mainly includes two parts: (1) H&E stained sections; (2) corresponding labels of different cell types. The prior art mainly relies on traditional manual annotation or registration of adjacent sections after staining adjacent sections into H&E sections and IHC sections, and obtains annotation information of various cell types by corresponding IHC sections. For traditional manual annotation, the main problems are too time-consuming and labor-intensive, and accurate pixel-level annotation cannot be obtained. For the method of registering adjacent sections, since adjacent sections are not completely consistent, the accuracy of registration is greatly reduced, so accurate pixel-level annotation cannot be obtained.

[0044] To this end, the present application provides an image segmentation model training scheme for stained images, which can obtain accurate HE image annotation information, and train an accurate image segmentation model based on HE images and annotation information, which is a technical problem to be solved in the art.

[0045] With reference to Figure 1 As shown in the drawings, the embodiments of the present application disclose an image segmentation model training method for stained images, comprising:

[0046] Step S11: Obtain a plurality of pairs of HE stained images and mIHC stained images of the same tissue section, and perform pixel-by-pixel image registration processing of the staining mode on each pair of the HE stained image and the mIHC stained image, to obtain target mask information of the HE stained image according to the image registration result, and construct an image training set based on the HE stained image and the corresponding target mask information; wherein the target mask information is label information marking different cell population types and cell population image positions in the stained image.

[0047] In this embodiment, after HE staining of a tissue section, the HE stained tissue section is scanned by a scanning machine to obtain an HE stained image of the tissue section. The HE stained tissue section is subjected to decolorization processing and mIHC staining, and scanned by the scanning machine to obtain an mIHC stained image of the same tissue section. It can be understood that a large number of HE stained images are collected as part of the training images of the image training set. These images should cover various different tissue sample conditions, including different cell distribution, morphology and other characteristics, to ensure that the model can learn a variety of sample patterns, such as images of tissue sections from different patients and different lesion sites after HE staining. Then the same tissue section of the above part of the training images is subjected to decolorization processing and mIHC staining to obtain an mIHC stained image of the same tissue section. Specifically, during mIHC staining, different cell types or cell markers can be labeled, so that annotation information about different cell populations, i.e. label information, can be obtained at the same time during this process. The annotation information marks the specific positions of different cell populations (such as tumor cells, T cells, B cells, macrophages, etc.) in each section, and the value of each pixel point in the annotation information represents the cell type to which it belongs. These annotation information is also subjected to feature processing, such as conversion into a specific vector representation, so as to better match and associate with image features in the subsequent operation.

[0048] In this embodiment, the preset multi-cell group annotation registration algorithm is used to perform structural registration processing on the HE staining image and the mIHC staining image to obtain a registered HE staining image; and the visualization information of the mIHC staining image is used as mask information of a corresponding image region of the registered HE staining image to obtain the target mask information. It can be understood that, for each pair of HE staining image and mIHC staining image, pixel-by-pixel image registration processing is performed to find initial matching points in the HE staining image and the mIHC staining image. These matching points can be specific positions of cells (such as center positions of cell groups, positions of single cells with obvious features, etc.). For example, the approximate center position of a tumor cell group in the HE staining image is found, and then the center position of the corresponding labeled tumor cell group in the mIHC staining image is found as an initial matching point. Based on the determined initial matching point, the transformation parameters between the two images are calculated to achieve registration. Since the HE staining image and the mIHC staining image are of the same tissue section, in the registration process, the image transformation only includes translation and rotation.

[0049] For translation transformation, the displacement amount of the two images in the horizontal and vertical directions is calculated so that the matching points can be aligned in space. For example, if the center position of a cell group in the HE staining image and the center position of the corresponding cell group in the mIHC staining image differ by a certain number of pixels in the horizontal direction, the horizontal translation parameter can be obtained by calculating the difference.

[0050] For rotation transformation, the angle of rotation of the two images is calculated so that the directions of the images can be matched. For example, by analyzing the arrangement direction of cells and other features, the angle of rotation between the two images is determined to achieve the alignment of the directions of cell groups and other structures.

[0051] In order to obtain more accurate registration results, the calculated transformation parameters need to be optimized. Least squares method, gradient descent method and other optimization algorithms can be used to adjust the transformation parameters with the goal of minimizing the difference between the two images (such as the distance based on feature points, the difference of cell group annotation, etc.). Through continuous iteration and optimization, the two images can be optimally matched at the pixel level, and the output registered image is the registered HE staining image.

[0052] After completing the pixel-level registration of the HE staining image and the mIHC staining image, the existing cell group annotation information (different cell types or cell markers labeled in the mIHC staining process) in the mIHC staining image is used to correspond to the registered HE staining image. Because the two images have been registered at the pixel level, the cell group annotation in the mIHC staining image can be accurately transferred to the HE staining image.

[0053] According to the cell class group annotation information transferred to the HE staining image, a mask is generated. Each pixel point in the mask corresponds to a position in the HE staining image, and the value thereof represents a cell type to which the position belongs. For example, if a certain pixel point belongs to a tumor cell type in the transferred annotation, the value of the pixel point in the mask is set to a specific value representing a tumor cell. In this way, the mask annotated with the position information and the category information of different cell class groups in the HE staining image is obtained, and can be used for subsequent image segmentation and other related operations.

[0054] Step S12: training the U-Net network structure by using the image training set, so that each convolutional layer of the U-Net network extracts the feature of each slice image of the HE staining image to obtain a corresponding feature map, and converts the feature map into an initial probability prediction result of different cell class groups.

[0055] In this embodiment, a U-Net network structure including a contracting path and an expanding path is constructed; wherein a plurality of convolutional blocks composed of two convolutional layers and a ReLU activation function and a maximum pooling layer for downsampling are arranged in the contracting path; and a deconvolutional layer for upsampling and an output layer including a convolutional layer and a softmax activation function are arranged in the expanding path. It can be understood that the U-Net network structure is built, including its contracting path (Contracting Path) and expanding path (Expanding Path). In the contracting path, a plurality of convolutional blocks composed of two 3x3 convolutional layers and a ReLU activation function, and a 2x2 maximum pooling layer for downsampling are arranged; in the expanding path, a transposed convolutional (deconvolutional) layer is arranged for upsampling, and a cropping and splicing operation with the feature map of the contracting path is performed at an appropriate position, and finally an output layer is arranged as a 1x1 convolutional layer with a softmax activation function. In medical image segmentation, U-Net learns the feature representation of different cells in the input image (such as the HE staining image) through the convolutional layers and the downsampling operation of the contracting path. For example, for tumor cells, the network learns their irregular shape, high nuclear-cytoplasmic ratio, and other features; for immune cells, it learns their relatively small size, specific staining patterns, and other features. These feature representations are automatically learned by the network and do not require manual design of feature extractors. The design of the contracting path and the expanding path enables U-Net to fuse multi-scale information. In the contracting path, the network learns global and abstract features; in the expanding path, by splicing the feature maps of the contracting path, the network can combine these global features with local detailed features. For example, when segmenting cells, the network can not only use the global features such as the shape of the whole cell to determine the cell type, but also use the detailed features such as the nuclear structure inside the cell to improve the accuracy of segmentation.

[0056] In this embodiment, the HE stained image is input into the U-Net network structure, the convolutional layers of the contraction path of the U-Net network structure are used to extract the slice image features of the image, to obtain the corresponding initial feature map, the initial feature map is processed by the ReLU activation function for nonlinear feature processing, to obtain the process feature map, and the process feature map is processed by the max pooling layer for down-sampling processing, to obtain the target feature map; the target feature map is input into the expansion path of the U-Net network structure, the target feature map is processed by the deconvolutional layer for up-sampling processing, the feature map size of the target feature map is gradually restored to the size of the HE stained image, to obtain the restored target feature map; and the restored target feature map is input into the output layer, the restored target feature map is converted into the probability prediction of different cell groups by the output layer, to obtain the initial probability prediction result of semantic segmentation. It can be understood that the HE stained image in the training set is then input into the U-Net network structure, and then in the U-Net network structure, the input HE stained image first passes through the contraction path. The convolutional layers of the contraction path of the U-Net network structure are used to extract the shape features, size features and texture features of the cells in the image, to obtain the slice image features. In this way, the ReLU activation function is used to increase the nonlinear expression ability after each convolutional layer. With the down-sampling, the receptive field of the network can be expanded, so that the network can learn more macro features, while the calculation amount and the number of parameters are reduced. In this way, the network gradually extracts more macro features, and the number of feature channels gradually increases. Then, through the expansion path, the feature map size is gradually restored to the size close to the input image by transposed convolution for up-sampling, and the feature maps of the contraction path are spliced in the process to fuse multi-scale information, so that the network can use global features and combine detail features for segmentation. Finally, through the 1x1 convolutional layer and the softmax activation function of the output layer, the feature map generated by the expansion path is converted into the probability prediction of different cell groups, to obtain the preliminary result of semantic segmentation, that is, the initial probability prediction result. It should be noted that the output layer of the U-Net is a 1x1 convolutional layer, which converts the target feature map generated by the expansion path into the required segmentation result. For example, if different cell groups in the image are to be segmented, the number of channels of the output layer is the same as the number of types of cell groups, and each channel represents the probability distribution of a cell group. Through a softmax activation function, these probability distributions can be converted into the probability of each pixel belonging to different cell groups, thereby obtaining the final initial probability prediction result.

[0057] Step S13: updating the network parameters of the U-Net network according to the initial probability prediction result and the label information in the corresponding target mask information, so as to update the U-Net network, and obtaining the updated U-Net network as a preset image segmentation model.

[0058] In this embodiment, the preset loss function is used to calculate the loss function value of the initial probability prediction result and the label information in the corresponding target mask information. The gradient information of different network structures is calculated in turn from the output layer by using the back propagation algorithm and the loss function value, so as to update the U-Net network according to the gradient information, and obtain the updated U-Net network as a preset image segmentation model. It can be understood that the initial probability prediction result obtained by forward propagation is compared with the corresponding mask information (i.e. the real annotation information), and the loss function is calculated. The commonly used preset loss function is, for example, the cross-entropy loss function, which can measure the difference between the prediction result and the real result. For example, if the probability of a pixel in the prediction result is predicted as a tumor cell and does not match the actual cell type (assuming it is a T cell) in the mask, the value of the loss function will reflect this difference. According to the calculated loss function, the gradient of each neuron is calculated in turn from the output layer to the input layer by using the back propagation algorithm. These gradients represent the influence of each parameter (such as the weights and biases of the convolutional layer) on the loss function. Then, the parameters of the network are updated according to these gradients, and an optimization algorithm such as the stochastic gradient descent algorithm and its variants (such as Adagrad (Adaptive Gradient Algorithm), Adadelta (Adaptive Delta), Adam (Adaptive Moment Estimation), etc.) is used to adjust the parameters, so that the network can output a segmentation result closer to the real mask information when the same or similar HE staining image is input next time. The above input data, forward propagation, loss function calculation, back propagation and parameter updating process are repeated for the HE staining images of the entire training data set for multiple rounds of iterative training. In each round of training, the network adjusts its parameters according to the new gradient information, and continuously optimizes the segmentation ability of different cell groups. After a certain number of training rounds, the trained model needs to be evaluated. A part of the validation data set (which also consists of HE staining images and corresponding mask information, but does not participate in the training process) can be used to input the HE staining images of the validation data set into the trained model to obtain the predicted segmentation result, and then compare it with the corresponding mask information. The performance of the model is measured by calculating the accuracy, recall rate, F1 value and other evaluation indicators. If the model performance does not meet the expectations, the training can be continued, and the training parameters (such as learning rate, training rounds, etc.) can be adjusted until a satisfactory trained semantic segmentation model is obtained as a preset image segmentation model.

[0059] The untrained semantic segmentation model (U-Net network) first learns the visual features of the HE-stained slices in the dataset. In the contracting path, these features are extracted through a series of 3x3 convolutional layers. For example, for cells in the HE-stained slices, the model learns the shape, size, texture, and other features of the cells. The ReLU activation function after each convolutional layer increases the network's non-linear expression ability, allowing it to learn more complex cell feature patterns. At the same time, the model learns the mask information labeled with different cell types corresponding to the HE-stained slices. These masks contain the location and class information of each cell in the image, which is the goal of the model's learning. For example, the mask clearly identifies the specific locations of tumor cells, T cells, B cells, and macrophages in the HE-stained image.

[0060] U-Net is an end-to-end neural network that automatically learns all parameters from the input raw image to the output segmentation result through training data. During the training process, the network's parameters are adjusted according to the loss function (such as the cross-entropy loss function, which measures the difference between the segmentation result and the true label) through the backpropagation algorithm. In this way, the network can continuously optimize its parameters to adapt to different medical image segmentation tasks, such as accurately segmenting tumor cells, T cells, B cells, and macrophages in HE-stained images. Therefore, the pre-set image segmentation model obtained after training is a model that learns the correlation between the features of each slice image in the HE-stained image and the corresponding mask information in the corresponding mIHC-stained image; the mask information includes the location information and class information of each cell group in the HE-stained image.

[0061] It can be seen that the present application discloses a training method of an image segmentation model of a stained image, comprising: acquiring a plurality of pairs of HE stained images and mIHC stained images of the same tissue section, and performing pixel-by-pixel image registration processing of the staining mode on each pair of the HE stained image and the mIHC stained image to acquire target mask information of the HE stained image according to the image registration result, and constructing an image training set based on the HE stained image and the corresponding target mask information; wherein the target mask information is label information marking different cell group types and cell group image positions in the stained image; training a U-Net network structure using the image training set so that each convolutional layer of the U-Net network extracts slice image features of the HE stained image to obtain corresponding feature maps, and converts the feature maps into initial probability prediction results of different cell groups; and updating network parameters of the U-Net network according to the initial probability prediction results and the label information in the corresponding target mask information to update the U-Net network to obtain an updated U-Net network as a preset image segmentation model. As can be seen, by acquiring the HE stained image and the mIHC stained image of the same tissue section and performing pixel-by-pixel image registration processing, the information of the two stained images is fully utilized. The mIHC stained image can provide more detailed marking information about different cell groups, and after registration with the HE stained image, the target mask information of the HE stained image can be accurately acquired according to the registration result, which accurately marks different cell group types and cell group image positions in the stained image. Training the model based on such accurate marking information enables the model to more accurately identify and segment different cell groups, improving the accuracy of cell group segmentation. Moreover, the U-Net network structure is used for training, and each convolutional layer thereof can effectively extract slice image features of the HE stained image. The contraction path of the U-Net gradually extracts features of different scales, from local cell details to more macroscopic cell group distribution features, which can be effectively captured. The expansion path can fuse these features and restore them to the original image resolution, so that the model can fully utilize image features for cell group segmentation, further improving segmentation accuracy.

[0062] Referring to Figure 2 The present application also specifically discloses an image segmentation method of a stained image, comprising:

[0063] Step S21: when there is an image segmentation request of an HE stained image, determining a HE stained image to be segmented based on the image segmentation request.

[0064] In this embodiment, when there is an image segmentation request of an HE stained image, the HE stained image involved in the request is taken as a HE stained image to be segmented.

[0065] Step S22: input the HE staining image to be segmented into a preset image segmentation model, perform tumor cell class group annotation processing, T lymphocyte class group annotation processing, B lymphocyte class group annotation processing, and macrophage class group annotation processing on the HE staining image to be segmented by using the preset image segmentation model, segment the image region where each cell class group is located, and output corresponding tumor cell class group category information and corresponding image region position information, T lymphocyte class group category information and corresponding image region position information, B lymphocyte class group category information and corresponding image region position information, and macrophage class group category information and corresponding image region position information.

[0066] In this embodiment, the model identifies the characteristics of the tumor cell class group in the input HE staining image to be segmented based on the characteristic patterns and correlation relationships learned by the model. These characteristics can include the unique nucleus morphology of tumor cells (such as irregular shape, large nucleus-cytoplasm ratio, etc.), the staining characteristics of the cytoplasm, etc. Through a series of neural network operations, the model performs annotation processing on the regions in the image that meet the characteristics of the tumor cell class group, and determines which pixel regions belong to the tumor cell class group. Then, the regions annotated as the tumor cell class group are segmented from the entire image to determine their specific position range in the image, thereby achieving segmentation of the image region where the tumor cell class group is located.

[0067] T lymphocyte class group annotation processing and region segmentation:

[0068] For the T lymphocyte class group, the model also identifies its characteristics based on the learned knowledge. T lymphocytes usually have relatively small cell volume, specific cell nucleus morphology, and specific staining patterns under HE staining, etc.

[0069] Using these characteristics, the T lymphocyte class group in the image is identified and annotated, and the pixel regions belonging to the T lymphocyte class group are accurately annotated. Subsequently, the annotated regions are segmented to determine their position information in the image, and the segmentation of the image region where the T lymphocyte class group is located is completed.

[0070] B lymphocyte class group annotation processing and region segmentation:

[0071] The B lymphocyte class group also has its unique characteristics in the HE staining image, such as different cell nucleus shapes and cytoplasm staining conditions, etc. The model learns these characteristics and accurately finds the pixel regions corresponding to the B lymphocyte class group in the input image. After annotation processing, these regions are segmented from the image to determine their specific positions in the image, and the segmentation of the image region where the B lymphocyte class group is located is achieved.

[0072] Macrophage class group annotation processing and region segmentation:

[0073] The macrophage cell group has some obvious characteristics, such as larger cell volume, irregular cell shape, and specific staining performance under HE staining, etc. The model identifies the macrophage cell group in the HE-stained image to be segmented according to these characteristics. Then it is labeled and processed to divide the pixel area belonging to the macrophage cell group, and the location of these areas in the image is determined to realize the segmentation of the image area where the macrophage cell group is located.

[0074] After completing the labeling and region segmentation of tumor cell groups, T lymphocyte groups, B lymphocyte groups, and macrophage groups, the preset image segmentation model will output the following corresponding information:

[0075] Tumor cell group information output:

[0076] The class information of the tumor cell group is output, which clearly identifies that these segmented regions belong to the tumor cell group. At the same time, the corresponding image region position information is output, which details the specific position range of the tumor cell group in the image, such as giving the position of each pixel region belonging to the tumor cell group in the form of coordinates or relative position description.

[0077] T lymphocyte group information output:

[0078] The class information of the T lymphocyte group is output, indicating that these segmented regions are T lymphocyte groups. The corresponding image region position information is also output, which accurately presents the specific position of the T lymphocyte group in the image, facilitating subsequent analysis of the distribution of T lymphocytes, etc.

[0079] B lymphocyte group information output:

[0080] The class information of the B lymphocyte group and the corresponding image region position information are output, which clearly defines the position range of the B lymphocyte group in the image, providing data support for studying the distribution of B lymphocytes in the tissue.

[0081] Macrophage group information output:

[0082] Finally, the class information and corresponding image region position information of the macrophage group are output, so that the distribution of the macrophage group in the HE-stained image to be segmented can be accurately understood, which is helpful for further analyzing the role of macrophages in related pathological processes.

[0083] The HE staining image to be segmented is input into a preset image segmentation model for processing. The model is obtained by learning the correlation between the features of each slice image in the HE staining image and the corresponding mIHC staining image and the corresponding mask information. This means that the model has fully learned the features of different staining images and accurate cell group annotation information (mask information), can accurately identify the feature patterns of each cell group in the HE staining image, and thus realizes high-precision segmentation of the image region where each cell group is located.

[0084] As can be seen, by inputting the HE staining slice image to be segmented into a preset image segmentation model for processing, the model is obtained by learning the correlation between the features of each slice image in the HE staining image and the corresponding mIHC staining image and the corresponding mask information. This means that the model has fully learned the features of different staining images and accurate cell group annotation information, can accurately identify the feature patterns of each cell group in the HE staining image, and thus realizes high-precision segmentation of the image region where each cell group is located.

[0085] Referring to Figure 3 The application also discloses a training device of an image segmentation model of a staining image, which comprises:

[0086] The training set acquisition module 11 is configured to acquire a plurality of pairs of HE staining images and mIHC staining images of the same tissue slice, perform pixel-by-pixel image registration processing of the staining modalities on each pair of HE staining image and mIHC staining image, acquire target mask information of the HE staining image according to the image registration result, and construct an image training set based on the HE staining image and the corresponding target mask information. The target mask information is label information marking different cell group types and cell group image positions in the staining image.

[0087] The model training module 12 is configured to train a U-Net network structure using the image training set, so that each convolutional layer of the U-Net network extracts the features of each slice image of the HE staining image to obtain a corresponding feature map, and converts the feature map into an initial probability prediction result of different cell groups.

[0088] The model updating module 13 is configured to perform backpropagation updating on the network parameters of the U-Net network according to the initial probability prediction result and the label information in the corresponding target mask information, update the U-Net network, and obtain an updated U-Net network as a preset image segmentation model.

[0089] It can be seen that the application discloses obtaining a plurality of pairs of HE staining images and mIHC staining images of the same tissue slice, performing pixel-by-pixel image registration processing on each pair of the HE staining image and the mIHC staining image, obtaining target mask information of the HE staining image according to an image registration result, and constructing an image training set based on the HE staining image and corresponding target mask information; wherein the target mask information is label information marking different cell group types and cell group image positions in the staining image; a U-Net network structure is trained by using the image training set, so that each convolutional layer of the U-Net network extracts slice image features of the HE staining image to obtain corresponding feature maps, and the feature maps are converted into initial probability prediction results of different cell groups; the network parameters of the U-Net network are updated by backward propagation according to the initial probability prediction results and the label information in the corresponding target mask information, so as to update the U-Net network, and the updated U-Net network is used as a preset image segmentation model. It can be seen that by obtaining the HE staining image and the mIHC staining image of the same tissue slice and performing pixel-by-pixel image registration processing, the information of the two kinds of staining images is fully utilized. The mIHC staining image can provide more detailed marking information about different cell groups. After registration with the HE staining image, the target mask information of the HE staining image can be accurately obtained according to the registration result, and the information accurately marks different cell group types and cell group image positions in the staining image. Training the model based on such accurate marking information enables the model to more accurately identify and segment different cell groups, and improves the accuracy of cell group segmentation. Moreover, the U-Net network structure is used for training, and each convolutional layer of the U-Net network can effectively extract slice image features of the HE staining image. The contraction path of the U-Net gradually extracts features of different scales, from local cell details to more macroscopic cell group distribution features, which can be effectively captured. The expansion path can fuse these features and restore them to the original image resolution, so that the model can fully utilize the image features for cell group segmentation, and further improve the segmentation accuracy.

[0090] Further, the embodiment of the application further discloses an electronic device, Figure 4 is a structural diagram of an electronic device 20 according to an exemplary embodiment, and the contents in the figure cannot be considered as any limitation on the use range of the application.

[0091] Figure 4A structural schematic diagram of an electronic device 20 is provided in the embodiments of the present application. The electronic device 20 can specifically include at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25 and a communication bus 26. The memory 22 is configured to store a computer program, and the processor 21 is configured to load and execute the computer program to implement the related steps in the image segmentation model training method of the stained image disclosed in any of the foregoing embodiments. In addition, the electronic device 20 in the embodiments of the present application can be specifically an electronic computer.

[0092] In the embodiments of the present application, the power supply 23 is configured to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 is capable of creating a data transmission channel between the electronic device 20 and external devices, and the communication protocol followed by the communication interface 24 can be any communication protocol applicable to the technical solutions of the present application, which is not specifically limited herein; the input / output interface 25 is configured to obtain external input data or output data to the outside, and the specific interface type can be selected according to the specific application needs, which is not specifically limited herein.

[0093] The processor 21 can include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 21 can be implemented in at least one of a hardware form of a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array) and a PLA (Programmable Logic Array). The processor 21 can also include a main processor and a coprocessor. The main processor is a processor for processing data in a wake-up state, also known as a CPU (Central Processing Unit). The coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 21 can be integrated with a GPU (Graphics Processing Unit) that is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 21 can further include an AI (Artificial Intelligence) processor configured to process machine learning-related computing operations.

[0094] In addition, the memory 22 as a carrier for resource storage can be a read-only memory, a random access memory, a magnetic disk or an optical disk, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage mode can be temporary storage or permanent storage.

[0095] The operating system 221 is used to manage and control each hardware device on the electronic device 20 and the computer program 222, so as to realize the operation and processing of the processor 21 on the mass data 223 in the memory 22, which can be Windows Server, Netware, Unix, Linux, etc. In addition to the computer program capable of completing the image segmentation model training method of the stained image disclosed by the electronic device 20, the computer program 222 can further include a computer program capable of completing other specific work. The data 223 can include the data received by the electronic device from the external device, and can also include the data collected by the self input and output interface 25.

[0096] Further, the application also discloses a computer readable storage medium for storing a computer program; wherein the computer program is executed by the processor to realize the image segmentation model training method of the stained image disclosed above. For the specific steps of the method, please refer to the corresponding content disclosed in the foregoing embodiments, which will not be repeated here.

[0097] Each embodiment in the specification is described in a progressive manner, and each embodiment focuses on the difference from other embodiments. For the same or similar parts between each embodiment, please refer to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant part is described in the method part.

[0098] Those skilled in the art will further appreciate that the units and algorithm steps of the examples described in connection with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or any combination thereof. To clearly illustrate the interchangeability of hardware and software, various components have been described above generally in terms of their functionality, without limitation. The handwiring and software implementations of the examples described herein could be accomplished using any number of microprocessors, microcontrollers, programmable consumption logic devices, application-specific integrated circuits, or general-purpose computers with interconnecting circuits that either run software programs or use opencircuit or other hardware components that are designed to perform the functions described herein. The embodiments described herein can be implemented along with software modules, and the software modules can be stored on any of a variety of non-transitory machine-readable media. A non-transitory machine-readable medium includes any medium that participates in providing instructions to a processor for execution. Such a medium can take many forms, including but not limited to, non-volatile media, volatile media, and transmission media. Non-volatile media includes, for example, optical or magnetic disks and other persistent memory. Volatile media includes dynamic memories, and physical registers. Transmission media includes coaxial cables, copper wires and fiber optic cables, including wires that comprise bus conductors. Transmission media also can also take the form of acoustic or light waves, such as those generated during radio frequency (RF) and infrared (IR) data communications. Common forms of computer-readable media include, for example, a floppy disk, a flexible disk, hard disk, solid-state drive, magnetic tape, or any other magnetic data storage medium, a Compact Disc - Read Only Memory (CD-ROM), any other optical medium, punch cards, paper tape, any other physical medium with patterns of holes, a RAM, a programmable ROM (PROM), an erasable PROM (EPROM), a FLASH-EPROM, any other memory chip or cartridge, a carrier wave, a

[0099] Finally, it should also be noted that, in the present text, relational terms such as first and second and the like can only be used to distinguish one entity or operation from another entity or operation, without necessarily implying any such actual relationship or order between such entities or operations. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by an occurrence of the phrase "comprising a" does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0100] The above has carried on the detailed introduction to the scheme provided by the present application, the principle and implementation mode of the present application are described by applying the specific examples in the present text, the above example explanation is only for helping the understanding of the method and core idea of the present application; simultaneously, for the general technical personnel of the field, according to the idea of the present application, there will be the change in the specific implementation mode and application range, the above-mentioned content should not be understood as the limitation of the present application.

Claims

1. A method for training an image segmentation model for a stained image, characterized in that, include: Acquire several pairs of HE-stained images and mIHC-stained images of the same tissue sections, and perform pixel-by-pixel image registration processing on each pair of HE-stained images and mIHC-stained images to obtain the target mask information of the HE-stained images based on the image registration results, including: The HE-stained image and the mIHC-stained image are structurally registered using a preset multi-cell group labeling and registration algorithm to obtain a registered HE-stained image. The visualization information of the mIHC-stained image is used as mask information for the corresponding image region of the registered HE-stained image to obtain the target mask information. An image training set is constructed based on the HE-stained image and the corresponding target mask information. The target mask information consists of labels indicating different cell group types and cell group image locations in the stained image. The U-Net network structure is trained using the image training set so that each convolutional layer of the U-Net network extracts the features of each slice of the HE-stained image to obtain the corresponding feature map, and the feature map is converted into the initial probability prediction results of different cell groups, including; A U-Net network structure including a shrinking path and an expanding path is constructed; wherein, the shrinking path is provided with multiple convolutional blocks consisting of two convolutional layers and a ReLU activation function, as well as a max pooling layer for downsampling; the expanding path is provided with a deconvolutional layer for upsampling and an output layer containing a convolutional layer and a softmax activation function; The HE-stained image is input into the U-Net network structure to extract the image features of each slice through the convolutional layer of the contraction path of the U-Net network structure to obtain the corresponding initial feature map. The initial feature map is then subjected to nonlinear feature processing using the ReLU activation function to obtain the process feature map. Finally, the process feature map is downsampled through the max pooling layer to obtain the target feature map. The target feature map is input into the extension path of the U-Net network structure to perform upsampling processing on the target feature map through the deconvolution layer, so as to gradually restore the feature map size of the target feature map to the size of the HE-stained image, so as to obtain the restored target feature map; The restored target feature map is input into the output layer, so that the restored target feature map is converted into probability predictions for different cell groups through the output layer, and the initial probability prediction results of semantic segmentation are obtained. Based on the initial probability prediction results and the label information in the corresponding target mask information, the network parameters of the U-Net network are updated by backpropagation to update the U-Net network, and the updated U-Net network is used as the preset image segmentation model.

2. The image segmentation model training method for stained images according to claim 1, characterized in that, Obtain HE-stained and mIHC-stained images of the same tissue section, including: After staining a tissue section with hematoxylin and eosin (HE), the HE-stained tissue section is scanned using a scanning machine to obtain an HE-stained image of the tissue section. The HE-stained tissue sections were destaining and mIHC-stained, and then scanned using the scanning machine to obtain mIHC-stained images of the same tissue sections.

3. The image segmentation model training method for stained images according to claim 1, characterized in that, The step of backpropagating and updating the network parameters of the U-Net network based on the initial probability prediction result and the label information in the corresponding target mask information to update the U-Net network and obtain the updated U-Net network as a preset image segmentation model includes: The initial probability prediction result and the label information in the corresponding target mask information are calculated using a preset loss function to obtain the corresponding loss function value. The gradient information of different network structures is calculated sequentially from the output layer using the backpropagation algorithm and the loss function value. The U-Net network is then updated based on the gradient information to obtain the updated U-Net network as a preset image segmentation model.

4. The image segmentation model training method for stained images according to claim 3, characterized in that, The extraction of image features for each slice of the image through the convolutional layers of the contraction path of the U-Net network structure includes: The convolutional layers of the contraction path of the U-Net network structure extract the shape, size, and texture features of cells in the image to obtain the features of each slice image.

5. The image segmentation model training method for stained images according to claim 3, characterized in that, After obtaining the updated U-Net network as the preset image segmentation model, the process further includes: When there is an image segmentation request for an HE-stained image, the HE-stained image to be segmented is determined based on the image segmentation request; The HE-stained image to be segmented is input into a preset image segmentation model. The preset image segmentation model performs tumor cell group annotation processing, T lymphocyte group annotation processing, B lymphocyte group annotation processing, and macrophage group annotation processing on the HE-stained image to be segmented, so as to segment the image region where each cell group is located and output the corresponding tumor cell group category information and corresponding image region location information, T lymphocyte group category information and corresponding image region location information, B lymphocyte group category information and corresponding image region location information, and macrophage group category information and corresponding image region location information.

6. A training device for an image segmentation model of a stained image, characterized in that, include: The training set acquisition module is used to acquire several pairs of HE-stained images and mIHC-stained images of the same tissue slices, and to perform pixel-by-pixel image registration processing on each pair of HE-stained images and mIHC-stained images to obtain the target mask information of the HE-stained images based on the image registration results, so as to construct an image training set based on the HE-stained images and the corresponding target mask information; wherein, the target mask information is label information that marks different cell population types and cell population image positions in the stained images; The model training module is used to train the U-Net network structure using the image training set, so that each convolutional layer of the U-Net network can extract the features of each slice of the HE-stained image to obtain the corresponding feature map, and convert the feature map into the initial probability prediction results of different cell groups. The model update module is used to backpropagate and update the network parameters of the U-Net network according to the initial probability prediction results and the label information in the corresponding target mask information, so as to update the U-Net network and obtain the updated U-Net network as the preset image segmentation model. The training set acquisition module is specifically used to perform structural registration processing on the HE staining image and the mIHC staining image using a preset multi-cell group annotation and registration algorithm to obtain a registered HE staining image; and to use the visualization information of the mIHC staining image as the mask information of the corresponding image region of the registered HE staining image to obtain the target mask information. The model training module is specifically used to construct a U-Net network structure including a contraction path and an expansion path. The contraction path includes multiple convolutional blocks consisting of two convolutional layers and a ReLU activation function, as well as a max-pooling layer for downsampling. The expansion path includes a deconvolutional layer for upsampling and an output layer containing convolutional layers and a softmax activation function. The HE-stained image is input into the U-Net network structure to extract image features from each slice of the image through the convolutional layers of the contraction path of the U-Net network structure, obtaining corresponding initial feature maps. The ReLU activation function is then used to apply these initial feature maps. The image undergoes nonlinear feature processing to obtain a process feature map. This process feature map is then downsampled using the max-pooling layer to obtain a target feature map. The target feature map is input to the extension path of the U-Net network structure, where it is upsampled using the deconvolution layer to gradually restore its size to that of the HE-stained image, resulting in a restored target feature map. This restored target feature map is then input to the output layer, which converts it into probability predictions for different cell groups, yielding initial probability prediction results for semantic segmentation.

7. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the image segmentation model training method for stained images as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, Used to store a computer program; wherein, when the computer program is executed by a processor, it implements the steps of the image segmentation model training method for stained images as described in any one of claims 1 to 5.

Images