Method for assisting positioning analysis of lymphoma immunohistochemical positive cells

Abstract

The invention relates to the technical field of auxiliary diagnosis, in particular to a method for assisting positioning analysis of lymphoma immunohistochemical positive cells. The method comprises the following steps: collecting pathological section data of a large class of lymphoma patients, and preprocessing the pathological section data; the method comprises the following steps: constructing a plurality of immunohistochemical cell segmentation models based on an nnUnet segmentation network, and training the plurality of immunohistochemical cell segmentation models; performing rigid registration and non-rigid registration on the HE slice image and the IHC slice image; and inputting the registered slice images into a plurality of immunohistochemical cell segmentation models, outputting segmentation results of the IHC positive cells, and visually displaying the segmentation results of the IHC positive cells. According to the method, the images of the IHC positive cells of the lymphoma are automatically segmented, and the obtained information of the IHC positive cells is registered with the HE slices, so that the working efficiency of a doctor can be improved, the doctor can more quickly observe the type and the form of the positive cells, and the efficiency of diagnosing and typing the lymphoma of a patient is improved.

Description

Technical Field

[0001] The present invention relates to the field of auxiliary diagnosis technology, and in particular to a method for auxiliary lymphoma immunohistochemical positive cell localization analysis. Background Art

[0002] Lymphoma is a clonal malignant tumor of lymphocytes. In the World Health Organization's classification of hematopoietic and lymphoid tissue tumors, there are nearly 100 known pathological types of lymphoma. Compared with other malignant tumors, the cure rate of lymphoma is as high as 90%. Its accurate diagnosis is crucial for formulating a correct treatment plan. Pathological diagnosis is the "gold standard" for judging malignant tumors. Traditional pathological sections still rely on doctors' years of experience in positioning, analysis and diagnosis. The diagnostic process lacks accuracy and quantitative information. In clinical practice, pathologists make pathological diagnoses of lymphoma by observing serial sections of the same tissue with the naked eye. It is necessary to judge the abnormal pathological changes such as general cell morphology, tissue cell necrosis, edema, deformation and inflammatory cell infiltration through hematoxylin and eosin (HE) staining sections, and combine immunohistochemical (IHC) staining sections to detect specific cell markers and changes in the expression of some specific proteins in the tissue to make a final diagnosis. With the development of digital pathological image technology, digital whole slide imaging (WSI) captures histological details on tissue sections with high resolution, providing a good data basis for the automatic analysis of pathological images. Digital pathology and microscopic images play an important role in disease diagnosis and decision-making to ensure that clinicians can choose better treatment measures. With the continuous advancement of computer hardware and software and the rapid development of data-driven artificial intelligence methods, the application of deep learning in digital pathological diagnosis has become one of the research hotspots in the field of pattern recognition and artificial intelligence. Therefore, a new analysis method based on deep learning is needed to improve the accuracy of diagnosing lymphoma.

[0003] During the diagnostic workup of lymphoma patients, tissue samples are usually obtained by biopsy or surgical resection, followed by pathological preparation and, in most cases, staining with hematoxylin and eosin (HE). Therefore, HE slides are available for almost all lymphoma patients, and slice data are an easily accessible, information-rich data source for deep learning methods, which also explains the previous focus of research on these types of images. However, deep learning is a tool that is applicable to different types of histological staining, such as immunohistochemistry (IHC). Scanned whole-slide histology images include a huge amount of data, which poses a great challenge to the analysis of the images. Their large files make it impossible to fully load them into the memory of a graphics processing unit (GPU). The current diagnosis of lymphoma is mainly determined by the morphological pattern of WSI. Digital pathology, based on the latest advances in artificial intelligence technology, provides reliable performance for auxiliary diagnosis and prediction of clinical outcomes.

[0004] At present, there are studies based on deep learning algorithms to achieve automatic diagnosis of lymphoma. Steinbuss et al. used labeled HE-stained pathological image blocks to train and optimize the Efficient-Net convolutional neural network algorithm to achieve automatic classification of non-Hodgkin's lymphoma with an accuracy of 95.56%. Lymphoma can usually be divided into two categories: Hodgkin's lymphoma and non-Hodgkin's lymphoma (NHL). NHL accounts for about 80%-90% of all lymphomas. Zhang et al. combined transfer learning and principal component analysis to classify pathological images of non-Hodgkin's lymphoma. The classification model achieved a prediction accuracy of 98.93% for chronic lymphocytic leukemia, follicular lymphoma, and mantle cell lymphoma. Fassler DJ et al. proposed a new integrated method: ColorAE-U-Net, which segmented six immunohistochemically positive cells in pancreatic ductal adenocarcinoma (PDAC) slice images, and proved that the segmentation effect of Unet using ColorAE was better than that of Unet alone. However, the performance of segmenting six types of immunohistochemical positive cells using different segmentation methods in the article is not high, and the dice is concentrated between 0.5 and 0.65. It is still difficult to use a model to accurately segment different immunohistochemical positive cells. Using registration technology and segmentation technology at the same time, Feng et al. proposed a Ki-67 image accurate registration method based on WTS and Ki-67 automatic recognition and counting software based on deep learning. The project uses the GoogleNet Inception V1 model to automatically locate the breast cancer area, uses the elastic toolbox to register HE and IHC, and finally uses the algorithm provided by Image J to automatically extract the structure, morphology, color and other features of positive and negative cells, and trains the random forest classifier to identify Ki-67 positive or negative cells. However, these studies did not segment multiple pathological cells at the same time, but only segmented a single type of pathological cells and did not register immunohistochemical cells with HE sections, which is not conducive to doctors being able to better diagnose lymphoma typing. Summary of the invention

[0005] In order to solve the technical problems existing in the prior art, the present invention proposes a method for computer-assisted lymphoma immunohistochemical positive cell localization, which automatically segments various IHC positive cell images of lymphoma by constructing various immunohistochemical cell segmentation models, and fuses the information of positive cells in the immunohistochemical slices with the hematoxylin and eosin stained slices by rigidly registering and non-rigidly registering the HE slice images and the IHC slice images, which can help improve the work efficiency of doctors, help doctors observe the type and morphology of positive cells more quickly, and improve the efficiency of diagnosing and typing patients' lymphomas.

[0006] The first object of the present invention can be achieved by adopting the following technical solutions:

[0007] A method for assisting lymphoma immunohistochemistry positive cell localization analysis, the method comprising:

[0008] S1. Data collection and preprocessing: collecting pathological section data of lymphoma patients, preprocessing the pathological section data, and constructing a segmentation model data set;

[0009] S2. Build multiple immunohistochemical cell segmentation models based on the nnUnet segmentation network and train multiple immunohistochemical cell segmentation models;

[0010] S3, performing rigid registration and non-rigid registration on the HE slice image and the IHC slice image to obtain the registered HE slice image and the IHC slice image;

[0011] S4. Input the registered HE slice image and IHC slice image into multiple immunohistochemical cell segmentation models, output the segmentation results of IHC positive cells, and visualize the segmentation results of the registered HE slice image and IHC positive cells.

[0012] Specifically, the pathological section data of the lymphoma patients include multiple IHC type sections and one HE tissue staining section of the lymphoma patients.

[0013] Specifically, the preprocessing of the pathological slice data to construct a segmentation model data set includes:

[0014] Performing format conversion on the pathological slice data to obtain format-converted pathological slice data, and dividing a single format-converted pathological slice data into a plurality of image blocks;

[0015] For the labeling of positive and negative cells of immunohistochemical pathology in each image block, image blocks with tissue area less than the preset threshold and image blocks containing more slice edge information are excluded to obtain image blocks with positive cells labeled.

[0016] Specifically, the nnUnet segmentation network is used to construct a variety of immunohistochemical cell segmentation models, including:

[0017] The nnUnet segmentation network was trained using the immunohistochemical image data of the segmentation model dataset. The cross entropy loss function was used as the loss function of various immunohistochemical cell segmentation models. The back propagation algorithm was used to calculate the gradient and update the parameters of various immunohistochemical cell segmentation models. The training was iterated continuously until the nnUnet segmentation network converged to obtain various immunohistochemical cell segmentation models after training.

[0018] Specifically, the step S3 includes:

[0019] S31, preprocessing the HE slice image and the IHC slice image to obtain a preprocessed HE slice image and an IHC slice image;

[0020] S32, performing exhaustive rotation angle search on the preprocessed HE slice image and the IHC slice image, obtaining the optimal rotation angle according to the image similarity index Dice and information entropy, and rotating the IHC slice image according to the optimal rotation angle;

[0021] S33, performing rigid registration, segmenting the HE slice image and the IHC slice image into a background part and a tissue part through a semantic segmentation network SSN, matching the best matching feature points of the tissue part of the HE slice image and the tissue part of the IHC slice image, and performing registration on the tissue part of the HE slice image and the tissue part of the IHC slice image according to the best matching feature points to obtain a pre-registered image;

[0022] S34 , performing non-rigid registration on the pre-registered images to obtain registered HE slice images and IHC slice images.

[0023] Specifically, the preprocessing of the HE slice image and the IHC slice image includes: performing image normalization, image grayscale, histogram matching, and filtering and denoising processing on the HE slice image and the IHC slice image.

[0024] Specifically, the step S32 includes:

[0025] The HE slice image is used as the fixed image F and the IHC slice image is used as the moving image M. An exhaustive rotation search is performed, and the operation is iterative: each round of clockwise rotation is performed by a preset transformation angle, and the image similarity index Dice is recalculated according to the formula after each round of transformation;

[0026] Calculate the information entropy of the fixed image F and the moving image M tissue region images;

[0027] The original input HE slice image, IHC slice image and the rotated HE slice image, IHC slice image are compared, and the image similarity index Dice and information entropy are used as the similarity measure of the fixed image F and the moving image M, and the optimal rotation angle is selected to rotate the moving image M.

[0028] Specifically, the step S33 includes:

[0029] S331. Use the complete WSI in the ANHIR data to train the semantic segmentation network SSN. Based on the pre-trained parameters of the semantic segmentation network SSN, use the data of the segmentation model dataset to train the semantic segmentation network SSN. Use the trained semantic segmentation network SSN to segment the HE slice image and the IHC slice image into background and tissue parts.

[0030] S332, generating feature descriptors of the tissue part of the HE slice image and the tissue part of the IHC slice image, and automatically searching for feature points using a scale-invariant feature transformation algorithm to obtain the best matching feature points;

[0031] S333 : Perform affine transformation on the tissue part of the HE slice image and the tissue part of the IHC slice image according to the feature descriptor and the matched feature points.

[0032] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0033] The present invention proposes a computer-assisted method for localizing immunohistochemical positive cells in lymphoma, which automatically segments multiple IHC positive cell images of lymphoma by acquiring continuous slices of diseased tissues of the same patient and constructing multiple immunohistochemical cell segmentation models, obtains registered HE slice images and IHC slice images by rigid registration and non-rigid registration of HE slice images and IHC slice images, fuses the information of positive cells in the acquired IHC slice images with the HE slices, and can obtain quantifiable and highly repeatable positive cell localization information to determine the area of ​​positive cells. The present invention can assist in improving the work efficiency of doctors, reduce the time and energy of doctors in manually analyzing cell images, and help doctors observe the type and morphology of positive cells more quickly, thereby improving the efficiency of diagnosing and typing patients' lymphomas. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.

[0035] Figure 1 is a flow chart of a method for assisting lymphoma immunohistochemistry positive cell localization analysis in an embodiment of the present invention;

[0036] Figure 2 is a schematic diagram of the immunohistochemical pathology positive cell delineation results in an embodiment of the present invention;

[0037] Figure 3is a training flow chart of various immunohistochemical cell segmentation models in an embodiment of the present invention;

[0038] Figure 4 is a schematic diagram of the preprocessing process results in an embodiment of the present invention;

[0039] Figure 5 is a schematic diagram of an exhaustive rotation search registration result in an embodiment of the present invention;

[0040] Figure 6 It is a semantic segmentation network SSN diagram based on CNN and Transformer in an embodiment of the present invention;

[0041] Figure 7 is the best matching graph for SIFT feature point detection in the embodiment of the present invention;

[0042] Figure 8 It is a diagram of the registration effect in the embodiment of the present invention. DETAILED DESCRIPTION

[0043] The technical solution of the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is obvious that the described embodiments are part of the embodiments of the present invention, not all of the embodiments, and the implementation of the present invention is not limited to this. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0044] Embodiment 1:

[0045] like Figure 1 As shown, a flow chart of a method for assisting lymphoma immunohistochemistry positive cell localization analysis, the method of assisting lymphoma immunohistochemistry positive cell localization analysis of the present invention comprises the following steps:

[0046] S1. Data collection and preprocessing: Collect pathological section data of lymphoma patients, preprocess the pathological section data, and build a segmentation model data set.

[0047] Specifically, pathological section data of patients with lymphoma are collected, and the pathological section data include a variety of different types of IHC sections and a HE section of patients with lymphoma. In this embodiment, the lymph node biopsy tissue can be selected from the pathology department of the hospital for examination. The diagnosis result is that the patient belongs to the lymphoma category, and there are more than 10 types of continuous immunohistochemistry slides. All cases are scanned in high definition using the Jiangfeng slide scanner, and the complete data set included in the project is uniformly named. The data was collected from 2019 to 2023, with a total of 97 cases, each with 8 types of immunohistochemical sections and one HE-stained section. The original data size for 19-21 is about 90G, the original data size for 21-22 is 79G, and the data size for 22-23 is about 100G. Three physicians with pathological diagnosis qualifications manually annotate immunohistochemical positive cells, and select at least 10 patients each year for image preprocessing of their complete sections.

[0048] Specifically, the pathological slice data are preprocessed, including: converting the format of the pathological slice data to obtain the converted pathological slice data, segmenting the converted pathological slice data of a single patient into multiple image blocks, annotating the positive cells and negative cells of the immunohistochemical pathology of each image block, excluding image blocks with a tissue area ratio less than a preset threshold, and excluding image blocks containing more slice edge information, to obtain a segmentation model data set.

[0049] In this embodiment, a full set of pathological slice data of more than 10 patients were randomly selected for format conversion, and all data were converted from .kfb format to .tif format. The single immunohistochemical slice .tif format data of a single patient was divided into image blocks (patches) of 512*512 size, and 10 relatively clear patches were selected. Patches whose tissue areas did not fill the entire patch field of view were not included in the experiment. A total of 5+2 immunohistochemical cytopathological images were used, CD3 (Tcell: positive), CD5 (Tcell: positive, identifying mantle cell lymphoma), CD20 (Bcell: positive), Ki-67 (malignant cell proliferation index, usually expressed as a percentage), Bc1-2 (B cell: positive; often used to distinguish reactive hyperplasia and lymphoma, and observe changes in germinal centers); CD79a (Bcell: positive), cyclind1 (diagnosis of mantle cell lymphoma). The complete immunohistochemical slices were divided into 512×512 patches using a python script. The positive and negative cells of immunohistochemical pathology in each image block were annotated using the Qupath image segmentation and annotation tool. Image blocks with tissue areas less than the preset threshold and image blocks containing more slice edge information were excluded to obtain the segmentation model dataset.

[0050] like Figure 2Shown is a schematic diagram of the immunohistochemical pathology positive cell delineation results, with immunohistochemical positive cells and negative cells manually delineated by a physician. Figure 2 a, b, c are cropped image patches of size 512×512. Figure 2 Figures d, e, and f are the corresponding manual outlining results. The red part is for immunohistochemically positive cells, and the blue part is for immunohistochemically negative cells. During the outlining process, image blocks with a tissue area of ​​less than 60% were excluded, and image blocks containing more slice edge information in an image block were also not included in the data set. The reason is that when performing immunohistochemical staining, the edges or areas near the edges may be darkly stained or all positive. This is because the tissue sections and glass slides are not firmly attached, causing the tissue at the edges to loosen and float on the liquid, and the dye is not washed thoroughly enough; there is also the effect of liquid surface tension, which will make the volatilization effect of the middle area better and the staining more uniform. During the diagnosis process, the edge is not used as the main area for diagnosis.

[0051] The segmentation model dataset is divided into a training set and a validation set. Table 1 is the image segmentation training dataset division table B. There are 500 images of each type of immunohistochemistry data, 300 of which are used as training sets, 100 as validation sets, and 100 as test sets.

[0052] The training set is used to train the target detection model, mainly to determine the model's parameters such as weights and biases, which are usually called learning parameters. These parameters are continuously adjusted through the training process so that the model can better learn and identify targets. The validation set is used to evaluate the performance of the model after each round of training to monitor the overfitting of the model and help adjust the model parameters and hyperparameters.

[0053] Table 1

[0054]

[0055] S2. Based on the nnUnet segmentation network, multiple immunohistochemical cell segmentation models are constructed and trained. The trained multiple immunohistochemical cell segmentation models can be used to automatically segment multiple IHC positive cell images of lymphoma.

[0056] Specifically, the nnUnet segmentation network was trained using the data of immunohistochemical type positive cells in the segmentation model data set, and the Cross entropy Loss loss function was used as the loss function of various immunohistochemical cell segmentation models. The back propagation algorithm was used to calculate the gradient and update the parameters of various immunohistochemical cell segmentation models. The training was iterated continuously until the nnUnet segmentation network converged to obtain the trained various immunohistochemical cell segmentation models.

[0057] like Figure 3 As shown in the figure, the training flow chart of various immunohistochemical cell segmentation models, in the process of applying nnUnet segmentation network training, the learning rate is adaptively adjusted by dynamically adjusting the optimization function, cross entropy loss function, etc. The nnUnet segmentation network will perform forward propagation calculations to obtain the predicted output, and then compare the predicted value with the true value to calculate the cross entropy loss CrossEntropyLoss. Finally, the back propagation algorithm is used to calculate the gradient and update the parameters, and the training is continuously iterated until the network converges. The cross entropy loss function CrossEntropyLoss is expressed as:

[0058]

[0059] In the formula, y i is the predicted probability of the i-th pixel, p i is the actual label value of the i-th pixel, and N is the total number of sample pixels. The cross entropy loss function is achieved by adjusting y i 、p i The value of makes the predicted probability distribution closer and closer to the actual label distribution, thereby improving the classification accuracy of the model.

[0060] In this example, when the data of immunohistochemical positive cells in the segmentation model dataset are used to train the nnUnet segmentation network, the data of multiple immunohistochemical positive cells are integrated, and the model is trained with the integrated data for 1000 epochs. The Adam optimizer is used, and the initial learning rate is set to 3e-4. The learning rate adjustment strategy is: calculate the exponential sliding average loss of the training set and the validation set. If the exponential sliding average loss of the training set decreases by less than 5e-3 within 30 epochs, the learning rate is decayed by 5 times. The excellent performance of constructing multiple immunohistochemical cell segmentation models based on the nnUnet segmentation network in the immunohistochemical positive cell segmentation task is demonstrated. The over-segmentation and under-segmentation areas generated by the nnUnet network during the segmentation process are significantly reduced, which greatly improves the accuracy and stability of the segmentation.

[0061] S3. Perform rigid registration and non-rigid registration on the HE slice image and the IHC slice image to obtain the registered HE slice image and IHC slice image.

[0062] Pathological sections include HE sections and IHC sections. Clinically, pathologists first perform an optical microscope examination of HE tissue-stained sections to make a preliminary diagnosis of different lymphoma entities based on the growth pattern of lymphoid tissue and the cytological characteristics of abnormal cells. Different types of lymphoma treatment and prognosis schemes are also different. Misdiagnosis of lymphoma types can lead to worsening of the disease. Correct diagnosis and early treatment can effectively improve the cure rate of lymphoma. Since lymphoma does not have specific changes in tissue structure and cell morphology, the histological characteristics of different lymphoma types are very different. It is difficult to accurately diagnose lymphoma by observing HE-stained pathological sections with the naked eye. The entire diagnostic process requires doctors to switch sections at any time and diagnose in combination with IHC immunohistochemistry. This process is very time-consuming and labor-intensive, with poor repeatability, and no quantitative information is saved. In order to solve the problems encountered by pathologists in the diagnosis of such diseases, this embodiment provides a registration method for HE slice images and IHC slice images. By registering the HE slices with the IHC slices, the information provided to the doctor on the two slices can be quickly loaded, and the performance of the computer can be fully utilized to provide visual and quantifiable data.

[0063] S31 , image preprocessing, preprocessing the HE slice image and the IHC slice image to obtain the preprocessed HE slice image and the IHC slice image.

[0064] Specifically, the HE slice image and the IHC slice image are preprocessed, including image normalization, image grayscale, histogram matching, and filtering and denoising operations on the HE slice image and the IHC slice image to obtain the preprocessed HE slice image and the IHC slice image. Preprocessing the HE slice image and the IHC slice image can make the registration process as simple as possible. The image is grayscaled, and the data is converted into grayscale images according to different levels, smoothed and downsampled to a lower resolution. The image is converted to grayscale, which reduces the number of pixels represented by only a single channel by at least 10,00 times. Then, the image is read at the required level according to the operation and then downsampled once. It is worth noting that although these downsampled images provide a manageable image size to perform WSI alignment, all steps are performed on the full-resolution version, histogram matching, calculating the image entropy of the two images, and matching the histogram of the low entropy image with the histogram of the high entropy image.

[0065] like Figure 4 As shown in Figure 1, a schematic diagram of the preprocessing process results, a and b are the input HE slice image (fixed image) and IHC slice image (transformed image), and grayscale processing, histogram matching and denoising operations are performed according to the preprocessing process. c and d are grayscale images of HE and IHC. The blue lines in the histogram indicate the pixel distribution after the IHC image is matched with the HE histogram. Figure e is the result after histogram matching. f and g are the denoising operations performed on the results of the previous step. Figure 4 As shown in Figure e, there is a lot of edge noise after matching, and denoising is performed through median filtering to complete image preprocessing.

[0066] S32, performing a rigid transformation, performing an exhaustive rotation angle search on the preprocessed HE slice image and the IHC slice image, obtaining an optimal rotation angle according to the image similarity index Dice and information entropy, and rotating the IHC slice image according to the optimal rotation angle.

[0067] S321, using the HE slice image as the fixed image F and the IHC slice image as the moving image M, perform exhaustive rotation search, iterative operation: each round of transformation angle rotates 5 degrees clockwise, and after each round of transformation, recalculates the image similarity index Dice according to the formula, and the formula used is as follows:

[0068]

[0069] Wherein, Mi and Fi are the i-th segmented regions in the moving image M and the fixed image F, respectively.

[0070] S322, calculating the information entropy of the fixed image F and the moving image M tissue region images, the information entropy calculation formula is as follows:

[0071] MI(F,M)=H(F)+H(M)-H(F,M);

[0072] Among them, H(F) and H(M) represent the information entropy of images F and M respectively, and H(F,M) is the joint entropy of F and M.

[0073]

[0074] Where N is the number of different grayscale values ​​in the image, and the grayscale value range is generally (0-255). Pi is the probability that a point with grayscale value i appears in the image, and p FM (a,b) is the probability that the grayscale value of a pixel at the same position in the F image is a and the grayscale value in M ​​is b.

[0075] S323, using the image similarity index Dice and information entropy as similarity metrics between the fixed image F and the moving image M, comparing the original input HE slice image, IHC slice image and the HE slice image, IHC slice image after rotation transformation, and selecting the optimal rotation angle to rotate the rotated image M.

[0076] like Figure 5As shown in the figure, the schematic diagram of the registration result of exhaustive rotation search is shown. The similarity measurement of the two images is optimal when the rotation matching search is performed at 180 degrees. The rotation registration is performed after a 180-degree rotation. The IHC registration result without rotation registration is poor. The figure includes the registration results before and after rotation. The image similarity index of the two images is calculated. The optimal rotation angle is determined by exhaustively searching the rotation angle of the image and the entropy of the image based on the image similarity index, which provides a good registration basis for subsequent rigid registration and non-rigid registration.

[0077] When performing image registration, the impact of the pre-rotation operation on the final result cannot be ignored. Comparing the results without pre-rotation and with pre-rotation, it was found that there was a significant difference between the two. In the experiment, the total number of IHC slices was about 990. Among them, the angle deviation between the tissue part in the IHC and the tissue part in the HE slice was 45° to 90°, accounting for about 20% of the total, and the deviation was 90° to 180°, accounting for about 36% of the total. This data shows that large angle deviation is a common problem, which brings considerable challenges to the registration work. Experimental results show that pre-rotation operation in the preprocessing stage can effectively improve the registration rate. Through this step, image registration can be performed faster and more intuitively, so as to better meet the needs of subsequent analysis.

[0078] In this embodiment, the en evaluation index is used to calculate the entropy difference of the two images. For fixed images and moving images, it is ensured that when selecting the optimal rotation angle, the rotation angle that loses too much effective information will not be selected. It can be seen from the table that when the rotation angle is 90 degrees and 270 degrees, more information is lost, and the entropy mean of other angles is about 6.5. In this step, the SSIM structural similarity index is used as an index for selecting the optimal angle. As shown in Table 1, the exhaustive rotation search evaluation index and the rotation angle are used to perform an exhaustive rotation search (times / angle 5°), and en, SSIM, and MSE are used as evaluation indicators, EN is image entropy, SSIM is an image structure similarity evaluation index, and MSE is the mean square error.

[0079] Table 2

[0080]

[0081] S33, performing rigid registration, segmenting the HE slice image and the IHC slice image into a background part and a tissue part through a semantic segmentation network SSN, matching the best matching feature points of the tissue part of the HE slice image and the tissue part of the IHC slice image, and performing registration on the tissue part of the HE slice image and the tissue part of the IHC slice image according to the best matching feature points to obtain a pre-registered image;

[0082] S331. Use the complete WSI in the ANHIR data to train the semantic segmentation network SSN. Based on the pre-trained parameters of the semantic segmentation network SSN, use the data of the segmentation model data set to continue training the semantic segmentation network SSN. The HE slice image and the IHC slice image are segmented into background and tissue parts by the trained semantic segmentation network SSN. In this embodiment, the WSI images in the ANHIR data are used for training, and 50 complete WSI images are obtained. 40 slides are used to train the semantic segmentation network SSN, and the remaining 10 are used for testing. 150 patches randomly cropped from the test slides are used as a small validation set to select the best model during training, compressed to an SVS format of 100,000×100,000 pixels. The entire WSI image I is tiled into 20 times smaller patches of 512×512 size. During the training phase, the patch collector randomly crops training patches from the ANHIR dataset.

[0083] like Figure 6 As shown in the figure, the semantic segmentation network SSN diagram based on CNN and Transformer includes the convolution layer and pooling layer of the decoder part in the semantic segmentation network SSN, pooling layers 3, 4, and 5, and the output size of Pool3 is 28×28×256, which means that it generates 28×28 feature descriptors for the input image. Each descriptor is normalized to unit variance, so that these feature descriptors have a consistent scale in subsequent processing. Each descriptor in pooling layer 3 has 256 feature values, and each descriptor has a receptive field of size 8×8, which means that each feature point (located at the center of the 28×28 grid) has a 9×9 pixel area for feature extraction, and the center of each grid unit is considered to be the feature point of the respective descriptor. In pooling layer 4, the output size is 14×14×512, and each descriptor includes a 16×16 receptive field; in pooling layer 5, each descriptor has a 32×32 receptive field shared by 16 feature points from pooling layer 3.

[0084] S332, generating feature descriptors of the tissue part of the HE slice image (fixed image) and the tissue part of the IHC slice image (moving image), and automatically searching for feature points using a scale-invariant feature transform (SIFT) algorithm to obtain the best matching feature points.

[0085] HE slice images and IHC slice images generate feature descriptors of the tissue part of the HE slice image (fixed image) and the tissue part of the IHC slice image (moving image) during tissue segmentation. The feature descriptors include information on the shape, edge, texture and other features of the object and parameters such as feature vectors, which are used as input to restrict the affine transformation operation in the subsequent affine transformation. In the decoding stage, the features of the image pair are extracted through different layers. Considering that the feature points are considered at a small feature distance, the pooling layer 3 to 5 features of blocks 3 to 5 are retained, and the descriptors are calculated for each window in a sliding manner in the small block image. Once the feature descriptor is formed, the Euclidean distance between all possible feature points is calculated. The formula is as follows:

[0086]

[0087] in, are feature descriptors calculated for the tissue part of the HE slice image and the tissue part of the IHC slice image, respectively. F ,P M They are fixed images and moving images;

[0088] The tissue part of the preprocessed HE slice image and the tissue part of the IHC slice image are used as input for SIFT feature point detection. Data-driven descriptors are used to estimate global transformations, a deep feature-based registration. SIFT is used to find the best matches and feature descriptors for these downsampled preprocessed tissues, and then the fast approximate nearest neighbor library (FLANN) is used to match the best matching feature points. Lowe's ratio test is used to filter out bad matches. The following three conditions must be met to create a match, including: more than 5 matching points; random sample consistency estimates of the homography matrix; and the consistency score is higher than the critical value of 71.4% when evaluated using the Dice and SSIM indicators. Figure 7 As shown in Figure 1, it is the best matching graph for SIFT feature point detection.

[0089] Calculate the homography matrix H of the tissue part of the small patch image so that it is properly scaled on the original image. In order to determine the position of a given patch in the moving image in the fixed image, take the coordinates of the center of the moving image and apply H'. The calculation formula is H'=S×H×S-1, where S is as follows:

[0090]

[0091] The transformed coordinates are the center coordinates of the fixed image patch. By using the center coordinates, the robustness of the transformation can be maximized when the image is rotated and / or otherwise changed. Any transformed coordinates outside the boundaries or empty space in the moving image will be ignored.

[0092] S333 : Perform affine transformation on the tissue part of the HE slice image and the tissue part of the IHC slice image according to the feature descriptor and the best matching feature point.

[0093] The transformation parameters are estimated using the matched feature points as control points, and the affine transformation is calculated using the least squares method to find an optimal rigid transformation that best aligns these matching points. The rigid registration process requires applying this method twice. First, it is applied to the entire preprocessed WSI to find the best matching point globally, which is called the "global transformation"; second, the tissue area is divided into 100×100 pixel image parts, called sliding retrieval blocks. The sliding retrieval blocks are small enough to ensure that the pixel blocks can completely include the blank and tissue areas. The feature matching method is applied to each part separately, which is called the "tissue transformation". This process produces more matching feature points and has the potential to further improve the alignment. Define a: p1 and p2 as blank; b: p1 and p2 as tissue area; c: p1 as blank and p2 as tissue area, where p1 and p2 are the coordinates of the sliding small image at the current position on the fixed image and the transformed image, and evaluate the overlap matching degree of the global image and the tissue image. The formula is as follows:

[0094]

[0095] The overlap match is used to measure whether to perform a rigid registration of the tissue part. Rigid transformation is applied to the moving image only when the global transformation and tissue transformation tend to improve the DICE overlap. The Youden index is used as the critical value for finding the correct alignment, which is 71.4%. When the match is higher than this value, it means that the changes in the alignment are effective.

[0096] S34, performing non-rigid registration on the pre-registered images to obtain registered HE slice images and IHC slice images, thereby achieving registration of continuous HE slice images and IHC slice images.

[0097] Taking the pre-registered image as input, perform non-rigid registration. In non-rigid registration, use a non-linear registration using one of the 3-step registration pipelines. Transform y to:

[0098]

[0099] Where u is the transformation field, and the regularizer is used to control the smoothness of the calculated deformation. The curvature regularization formula is as follows:

[0100]

[0101] Then minimize the following objective functions for transformation y and displacement u respectively:

[0102] J(R,T,y):=NGF(R,T,y)+αCURV(y)→min

[0103] NGF(R,T,y) is the normalized gradient field, and the regularization parameter α>0 is manually selected to provide smooth deformation and avoid topological changes (mesh collapse), while being flexible enough to correct local changes and improve image similarity. For the numerical implementation, the displacement field u is discretized using first-order b-splines, defined on a uniform control point grid with N points, and then the non-parametric objective function is optimized with respect to the control points.

[0104] like Figure 8 As shown in the figure, the registration effect diagram, Figure 8 It includes five groups (1-5) of fixed images and moving images, af represents: HE slice, IHC slice, overlap effect of IHC and HE after registration, transformation interpolation effect after IHC slice registration, and 10 times magnified detail display after HE and IHC registration, and the cell tissue or blood vessel section at the same position is marked in the yellow box as a registration control. The registration process includes five groups of slices to be registered, a column is a fixed image-HE, b is an IHC slice that needs to be registered with HE, IHC and HE are registered using the method of the present invention, c column is the information matching effect of the two slices after the transformation, d column is the transformation map of IHC after registration, the registration steps include rigid registration and non-rigid registration, e, f are the display of detailed information after registration, and the yellow box in the figure indicates the cell alignment or blood vessel section alignment after the registration is completed (5e, f).

[0105] S4. Input the registered HE slice image and IHC slice image into multiple immunohistochemical cell segmentation models, output the segmentation results of IHC positive cells, and visualize the segmentation results of the registered HE slice image and IHC positive cells.

[0106] Specifically, the registered slice images are input into a variety of immunohistochemical cell segmentation models, and the multiple immunohistochemical cell segmentation models output the segmentation results of positive cells, and the registered slice images and the segmentation results of positive cells can be visualized by computer software. The computer software can create an easy-to-use graphical user interface (GUI) using Python language to display the registered slice images and the segmentation results of positive cells through the software, so that doctors can understand the situation of the lesion area more intuitively and improve the accuracy and efficiency of diagnosis.

[0107] In the process of clinical case diagnosis, the number of histological sections that pathologists need to process every day may be as high as 600. In addition, they also need to deal with other tasks such as cell freezing. Relying solely on the naked eye and focused brainpower to cope with such a huge workload, pathologists are bound to make misjudgments. In pathological diagnosis, the diagnosis of lymphoma is particularly difficult and can be said to be one of the most challenging diseases. There are more than 100 types of lymphoma, and each type has its own unique pathological characteristics, which makes the diagnostic process extremely complicated. What is even more difficult is that the diagnosis of lymphoma is easily confused with inflammation of the lymphatic system because the pathological manifestations of the two are sometimes very similar. The present embodiment proposes a method for computer-assisted immunohistochemical positive cell localization of lymphoma, which automatically segments multiple IHC positive cell images of lymphoma by acquiring continuous sections of the same patient's diseased tissue, constructing multiple immunohistochemical cell segmentation models, and obtaining registered HE slice images and IHC slice images by rigid registration and non-rigid registration of HE slice images and IHC slice images, and merging the information of positive cells in the immunohistochemical slices with the hematoxylin and eosin stained slices to obtain quantifiable and highly repeatable positive cell localization information to determine the area of ​​positive cells. The present invention can assist in improving the work efficiency of doctors, reduce the time and energy of doctors in manually analyzing cell images, and help doctors observe the type and morphology of positive cells more quickly, thereby improving the efficiency of diagnosing and typing patients' lymphomas. It can provide assistance to doctors to help them diagnose lymphomas faster and more accurately.

[0108] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A method for assisting lymphoma immunohistochemistry positive cell localization analysis, characterized in that: The following steps are involved: S1. Data collection and preprocessing: collecting pathological section data of lymphoma patients, preprocessing the pathological section data, and constructing a segmentation model data set; S2. Build multiple immunohistochemical cell segmentation models based on the nnUnet segmentation network and train multiple immunohistochemical cell segmentation models; S3, performing rigid registration and non-rigid registration on the HE slice image and the IHC slice image to obtain the registered HE slice image and the IHC slice image; S4. Input the registered HE slice image and IHC slice image into multiple immunohistochemical cell segmentation models, output the segmentation results of IHC positive cells, and visualize the segmentation results of the registered HE slice image and IHC positive cells.

2. A method for assisting lymphoma immunohistochemistry positive cell localization analysis according to claim 1, characterized in that: The pathological section data of the lymphoma patients include a plurality of different types of IHC sections and one HE section of the lymphoma patients.

3. A method for assisting lymphoma immunohistochemistry positive cell localization analysis according to claim 2, characterized in that: The preprocessing of the pathological slice data to construct a segmentation model data set includes: Performing format conversion on the pathological slice data to obtain format-converted pathological slice data, and dividing a single format-converted pathological slice data into a plurality of image blocks; For the labeling of positive and negative cells of immunohistochemical pathology in each image block, image blocks with tissue area less than the preset threshold and image blocks containing more slice edge information are excluded to obtain image blocks with positive cells labeled.

4. A method for assisting lymphoma immunohistochemistry positive cell localization analysis according to claim 3, characterized in that: The nnUnet segmentation network is used to construct a variety of immunohistochemical cell segmentation models, including: The nnUnet segmentation network was trained using the immunohistochemical image data of the segmentation model dataset. The cross entropy loss function was used as the loss function of various immunohistochemical cell segmentation models. The back propagation algorithm was used to calculate the gradient and update the parameters of various immunohistochemical cell segmentation models. The training was iterated continuously until the nnUnet segmentation network converged to obtain various immunohistochemical cell segmentation models after training.

5. A method for assisting lymphoma immunohistochemistry positive cell localization analysis according to claim 4, characterized in that: The step S3 comprises: S31, preprocessing the HE slice image and the IHC slice image to obtain a preprocessed HE slice image and an IHC slice image; S32, performing exhaustive rotation angle search on the preprocessed HE slice image and the IHC slice image, obtaining the optimal rotation angle according to the image similarity index Dice and information entropy, and rotating the IHC slice image according to the optimal rotation angle; S33, performing rigid registration, segmenting the HE slice image and the IHC slice image into a background part and a tissue part through a semantic segmentation network SSN, matching the best matching feature points of the tissue part of the HE slice image and the tissue part of the IHC slice image, and performing registration on the tissue part of the HE slice image and the tissue part of the IHC slice image according to the best matching feature points to obtain a pre-registered image; S34 , performing non-rigid registration on the pre-registered images to obtain registered HE slice images and IHC slice images.

6. The method for assisting lymphoma immunohistochemistry positive cell localization analysis according to claim 5, characterized in that: The preprocessing of the HE slice image and the IHC slice image includes: performing image normalization, image grayscale, histogram matching, and filtering and denoising processing on the HE slice image and the IHC slice image.

7. The method for assisting lymphoma immunohistochemistry positive cell localization analysis according to claim 5, characterized in that: The step S32 comprises: The HE slice image is used as the fixed image F and the IHC slice image is used as the moving image M. An exhaustive rotation search is performed, and the iterative operation is performed: each round of clockwise rotation is performed by a preset transformation angle, and the image similarity index Dice is recalculated after each round of transformation; Calculate the information entropy of the fixed image F and the moving image M tissue region images; The image similarity index Dice and information entropy are used as the similarity metrics between the fixed image F and the moving image M. The original input HE slice image, IHC slice image and the rotated HE slice image, IHC slice image are compared, and the optimal rotation angle is selected to rotate the moving image M.

8. The method for assisting lymphoma immunohistochemistry positive cell localization analysis according to claim 5, characterized in that: The step S33 comprises: S331, using the complete WSI in the ANHIR data to train a semantic segmentation network SSN, using the data of the segmentation model dataset to train the semantic segmentation network SSN based on the pre-trained parameters of the semantic segmentation network SSN, and segmenting the HE slice image and the IHC slice image into a background part and a tissue part through the trained semantic segmentation network SSN; S332, generating feature descriptors of the tissue part of the HE slice image and the tissue part of the IHC slice image, and automatically searching for feature points using a scale-invariant feature transformation algorithm to obtain the best matching feature points; S333 : Perform affine transformation on the tissue part of the HE slice image and the tissue part of the IHC slice image according to the feature descriptor and the best matching feature point.

Images