CD34 immunohistochemical image recognition and microvessel density calculation method for hepatocellular carcinoma

Abstract

The present invention discloses a method for recognizing CD34 immunohistochemical images of hepatocellular carcinoma and calculating microvessel density. This method addresses the problems of manual diagnostic methods, which are complex, time-consuming, and labor-intensive, and rely on extensive diagnostic experience. In order to more comprehensively assess the severity of hepatocellular carcinoma patients and guide subsequent treatment, the present invention employs a deep learning network to perform binary classification on images, which has strong generalization capabilities and high robustness. The generated probability heat map intuitively reflects the model recognition results. The calculation of microvessel density is highly interpretable, simple, and convenient, making it easy for doctors to operate and adjust.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and in particular to a method for hepatocellular carcinoma CD34 immunohistochemical image recognition and microvessel density calculation. Background Art

[0002] Hepatocellular carcinoma (HCC) is a common malignant tumor that can occur at any age. The incidence of HCC is very high in China. The annual mortality rate for HCC ranks high among the causes of death from malignant tumors, and its incidence continues to rise. Therefore, early diagnosis is crucial for treatment. Research on HCC recognition and segmentation technologies is crucial for reducing HCC mortality.

[0003] Hepatocellular carcinoma (HCC) is a rare disease. According to statistics, approximately 70% of HCC patients in China are diagnosed at an advanced stage. Early diagnosis of HCC has a relatively high cure rate. Therefore, early intelligent diagnosis of HCC is crucial for patient treatment and prognosis. Computer-assisted diagnosis (CAD) systems are rapidly developing in the medical field. Using computers to objectively and quantitatively analyze medical imaging data aids physicians in clinical diagnosis, improving diagnostic accuracy and efficiency.

[0004] CD34 is a 105-115 kDa glycoprotein involved in cell adhesion and initial angiogenesis. Its expression is low in mature cells. Studies have shown that positive CD34 expression is positively correlated with hepatocellular carcinoma. Physicians observe CD34 immunohistochemically stained sections to help diagnose patients. Microvascular formation is closely related to the occurrence and progression of tumors. Therefore, when diagnosing CD34 cancerous areas, it is necessary to calculate microvessel density (MVD). The MVD value represents the expression of CD34 in the tissue, thereby quantitatively reflecting the proliferation of microvessels.

[0005] Traditional biopsy diagnosis is performed entirely manually by doctors, a complex, time-consuming, and labor-intensive process that relies on extensive diagnostic experience. To address these shortcomings and deficiencies, the present invention proposes a deep learning-based method for CD34 pathology image recognition and microvessel density calculation in hepatocellular carcinoma. This method combines deep learning with digital pathology slide image recognition and analysis, and uses visualization to generate probability heatmaps. This method effectively assists doctors in diagnosing hepatocellular carcinoma and has important clinical implications. Summary of the Invention

[0006] In view of the shortcomings of the prior art described above, the present invention aims to provide a method for recognizing CD34 immunohistochemical images of hepatocellular carcinoma and calculating microvascular density. This method addresses the complex, time-consuming, and labor-intensive diagnostic process of manual methods, relying on extensive diagnostic experience, and provides a more comprehensive assessment of the severity of hepatocellular carcinoma in patients, guiding subsequent treatment. This method utilizes a deep learning network for binary image classification, demonstrating strong generalization and robustness. The generated probability heatmap intuitively reflects the model's recognition results. The microvascular density calculation is highly interpretable, simple, and convenient, making it easy for physicians to adjust and operate.

[0007] Therefore, the technical solution of the present invention is:

[0008] According to a first aspect of the present invention, a method for CD34 immunohistochemical image recognition and microvessel density calculation in hepatocellular carcinoma is provided, the method comprising:

[0009] Acquiring an original data set, wherein the original data set includes a CD34 full digital slice pathology scan image data set;

[0010] Downsampling the original data set and performing tissue region extraction on the data set to obtain a tissue region coordinate list, and dividing the original data set into a training set and a test set;

[0011] Generate cancer foci and normal region masks in the tissue region of the training set slice, and extract small images of the cancer foci and normal regions based on the cancer foci and normal region masks and the tissue region coordinate list;

[0012] Performing data augmentation and color standardization on the training set;

[0013] Divide the training set into training subset, validation subset, and test subset;

[0014] The training subset and the validation subset are placed in different convolutional neural networks for deep learning model training, and the trained convolutional neural network model is evaluated using the test subset to determine the convolutional neural network model with the best training effect;

[0015] Obtain a small image of the original slice corresponding to each coordinate in the tissue region coordinate list of the test set, normalize the color of each small image, and then put it into the optimal convolutional neural network model to obtain the corresponding prediction classification confidence;

[0016] Concatenate the predicted classification confidences of the test set in order and draw a probability heat map;

[0017] Based on the predicted classification confidence of the test set, the threshold is used for segmentation, the holes are filled, and the classification prediction results are plotted;

[0018] The small image predicted to be a cancerous area was converted into color space, the brown part was extracted, and the ratio of the extracted brown area to the total area was calculated to obtain the microvascular density.

[0019] Furthermore, the obtaining of the original data set specifically includes:

[0020] Acquire a CD34 full digital slice pathology scan image dataset, wherein the CD34 full digital slice pathology scan image dataset includes CD34 full digital slice pathology scan images and annotations of all cancerous lesions in each scan image, and the data has been desensitized;

[0021] The CD34 full-digital slide pathology scan image dataset was subjected to quality control screening to eliminate slides with quality control issues. The quality control issues included one or a combination of out-of-focus scanned slides, staining that was not within a preset range, stains that exceeded a preset area, and tissue cell overlap that exceeded a preset number.

[0022] Furthermore, the original data set is downsampled at a downsampling ratio of 2^7 or 2^6, where the downsampling ratio is a ratio of the length or width of the image before downsampling to the length or width of the image after downsampling.

[0023] Furthermore, the step of generating cancerous foci and normal region masks in the tissue region of the training set slices, and extracting small images of the cancerous foci and normal regions based on the cancerous foci and normal region masks and the tissue region coordinate list, specifically includes:

[0024] When extracting the cancerous area and the tissue area, the thumbnail size is larger than 244px*244px, and the layer where the thumbnail is located is the layer where the original image of the scanned image is located.

[0025] Furthermore, the data augmentation and color standardization of the training set specifically includes:

[0026] Augment the small image by random rotation and random flipping;

[0027] Normalizes the color of an image using the color normalization method.

[0028] Furthermore, the training subset and the validation subset are placed in different convolutional neural networks for deep learning model training, and the trained convolutional neural network model is evaluated using the test subset to determine the convolutional neural network model with the best training effect, specifically including:

[0029] Different convolutional neural network models include at least two of ResNet34, ResNet50, VGG19, MobileNet v2, and InceptionNet v3;

[0030] During model training, the image is normalized using the following formula:

[0031] dst=(src-128) / 128

[0032] Where src is the small image before conversion, and dst is the small image after conversion. Both are in the form of two-dimensional arrays and are in the range [0,255].

[0033] The output loss value during training adopts cross entropy loss; the learning rate during training adopts a piecewise decay strategy, setting the gamma value and step_size value. Every step_size rounds of iterative training, the learning rate is multiplied by the gamma value.

[0034] When testing and evaluating the model, the accuracy, precision, recall, and F1 score of the test set are calculated, and the area under the receiver operating characteristic curve is calculated for comprehensive evaluation:

[0035]

[0036]

[0037]

[0038]

[0039] Among them, TP is the positive sample predicted by the model as positive; TN is the negative sample predicted by the model as negative; FP is the negative sample predicted by the model as positive; FN is the positive sample predicted by the model as negative.

[0040] Furthermore, the predicted classification confidence based on the test set is divided using a threshold, and holes are filled to draw the classification prediction results, which specifically includes:

[0041] The confidence threshold is calculated using the following formula:

[0042]

[0043]

[0044] Y=Sensitivity+Specificity-1

[0045] Among them, Sensitivity represents sensitivity, Specificity represents specificity, and Y represents the confidence threshold;

[0046] Set the values ​​of all points in the area with a connected area less than the threshold th1 to zero; set the values ​​of all points in the area with a connected area less than the threshold th2 to one; in the area where the tissue area is 1 but the predicted area is still 0, recursively search from the inside to the outside for the category with the largest number of points in the enclosed area, and set it to that category.

[0047] Furthermore, the color space conversion is performed on the small image whose classification prediction result is a cancerous area to extract the brown part, which specifically includes:

[0048] Based on the following formula, the original BGR color space of the small image is converted to the HSV color space:

[0049] dst=cv2.cvtColor(src,cv2.COLOR_BGR2HSV)

[0050] Where src is the small image before conversion, and dst is the small image after conversion;

[0051] The upper bound thup and lower bound thdown of the HSV space should be set to extract the brown part.

[0052] According to a second aspect of the present invention, a device for CD34 immunohistochemical image recognition and microvessel density calculation of hepatocellular carcinoma is provided, the device comprising a processor configured to:

[0053] an acquisition module configured to acquire an original data set, wherein the original data set includes a CD34 full digital slice pathology scan image data set;

[0054] a list extraction module configured to downsample the original data set and perform tissue region extraction on the original data set to obtain a tissue region coordinate list, and divide the original data set into a training set and a test set;

[0055] a thumbnail extraction module configured to generate masks of cancerous foci and normal regions in tissue regions of a training set slice, and extract thumbnails of the cancerous foci and normal regions based on the masks and the tissue region coordinate list;

[0056] a training set processing module, configured to perform data augmentation and color normalization on the training set;

[0057] A partitioning module is configured to partition the training set into a training subset, a validation subset, and a test subset;

[0058] A training module is configured to place the training subset and the validation subset into different convolutional neural networks for deep learning model training, and use the test subset to evaluate the trained convolutional neural network model to determine the convolutional neural network model with the best training effect;

[0059] The prediction module is configured to obtain a small image of the original slice corresponding to each coordinate in the tissue region coordinate list of the test set, color-normalize each small image, and then put it into the optimal convolutional neural network model to obtain the corresponding prediction classification confidence;

[0060] A heat map drawing module is configured to sequentially concatenate the predicted classification confidences of the test set and draw a probability heat map;

[0061] A prediction result drawing module is configured to divide the test set based on the prediction classification confidence, fill the holes, and draw the classification prediction results;

[0062] The extraction and calculation module is configured to perform color space conversion on the small image whose classification prediction result is a cancerous area, extract the brown part, and calculate the ratio of the extracted brown area to the total area to obtain the microvascular density.

[0063] According to a third aspect of the present invention, a non-transitory computer-readable storage medium storing instructions is provided. When the instructions are executed by a processor, the method for CD34 immunohistochemical image recognition and microvessel density calculation of hepatocellular carcinoma according to any embodiment of the present invention is executed.

[0064] Compared with the prior art, the present invention has the following beneficial effects:

[0065] 1. The present invention is a deep learning-based method for recognizing CD34 immunohistochemical pathological images of hepatocellular carcinoma and calculating microvessel density. This method can reduce the pressure on doctors, give full play to the advantages of deep learning in the field of image processing, and accurately determine whether a tissue area has cancerous changes and calculate them, thereby assisting doctors in diagnosing CD34 immunohistochemical pathological sections.

[0066] 2. This paper, incorporating the characteristics of CD34 immunohistochemistry pathological sections of hepatocellular carcinoma, performs data augmentation and color normalization on small images during the deep learning image processing process, improving model generalization and classification accuracy. Furthermore, for the model output probability heatmap, this paper post-processes the binary classification results while ensuring classification accuracy to enhance the effect on the sections.

[0067] 3. The present invention combines the distinction between microvessels and tissues and the actual situation of doctors when calculating MVD, and adopts a color space conversion method to extract and obtain the MVD value, which has good interpretability, is easy to adjust, and is easily accepted by doctors. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In the drawings, which are not necessarily drawn to scale, the same reference numerals may describe similar components in different views. The same reference numerals with letter suffixes or different letter suffixes may represent different instances of similar components. The accompanying drawings generally illustrate various embodiments by way of example and not limitation, and together with the description and claims, serve to illustrate the embodiments of the invention. Where appropriate, the same reference numerals are used throughout the drawings to refer to the same or similar parts. Such embodiments are illustrative and are not intended to be exhaustive or exclusive of the embodiments of the present apparatus or method.

[0069] Figure 1 Flowchart of a method for CD34 immunohistochemical image recognition and microvessel density calculation in hepatocellular carcinoma according to an embodiment of the present invention.

[0070] Figure 2 Schematic diagram of image augmentation and color normalization according to an embodiment of the present invention.

[0071] Figure 3 Schematic diagram of CD34 immunohistochemistry pathological section of primitive hepatocellular carcinoma containing cancerous foci areas marked by doctors according to an embodiment of the present invention.

[0072] Figure 4 This is a probability heat map output by the model of the present invention according to an embodiment of the present invention.

[0073] Figure 5 This is a binary prediction graph output by the model of the present invention according to an embodiment of the present invention.

[0074] Figure 6 This is a binary prediction image using a post-processing method according to an embodiment of the present invention.

[0075] Figure 7 This is a superimposed image of the outline of the cancerous area in the binary prediction image and the original image according to an embodiment of the present invention.

[0076] Figure 8 This is a small picture of CD34 immunohistochemical pathological section containing microvessels according to an embodiment of the present invention.

[0077] Figure 9 This is a binary image obtained by using the microvessel extraction method of the present invention according to an embodiment of the present invention.

[0078] Figure 10 2 is a structural diagram of a device for hepatocellular carcinoma CD34 immunohistochemistry image recognition and microvessel density calculation according to an embodiment of the present invention. DETAILED DESCRIPTION

[0079] The following examples are merely intended to better illustrate the present invention, but the present invention is not limited to the examples set forth herein. Therefore, non-essential improvements and adjustments to the embodiments made by those skilled in the art based on the above-described invention and applied to other embodiments are still within the scope of the present invention.

[0080] The present invention will now be further described with reference to the accompanying drawings.

[0081] The embodiment of the present invention provides a method for CD34 immunohistochemical image recognition and microvessel density calculation of hepatocellular carcinoma. The CD34 immunohistochemical pathological section image data of hepatocellular carcinoma obtained by a hospital within half a year is used as an example to construct an implementation case of the present invention. The present invention first uses a deep learning network to complete the binary classification task of cancerous area and normal area, and then extracts microvessels from the cancerous area to calculate MVD. The flowchart of the present invention based on the implementation example is as follows Figure 1 As shown, the following steps are included:

[0082] S1. Obtain CD34 full digital slide pathology scan image dataset and perform image quality control screening.

[0083] In some embodiments, step S1 specifically includes the following steps:

[0084] S1-1. Obtain a dataset of CD34 full-digital slice pathology scan images of hepatocellular carcinoma from a hospital within six months. The dataset should include CD34 full-digital slice pathology scan images and accurate annotation of all cancerous lesions in each image. The data has been desensitized. For a detailed slice diagram, please refer to Figure 3 shown.

[0085] S1-2. During the quality control screening process, scanned sections with problems such as out-of-focus, too light or too dark staining, large stains, or large overlapping tissue cells should be excluded.

[0086] S2. Downsample the original dataset and perform tissue region extraction to obtain a list of tissue region coordinates, and divide the original dataset into a training set and a test set at a ratio of 5:1.

[0087] In some embodiments, step S2 specifically includes the following steps:

[0088] S2-1. The downsampling ratio (i.e., the length or width of the image before downsampling: the length or width of the image after downsampling) is usually 2^7 or 2^6. A downsampling ratio of 2^7 is preferred because it is about 4 times faster than 2^6 without significantly affecting accuracy.

[0089] S3. Generate masks of the cancerous foci and normal regions in the tissue regions of the training set slices, and extract small images of the cancerous foci and normal regions based on the masks and the tissue region coordinate list in S2.

[0090] In some embodiments, step S3 specifically includes the following steps:

[0091] S3-1, when extracting the cancerous area and tissue area, the thumbnail size should be larger than 244px*244px, preferably, the thumbnail size is 256px*256px. The layer where the thumbnail is located should be the layer where the original image of the scanned image is located, generally an image at 40x magnification. Figure 8 Shown is a small image of a CD34 immunohistochemical pathological section containing microvessels.

[0092] S4. Perform data augmentation and color standardization on the training set, such as Figure 2 shown.

[0093] In some embodiments, step S4 specifically includes the following steps:

[0094] S4-1, augment the small image by random rotation (rotation by positive integer multiples of 90°) and random flipping (up and down flipping, left and right flipping);

[0095] S4-2. Perform color standardization on the image using a color standardization method. It is recommended to use the Macenko or Vahadane method, with the Macenko method being preferred because it has a faster processing speed and relatively better processing results.

[0096] S5. Divide the training set into training subset, validation subset, and test subset in a ratio of 7:1:2.

[0097] S6: Train the classification model. Put the training subset and validation subset in S5 into different convolutional neural networks for deep learning model training. Use the test subset to evaluate the model to select the model with the best training effect.

[0098] In some embodiments, step S6 specifically includes the following steps:

[0099] S6-1. Using the PyTorch deep learning framework to build this model training method;

[0100] S6-2. For pre-trained backbone network models, ResNet34, ResNet50, VGG19, MobileNet v2, InceptionNet v3, or other classification models can be used. Based on actual training results, the recommended order is: ResNet34, ResNet50, MobileNet v2, InceptionNet v3, VGG19. Based on actual training results, the ImageNet dataset is preferred for pre-training backbone network models, followed by the CIFAR series.

[0101] S6-3. Whether or not the model is batch normalized during training has no significant impact on the model results. In terms of training efficiency, batch normalization is not necessary;

[0102] S6-4. When training the model, the image needs to be normalized to make the model converge faster. For a new dataset, it is not necessary to calculate the mean and standard deviation of the dataset based on the dataset. Only the most basic normalization method can be used, namely:

[0103] dst=(src-128) / 128

[0104] Where src is the image before conversion, and dst is the image after conversion. Both are in the form of two-dimensional arrays in the range [0, 255]. The output loss value during training uses cross-entropy loss; the optimizer used during training is SGD or Adam, with SGD being the preferred optimizer. The learning rate during training uses a piecewise decay strategy, with appropriate gamma and step_size values ​​set. After every step_size rounds of training, the learning rate is multiplied by gamma.

[0105] S6-5. When testing and evaluating the model, the accuracy, precision, recall, and F1 score of the test set should be calculated, and the area under the receiver operating characteristic (ROC) curve (AUC) should be calculated for comprehensive evaluation. The formula is as follows:

[0106]

[0107]

[0108]

[0109]

[0110] TP is the positive sample predicted by the model as positive (cancer area); TN is the negative sample predicted by the model as negative (normal area); FP is the negative sample predicted by the model as positive; FN is the positive sample predicted by the model as negative.

[0111] S7. Obtain a small square image of the original slice corresponding to each coordinate in the tissue region coordinate list of the test set in S2. After color standardization, each small image is put into the convolutional neural network with the best effect in S6 to obtain the corresponding prediction classification confidence.

[0112] S8, concatenate the predicted classification confidence of the test set in S7 in order and draw a probability heat map, such as Figure 4 shown.

[0113] S9, divide the test set probability heat map obtained in S8 into confidence levels using a threshold, and use post-processing methods to fill in the holes and draw the classification prediction results, such as Figure 5 shown.

[0114] In some embodiments, step S9 specifically includes the following steps:

[0115] S9-1. The confidence interval is [0,1]. The confidence threshold is preferably 0.7. You can also use the Youden index for more accurate calculation. The Youden index calculation requires sensitivity and specificity. The calculation formula is as follows:

[0116]

[0117]

[0118] Y=Sensitivity+Specificity-1

[0119] S9-2. The post-processing method consists of three parts, which are operated in series from the beginning to the end. They are: the elimination algorithm, the filling algorithm, and the expansion algorithm. The elimination algorithm sets the values ​​of all points in the area with a connected area less than the threshold th1 to zero; the filling algorithm sets the values ​​of all points in the area with a connected area less than the threshold th2 to one; the expansion algorithm operates after the filling algorithm and is applied to areas where the tissue area is 1 but the predicted area is still 0. The point is recursively searched from the inside out to find the category with the largest number of points in the enclosed area, and is placed in that category.

[0120] Preferably, the value of th1 in the post-processing algorithm is 5, and the value of th2 is 20.

[0121] like Figure 6 As shown in , it is a binary prediction map obtained by post-processing method, such as Figure 7As shown in FIG, the outline of the cancer area in the binary prediction image is superimposed with the original image.

[0122] S10. Perform color space conversion on the small square image of the cancerous area whose classification prediction result of the test set in S2 is extracted, extract the brown part, and calculate the ratio of the extracted brown area to the total area, which is MVD.

[0123] In some embodiments, step S10 specifically includes the following steps:

[0124] S10-1. Use OpenCV to read a small image. Its original color space is BGR and should be converted to HSV color space. You need to call the OpenCV function:

[0125] dst=cv2.cvtColor(src,cv2.COLOR_BGR2HSV)

[0126] Where src is the small image before conversion, and dst is the small image after conversion;

[0127] S10-2, when extracting the brown part, the upper bound of the HSV space should be set. up and the lower bound th down . Preferably, the upper bound th up The value of is [11,70,46], the lower bound th down is [20,255,255].

[0128] like Figure 9 As shown, it is a binary image using the microvessel extraction method of the present invention. The present invention uses a deep learning network to perform binary classification on the image, which has strong generalization ability and high robustness; the generated probability heat map intuitively reflects the model recognition result; the calculation of microvessel density is highly interpretable, simple and convenient, and easy for doctors to operate and adjust.

[0129] The embodiment of the present invention also provides a device for hepatocellular carcinoma CD34 immunohistochemistry image recognition and microvessel density calculation, such as Figure 10 As shown, the device 1000 includes:

[0130] An acquisition module 1001 is configured to acquire an original data set, wherein the original data set includes a CD34 full digital slide pathology scan image data set;

[0131] A list extraction module 1002 is configured to downsample the original data set and perform tissue region extraction on the original data set to obtain a tissue region coordinate list, and divide the original data set into a training set and a test set;

[0132] A thumbnail extraction module 1003 is configured to generate a mask of the cancerous foci region and the normal region in the tissue region of the training set slice, and extract thumbnails of the cancerous foci region and the normal region based on the mask of the cancerous foci region and the normal region and the tissue region coordinate list;

[0133] A training set processing module 1004 is configured to perform data augmentation and color normalization on the training set;

[0134] A partitioning module 1005 is configured to partition the training set into a training subset, a validation subset, and a test subset;

[0135] The training module 1006 is configured to place the training subset and the validation subset into different convolutional neural networks for deep learning model training, and use the test subset to evaluate the trained convolutional neural network model to determine the convolutional neural network model with the best training effect;

[0136] Prediction module 1007 is configured to obtain a small image of the original slice corresponding to each coordinate in the tissue region coordinate list of the test set, perform color standardization on each small image, and then input it into the optimal convolutional neural network model to obtain the corresponding prediction classification confidence;

[0137] The heat map drawing module 1008 is configured to sequentially concatenate the predicted classification confidences of the test set and draw a probability heat map;

[0138] The prediction result drawing module 1009 is configured to divide the test set based on the prediction classification confidence, fill the holes, and draw the classification prediction results;

[0139] The extraction and calculation module 1010 is configured to perform color space conversion on the small image whose classification prediction result is a cancerous area, extract the brown part, and calculate the ratio of the extracted brown area to the total area to obtain the microvessel density.

[0140] In some embodiments, the acquisition module is further configured to:

[0141] Acquire a CD34 full digital slice pathology scan image dataset, wherein the CD34 full digital slice pathology scan image dataset includes CD34 full digital slice pathology scan images and annotations of all cancerous lesions in each scan image, and the data has been desensitized;

[0142] The CD34 full-digital slide pathology scan image dataset was subjected to quality control screening to eliminate slides with quality control issues. The quality control issues included one or a combination of out-of-focus scanned slides, staining that was not within a preset range, stains that exceeded a preset area, and tissue cell overlap that exceeded a preset number.

[0143] In some embodiments, the list extraction module is further configured to: downsample the original data set at a downsampling ratio of 2^7 or 2^6, where the downsampling ratio is the ratio of the length or width of the image before downsampling to the length or width of the image after downsampling.

[0144] In some embodiments, the thumbnail extraction module is further configured to:

[0145] When extracting the cancerous area and the tissue area, the thumbnail size is larger than 244px*244px, and the layer where the thumbnail is located is the layer where the original image of the scanned image is located.

[0146] In some embodiments, the training set processing module is further configured to:

[0147] Augment the small image by random rotation and random flipping;

[0148] Normalizes the color of an image using the color normalization method.

[0149] In some embodiments, the training module block is further configured to:

[0150] Different convolutional neural network models include at least two of ResNet34, ResNet50, VGG19, MobileNet v2, and InceptionNet v3;

[0151] During model training, the image is normalized using the following formula:

[0152] dst=(src-128) / 128

[0153] Where src is the small image before conversion, and dst is the small image after conversion. Both are in the form of two-dimensional arrays and are in the range [0,255].

[0154] The output loss value during training adopts cross entropy loss; the learning rate during training adopts a piecewise decay strategy, setting the gamma value and step_size value. Every step_size rounds of iterative training, the learning rate is multiplied by the gamma value.

[0155] When testing and evaluating the model, the accuracy, precision, recall, and F1 score of the test set are calculated, and the area under the receiver operating characteristic curve is calculated for comprehensive evaluation:

[0156]

[0157]

[0158]

[0159]

[0160] TP is the positive sample predicted by the model as positive; TN is the negative sample predicted by the model as negative; FP is the negative sample predicted by the model as positive; and FN is the positive sample predicted by the model as negative.

[0161] In some embodiments, the prediction result drawing module is further configured to:

[0162] The confidence threshold is calculated using the following formula:

[0163]

[0164]

[0165] Y=Sensitivity+Specificity-1

[0166] Among them, Sensitivity represents sensitivity, Specificity represents specificity, and Y represents the confidence threshold;

[0167] Set the values ​​of all points in the area with a connected area less than the threshold th1 to zero; set the values ​​of all points in the area with a connected area less than the threshold th2 to one; in the area where the tissue area is 1 but the predicted area is still 0, recursively search from the inside to the outside for the category with the largest number of points in the enclosed area, and set it to that category.

[0168] In some embodiments, the extraction and calculation module is further configured to:

[0169] Based on the following formula, the original BGR color space of the small image is converted to the HSV color space:

[0170] dst=cv2.cvtColor(src,cv2.COLOR_BGR2HSV)

[0171] Where src is the small image before conversion, and dst is the small image after conversion;

[0172] The upper bound thup and lower bound thdown of the HSV space should be set to extract the brown part.

[0173] It should be noted that the device for hepatocellular carcinoma CD34 immunohistochemical image recognition and microvessel density calculation provided in this embodiment shares the same technical concept as the previous method for hepatocellular carcinoma CD34 immunohistochemical image recognition and microvessel density calculation, and can achieve the same beneficial effects, which will not be described in detail here.

[0174] An embodiment of the present invention further provides a non-transitory computer-readable storage medium storing instructions, which, when executed by a processor, executes the method for CD34 immunohistochemical image recognition and microvessel density calculation of hepatocellular carcinoma as described in any of the above embodiments.

[0175] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some or all of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and description of the present invention.

Claims

1. A method for CD34 immunohistochemical image recognition and microvessel density calculation in hepatocellular carcinoma, characterized in that: The method comprises: Acquiring an original data set, wherein the original data set includes a CD34 full digital slice pathology scan image data set; Downsampling the original data set and performing tissue region extraction on the data set to obtain a tissue region coordinate list, and dividing the original data set into a training set and a test set; Generate cancer foci and normal region masks in the tissue region of the training set slice, and extract small images of the cancer foci and normal regions based on the cancer foci and normal region masks and the tissue region coordinate list; Performing data augmentation and color standardization on the training set; Divide the training set into training subset, validation subset, and test subset; The training subset and the validation subset are placed in different convolutional neural networks for deep learning model training, and the trained convolutional neural network model is evaluated using the test subset to determine the convolutional neural network model with the best training effect; Obtain a small image of the original slice corresponding to each coordinate in the tissue region coordinate list of the test set, normalize the color of each small image, and then put it into the optimal convolutional neural network model to obtain the corresponding prediction classification confidence; Concatenate the predicted classification confidences of the test set in order and draw a probability heat map; Based on the predicted classification confidence of the test set, the threshold is used for segmentation, the holes are filled, and the classification prediction results are plotted; The small image predicted as a cancerous area was converted into a color space, the brown part was extracted, and the ratio of the extracted brown area to the total area was calculated to obtain the microvascular density. The predicted classification confidence based on the test set is divided using a threshold, and holes are filled, and the classification prediction results are drawn, specifically including: The confidence threshold is calculated using the following formula: Y=Sensitivity+Specificity-1 Among them, Sensitivity represents sensitivity, Specificity represents specificity, and Y represents the confidence threshold; Set the values ​​of all points in the area with a connected area less than the threshold th1 to zero; set the values ​​of all points in the area with a connected area less than the threshold th2 to one; in the area where the tissue area is 1 but the predicted area is still 0, recursively search from the inside out for the category with the largest number of points in the enclosed area, and set it to that category; The color space conversion is performed on the small image whose classification prediction result is a cancerous area to extract the brown part, specifically including: Based on the following formula, the original BGR color space of the small image is converted to the HSV color space: dst=cv2.cvtColor(src,cv2.COLOR_BGR2HSV) Where src is the small image before conversion, and dst is the small image after conversion; Set the upper bound thup and lower bound thdown of the HSV space to extract the brown part.

2. The method for hepatocellular carcinoma CD34 immunohistochemical image recognition and microvessel density calculation according to claim 1, characterized in that: The obtaining of the original data set specifically includes: Acquire a CD34 full digital slice pathology scan image dataset, wherein the CD34 full digital slice pathology scan image dataset includes CD34 full digital slice pathology scan images and annotations of all cancerous lesions in each scan image, and the data has been desensitized; The CD34 full-digital slide pathology scan image dataset was subjected to quality control screening to eliminate slides with quality control issues. The quality control issues included one or a combination of out-of-focus scanned slides, staining that was not within a preset range, stains that exceeded a preset area, and tissue cell overlap that exceeded a preset number.

3. The method for hepatocellular carcinoma CD34 immunohistochemical image recognition and microvessel density calculation according to claim 1, characterized in that: The original data set is downsampled at a downsampling ratio of 2^7 or 2^6, where the downsampling ratio is a ratio of the length or width of the image before downsampling to the length or width of the image after downsampling.

4. The method for hepatocellular carcinoma CD34 immunohistochemical image recognition and microvessel density calculation according to claim 1, characterized in that: Generating cancer foci and normal region masks in the tissue region of the training set slice, and extracting small images of the cancer foci and normal regions according to the cancer foci and normal region masks and the tissue region coordinate list, specifically includes: When extracting the cancerous area and the tissue area, the thumbnail size is larger than 244px*244px, and the layer where the thumbnail is located is the layer where the original image of the scanned image is located.

5. The method for hepatocellular carcinoma CD34 immunohistochemical image recognition and microvessel density calculation according to claim 1, characterized in that: The data augmentation and color standardization of the training set specifically include: Augment the small image by random rotation and random flipping; Normalizes the color of an image using the color normalization method.

6. The method for hepatocellular carcinoma CD34 immunohistochemical image recognition and microvessel density calculation according to claim 1, characterized in that: The step of placing the training subset and the validation subset into different convolutional neural networks for deep learning model training, and using the test subset to evaluate the trained convolutional neural network model to determine the convolutional neural network model with the best training effect, specifically includes: Different convolutional neural network models include at least two of ResNet34, ResNet50, VGG19, MobileNet v2, and InceptionNet v3; During model training, the image is normalized using the following formula: dst=(src-128) / 128 Where src is the small image before conversion, and dst is the small image after conversion. Both are in the form of two-dimensional arrays and are in the range [0,255]. The output loss value during training adopts cross entropy loss; the learning rate during training adopts a piecewise decay strategy, setting the gamma value and step_size value. Every step_size rounds of iterative training, the learning rate is multiplied by the gamma value. When testing and evaluating the model, the accuracy, precision, recall, and F1 score of the test set are calculated, and the area under the receiver operating characteristic curve is calculated for comprehensive evaluation: TP is the positive sample predicted by the model as positive; TN is the negative sample predicted by the model as negative; FP is the negative sample predicted by the model as positive; and FN is the positive sample predicted by the model as negative.

7. A device for CD34 immunohistochemical image recognition and microvessel density calculation of hepatocellular carcinoma, characterized in that: The device comprises: an acquisition module configured to acquire an original data set, wherein the original data set includes a CD34 full digital slice pathology scan image data set; a list extraction module configured to downsample the original data set and perform tissue region extraction on the original data set to obtain a tissue region coordinate list, and divide the original data set into a training set and a test set; a thumbnail extraction module configured to generate masks of cancerous foci and normal regions in tissue regions of a training set slice, and extract thumbnails of the cancerous foci and normal regions based on the masks and the tissue region coordinate list; a training set processing module, configured to perform data augmentation and color normalization on the training set; A partitioning module is configured to partition the training set into a training subset, a validation subset, and a test subset; A training module is configured to place the training subset and the validation subset into different convolutional neural networks for deep learning model training, and use the test subset to evaluate the trained convolutional neural network model to determine the convolutional neural network model with the best training effect; The prediction module is configured to obtain a small image of the original slice corresponding to each coordinate in the tissue region coordinate list of the test set, color-normalize each small image, and then put it into the optimal convolutional neural network model to obtain the corresponding prediction classification confidence; A heat map drawing module is configured to sequentially concatenate the predicted classification confidences of the test set and draw a probability heat map; A prediction result drawing module is configured to divide the test set based on the prediction classification confidence, fill the holes, and draw the classification prediction results; The extraction and calculation module is configured to perform color space conversion on the small image that is classified as a cancerous area, extract the brown part, and calculate the ratio of the extracted brown area to the total area to obtain the microvascular density; The prediction result drawing module is further configured to: The confidence threshold is calculated using the following formula: Y=Sensitivity+Specificity-1 Among them, Sensitivity represents sensitivity, Specificity represents specificity, and Y represents the confidence threshold; Set the values ​​of all points in the area with a connected area less than the threshold th1 to zero; set the values ​​of all points in the area with a connected area less than the threshold th2 to one; in the area where the tissue area is 1 but the predicted area is still 0, recursively search from the inside out for the category with the largest number of points in the enclosed area, and set it to that category; The extraction and calculation module is further configured to: Based on the following formula, the original BGR color space of the small image is converted to the HSV color space: dst=cv2.cvtColor(src,cv2.COLOR_BGR2HSV) Where src is the small image before conversion, and dst is the small image after conversion; Set the upper bound thup and lower bound thdown of the HSV space to extract the brown part. 8 . A non-transitory computer-readable storage medium storing instructions, which, when executed by a processor, executes the method according to claim 1 .

Images