Cushing disease para-tumor ACTH expression and prognosis measurement method based on deep learning

Abstract

The invention discloses a Cushing disease para-tumor ACTH expression and prognosis measurement method based on deep learning, and the method comprises the steps: segmenting clustering blocks from two input images with different amplification factors through employing a single-link clustering algorithm, and carrying out the cross-modal pairing and registration of the clustering blocks; inputting one type of clustering blocks into a trained multi-instance learning (MIL) classifier to obtain a positive region, namely a cell nucleus dense deep staining region in a T-PIT staining image, and projecting the positive region to the other type of clustering blocks according to a transformation matrix obtained by registration; and finally, carrying out image segmentation on the projection area of the other type of clustering blocks and analyzing the hormone level to realize prognosis measurement. According to the method, the distance matrix can be extracted from the original image only through one-time distance transformation, and various similarity measures and various optimization algorithms are combined for use according to a coarse-to-fine thought, so that the calculation amount of image segmentation is remarkably reduced, and meanwhile, the various similarity measures and the various optimization algorithms can be combined for use; and performing step-by-step registration on the images under different resolutions.

Description

Technical Field

[0001] The present invention relates to a technology in the field of image processing, specifically a method for measuring the expression and prognosis of adreno-cortico-tropic-hormone (ACTH) adjacent to a Cushing's disease tumor based on deep learning. Background Art

[0002] Existing image processing technology will produce images with severe noise and large differences between modalities when processing Cushing's disease images. It has problems such as inability to complete registration, excessive calculation, and inability to correctly distinguish between foreground and background, positive and negative areas. Summary of the invention

[0003] In view of the difficulties of the prior art, the present invention proposes a method for measuring peritumoral ACTH and prognosis of Cushing's disease based on deep learning. Through single-link clustering, the distance matrix can be extracted from the original image through only one distance transformation, and multiple similarity metrics and multiple optimization algorithms can be combined in a coarse-to-fine manner, which significantly reduces the computational complexity of image segmentation and can combine multiple similarity metrics and multiple optimization algorithms to perform step-by-step registration of images at different resolutions.

[0004] The present invention is achieved through the following technical solutions:

[0005] The present invention relates to a method for peritumoral ACTH expression and prognosis measurement of Cushing's disease based on deep learning. After cluster blocks are segmented from two input images with different magnifications by using a single-link clustering algorithm, cross-modal pairing and registration are performed on them respectively; one type of cluster blocks is input into a trained multiple instance learning (MIL) classifier to obtain a positive area, that is, a densely dark-stained area of ​​cell nuclei in a T-PIT staining image, and the positive area is projected onto another type of cluster blocks according to a transformation matrix obtained by registration; finally, image segmentation is performed on the projected area of ​​the other type of cluster blocks, and hormone levels are analyzed to achieve prognosis measurement.

[0006] The present invention relates to a system for implementing the above method, comprising: a hierarchical clustering unit, an image registration unit, an image classification unit, a transformation mapping unit and an image segmentation and data processing unit, wherein: the hierarchical clustering unit performs an improved single-link hierarchical clustering process according to a pathological whole slide image (WSI), and obtains a plurality of cluster blocks arranged from large to small in area in two images to be registered; the image registration unit performs an integrated multi-modal image registration process from coarse to fine flow according to the information of the cluster blocks corresponding to each other obtained by the hierarchical clustering unit, and obtains the rigid transformation matrix of each cluster block; the image classification unit performs a convolutional neural network-based multi-modal image registration process according to the information of each cluster block belonging to the motion image. The image classification processing of multiple instance learning is performed to obtain tumor-positive areas and tumor-negative areas; the transformation mapping unit projects the positive area and negative area information on the moving image obtained by the image classification unit onto the fixed image through the transformation matrix obtained by the image registration unit, and obtains the result of segmenting the fixed image into positive areas and negative areas; the image segmentation and data processing unit performs threshold-based image segmentation on the fixed image after white balance processing according to the positive area and negative area information in the fixed image obtained by the transformation mapping unit, and obtains the graded hormone expression effects of the positive area and negative area in the fixed image, and displays the specific values ​​and their weighted averages to the user in the form of a list. Technical Effects

[0007] Compared with the existing technology, the present invention significantly reduces the computational complexity of the pathological image segmentation task based on single-link hierarchical clustering, achieves a higher accuracy than the existing technology in the multimodal Cushing's disease pathological image registration task with severe noise interference, and achieves a higher accuracy than the existing technology in the classification task of t-pit (or denoted as Tpit) stained plaque images. It is the first end-to-end Cushing's disease multimodal pathological section image data analysis system. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 It is a flow chart of the present invention;

[0009] Figure 2 Schematic diagram of WSI level image clustering;

[0010] In the figure: (a) and (b) are the original WSI images stained with adreno-cortico-tropic-hormone (ACTH) and t-pit (the transcription of ACTH-secreting tumor cells), respectively; (c) and (d) are abstract representations of the shape, angle, and position relationship of each cluster block in (a) and (b), indicating the necessity of registering cluster blocks separately; (e) is the detailed process of improved single-link hierarchical clustering for the purpose of extracting the valid registration area;

[0011] Figure 3 Schematic diagram of cluster block level image registration;

[0012] In the figure: (a) are three typical pairs of images to be registered (fixed images, moving images); (b) is a detailed flowchart of the two-stage intensity-based integrated registration. Each optimization module (gradient descent optimization or on-plus-one evolutionary optimization) has a separate back-propagation metric (yellow box), except for the two marked ones, the rest are omitted in the figure; (c)-(e) are the registration results of the three image pairs shown in (a); (f) is a typical example where internal gaps are conducive to registration; (g) is a typical example where internal gaps interfere with registration;

[0013] Figure 4 This is a comparison between the registration method of the present invention and other existing technologies on a Cushing's disease dataset, specifically the registration results of 8 feature point algorithms for 3 groups of typical sample pairs.

[0014] In the figure: (a) is the SURF method; (b) is the MSER method; (c) is the FAST method; (d) is the BRISK method; (e) is the Harris method; (f) is the MinEigen method; (g) is the KAZE method; (h) is the ORB method;

[0015] Figure 5 Flowchart for patch-level image classification and statistical plots of convergence performance of different optimization algorithm variants;

[0016] In the figure: (a) is the model architecture designed by the present invention; (b)-(c) are performance and convergence speed comparison diagrams of 10 optimization algorithms;

[0017] Figure 6 Display the user interface. DETAILED DESCRIPTION

[0018] like Figure 1 As shown, this embodiment relates to a method for measuring ACTH expression and prognosis of Cushing's disease adjacent to the tumor based on deep learning, including:

[0019] The first step is to use the single-link clustering method to achieve image-level clustering, which includes:

[0020] 1.1) Find all n connected domains in the input image I and number them from 1 to n, fill each connected domain with its number as the gray value, and get the filled image I N , where the gray value is greater than 0 in area I fg , which is the foreground area of ​​the input image I.

[0021] 1.2) Use Matlab's bwist function to obtain the binary image binary I from the filled imageN Get the input image I to region I fg The nearest distance field D and the nearest pixel index map IDX are recorded for each point I on the map I i (For simplicity, one dimension is used to represent two-dimensional coordinates) The output of this link [D i IDX i ], specifically: Where: Distance(I i ,I k ) refers to two points I i with I k The Euclidean distance between i Point I i The shortest distance to the nearest connected domain, IDX i are the coordinates of the points closest to this point in all connected domains. The width and height of D and IDX are the same as those of the input image I.

[0022] 1.3) Take the IDX calculated by step 1.2 for each point i As coordinates, search for the label of the area closest to it in the filled connected domain, and set the value N i =I N (IDX i ) is assigned as a feature to each point, and the nearest area field image N is obtained. The gray value of the pixel with coordinate i in image N is N i =I N (IDX i ),i∈I.

[0023] 1.4) Perform a dilation operation on the nearest region field image N with a scale of (3, 3) to obtain the dilated image N dilated , the difference line with a width of 1 pixel obtained by comparing the two is the boundary line of the region field, the boundary source information, that is, the number of the two regions and the distance information of the boundary bd = 2*D(j), where: j∈(N dilated -N dilated ∩N);

[0024] The boundary source information includes: the pixel value before expansion is the smaller area number of the two adjacent areas, that is, label1 i = N(i), the pixel value after expansion is the larger area number of the two adjacent areas, that is, label2 i =N dilated (i), i∈(N dilated -N dilated ∩N).

[0025] 1.5) Substitute the distance information bd and the boundary source information into the distance matrix DM whose diagonal elements are all 0, specifically: Perform hierarchical clustering on the distance matrix DM to obtain clustering trees for the two images, such as Figure 2 (e) The last row shows.

[0026] 1.6) According to the observation of the input image I, the cluster tree is uniformly cut at a tree height of 100. Specifically, the ACTH staining image is cut into 5 classes, and the t-pit staining image is cut into 2 classes. It can be seen that the former has 3 more impurities than the latter. Since the ACTH staining image and the t-pit staining image are made by staining adjacent thin slices on each fragment in the same sample, the area of ​​each fragment on different staining images will not change significantly. Therefore, the order of arrangement of the fragments in the same image by area size will not change in another image. After arranging by area size, the first two classes are matched one by one, and the preprocessing work of registration, that is, registration ROI extraction, is completed.

[0027] The second step is cluster block level registration, which is to extract the image foreground to screen the key features, weaken the texture features and strengthen the color features.

[0028] The extracted image foreground includes:

[0029] ① Taking advantage of the fact that there are gaps in the stained image and the background has little difference except for a small amount of impurities, a disk with a radius of 4 is used to traverse the image and remove the part with small variance, specifically: Where: disk(I i ) indicates that I i is a disk with a radius of 4 and a center. t represents the threshold for screening variance. distance(I i ,I k ) represents the Euclidean distance between two points;

[0030] In this embodiment, the threshold is 400 pixels (applicable to Cushing's disease pathology images).

[0031] ② Close the remaining connected areas and remove the areas with an area less than 400 pixels squared. This will remove the impurities in the foreground area. Since the foreground image has almost only contour features, taking the minimum resolution of 0.625 times can cover the basic information of the image.

[0032] ③ Such as Figure 3As shown in (b), the intensity-based integrated registration method is used. First, the image with a magnification of 0.625 times (scale = 0.625) is initially unimodally registered based on the affine transformation in the lower section to find the approximate rotation angle and displacement. This is then used as the initial matrix to continue multimodally registering the image with a resolution of 5 times. Finally, the integrated learning model is used to try a variety of initial conditions and image preprocessing methods, and the transformation matrix with the best result is selected until the two images are completely corresponding, as shown in Figure 2. Figure 3 (c) and Figure 3 (e), specifically:

[0033] The coordinates before the transformation are (x, y), and the coordinates after the transformation are (x', y'). For rigid registration, the affine transformation matrix is ​​as follows: Where: θ represents the rigid rotation angle of the image. w and h determine the translation and scaling process of the image. When the program uses the result matrix of the registration under scale = 0.625 as the initial matrix of the registration under scale = 5, the w and h values ​​in the result matrix are automatically multiplied by 8. When the result matrix of the registration under scale = 5 is applied to the image transformation under scale = 10, the w and h values ​​in the result matrix are automatically multiplied by 2.

[0034] In addition, in this embodiment, the following two situations are taken into consideration when processing tissue gaps in an image. Figure 3 As shown in (f), the gap is feature information that is helpful for registration. Figure 3 As shown in (g), the gap is redundant information that interferes with the registration. In order to take both situations into account, in the registration based on affine transformation, each pair of registered images is registered twice using image processing technology to remove the gap and retain the gap, and the transformation matrix with the highest degree of fit is selected based on the variance after pixel-by-pixel subtraction between the two images, and then connected to the downstream task.

[0035] Because image registration is only one of the steps of this software, for the image group that failed to be registered, the software requires that the doctor can perform manual registration and further execute the subsequent steps. This embodiment uses the app.UIAxes module in MATLAB to implement interactive translation operations, and makes an angleSlider module to implement rotation operations. The user clicks the moving image to start controlling the translation movement of the image, and stops moving after clicking again. The movement trajectory of the image is the same as the real-time movement trajectory of the mouse. At the same time, the user can adjust the rotation angle of the moving image by dragging the slider angleSlider until the moving image completely overlaps with the fixed image. After the manual registration is completed, click the completion button above to save the corresponding transformation matrix at this moment as the result output of the registration stage. This operation is performed after artificially discovering that the registration based on affine transformation fails, so as to improve the fault tolerance of this embodiment.

[0036] Step 3: After constructing and training a multi-instance learning (MIL) classifier, perform patch-level image classification on the clustering blocks of the motion images obtained in Step 1:

[0037] As Figure 5 (a) shows, the MIL classifier is a convolutional neural network (ConvNet), including: a convolutional layer (Conv), a pooling layer (pool), an activation function (Relu), and an average layer, where: ConvNet applies four convolutional layers with padding. The operators of the first two layers are 5×5, and the others are 3×3; the average layer calculates the output of each batch of samples, that is, maps the input features to a single scalar value. Except for the last layer, batch normalization and rectified linear activation function (RELU) are utilized throughout ConvNet.

[0038] The MIL classifier is trained using the Adam optimization algorithm, and the recognition rate reaches 97.8%.

[0039] Step 4: Pixel-level image segmentation:

[0040] In this embodiment, with four thresholds as boundaries, the normal cell regions segmented from the ACTH staining images in the previous step are divided into three different sub-regions: positive + region, positive ++ region, and positive +++ region. By correcting using the background average intensity of all images, the positive threshold criteria for calibrating the images are set as: positive grade I with an intensity of 180 - 120, positive grade II with an intensity of 120 - 70, and positive grade III with an intensity of 70 - 40, and the weighted sum result of each region is used as the quantitative analysis result of the entire image.

[0041] It is observed that there are significant differences in the background colors of different images, which are caused by inconsistent exposure times during shooting. To eliminate the influence of exposure time on subsequent quantification and grading, in this embodiment, before pixel-level image segmentation in Step 4, the background region is found through significant region segmentation technology, the average pixel value of the background regions of all images is calculated, and the intensity of the entire image is calibrated by using the difference between this value and the average value of the background regions of each image.

[0042] After specific actual experiments, on some of the Cushing's disease pathology dataset samples provided at github.com / lhy2024 / CRCS, through Figure 6The program shown implements the above method. After startup, the user needs to click "Select Folder" in the operation interface to select the folder named "sample_block" where the data set samples are stored. The program will automatically read all subfolder names (i.e., sample numbers) and display them in the list on the left. Click any number in the list on the left to start automatic registration in the default mode, and the registration results will be displayed on the canvas in the middle. After confirming that the registration is correct by visual observation, click the "Accept" button to view the graded quantitative data of ACTH hormone expression in the tumor and peritumoral area in the list on the right. If the pathological image of the sample contains multiple cluster blocks, click "Accept" to register the next cluster block until all cluster blocks are registered, and click "Accept" to view the data. The user can choose to check the "Manual Registration" button in the upper left corner to start the manual registration mode. After the manual registration is completed, click the "Accept" button to start the registration of the next cluster block or view the quantitative data.

[0043] The following data is obtained by running the above method:

[0044] Table 1

[0045] 1) Use the WSI-level image clustering algorithm to find the cluster block with a magnification of 0.625 on the WSI (abbreviated as Block_0.625), and imcrop the cluster blocks with magnifications of 5 and 10 respectively (abbreviated as Block_5 and Block_10), which are the inputs required for subsequent registration and classification.

[0046] 2) Calculate the black and white binary image of each Block_0.625.

[0047] 3) Image registration. Figure 3 As shown, one path is to first perform single-modality registration on the black-and-white binary image of Block_0.625, and then perform multi-modality registration on Block_5. The other path is to directly perform single-modality and multi-modality registration on Block_5 and save the transformation matrix tform.

[0048] 4) Cut all Block_Tpit_10 into 100x100 pixel patches, find all positive patches through the supervised trained MIL-CNN model, and record the serial numbers.

[0049] 5) Put the positive plaques back together according to the recorded cutting order to obtain the positive area segmentation result of the t-pit image, and use the previously saved transformation matrix to calculate the corresponding positive area on ACTH.

[0050] 6) Quantitative grading of ACTH tumor and adjacent areas is performed, showing the weighted average total value and each sub-item.

[0051] like Figure 4 As shown in Table 1, the present invention is compared with the existing image registration method. As shown in Table 2, the present invention is compared with the existing image classification method.

[0052] Table 2 Comparison between the present invention and the existing image registration method

[0053] Table 3 Comparison between the present invention and existing image classification methods

[0054] The present invention is superior to the existing technologies in both registration and classification on the test set of Cushing's disease pathological images (the registration accuracy is 82.6% and the classification accuracy is 97.8%). Compared with the prior art, the present method can extract accurate affine transformation from such multimodal images with severe noise through an integrated two-stage image registration scheme, and the convolutional neural network image classifier based on multi-instance learning of the present invention has a higher classification accuracy for such plaque images with severe noise.

[0055] The above-mentioned specific implementation can be partially adjusted in different ways by those skilled in the art without departing from the principle and purpose of the present invention. The protection scope of the present invention shall be based on the claims and shall not be limited by the above-mentioned specific implementation. Each implementation scheme within its scope shall be subject to the constraints of the present invention.

Claims

1. A method for measuring the expression and prognosis of ACTH adjacent to the tumor in Cushing's disease based on deep learning, characterized in that, After the cluster blocks were segmented from the input images of two different magnifications using the single-link clustering algorithm, they were paired and registered across modalities. One of the cluster blocks was input into the trained multiple instance learning (MIL) classifier to obtain the positive area, i.e., the densely stained area of ​​the cell nucleus in the T-PIT staining image. The positive area was then projected onto the other cluster block according to the transformation matrix obtained by the registration. Finally, the projected area of ​​the other cluster block was segmented and the hormone level was analyzed to achieve prognostic measurement.

2. The method for measuring the expression and prognosis of ACTH adjacent to the tumor in Cushing's disease based on deep learning according to claim 1, characterized in that specifically include: The first step is to use the single-link clustering method to achieve image-level clustering, which includes: 1.1) Find all n connected components in the input image I and number them from 1 to n. Fill each connected component with its number as the gray value to obtain the filled image I N , where the area I with gray value greater than 0 fg , that is, the foreground area of the input image I; 1.2) Obtain the distance field D and the nearest pixel index map IDX to the region I in the input image I from the binary image binary I of the filled image through the bwist function of Matlab N in the input image I fg The nearest distance field D and the nearest pixel index map IDX are recorded for each point I on the graph I i (For simplicity, one-dimensional representation is used for two-dimensional coordinates) The output [D i IDX i in this step is specifically as follows: Where: Distance(I i ,I k ) refers to the Euclidean distance between two points I i and I k . D i is the nearest distance from point I i to the nearest connected domain, and IDX i is the coordinate of the nearest point to this point among all connected domains. The width and height of D and IDX are the same as those of the input image I; 1.3) Using the IDX calculated for each point in step 1.2 i as the coordinates, search for the label of the region closest to it in the filled connected component respectively, and assign the value N i = I N (IDX i ) as a feature to each point, obtaining the nearest region field image N. The gray value of the pixel with coordinate i in image N is N i = I N (IDX i ), i ∈ I; 1.4) Perform a dilation operation with a scale of (3, 3) on the nearest region field image N to obtain the dilated image N dilated , the difference line with a width of 1 pixel obtained by comparing the two is the region field boundary line and the boundary source information, that is, the numbers of the two regions and the boundary distance information bd = 2 * D(j), where: j ∈ (N dilated -N dilated ∩N); 1.5) Substitute the distance information bd and the boundary source information into the distance matrix DM whose diagonal elements are all 0, specifically: Perform hierarchical clustering on the distance matrix DM to obtain the clustering trees of the two images respectively; 1.6) According to the observation of input image I, the cluster tree is uniformly cut at a tree height of 100. Specifically, the ACTH staining image is cut into 5 classes, and the t-pit staining image is cut into 2 classes. It can be seen that the former has 3 more impurities than the latter. Since the ACTH staining image and the t-pit staining image are made by staining adjacent thin slices on each fragment in the same sample, the area of ​​each fragment on different staining images will not change significantly. Therefore, the order of arrangement of the fragments in the same image by area size will not change in another image. After arranging by area size, the first two classes are matched one by one, and the preprocessing work of registration, that is, registration ROI extraction, is completed. The second step is cluster block level registration, which uses the method of extracting the image foreground to screen the key features, weaken the texture features and strengthen the color features; The third step is to construct and train a multiple instance learning (MIL) classifier, and then perform patch-level image classification on the clustered blocks of the motion image obtained in the first step; Step 4: Pixel-level image segmentation: Using four thresholds as boundaries, the normal cell area segmented in the ACTH-stained image in the previous step was divided into three different sub-areas: positive + area, positive ++ area, and positive ++ area; by using the background average intensity of all images for correction, the positive threshold standard of the calibration image was set to: positive level I with an intensity of 180-120, positive level II with an intensity of 120-70, and positive level III with an intensity of 70-40, and the weighted sum result of each area was used as the quantitative analysis result of the entire image.

3. The method for measuring the expression and prognosis of ACTH adjacent to the tumor in Cushing's disease based on deep learning according to claim 2, wherein, The described boundary source information includes: the pixel value before dilation is the region number smaller among the adjacent two regions, i.e., label1 i = N(i), and the pixel value after dilation is the region number larger among the adjacent two regions, i.e., label2 i = N dilated (i), i ∈ (N dilated - N dilated ∩ N).

4. The method for measuring the expression and prognosis of ACTH adjacent to the tumor in Cushing's disease based on deep learning according to claim 2, characterized in that, The extracted image foreground includes: ① Taking advantage of the fact that there are gaps in the stained image and the background has little difference except for a small amount of impurities, a disk with a radius of 4 is used to traverse the image and remove the part with small variance, specifically: Where: disk(I i ) indicates that I i is a disk with a radius of 4 and a center. t represents the threshold for screening variance. distance(I i ,I k ) represents the Euclidean distance between two points; ② Close the remaining connected areas and remove the areas with an area less than 400 pixels squared. This will remove the impurities in the foreground area. Since the foreground image has almost only contour features, taking the minimum resolution of 0.625 times can cover the basic information of the image. ③ Using the intensity-based integrated registration method, first perform a preliminary single-modal registration based on the affine transformation in the lower section on the image with a magnification of 0.625 times (scale = 0.625), find the approximate rotation angle and displacement, and then use this as the initial matrix to continue multi-modal registration on the image with a resolution of 5 times. Finally, use the integrated learning mode, try a variety of initial conditions and image preprocessing methods, and take the transformation matrix with the best result until the two images are completely corresponding, specifically: The coordinates before transformation are (x, y), and the coordinates after transformation are (x', y'). For rigid registration, the affine transformation matrix is as follows: Where: θ represents the rigid rotation angle of the image, and w and h determine the translation and scaling process of the image. When the program uses the result matrix registered at scale = 0.625 as the initial matrix for registration at scale = 5, the values of w and h in the result matrix will be automatically multiplied by 8. When applying the result matrix registered at scale = 5 to the image transformation at scale = 10, the values of w and h in the result matrix will be automatically multiplied by 2.

5. The method for measuring the expression and prognosis of ACTH adjacent to the tumor in Cushing's disease based on deep learning according to claim 4, wherein For each pair of registered images, image processing technology is used to remove the gap and retain the gap to perform two registrations. The variance after pixel-by-pixel subtraction of the two images is used as the basis to select the transformation matrix with the highest degree of fit, and then connect it to the downstream tasks.

6. The method for measuring the expression and prognosis of ACTH adjacent to the tumor in Cushing's disease based on deep learning according to claim 2, characterized in that, The MIL classifier is a convolutional neural network (ConvNet), including: convolutional layer (Conv), pooling layer (pool), activation function (Relu) and average layer, wherein: ConvNet applies four convolutional layers with padding; the operators of the first two layers are 5×5, and the others are 3×3; the average layer calculates the output of each batch of samples, that is, the input feature map is calculated as a single scalar value; except for the last layer, batch normalization and rectified linear activation function (RELU) are used in the entire ConvNet.

7. The method for measuring the expression and prognosis of ACTH adjacent to the tumor in Cushing's disease based on deep learning according to claim 2, characterized in that In the early stage of the fourth step of pixel-level image segmentation, the background area is first found through the salient region segmentation technique, the average pixel value of all image background areas is calculated, and the intensity of the entire image is calibrated by using the difference between this value and the average value of each image background area.

8. A deep learning-based Cushing's disease peritumoral ACTH expression and prognosis measurement system for implementing the method according to any one of claims 1-7, characterized in that, include: A hierarchical clustering unit, an image registration unit, an image classification unit, a transformation mapping unit, and an image segmentation and data processing unit, wherein: the hierarchical clustering unit performs an improved single-link hierarchical clustering process according to a pathological whole slide image (WSI), and obtains a plurality of cluster blocks arranged from large to small in area in two images to be registered; The image registration unit performs an integrated multimodal image registration process from coarse to fine based on the information of the pairwise corresponding cluster blocks obtained by the hierarchical clustering unit, and obtains the rigid transformation matrix of each cluster block respectively; the image classification unit performs image classification based on multi-instance learning of a convolutional neural network based on the information of each cluster block belonging to the moving image, and obtains tumor-positive areas and tumor-negative areas; the transformation mapping unit projects the positive area and negative area information on the moving image obtained by the image classification unit onto the fixed image through the transformation matrix obtained by the image registration unit, and obtains the result of segmenting the fixed image into positive areas and negative areas; the image segmentation and data processing unit performs threshold-based image segmentation on the fixed image after white balance processing based on the information of the positive area and negative area in the fixed image obtained by the transformation mapping unit, and obtains the graded hormone expression effects of the positive area and negative area in the fixed image, and displays the specific values ​​and their weighted mean values ​​to the user in the form of a list.

Images