Method, apparatus and storage medium for image processing
By employing multi-resolution feature extraction and various downsampling methods, combined with thresholding, the problem of incomplete feature extraction in WSI image processing was solved, achieving more efficient and accurate image classification.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- FUJITSU LTD
- Filing Date
- 2021-04-16
- Publication Date
- 2026-07-10
AI Technical Summary
Existing WSI image processing methods fail to fully utilize the multi-resolution information of images during feature extraction, resulting in incomplete feature extraction and susceptibility to noise, which affects diagnostic accuracy and efficiency.
By employing multi-resolution feature extraction and various downsampling methods, such as classical, max pooling, and min pooling, combined with thresholding, robust features are extracted and selected, and then optimized in the full-image classification model.
It improves the accuracy and efficiency of WSI image classification, reduces runtime, and is superior to existing methods.
Smart Images
Figure CN115294417B_ABST
Abstract
Description
Technical Field
[0001] This disclosure pertains to the field of image processing, and specifically to image classification, feature extraction, and feature selection. Background Technology
[0002] Whole-slide images (WSI) scan and digitize entire tissue sections at various resolutions, and are therefore widely used in histopathological tissue analysis. Tissue analysis in histopathology is the gold standard for cancer diagnosis and prognosis. WSI images are very large, while demand is growing rapidly. Therefore, relying solely on manual interpretation of WSI images is increasingly insufficient to meet practical needs. Figure 1 An example of such a WSI image is given. Figure 1 The WSI images in the dataset cover resolutions from 1x to 512x.
[0003] With the rapid development of deep learning in recent years, many automated diagnostic methods based on deep learning have been proposed. In response, the internationally renowned Camelyon competition was held in 2016 and 2017. This competition was the first to focus on the study of WSI images in the field of automated pathology diagnosis. The competition investigated the detection and classification of lymph node WSI images during the process of lymph node metastasis in breast cancer. In this competition, dozens, even hundreds, of teams developed and submitted their own deep learning-based methods.
[0004] Because WSI images are very large, these methods are not end-to-end frameworks. For example... Figure 2 As shown, they can be broadly divided into three parts: image patch-level image classification or segmentation algorithms, probabilistic map stitching and feature extraction algorithms, and slice-level (full image) image classification algorithms. Specifically, firstly, for image 201 at level 0 (e.g., ... Figure 1 The image (at 512x magnification) is trained at the image patch level using a classification or segmentation algorithm 200 to detect tumor regions. Then, the predicted probability map patches 202 are stitched together to form a complete image 203, and global morphological features are extracted 204 from it. Finally, a slice-level classification model 205 is trained based on these features to perform pathological classification of the slices, thereby diagnosing the patient's cancer progression based on the output 206.
[0005] However, these existing methods typically extract features from slice images at a fixed resolution without making full use of WSI images or offering targeted solutions for tumor region fragmentation and noise issues. As a result, these methods often extract missing or even incorrect features and fail to accurately reflect the characteristics of the prediction probability map.
[0006] To address this issue, a few methods have been developed that use clustering to extract features, but these methods are time-consuming and their performance is unsatisfactory. Summary of the Invention
[0007] A brief overview of this disclosure is provided below to provide a basic understanding of certain aspects of it. It should be understood that this overview is not an exhaustive summary of the disclosure. It is not intended to identify key or essential parts of the disclosure, nor is it intended to limit its scope. Its purpose is merely to present certain concepts in a simplified form as a prelude to the more detailed description that follows.
[0008] According to one aspect of the present invention, a method for image processing is provided, comprising: obtaining a probability map of an image through preprocessing; downsampling the probability map and extracting features in multiple dimensions from the downsampled probability map based on combinations of different thresholds and different resolutions; inputting the extracted features into a classification model for the entire image and obtaining the top N parameter combinations with the highest classification accuracy among all parameter combinations with different thresholds and different resolutions using a validation set, where N is an integer greater than zero; selecting the top M features with the highest importance scores in the first M dimensions from the extracted features using the classification model for the entire image based on the top N parameter combinations with the highest classification accuracy, where M is an integer greater than zero; selecting the top N' parameter combinations with the highest classification accuracy from all parameter combinations using the classification model for the entire image based on the features with the highest importance scores in the first N' parameter combinations with the highest classification accuracy, where N' is an integer greater than zero; and classifying the image using the classification model for the entire image based on the features with the highest importance scores in the first M' parameter combinations with the highest classification accuracy.
[0009] Preferably, the method further includes: after selecting the features with the highest importance scores in the top M dimensions, selecting the top K parameter combinations with the greatest parameter differences from the top N' parameter combinations with the highest classification accuracy, where K is an integer greater than zero and less than N; and using a classification model for the entire image, selecting the features with the highest importance scores in the top P dimensions from the M×K dimensional features composed of the top K parameter combinations with the greatest parameter differences and the features with the highest importance scores in the top M dimensions, where P is an integer greater than zero, wherein classifying the image using a classification model for the entire image includes classification based on the features with the highest importance scores in the top P dimensions.
[0010] Preferably, the method further includes downsampling the probability map using different downsampling methods, wherein obtaining the top N parameter combinations with the highest classification accuracy includes: after downsampling the probability map using different downsampling methods: performing multi-dimensional feature extraction on the downsampled probability map based on combinations of different thresholds, different downsampling methods, and different resolutions; and inputting the extracted features into a classification model for the entire map, and using a validation set to obtain the top N parameter combinations with the highest classification accuracy among all parameter combinations of different thresholds, different resolutions, and different downsampling methods.
[0011] Preferably, the parameter combinations with the greatest differences among the top K parameters are selected from the parameter combinations with the highest classification accuracy among the top N' parameters. For each downsampling method, the parameter combination with the greatest difference in resolution or the greatest difference in threshold is selected, wherein the sum of the number of parameter combinations selected for each downsampling method is K.
[0012] Preferably, resolution differences are given higher priority than threshold differences.
[0013] Preferably, the threshold ranges from 0 to 1.
[0014] Preferably, preprocessing includes inputting the image into a pre-trained classification or segmentation model for image patches to obtain a probability map.
[0015] Preferably, selecting the top M features with the highest importance scores from the extracted features based on the top N parameter combinations with the highest classification accuracy using a classification model for the entire image includes: First, for each of the top N parameter combinations with the highest classification accuracy, training a classification model for the entire image using the corresponding extracted features to obtain the importance score of each feature dimension under each parameter combination; Second, calculating the mean of the importance scores of each feature dimension under each parameter combination, and selecting the top Q percent of features with the highest importance scores, where Q is greater than 0 and less than 100; Third, for each of the top N parameter combinations with the highest classification accuracy, inputting the top Q percent of features with the highest importance scores into the classification model for the entire image, to obtain the importance score of each feature dimension under each parameter combination; and repeating steps two and three until the top M features with the highest importance scores are selected.
[0016] Preferably, using a classification model for the entire image, selecting the top P-dimensional features with the highest importance scores from among the M×K-dimensional features composed of the top K parameter combinations with the greatest parameter differences and the top M-dimensional features with the highest importance scores includes: First, selecting the top M-dimensional features with the highest importance scores for each parameter combination among the top K parameter combinations with the greatest parameter differences, and using these M×K-dimensional features as input to train the classification model for the entire image to obtain the importance score for each dimension; Second, based on the obtained importance scores for each dimension, selecting the top Q-dimensional features with the highest importance scores from the M×K-dimensional features, where Q is greater than 0 and less than 100; Third, using the top Q-dimensional features with the highest importance scores as input to retrain the classification model for the entire image to obtain updated importance scores for each dimension of the top Q-dimensional features with the highest importance scores; and repeating steps two and three until the top P-dimensional features with the highest importance scores are selected.
[0017] According to another aspect of the present invention, an apparatus for image processing is provided, comprising: a preprocessing unit configured to obtain a probability map of an image through preprocessing; a multidimensional feature extraction unit configured to downsample the probability map and extract features of multiple dimensions from the downsampled probability map based on combinations of different thresholds and different resolutions; a first parameter selection unit configured to input the extracted features into a classification model for the entire image and obtain the top N parameter combinations with the highest classification accuracy from all parameter combinations with different thresholds and different resolutions using a validation set, where N is an integer greater than zero; and a feature dimensionality reduction unit configured to... The system comprises: a first parameter selection device configured to select the top M-dimensional importance scores from the extracted features based on the top N parameter combinations with the highest classification accuracy, where M is a positive integer; a second parameter selection device configured to select the top N' parameter combinations with the highest classification accuracy from all parameter combinations based on the M-dimensional importance scores, where N' is a positive integer; and an image classification device configured to classify the image based on the top M-dimensional importance scores of one of the top N' parameter combinations with the highest classification accuracy, using a classification model for the entire image.
[0018] According to other aspects of the invention, corresponding computer program code, computer-readable storage media, and computer program products are also provided.
[0019] The method and apparatus for image processing of the present invention enable accurate and rapid image classification.
[0020] These and other advantages of the invention will become more apparent from the following detailed description of preferred embodiments of the invention in conjunction with the accompanying drawings. Attached Figure Description
[0021] To further illustrate the above and other advantages and features of this disclosure, the specific embodiments of this disclosure will be described in more detail below with reference to the accompanying drawings. These drawings, together with the following detailed description, are included in and form a part of this specification. Elements having the same function and structure are indicated by the same reference numerals. It should be understood that these drawings only depict typical examples of this disclosure and should not be considered as limiting the scope of this disclosure. In the drawings:
[0022] Figure 1 An example of a WSI image is shown;
[0023] Figure 2 This is a block diagram of existing methods for automatic diagnosis of WSI images;
[0024] Figure 3 A schematic flowchart of the method according to the present invention is shown.
[0025] Figure 4 This is a flowchart of a method for image processing according to an embodiment of the present invention;
[0026] Figure 5 The results of downsampling and denoising a probability map using different downsampling methods are illustrated.
[0027] Figure 6 This is a flowchart illustrating how feature dimensionality reduction is performed according to an embodiment of the present invention;
[0028] Figure 7 This is a flowchart of a method for image processing according to another embodiment of the present invention;
[0029] Figure 8 This is a flowchart illustrating how feature dimensionality reduction is performed according to another embodiment of the present invention;
[0030] Figure 9 This is a block diagram of an image processing apparatus according to one embodiment of the present invention;
[0031] Figure 10 This is a block diagram of an exemplary structure of a general-purpose personal computer in which the methods and / or devices according to embodiments of the present invention can be implemented. Detailed Implementation
[0032] Exemplary embodiments of this disclosure will be described below with reference to the accompanying drawings. For clarity and brevity, not all features of actual embodiments are described in the specification. However, it should be understood that many implementation-specific decisions must be made in the development of any such actual embodiment to achieve the developer's specific goals, such as complying with constraints related to the system and business, and these constraints may vary depending on the implementation. Furthermore, it should be understood that while development work can be very complex and time-consuming, such development work is merely a routine task for those skilled in the art who benefit from this disclosure.
[0033] It should also be noted that, in order to avoid obscuring this disclosure with unnecessary details, only the equipment structure and / or processing steps closely related to the solution according to this disclosure are shown in the accompanying drawings, while other details that are not closely related to this disclosure are omitted.
[0034] As mentioned earlier, existing methods for histopathological tissue analysis based on WSI images are not only time-consuming but also have unsatisfactory performance. To address this issue, this invention proposes a simple and effective feature extraction method that can be easily applied to most automated pathological diagnosis frameworks for WSI images. Furthermore, the method of this invention can significantly reduce runtime while maintaining or even improving performance. Compared with other methods, the method of this invention has many advantages.
[0035] First, the method of this invention lies in multi-resolution feature extraction. That is, the method of this invention can extract features at multiple resolutions, thereby ensuring that all potentially useful features can be extracted.
[0036] Secondly, the method of this invention employs various downsampling techniques, such as classical downsampling, max pooling, and min pooling. These correspond to classical interpolation downsampling, clustering, and image erosion methods, respectively. Using these downsampling methods, fragmented tumors can be clustered together, and noise points in the probability map (heatmap) can be removed, resulting in more accurate features.
[0037] Finally, after extracting all possible features, the method of the present invention further selects and reduces the dimensionality of the extracted features to select robust and high-performance feature combinations.
[0038] In the Camelyon 17 competition mentioned above, the method of this invention achieved third place. Compared with existing methods, the method of this invention not only has superior performance but also significantly reduces the running time.
[0039] The following is combined with Figure 3The overall process of the method according to the present invention will be described below.
[0040] First, in step 301, a probability map is obtained. Specifically, in this embodiment, a large number of overlapping small patches are randomly extracted from the effective region of the input original image (e.g., a WSI image) to form a training set and a validation set, ensuring sample diversity while taking into account the ratio of positive to negative samples. Then, an image patch-level classification or segmentation model is trained based on this training set and validation set. Using the trained model, the effective region of the original image is segmented into patches using a sliding window approach, and the model is used to predict each image patch, thereby obtaining a probability map for each image patch. For example, the tumor probability of each pixel on the image patch is obtained. Then, based on the probability maps of each image patch, the probability map of the entire image is stitched together.
[0041] It should be understood that during stitching, the probability of pixels in the overlapping area is averaged, and pixels outside the receptive field of the image patch classification or segmentation model are cropped before stitching.
[0042] It should also be understood that the present invention is not limited to the methods of obtaining probability maps described above, and those skilled in the art can obtain probability maps as needed using any other known methods.
[0043] Next, in step 302, the probability map is downsampled to different resolutions using different downsampling methods. For example, in this embodiment, the probability map is downsampled using three downsampling methods: classical downsampling, max pooling, and min pooling. Classical downsampling methods include (but are not limited to) nearest neighbor interpolation, bilinear interpolation, bicubic interpolation, or Lanczos interpolation.
[0044] It should be understood that although three downsampling methods are used to downsample the probability map in this embodiment, the present invention is not limited to this, and one, two or more downsampling methods can be used to downsample the probability map.
[0045] Next, in step 303, threshold-based feature extraction is performed. The threshold is a predetermined value applied to the probability map, ranging from 0 to 1. That is, a threshold is applied to the downsampled probability map to convert it into a binary image, and then connected components are calculated from this binary image to extract various features. In this embodiment, the features can be, for example, morphological features. Next, step 304, feature selection, is performed. Step 304 consists of three steps. First, in step 3041, the optimal parameter combination is found in the validation set. Specifically, the optimal combination of, for example, threshold, resolution, and downsampling method, is found in the validation set.
[0046] It should be understood that the optimal parameter combination refers to the parameter combination that achieves high image classification accuracy on the validation set. The number of such optimal parameter combinations can be set as needed.
[0047] Next, in step 3042, the feature dimensionality is reduced under optimal parameters. Specifically, based on the optimal parameter combination selected in step 3041, a trained full-image (slide-level classification model) is used to select one or more features with the highest importance scores from the extracted features, thereby reducing the feature dimensionality.
[0048] It should be understood that the higher the importance score of a feature, the greater its influence on the prediction results of the classification model, and the stronger its correlation with the true value of the classification target.
[0049] Finally, in step 3043, the optimal combination of extracted features under different parameters is selected. Specifically, based on the features after dimensionality reduction, the classification accuracy of all parameter combinations is remeasured on the validation set using a full-image classification model. Among the various parameter combinations with the highest classification accuracy, several parameter combinations with the greatest parameter differences are selected, and the features extracted under these parameter combinations are mixed. Based on this, the optimal feature combination with the highest importance score is selected using a full-image classification model.
[0050] This completes step 304 of feature selection.
[0051] Finally, in step 305, the features selected in step 304 are used to train a full-image classification model. Based on this final full-image classification model, pathological classification of images such as WSI images can be performed, thereby diagnosing the patient's cancer progression.
[0052] The following is combined with Figures 4 to 8 The method 400 for image processing according to an embodiment of the present invention will be described in detail below.
[0053] like Figure 4As shown, in step 401, a probability map of the image is obtained through preprocessing. Specifically, in this embodiment, it can be based on, for example... Figure 2 Steps 201, 202, and 203 in the middle or Figure 3 Step 301 in the process is used to obtain the probability map of the original image.
[0054] Next, in step 402, the probability map is downsampled using different downsampling methods, and features of multiple dimensions are extracted from the downsampled probability map based on different combinations of thresholds and resolutions.
[0055] In this embodiment, three different downsampling methods are employed: classical interpolation downsampling, max pooling, and min pooling. The classical interpolation downsampling method allows for the acquisition of accurate features of the probability map at different resolutions. These accurate features faithfully reflect the true information of the probability map at different resolutions.
[0056] However, these precise features have certain problems. For example... Figure 5 As shown in the input probability map 501, the distribution of cancer cells is often fragmented and discontinuous. If we directly calculate the connected components and extract morphological features, such as the major axis length of the largest tumor region, from such a probability map, it will inevitably deviate significantly from the reality reflected by the probability map, leading to errors in the extracted feature values. Therefore, it is necessary to cluster the connected components (i.e., tumor regions) in the probability map. Due to the large size of WSI images and the considerable number of connected components, clustering algorithms are overly complex and slow, and the parameters of the clustering algorithm are difficult to determine in practical applications. Therefore, clustering algorithms lack robustness and universality.
[0057] To address this issue, this implementation also employs a max-pooling method for downsampling, such as... Figure 5 The probability diagram is shown in Figure 502. During downsampling, the maximum value within the neighborhood of a pixel is calculated as the downsampled value of that pixel. This method allows for the rapid connection of adjacent tumor regions, thereby extracting accurate feature values.
[0058] Furthermore, the probability graph inevitably contains some noise points or regions with calculation errors. To address this issue, this implementation also uses a minimum pooling method for downsampling, such as... Figure 5 The probability diagram is shown in Figure 503. During the downsampling process, the minimum value within the neighborhood of a pixel is calculated as the downsampled value of that pixel.
[0059] Figure 5The second row of images schematically illustrates the results of denoising images without tumor regions using the minimum pooling method. Typically, the input probability map contains some noise, which can affect the final classification result. For example... Figure 5 As shown in probability diagram 504, after downsampling using the minimum pooling method, most of the noise is filtered out.
[0060] In step 402, in addition to using different downsampling methods, different thresholds are also used for multidimensional feature extraction. First, a threshold is added to the downsampling probability map, with the threshold ranging from 0 to 1 (the optimal threshold is generally between 0.5 and 0.95). This produces a binary image. For example, in this binary image, regions with a value of 1 represent areas identified as cancer cells by the front-end algorithm, while regions with a value of 0 represent the distribution areas of normal cells.
[0061] Then, connected components are calculated for the binary image. For example, each connected component is considered a region of cancer cells. According to the definition in histopathology, a slice is negative if no cancer cells are present. If cancer cells are present, the long axis length of the largest cancer cell region is observed and classified. When the long axis length is less than 0.2 mm or the number of cancer cells in the region is less than 200, the slice is classified as isolated tumor cells (ITC). When the long axis length is greater than 0.2 mm or the number of cancer cells is greater than 200, and the long axis length is less than 2 mm, the slice is classified as micro-metastases. When the long axis length is greater than 2 mm, the slice is classified as macro-metastases. Thus, the classification of each slice image is closely related to the long axis length of the largest cancer cell region.
[0062] Finally, referring to the classification definition of histopathology described above, various morphological features are calculated for the set of connected components in the binary image, and these morphological features are used as input to a full-image classification model. This full-image classification model will classify images such as WSI images based on these morphological features.
[0063] In the Camelyon17 competition, approximately 40 features were extracted, including: the area of the largest tumor region, the length of the major axis of the largest tumor region, the average probability within the largest tumor region, the maximum probability within the largest tumor region, the tumor tissue ratio, etc. These features basically cover the vast majority of useful morphological features, and all useful information was extracted as much as possible.
[0064] Therefore, multidimensional features of probabilistic maps can be extracted based on various combinations of different downsampling methods, different thresholds, and different resolutions.
[0065] Next, in step 403, the extracted features are input into a classification model for the entire image, and the top N parameter combinations with the highest classification accuracy are obtained from all parameter combinations using the validation set, where N is a positive integer. Specifically, in this embodiment, the multidimensional features extracted in step 402 are used as input to the entire image classification model, and the corresponding entire image classification model under each parameter and the importance score of each morphological feature under that parameter are trained. The trained entire image classification model is then fed into the validation set to measure the classification accuracy (e.g., classification accuracy, classification kappa coefficient, etc.) of each entire image classification model. The set or sets of parameters with the highest accuracy are selected as the optimal parameter combination.
[0066] It should be understood that those skilled in the art can set the value of N as needed.
[0067] Next, in step 404, based on the top N parameter combinations with the highest classification accuracy, a full-image classification model is used to select the top M features with the highest importance scores from the extracted features, where M is a positive integer. It should be understood that those skilled in the art can set the value of M as needed.
[0068] Step 404 involves dimensionality reduction of the features to remove most unnecessary feature types, such as morphological features, thereby reducing the possibility of overfitting in the subsequent full-image classification model. With the optimal parameter combination obtained from step 403, the features are iteratively reduced based on the importance score of each morphological feature, thus improving the robustness of the features. The following section combines... Figure 6 A detailed implementation of step 404 is provided.
[0069] First, in step 4041, for each of the top N parameter combinations with the highest classification accuracy, a full-image classification model is trained using the corresponding extracted features, and the importance score of each feature dimension under each parameter combination is obtained. Specifically, the full-image classification model is trained based on the features corresponding to each optimal parameter combination, thereby obtaining the importance score of each feature dimension under each parameter combination.
[0070] Next, in step 4042, the importance score of each feature dimension under each parameter combination is averaged, and the features with the highest importance scores in the top 100% of Q dimensions are selected, where Q is greater than 0 and less than 100. It should be understood that those skilled in the art can set the value of Q as needed.
[0071] Next, in step 4043, the selected top 100% Q-dimensional features are re-inputted and the full-image classification model is trained to obtain the importance score of each feature in the top 100% Q-dimensional features under each parameter combination.
[0072] Finally, in step 4044, steps 4042 and 4043 are iterated until the selected feature dimensions, sorted by importance score, are less than or equal to M dimensions. It should be understood that after sorting the features based on importance score, features can be iteratively selected at different granularities as needed until the top M-dimensional features are selected, thereby achieving feature dimensionality reduction. For example, the first iteration can select the top Q-10% of features, the second iteration can select the top Q-20% of features, and so on, until the top M-dimensional features are selected.
[0073] In the Camelyon17 competition, the following four features were ultimately selected: the area of the largest tumor region, the length of the major axis of the largest tumor region, the average probability within the largest tumor region, and the maximum probability within the largest tumor region.
[0074] return Figure 4 In step 405, based on the feature with the highest importance score in M dimensions, the top N' parameter combinations with the highest classification accuracy are selected from all parameter combinations using a full-image classification model, where N' is an integer greater than zero.
[0075] It should be understood that those skilled in the art can set the value of N' as needed.
[0076] Finally, in step 406, the image is classified using a full-image classification model based on the feature with the highest importance score in the top M dimensions under one of the parameter combinations with the highest classification accuracy among the top N' parameter combinations.
[0077] Preferably, N' is an integer greater than zero.
[0078] It should be noted that the sorting in steps 403 to 405 only reflects the classification accuracy under each set of parameters. Although a satisfactory classification accuracy can be obtained under the optimal parameter combination, they do not fully utilize the diversity of features extracted in step 402. It should be understood that the features extracted under each set of parameters have their own different physical meanings and are often complementary. Therefore, preferably, feature selection can also be performed across parameters. The following section combines... Figure 7 and Figure 8 A method 700 for image processing according to another embodiment of the present invention is described in detail.
[0079] Steps 701 to 705 in method 700 and Figure 4 Steps 401 to 405 in method 400 shown are the same, so they will not be described again.
[0080] In step 706, among the first N' parameter combinations with the highest classification accuracy, the parameter combination with the largest resolution difference or the largest threshold difference is selected for each downsampling method. The sum of the number of parameter combinations selected for each downsampling method is K, where K is an integer greater than zero and less than N'.
[0081] Specifically, several candidate parameter combinations are selected from the top few parameter combinations with the highest classification accuracy in each downsampling method. The selection criteria are that the resolution difference or threshold difference between these candidates is large.
[0082] According to one implementation, parameter combinations that, for example, allow resolution differences or threshold differences to fall within a predetermined range can be selected. It should be understood that those skilled in the art can appropriately set this predetermined range as needed.
[0083] Next, in step 707, using the full-image classification model, the feature with the highest P-dimensional importance score is selected from the M×K-dimensional features composed of the first K parameter combinations with the largest parameter differences and the first M-dimensional features with the highest importance scores, where P is a positive integer. The following section combines... Figure 8 A detailed description of one implementation of step 707 is provided.
[0084] First, in step 7071, the feature with the highest importance score in the top M dimensions is selected from the parameter combinations with the greatest differences among the top K parameters, and these M×K dimensional features are used as input to train the whole image classification model to obtain the importance score of each dimension feature.
[0085] Next, in step 7072, based on the importance score of each dimension of the obtained feature, the top Q percent of the features with the highest importance scores are selected from the M×K dimension features, where Q is greater than 0 and less than 100.
[0086] Next, in step 7073, the top 100% of the features with the highest importance scores in the Q dimensions are used as input to retrain the whole image classification model to obtain the updated importance score of each feature in the top 100% of the Q dimensions.
[0087] Next, in step 7074, steps 7072 and 7073 are performed iteratively until the feature dimensions selected according to importance scores are less than or equal to P dimensions. As described above, after sorting the features based on importance scores, features can be iteratively selected at different granularities as needed until the top P-dimensional features are selected. Accordingly, robust and highly accurate cross-parameter feature combinations can be iteratively selected.
[0088] return Figure 7Finally, in step 708, the image is classified using a full-image classification model based on the features with the highest importance scores in the first P dimensions. For example, a full-image classification model can be used to classify WSI images pathologically based on the first P-dimensional features, thereby diagnosing the patient's cancer progression.
[0089] To verify the advantages of method 700 according to the implementation method, data from the Camelyon 16 and 17 competitions were used as comparative experimental data, divided into training and validation sets. During the experiment, the entire algorithm flow remained unchanged except for replacing the feature extraction and feature selection steps with the control group method. For each method, three metrics were used for evaluation: slice-level classification accuracy on the validation set, slice-level classification Kappa value, and patient cancer staging classification Kappa value. For each method, the parameters in the algorithm were brute-force iterated through, and the three accuracy metrics were used to select the optimal parameters for each method, and the three accuracy metrics for each parameter were calculated.
[0090] The comparative experiment used the top two feature extraction methods from Camelyon in 2017, which employed a DBSCAN-based connected component clustering method (first place) and a method that performed a closing operation on connected components (second place), respectively. The results of the comparative experiment are shown in Table 1 below.
[0091]
[0092] Table 1
[0093] As can be seen from Table 1, the method of the present invention can achieve better performance with the least amount of running time.
[0094] The above combination Figures 3 to 8 The methods according to various embodiments of the present invention have been described in detail. As can be seen from the above description, the methods according to the various embodiments enable accurate and rapid image classification.
[0095] The methods discussed above can be implemented entirely by a computer-executable program, or partially or entirely using hardware and / or firmware. When implemented in hardware and / or firmware, or when a computer-executable program is loaded into a hardware device capable of running the program, a device for processing transactions, as described below, is implemented. Hereinafter, an overview of these devices is given without repeating some details already discussed above; however, it should be noted that while these devices can perform the methods described above, the methods may not necessarily employ or be performed by those components of the described device.
[0096] Figure 9An image processing apparatus 900 according to one embodiment is shown, comprising a preprocessing unit 901, a multidimensional feature extraction unit 902, a first parameter selection unit 903, a feature dimensionality reduction unit 904, a second parameter selection unit 905, and an image classification unit 906. The preprocessing unit 901 is used to obtain a probability map of an image through preprocessing. The multidimensional feature extraction unit 902 is used to downsample the probability map and extract features in multiple dimensions from the downsampled probability map based on combinations of different thresholds and different resolutions. The first parameter selection unit 903 is used to input the extracted features into a classification model for the entire image and, using a validation set, obtain the top N parameter combinations with the highest classification accuracy among all parameter combinations with different thresholds and different resolutions, where N is a positive integer. The feature dimensionality reduction unit 904 is used to select the top M features with the highest importance scores in the extracted features based on the top N parameter combinations with the highest classification accuracy, using a classification model for the entire image, where M is a positive integer. The second parameter selection device 905 is used to select the top N' parameter combinations with the highest classification accuracy from all parameter combinations based on the features with the highest M-dimensional importance scores, using a classification model for the entire image, where N' is a positive integer. The image classification device 906 is used to classify the image based on the features with the highest M-dimensional importance scores under one of the top N' parameter combinations with the highest classification accuracy, using a classification model for the entire image.
[0097] Figure 9 The device 900 shown for image processing corresponds to Figure 4 The method 400 for image processing is shown. Therefore, details regarding the various devices in the apparatus 900 for image processing have already been provided in the [details omitted]. Figure 4 The method for image processing 400 is described in detail here and will not be repeated here.
[0098] Each component module and unit in the above-described device can be configured via software, firmware, hardware, or a combination thereof. Specific means or methods of configuration are well known to those skilled in the art and will not be elaborated upon here. When implemented via software or firmware, data can be transferred from a storage medium or network to a computer with a dedicated hardware architecture (e.g., ...). Figure 10 The general-purpose computer 100 shown is equipped with the programs that constitute the software, and when various programs are installed, the computer is able to perform various functions, etc.
[0099] Figure 10 This is a block diagram illustrating an exemplary structure of a general-purpose personal computer in which the methods and / or apparatuses according to embodiments of the present invention can be implemented. For example... Figure 10As shown, the Central Processing Unit (CPU) 101 performs various processes based on programs stored in the Read-Only Memory (ROM) 102 or programs loaded into the Random Access Memory (RAM) 103 from the Storage Section 108. The RAM 103 also stores data required as needed when the CPU 101 performs various processes, etc. The CPU 101, ROM 102, and RAM 103 are interconnected via a bus 104. An input / output interface 105 is also connected to the bus 104.
[0100] The following components are connected to the input / output interface 105: input section 106 (including keyboard, mouse, etc.), output section 107 (including display, such as cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.), storage section 108 (including hard disk, etc.), and communication section 109 (including network interface card, such as LAN card, modem, etc.). The communication section 109 performs communication processing via a network, such as the Internet. If necessary, drive 110 may also be connected to the input / output interface 105. Removable media 111, such as disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 110 as needed, so that computer programs read from them can be installed into storage section 108 as needed.
[0101] When the above series of processes are implemented through software, the program constituting the software is installed from a network such as the Internet or a storage medium such as removable media 111.
[0102] Those skilled in the art will understand that such storage media are not limited to Figure 10 The illustrated removable medium 111 stores a program and is distributed separately from the device to provide the program to the user. Examples of removable media 111 include disks (including floppy disks (registered trademark)), optical disks (including optical disc read-only memory (CD-ROM) and digital versatile disks (DVD)), magneto-optical disks (including mini-disk (MD) (registered trademark)), and semiconductor memory. Alternatively, the storage medium may be ROM 902, a hard disk included in storage section 908, etc., containing programs and distributed to the user along with the device containing them.
[0103] The present invention also provides corresponding computer program code and a computer program product storing machine-readable instruction code. When the instruction code is read and executed by a machine, the method described above according to the embodiments of the present invention can be performed.
[0104] Accordingly, storage media configured to carry the aforementioned program product storing machine-readable instruction code are also included in the disclosure of this invention. These storage media include, but are not limited to, floppy disks, optical disks, magneto-optical disks, memory cards, memory sticks, etc.
[0105] Based on the above description, the embodiments of this disclosure provide the following technical solutions, but are not limited thereto.
[0106] Appendix 1. A method for image processing, comprising:
[0107] A probability map of the image is obtained through preprocessing;
[0108] The probability map is downsampled, and features of multiple dimensions are extracted from the downsampled probability map based on different combinations of thresholds and resolutions.
[0109] The extracted features are input into a classification model for the entire image, and the top N parameter combinations with the highest classification accuracy are obtained from all parameter combinations with different thresholds and different resolutions using the validation set, where N is an integer greater than zero.
[0110] Based on the top N parameter combinations with the highest classification accuracy, the top M features with the highest importance scores are selected from the extracted features using a classification model for the entire image, where M is an integer greater than zero.
[0111] Based on the feature with the highest importance score in M dimensions, a classification model for the entire image is used to select the top N' parameter combinations with the highest classification accuracy from all parameter combinations, where N' is a positive integer; and
[0112] Based on the feature with the highest importance score in the top M dimensions under one of the parameter combinations with the highest classification accuracy among the top N' parameter combinations, the image is classified using a classification model for the entire image.
[0113] Appendix 2. Following the method in Appendix 1, this also includes, after selecting the feature with the highest importance score in the first M dimensions:
[0114] From the first N' parameter combinations with the highest classification accuracy, select the first K parameter combinations with the greatest parameter differences, where K is an integer greater than zero and less than N; and
[0115] Using a classification model for the entire image, we select the features with the highest importance scores in the top P dimensions from among the M×K dimensional features consisting of the top K parameter combinations with the largest parameter differences and the top M features with the highest importance scores, where P is a positive integer.
[0116] Among them, classifying images using a classification model for the entire image includes classification based on the features with the highest importance scores in the first P dimensions.
[0117] Note 3. According to the method in Note 1 or 2, it also includes downsampling the probability map using different downsampling methods.
[0118] Appendix 4. According to the method in Appendix 3, the parameter combinations that yield the highest classification accuracy for the top N include those obtained after downsampling the probability map using different downsampling methods:
[0119] Multi-dimensional feature extraction is performed on the downsampled probability map based on combinations of different thresholds, downsampling methods, and resolutions; and
[0120] The extracted features are input into a classification model for the entire image, and the top N parameter combinations with the highest classification accuracy are obtained from all parameter combinations with different thresholds, different resolutions, and different downsampling methods using the validation set.
[0121] Appendix 5. According to the method in Appendix 4, the different downsampling methods include the classical downsampling method, the max pooling method, and the min pooling method.
[0122] Note 6. According to the method in Note 4, the classical downsampling methods include nearest neighbor interpolation, bilinear interpolation, bicubic interpolation, or Lanczos interpolation.
[0123] Appendix 7. According to the method in Appendix 2, the parameter combination with the largest difference among the top K parameters is included among the parameter combinations with the highest classification accuracy among the top N' parameters. For each downsampling method, the parameter combination with the largest difference in resolution or the largest difference in threshold is selected, where the sum of the number of parameter combinations selected for each downsampling method is K.
[0124] Note 8. According to the method in Note 7, resolution difference has a higher priority than threshold difference.
[0125] Note 9. According to the method of Note 1 or 2, the preprocessing includes inputting the image into a trained classification or segmentation model for image patches to obtain a probability map.
[0126] Appendix 10. According to the method in Appendix 1, the top M features with the highest importance scores are selected from the extracted features using a classification model for the entire image, based on the top N parameter combinations with the highest classification accuracy. These features include:
[0127] The first step is to train a classification model for the entire image using the extracted features for each of the top N parameter combinations with the highest classification accuracy, so as to obtain the importance score of each feature under each parameter combination.
[0128] The second step is to calculate the mean of the importance score of each feature under each parameter combination, and select the top 100% of features with the highest importance scores in Q dimensions, where Q is greater than 0 and less than 100.
[0129] The third step involves, for each of the top N parameter combinations with the highest classification accuracy, inputting the top 100% of the features with the highest importance scores in each of the Q dimensions and training a classification model for the entire image. This yields the importance score for each dimension of the top 100% of the features with the highest importance scores in each parameter combination.
[0130] Repeat the second and third steps until the feature with the highest importance score in the first M dimensions is selected.
[0131] Appendix 11. According to the method in Appendix 7 or 8, where a classification model for the entire graph is used, the features with the highest importance scores in the top P dimensions are selected from the M×K dimensional features composed of the combination of the top K parameters with the greatest differences and the top M features with the highest importance scores. These features include:
[0132] The first step is to select the feature with the highest importance score in the top M dimensions under each parameter combination among the parameter combinations with the greatest difference in the top K parameters, and use these M×K dimensional features as input to train a classification model for the whole image to obtain the importance score of each dimension feature.
[0133] The second step is to select the top Q-th percentile features with the highest importance scores from the M×K-dimensional features based on the importance scores of each feature obtained, where Q is greater than 0 and less than 100.
[0134] The third step involves retraining the classification model for the entire graph using the top 100% of the Q-dimensional features with the highest importance scores as input, to obtain updated importance scores for each dimension of the top 100% of the Q-dimensional features; and
[0135] Repeat the second and third steps until the feature with the highest importance score in the first P dimensions is selected.
[0136] Note 12. According to the method of Note 1 or 2, the feature is the morphological feature in the image.
[0137] Note 13. According to the method of Note 1 or 2, the threshold ranges from 0 to 1.
[0138] Appendix 14. An apparatus for image processing, comprising:
[0139] A preprocessing unit configured to obtain a probability map of an image through preprocessing;
[0140] A multidimensional feature extraction device is configured to downsample a probability map and extract features from the downsampled probability map in multiple dimensions based on different combinations of thresholds and resolutions.
[0141] The first parameter selection device is configured to input the extracted features into a classification model for the whole image and use a validation set to obtain the top N parameter combinations with the highest classification accuracy among all parameter combinations with different thresholds and different resolutions, where N is an integer greater than zero.
[0142] The feature dimensionality reduction device is configured to select the top M features with the highest importance scores from the extracted features based on the top N parameter combinations with the highest classification accuracy, using a classification model for the entire image, where M is an integer greater than zero.
[0143] The second parameter selection device is configured to select the top N' parameter combinations with the highest classification accuracy from all parameter combinations based on the features with the highest importance scores in M dimensions, using a classification model for the entire graph, where N' is a positive integer; and
[0144] An image classification device is configured to classify an image using a classification model for the entire image, based on the feature with the highest importance score in the top M dimensions under one of the top N' parameter combinations with the highest classification accuracy.
[0145] Note 15. According to Note 14, the second parameter selection device is further configured to:
[0146] From the first N' parameter combinations with the highest classification accuracy, select the first K parameter combinations with the greatest parameter differences, where K is an integer greater than zero and less than N; and
[0147] Using a classification model for the entire image, we select the features with the highest importance scores in the top P dimensions from among the M×K dimensional features consisting of the top K parameter combinations with the greatest parameter differences and the top M features with the highest importance scores, where P is a positive integer.
[0148] The image classification device is also configured to classify based on the features with the highest importance scores in the first P dimensions.
[0149] Note 16. According to Note 14 or 15, the device, wherein the multidimensional feature extraction apparatus is further configured to downsample the probability map using different downsampling methods, and
[0150] The first parameter selection device is further configured to:
[0151] Multi-dimensional feature extraction is performed on the downsampled probability map based on combinations of different thresholds, downsampling methods, and resolutions; and
[0152] The extracted features are input into a classification model for the entire image, and the top N parameter combinations with the highest classification accuracy are obtained from all parameter combinations with different thresholds, different resolutions, and different downsampling methods using the validation set.
[0153] Note 17. According to the device of Note 15, wherein the selection of the top K parameter combinations with the greatest parameter differences is included among the top N' parameter combinations with the highest classification accuracy, and for each downsampling method, the selection of the parameter combination with the greatest resolution difference or the greatest threshold difference, wherein the sum of the number of parameter combinations selected for each downsampling method is K, and wherein the priority of resolution difference is higher than that of threshold difference.
[0154] Note 18. According to Note 14, the device, wherein the feature reduction apparatus is further configured to perform the following operations:
[0155] The first operation is to train a classification model for the entire image using the extracted features for each of the top N parameter combinations with the highest classification accuracy, so as to obtain the importance score of each feature under each parameter combination.
[0156] The second operation is to calculate the mean of the importance score of each feature under each parameter combination, and select the top 100 Q features with the highest importance scores, where Q is greater than 0 and less than 100.
[0157] The third operation involves, for each of the top N parameter combinations with the highest classification accuracy, inputting the top 100% of the features with the highest importance scores in each of the Q dimensions and training a classification model for the entire image. This yields the importance score of each dimension of the top 100% of the features with the highest importance scores in each parameter combination.
[0158] Repeat steps two and three until the feature with the highest importance score in the first M dimensions is selected.
[0159] Note 19. The device according to Note 17 or 18, wherein the second parameter selection device is further configured to perform the following operations:
[0160] The first step is to select the feature with the highest importance score in the top M dimensions under each parameter combination among the K parameter combinations with the greatest parameter differences, and use these M×K dimensional features as input to train a classification model for the entire image to obtain the importance score of each dimension feature.
[0161] The second operation is to select the top Q-th percentile features with the highest importance scores from the M×K-dimensional features based on the importance scores of each feature obtained, where Q is greater than 0 and less than 100.
[0162] The third operation involves retraining the classification model for the entire graph using the top 100% of the Q-dimensional features with the highest importance scores as input, to obtain updated importance scores for each dimension of the top 100% of the Q-dimensional features; and
[0163] Repeat steps two and three until the feature with the highest importance score in the first P dimensions is selected.
[0164] Appendix 20. A computer-readable storage medium storing a program that can be executed by a processor to perform the following operations:
[0165] A probability map of the image is obtained through preprocessing;
[0166] The probability map is downsampled, and features of multiple dimensions are extracted from the downsampled probability map based on different combinations of thresholds and resolutions.
[0167] The extracted features are input into a classification model for the entire image, and the top N parameter combinations with the highest classification accuracy are obtained from all parameter combinations with different thresholds and different resolutions using the validation set, where N is an integer greater than zero.
[0168] Based on the top N parameter combinations with the highest classification accuracy, the top M features with the highest importance scores are selected from the extracted features using a classification model for the entire image, where M is an integer greater than zero.
[0169] Based on the feature with the highest importance score in M dimensions, a classification model for the entire image is used to select the top N' parameter combinations with the highest classification accuracy from all parameter combinations, where N' is a positive integer; and
[0170] Based on the feature with the highest importance score in the top M dimensions under one of the parameter combinations with the highest classification accuracy among the top N' parameter combinations, the image is classified using a classification model for the entire image.
[0171] Finally, it should be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Furthermore, unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0172] While embodiments of the present invention have been described in detail above with reference to the accompanying drawings, it should be understood that the embodiments described above are merely illustrative of the invention and do not constitute a limitation thereof. Those skilled in the art can make various modifications and alterations to the above embodiments without departing from the spirit and scope of the invention. Therefore, the scope of the invention is defined only by the appended claims and their equivalents.
Claims
1. A method for image processing, comprising: A probability map of an image is obtained through preprocessing, wherein the preprocessing includes inputting the image into a trained classification or segmentation model for image patches to obtain the probability map; The probability map is downsampled using different downsampling methods, and features of multiple dimensions are extracted from the downsampled probability map based on different combinations of thresholds for binarization, the different downsampling methods, and different resolutions. The extracted features are input into a classification model for the entire image, and the top N parameter combinations with the highest classification accuracy are obtained from all parameter combinations of the different thresholds, the different downsampling methods, and the different resolutions using the validation set, where N is an integer greater than zero. Based on the N parameter combinations with the highest classification accuracy, the M features with the highest importance scores in the extracted features are selected using the classification model for the entire image, where M is an integer greater than zero. Based on the feature with the highest M-dimensional importance score, the top N' parameter combinations with the highest classification accuracy are selected from all parameter combinations using the classification model for the entire image, where N' is a positive integer; and The image is classified using the classification model for the entire image, based on the feature with the highest importance score in the top M dimensions under one of the parameter combinations with the highest classification accuracy among the top N' parameter combinations.
2. The method according to claim 1, further comprising, after selecting the feature with the highest importance score in the first M dimensions: From the N' parameter combinations with the highest classification accuracy, select the K parameter combinations with the greatest parameter differences, where K is an integer greater than zero and less than N; and Using the classification model for the entire image, the features with the highest importance scores in the top P dimensions are selected from the M×K dimensional features composed of the top K parameter combinations with the largest parameter differences and the top M features with the highest importance scores, where P is a positive integer. in, The classification of the image using the full-image classification model includes classification based on the features with the highest importance scores in the first P dimensions.
3. The method according to claim 2, wherein, The parameter combinations with the greatest differences among the top K parameters are selected from the top N' parameter combinations with the highest classification accuracy. For each downsampling method, the parameter combination with the greatest difference in resolution or the greatest difference in threshold is selected, where the sum of the number of parameter combinations selected for each downsampling method is K.
4. The method according to claim 3, wherein, The resolution difference has a higher priority than the threshold difference.
5. The method according to claim 1, wherein, Based on the top N parameter combinations with the highest classification accuracy, the top M features with the highest importance scores are selected from the extracted features using the classification model for the entire image, including: The first step is to train the classification model for the entire image using the extracted features for each of the top N parameter combinations with the highest classification accuracy, so as to obtain the importance score of each feature under each parameter combination. The second step is to calculate the mean of the importance score of each feature under each parameter combination, and select the top 100% of features with the highest importance scores in Q dimensions, where Q is greater than 0 and less than 100. The third step involves, for each of the top N parameter combinations with the highest classification accuracy, inputting the features with the highest importance scores in the top 100% of Q dimensions and training the classification model for the entire image, to obtain the importance score of each feature in each parameter combination among the top 100% of Q dimensions with the highest importance scores; and Repeat the second and third steps until the feature with the highest importance score in the first M dimensions is selected.
6. The method according to claim 3 or 4, wherein, Using the classification model for the entire image, the selection of the features with the highest importance scores in the top P dimensions from the M×K dimensional features composed of the parameter combinations with the largest differences in the top K parameters and the features with the highest importance scores in the top M dimensions includes: The first step is to select the feature with the highest importance score in the first M dimensions under each parameter combination among the parameter combinations with the greatest parameter differences, and use these M×K dimensional features as input to train the classification model for the whole image to obtain the importance score of each dimension feature. The second step is to select the top Q-th percentile features with the highest importance scores from the M×K-dimensional features based on the importance scores of each feature obtained, where Q is greater than 0 and less than 100. The third step involves retraining the classification model for the entire image using the features with the highest importance scores in the top 100% of the Q dimensions as input, to obtain updated importance scores for each dimension of the features with the highest importance scores in the top 100% of the Q dimensions; and Repeat the second and third steps until the feature with the highest importance score in the first P dimensions is selected.
7. An apparatus for image processing, comprising: A preprocessing apparatus configured to obtain a probability map of an image through preprocessing, wherein the preprocessing includes inputting the image into a trained classification or segmentation model for image patches to obtain the probability map; A multidimensional feature extraction device is configured to downsample the probability map using different downsampling methods and to extract features from the downsampled probability map in multiple dimensions based on different combinations of thresholds for binarization, the different downsampling methods, and different resolutions. A first parameter selection device is configured to input the extracted features into a classification model for the entire image, and to obtain the top N parameter combinations with the highest classification accuracy from all parameter combinations of the different thresholds, the different downsampling methods, and the different resolutions using a validation set, where N is an integer greater than zero. A feature dimensionality reduction device is configured to select the top M features with the highest importance scores from the extracted features based on the top N parameter combinations with the highest classification accuracy, using the classification model for the whole image, where M is an integer greater than zero. The second parameter selection device is configured to select the top N' parameter combinations with the highest classification accuracy from all parameter combinations based on the features with the highest importance scores in the M dimensions, using the classification model for the entire graph; and An image classification device is configured to classify the image using the classification model for the entire image, based on the feature with the highest importance score in the top M dimensions under one of the top N' parameter combinations with the highest classification accuracy.
8. A computer-readable storage medium having a computer program stored thereon, the computer program being able to implement the method according to any one of claims 1-6 when executed by a computer.