Classification method of multi-scale pathological full-slice images, storage medium and program product

Abstract

The invention discloses a multi-scale pathological full-slice image classification method, a storage medium and a program product, and the method comprises the steps: carrying out the cutting and resampling of an obtained pathological full-slice image, and obtaining three types of image blocks of different scales; respectively sending the three kinds of image blocks with different scales into corresponding image feature extraction models to obtain three-dimensional image features; performing connection operation on the image features of the three dimensions to form feature vectors; inputting the feature vectors into a full-connection neural network with a nonlinear activation function to obtain a classification result of each image block; and generating a classification result of the pathological full-slice image based on the classification result of each image block. According to the method, pathological image features under different scales can be integrated, the features of microscopic cells and macroscopic tissues are comprehensively considered, and a better classification effect is achieved.

Description

Technical Field

[0001] The present invention belongs to the field of image processing, and particularly relates to a classification method, a storage medium, and a program product for multi-scale pathological whole-slide images. Background Art

[0002] A pathological whole-slide image is obtained by taking a certain size of diseased tissue, making a pathological section (usually embedded in paraffin and then sectioned, and stained with hematoxylin-eosin), and then digitizing it through a microscopic scanner. Diagnosis based on pathological whole-slide images is an important means indispensable in the diagnosis and treatment cycles of various cancers. This method can classify different cancer types by the morphological changes of cells, tissues, etc. in the microscopic environment, and obtain important information such as the tumor type and grade of the patient.

[0003] Pathological whole-slide images have the characteristics of data scarcity and multi-scale. Among them, data scarcity is manifested in that due to the complexity of annotation, data collection and standardization, privacy and data security, it is often difficult to obtain high-quality pathological whole-slide images; for deep learning research in the field of pathology, a large amount of this type of data is often required to obtain satisfactory results and achieve better algorithm robustness. However, the scarcity of data makes it difficult to carry out such research on a small sample. In summary, due to the large amount of high-quality data required by deep learning, the further development of pathology-related research based on deep learning is restricted. The multi-scale characteristic is manifested in that pathological whole-slide images are usually observed at different magnifications (such as 10 times, 20 times, etc.), and different magnifications correspond to different field-of-view ranges and detail levels. Therefore, pathologists need to make comprehensive judgments at different scales, such as the infiltration pattern of tumors, the boundary between tumors and surrounding normal tissues, and the generation of microvessels. However, current research mainly focuses on pathological analysis at a fixed resolution, which makes it difficult for traditional models that rely on a single resolution to capture key features at different scales. Summary of the Invention

[0004] In view of the above problems, the present invention proposes a classification method, a storage medium, and a program product for multi-scale pathological whole-slide images. The present invention can synthesize the pathological image features at different scales, comprehensively consider the features of microscopic cells and macroscopic tissues, and achieve better classification effects.

[0005] In order to achieve the above technical objectives and reach the above technical effects, the present invention is realized through the following technical solutions:

[0006] In the first aspect, the present invention provides a classification method for multi-scale pathological whole-slide images, including:

[0007] Cropping and resampling the obtained pathological whole-slide image to obtain three different-scale image patches;

[0008] The image patches of three different scales are respectively fed into the corresponding image feature extraction models to obtain image features in three dimensions;

[0009] A concatenation operation is performed on the image features in the three dimensions to form a feature vector;

[0010] The feature vector is input into a fully connected neural network with a non-linear activation function to obtain the classification results of each image patch;

[0011] Based on the classification results of each image patch, the classification result of the whole pathological section image is generated.

[0012] Combined with the first aspect, optionally, the method for generating the image patches of the three different scales includes:

[0013] The obtained whole pathological section image is cropped in the form of a sliding window to obtain a number of non-overlapping image patches;

[0014] Threshold processing is sequentially performed on each non-overlapping image patch to obtain image patches meeting the requirements;

[0015] Each image patch meeting the requirements is resampled to obtain a number of non-overlapping first-scale image patches with the first scale;

[0016] Each image patch meeting the requirements is centrally cropped to obtain an intermediate region image, and then resampled to obtain second-scale image patches with the second scale;

[0017] Each intermediate region image is cropped, threshold processed, and resampled to obtain third-scale image patches with the third scale;

[0018] Wherein, the first scale is greater than the second scale, and the second scale is greater than the third scale.

[0019] Combined with the first aspect, optionally, the method for generating the first-scale image patches includes:

[0020] The obtained whole pathological section image is cropped to obtain a number of non-overlapping image patches;

[0021] The median value of the RGB channels of each image patch is calculated;

[0022] Based on the median value of the RGB channels of each image patch, the image patches meeting the requirements are retained by using the threshold method;

[0023] All the image patches are downsampled to 224×224 pixels by using the bicubic interpolation algorithm to obtain the first-scale image patches.

[0024] Combined with the first aspect, optionally, the calculation formula adopted by the threshold method is:

[0025] ;

[0026] Among them, is an image block that meets the requirements, , , are the median values in the RGB channels respectively, is an image block obtained by cropping the acquired whole-pathology section image;

[0027] Define ;

[0028] The first-scale image block is obtained through the following calculation formula:

[0029] ;

[0030] Among them, represents the first-scale image block, is a function for adjusting the resolution adopted by the bicubic interpolation algorithm, indicating downsampling to 224×224 pixels.

[0031] Combined with the first aspect, optionally, the intermediate-region image is obtained through the following calculation formula:

[0032] ;

[0033] The second-scale image block is obtained through the following calculation formula:

[0034]

[0035] Among them, is the intermediate-region image, is the second-scale image block, is a function for cropping the image, is the size of the image block obtained by cropping the whole-pathology section image.

[0036] Combined with the first aspect, optionally, cropping, threshold processing, and resampling the intermediate-region image to obtain a third-scale image block with a third scale includes:

[0037] Calculating the th non-downsampled image block with a third scale based on a preset cropping formula , and the expression of the preset cropping formula is:

[0038] ;

[0039] Among them, is the size of the image patch obtained by cropping the whole pathological section image, , each image patch has a size of , corresponding to the four sub-regions from left to right and top to bottom in the grid, is the row index, is the column index;

[0040] Calculate the median value of each RGB channel of the image patch with the third scale, and based on the median value of each RGB channel of the image patch, use the threshold method to retain the first image patch that meets the requirements ;

[0041] The third-scale image patch with the third scale is denoted as , and the calculation formula is:

[0042] .

[0043] Combined with the first aspect, optionally, the image feature extraction model is a large pathological model pre-trained on a large-scale pathological dataset;

[0044] Send the image patches of three different scales into the corresponding large pathological models respectively to obtain image features of three dimensions. The image features of the three dimensions are obtained through the following calculation formula:

[0045] ;

[0046] Among them, represents the function of the large pathological model, takes values of large, mid, and small respectively, representing different scales; is the image feature of the th scale; is the image patch of the th scale;

[0047] Perform a concatenation operation on the image features of the three dimensions to form a feature vector. The calculation formula used for the concatenation is:

[0048] ;

[0049] Among them, is the feature vector; is the concatenation function;

[0050] Input the feature vector into a fully connected neural network with a non-linear activation function to obtain the classification results of each image patch. The calculation formula used for the fully connected neural network is:

[0051] ;

[0052] ;

[0053] Among them, is the calculation result of the first fully connected layer, is the neural network dropout function, which is used to avoid overfitting, is the non-linear activation function, is the weight matrix of the first fully connected layer, is the bias vector of the first fully connected layer, is the weight matrix of the second fully connected layer, is the bias vector of the second fully connected layer, is the classification result.

[0054] Combined with the first aspect, optionally, the classification result of the pathological whole slide image is generated based on the classification results of each image patch, and the calculation formula used is:

[0055] ;

[0056] Among them, is the probability that the pathological whole slide image is predicted to be the th class, represents the total number of image patches, represents the output of the network corresponding to the image patch ; Among them, The function converts the input value into a probability distribution, making the prediction probability of each class between 0 and 1, and the sum of all class probabilities is 1.

[0057] In the second aspect, the present invention provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, it implements the classification method of the multi-scale pathological whole slide image according to any one of the first aspect.

[0058] In the third aspect, the present invention provides a computer program product, including a computer program / instructions, and when the computer program / instructions are executed by a processor, it implements the classification method of the multi-scale pathological whole slide image according to any one of the first aspect.

[0059] Compared with the prior art, the beneficial effects of the present invention:

[0060] The present invention proposes to crop and resample the pathological whole slide image to obtain three different scales of image patches, input the three different scales of image patches into the corresponding image feature extraction model to obtain image features, and send them into a fully connected neural network with a non-linear activation layer, which can comprehensively consider the pathological image features at different scales, take into account the features of microscopic cells and macroscopic tissues, and achieve better classification effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings, where:

[0062] Figure 1 It is a schematic flowchart of a classification method for multi-scale pathological whole-slide images according to an embodiment of the present invention;

[0063] Figure 2 It is a schematic flowchart of the processing of the first-scale image block, the second-scale image block, and the third-scale image block according to an embodiment of the present invention;

[0064] Figure 3 It is a schematic structural diagram of an image feature extraction model and a fully connected neural network according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0065] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0066] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present invention, the descriptions of "first", "second", etc. are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include at least one such feature. In addition, the technical solutions between the various embodiments can be combined with each other, but it must be based on the fact that those of ordinary skill in the art can implement them. When the combination of technical solutions results in contradictions or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0067] Embodiment 1

[0068] The embodiments of the present invention provide a classification method for multi-scale pathological whole-slide images, including the following steps:

[0069] (1) Crop and resample the obtained pathological whole-slide image to obtain three different-scale image blocks;

[0070] (2) Feed the image patches of three different scales into the corresponding image feature extraction models respectively to obtain image features in three dimensions;

[0071] (3) Perform a concatenation operation on the image features in the three dimensions to form a feature vector;

[0072] (4) Input the feature vector into a fully connected neural network with a non-linear activation function to obtain the classification results of each image patch;

[0073] (5) Generate the classification result of the pathological whole slide image based on the classification results of each image patch.

[0074] In a specific implementation manner of the embodiment of the present invention, the method for generating the image patches of the three different scales includes:

[0075] Crop the obtained pathological whole slide image in the form of a sliding window to obtain a number of non-overlapping image patches;

[0076] Sequentially perform threshold processing on each non-overlapping image patch to obtain image patches that meet the requirements;

[0077] Resample each image patch that meets the requirements to obtain a number of non-overlapping first-scale image patches with the first scale;

[0078] Centrally crop each image patch that meets the requirements to obtain an intermediate region image, and perform resampling to obtain second-scale image patches with the second scale;

[0079] Crop, perform threshold processing and resampling on each intermediate region image to obtain third-scale image patches with the third scale;

[0080] Wherein, the first scale is greater than the second scale, and the second scale is greater than the third scale.

[0081] In a specific implementation manner of the embodiment of the present invention, as Figure 2 shown, the method for generating the first-scale image patches includes:

[0082] Crop the obtained pathological whole slide image to obtain a number of non-overlapping image patches; In the specific implementation process, the size of each image patch is generally 1024*1024, and other values can also be taken, which are specifically set according to actual needs;

[0083] Calculate the median value of the RGB channels of each image patch;

[0084] Based on the median value of the RGB channels of each image patch, use the threshold method to retain the image patches that meet the requirements;

[0085] All image patches are downsampled to 224×224 pixels using the bicubic interpolation algorithm to obtain the first-scale image patches.

[0086] In a specific implementation manner of the embodiment of the present invention, the calculation formula adopted by the threshold method is:

[0087] ;

[0088] wherein, is the image patch that meets the requirements, , , are the median values in the RGB channels respectively, is the image patch obtained by cropping the acquired whole-pathology section image;

[0089] Define ;

[0090] The first-scale image patches are obtained through the following calculation formula:

[0091] ;

[0092] wherein, represents the first-scale image patches, is the function for adjusting the resolution adopted by the bicubic interpolation algorithm, indicating downsampling to 224×224 pixels.

[0093] In a specific implementation manner of the embodiment of the present invention, as Figure 2 shown, the intermediate-region image is obtained through the following calculation formula:

[0094] ;

[0095] The second-scale image patches are obtained through the following calculation formula:

[0096]

[0097] wherein, is the intermediate-region image, is the second-scale image patches, is the function for cropping the image, is the size of the image patch obtained by cropping the whole-pathology section image, and this step is actually to perform central cropping on the first-scale image patches.

[0098] In a specific implementation manner of the embodiment of the present invention, as Figure 2 shown, cropping, threshold processing, and resampling the intermediate-region image to obtain the third-scale image patches with the third scale, including

[0099] Calculate the th non-downsampled image patch with the third scale based on a preset cropping formula , and the expression of the preset cropping formula is:

[0100] ;

[0101] Among them, is the size of the image patch obtained by cropping the whole-pathology slide image, represents the th non-downsampled image patch obtained by cropping the intermediate-region image (the medium-magnification region can be regarded as 's grid, and the size of each high-magnification image patch is ), corresponding to the four sub-regions from left to right and top to bottom in the grid. is the row index, is the column index, .

[0102] Calculate the median value of the RGB channels of each image patch with the third scale, and based on the median value of the RGB channels of each image patch, use the threshold method to retain the first image patch that meets the requirements , and the third-scale image patch with the third scale is denoted as :

[0103] .

[0104] In a specific implementation manner of the embodiment of the present invention, the image feature extraction model is a large pathology model pre-trained on a large-scale pathology dataset;

[0105] Send image patches of three different scales into the corresponding large pathology models respectively to obtain image features in three dimensions, and the image features in the three dimensions are obtained through the following calculation formula:

[0106] ;

[0107] Among them, represents the function of the large pathology model, takes values of large, mid, and small respectively, representing different scales; is the image feature of the th scale; is the image patch of the th scale;

[0108] Perform a concatenation operation on the image features in the three dimensions to form a feature vector, and the concatenation adopts the following calculation formula:

[0109] ;

[0110] Among them, is the feature vector; is the connection function;

[0111] Inputting the feature vector into a fully connected neural network with a non-linear activation function to obtain the classification results of each image patch. The calculation formula adopted by the fully connected neural network is:

[0112] ;

[0113] ;

[0114] Among them, is the calculation result of the first fully connected layer, is the neural network dropout function for avoiding overfitting, is the non-linear activation function, is the weight matrix of the first fully connected layer, is the bias vector of the first fully connected layer, is the weight matrix of the second fully connected layer, is the bias vector of the second fully connected layer, is the classification result.

[0115] In a specific implementation manner of the embodiment of the present invention, the calculation formula for generating the classification result of the pathological whole slide image based on the classification results of each image patch is:

[0116] ;

[0117] Among them, is the probability that the pathological whole slide image is predicted to be the th class, represents the total number of image patches, represents the output of the network corresponding to the image patch ; Among them, The function converts the input value into a probability distribution, making the prediction probability of each class between 0 and 1, and the sum of all class probabilities is 1.

[0118] The classification method of the multi-scale pathological whole slide image in the embodiment of the present invention will be described in detail below in combination with a specific implementation manner.

[0119] As Figure 1 shown, the classification method of the multi-scale pathological whole slide image includes the following steps:

[0120] First: Preprocessing

[0121] (1.1) Generate image patches (low magnification, large scale), i.e., the first-scale image patches:

[0122] In the preprocessing stage, in the embodiments of the present invention, first, the obtained original pathological whole-slide image is cropped in the form of a sliding window at a magnification of 20 times or 40 times: that is, the whole high-resolution original pathological whole-slide image is cropped into multiple non-overlapping high-resolution image patches with a resolution of more than 1000×1000 pixels, so that the computer can process a single-slice image with hundreds of millions of pixels. To avoid the influence of blank areas of microscopic images and stains such as marker pens, the median value of the RGB channels of each image patch is calculated by the threshold method , and the valid image patches are retained according to the following rules:

[0123] ;

[0124] Among them, is the image patch that meets the requirements, , , are the median values of the RGB channels respectively, is the image patch obtained by cropping the obtained pathological whole-slide image. Based on this rule, only the image patches with the median values of all RGB channels within the range of 100 to 200 are retained to obtain high-resolution image patches.

[0125] Since the pathological large model can only input image patches of a relatively small size, usually 224×224 pixels, therefore, in the embodiments of the present invention, the high-resolution image patches are also resampled. The resampling process includes: downsampling the image patches with a relatively large field of view at the original resolution to 224×224 pixels through the bicubic interpolation algorithm, which contains macroscopic tissue morphological features and can reveal the spatial clustering relationship of cells or tissues. The image patches at this scale are helpful for analyzing the tissue structure and distribution pattern in a relatively large range. Define ;

[0126] The first-scale image patches are obtained through the following calculation formula:

[0127] ;

[0128] Among them, represents the first-scale image patches, is the function for adjusting the resolution adopted by the bicubic interpolation algorithm, indicating that is downsampled to 224×224 pixels.

[0129] (1.2) Generate image patches (medium magnification, medium scale), i.e., the second-scale image patches:

[0130] The obtained original whole-slide pathological image is centrally cropped to retain the middle region, and then downsampled to 224×224 pixels. The image patches at this resolution integrate macroscopic histological features and partial cytological features, providing relatively detailed tissue and cell information while retaining a certain context relationship. The image of the middle region is obtained through the following calculation formula:

[0131] ;

[0132] The second-scale image patch is obtained through the following calculation formula:

[0133]

[0134] where is the image of the middle region, is the second-scale image patch, is the function for cropping the image, is the size of the image patch obtained by cropping the whole-slide pathological image.

[0135] (1.3) Image patch (high magnification, small scale), i.e., the third-scale image patch:

[0136] The middle region retained by central cropping in step (1.2) is evenly cropped into 4 pieces, and the steps can be expressed as:

[0137] ;

[0138] The finally obtained 224×224 pixel image patch contains rich cell-level microscopic information at high magnification, thus providing high-resolution cell features and details.

[0139] The first image patch that meets the requirement at this scale is retained by the same threshold method , and the third-scale image patch with the third scale is denoted as :

[0140] .

[0141] The processing flow of the third-scale image patch, the third-scale image patch, and the third-scale image patch is as shown in Figure 2 , Figure 2 where "x" in it represents the image patch that does not meet the threshold requirement, and √ represents the image patch that meets the threshold requirement.

[0142] Second: Image patch classification

[0143] After cropping and resampling, the three different-scale image patches with a resolution of 224×224 will be separately fed into three trainable large models in parallel designed for different scales. The specific structure of the large model can be seen in Figure 3 The large model adopts the ViT (Vision Transformer) structure and consists of a linear projection layer, a position embedding layer, multiple ViT blocks, and a classification head. Among them, the number of ViT blocks is N, and each ViT block includes a normalization layer, a multi-head attention layer, a normalization layer, and a multi-layer perceptron arranged in sequence.

[0144] In the training stage, the above three trainable large models are fine-tuned with data of three scales for a specific task. Fine-tuning means further training for a specific task based on a pre-trained model. The fine-tuning of the proposed patent adopts the following strategy: freeze all the parameters of the linear projection layer and the position embedding layer, and freeze the parameters of some ViT blocks (such as the first 14 out of 24) to achieve a balance between the prior knowledge of pre-training and the knowledge in the field. The image feature extraction model is a large pathological model pre-trained on a large-scale pathological dataset;

[0145] The image patches of three different scales are separately fed into the corresponding large pathological models to obtain image features in three dimensions. The image features in the three dimensions are obtained through the following calculation formula:

[0146] ;

[0147] Among them, represents the function of the large pathological model, takes values of large, mid, and small respectively, representing different scales; is the image feature of the th scale; is the

[0148] image patch of the

[0149] ;

[0150] Among them, is the feature vector; is the connection function;

[0151] The feature vector is input into a fully connected neural network with a non-linear activation function to obtain the classification results of each image patch. The calculation formula adopted by the fully connected neural network is:

[0152] ;

[0153] ;

[0154] Among them, is the calculation result of the first fully connected layer, is the neural network dropout function, which is used to avoid overfitting, is the non-linear activation function, is the weight matrix of the first fully connected layer, is the bias vector of the first fully connected layer, is the weight matrix of the second fully connected layer, is the bias vector of the second fully connected layer, is the classification result.

[0155] In this way, the image patch-level classification result is finally obtained. This multi-scale feature fusion method utilizes information at different scales. By integrating macroscopic and microscopic tissue and cytological features, it can more comprehensively characterize and classify pathological images.

[0156] In order to balance the prior knowledge of the pre-trained large model and the specific task, by freezing some parameters in the model, the general features of the original model are maintained and space is left for fine-tuning of subsequent layers, enhancing the adaptability to specific task data. This step can ensure that while the model retains high-level feature representations, it can be more effectively fine-tuned on new data in downstream tasks, thus improving the overall performance and accuracy of the model.

[0157] Third: Classification of whole pathological slide images

[0158] By repeating the above image patch classification, the classification results of each image patch in a single slide can be obtained. To obtain the prediction result of a single slide, by averaging the classification probability distributions obtained from all the image patches of this slide, the classification probability distribution at the slide level can be obtained, and its specific calculation formula is:

[0159] ;

[0160] Among them, is the probability that this whole pathological slide image is predicted to be the th class, represents the total number of image patches, represents the image patch corresponding to the output of the network; among them, The function converts the input value into a probability distribution, making the prediction probability of each class between 0 and 1, and the sum of all class probabilities is 1.

[0161] This step calculates the probability of each classification in the whole slide pathological image by averaging the probabilities of all image patches (i.e., soft voting). Finally, the classification of the whole slide pathological image is completed. This classification method can integrate image information of different image patches and different spatial resolutions, and has good prediction ability and robustness.

[0162] Example 2

[0163] Based on the same inventive concept as in Example 1, an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, it implements the classification method of the multi-scale whole slide pathological image described in any one of Example 1.

[0164] Example 3

[0165] Based on the same inventive concept as in Example 1, an embodiment of the present invention provides a computer program product, including a computer program / instructions, and when the computer program / instructions are executed by a processor, it implements the classification method of the multi-scale whole slide pathological image described in any one of Example 1.

[0166] Those skilled in the art should understand that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0167] The present application is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram can be implemented by computer program instructions, and the combination of processes and / or blocks in the flowchart and / or block diagram can also be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for implementing the specified functions in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0168] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured product including an instruction device, and the instruction device implements the process in Figure 1One or more processes and / or blocks Figure 1 The functions specified in one or more blocks.

[0169] These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process. Thus, the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one Figure 1 One or more processes and / or blocks Figure 1 The steps of the functions specified in one or more blocks.

[0170] The embodiments of the present invention have been described above in conjunction with the accompanying drawings. However, the present invention is not limited to the above specific embodiments. The above specific embodiments are merely illustrative and not restrictive. Under the inspiration of the present invention, those of ordinary skill in the art can also make many forms without departing from the spirit and scope of the present invention as protected by the claims. All of these are within the protection scope of the present invention.

[0171] The basic principles, main features and advantages of the present invention have been shown and described above. Those skilled in the art should understand that the present invention is not limited by the above embodiments. The above embodiments and the descriptions in the specification only illustrate the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention will have various changes and improvements, and all of these changes and improvements fall within the scope of the present invention claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A classification method for multi-scale pathological full-slice images, characterized in that: include: The acquired pathological full-slice images were cropped and resampled to obtain image blocks of three different scales; The image blocks of three different scales are respectively sent to the corresponding image feature extraction model to obtain image features of three dimensions; Performing a connection operation on the image features of the three dimensions to form a feature vector; Inputting the feature vector into a fully connected neural network with a nonlinear activation function to obtain classification results of each image block; Based on the classification results of each image block, the classification results of the pathological full-slice image are generated.

2. The multi-scale pathological full-slice image classification method according to claim 1, characterized in that: The method for generating the image blocks of three different scales includes: The acquired pathological full-slice image is cropped in the form of a sliding window to obtain several non-overlapping image blocks; Performing threshold processing on each non-overlapping image block in sequence to obtain an image block that meets the requirements; Resampling each image block that meets the requirements into a plurality of non-overlapping first-scale image blocks having a first scale; Performing center cropping on each image block that meets the requirements to obtain a middle area image, and performing resampling to obtain a second-scale image block with a second scale; Performing cropping, threshold processing, and resampling on each intermediate region image to obtain a third-scale image block having a third scale; The first scale is larger than the second scale, and the second scale is larger than the third scale.

3. The multi-scale pathological full-slice image classification method according to claim 2, characterized in that: The method for generating the first-scale image block includes: The acquired pathological full-slice image is cropped to obtain several non-overlapping image blocks; Calculate the median value of the RGB channel of each image block; Based on the median value of the RGB channel of each image block, the threshold method is used to retain the image blocks that meet the requirements; The bicubic interpolation algorithm is used to downsample all image blocks to 224×224 pixels to obtain the first scale image blocks.

4. The multi-scale pathological full-slice image classification method according to claim 3, characterized in that: The calculation formula used in the threshold method is: ； in, To meet the requirements of the image block, , , The median values ​​of the RGB channels respectively, The image block is obtained by cropping the acquired pathological full-slice image; definition ; The first scale image block is obtained by the following calculation formula: ； in, represents the first scale image block, is the function used by the bicubic interpolation algorithm to adjust the resolution, indicating that Downsampled to 224×224 pixels.

5. The multi-scale pathological full-slice image classification method according to claim 4, characterized in that: The intermediate area image is obtained by the following calculation formula: ； The second scale image block is obtained by the following calculation formula: ； in, is the middle area image, is the second scale image block, is the function for cropping the image, is the size of the image block obtained by cropping the pathological full-slice image.

6. The multi-scale pathological full-slice image classification method according to claim 5, characterized in that: The step of clipping, thresholding and resampling the middle region image to obtain a third-scale image block having a third scale includes: Calculate the first Unsampled image patches with the third scale , the expression of the preset clipping formula is: ； in, is the size of the image block obtained by cropping the pathological full-slice image, , each image block The size is , corresponding to the four sub-areas from left to right and from top to bottom in the grid, is the row index, is the column index; Calculate the median value of the RGB channel of each image block with the third scale, and based on the median value of the RGB channel of each image block, use the threshold method to retain the first image block that meets the requirements ; The third-scale image block with the third scale is denoted as , the calculation formula is: 。 7. The multi-scale pathological full-slice image classification method according to claim 1, characterized in that: The image feature extraction model is a large pathology model pre-trained on a large-scale pathology dataset; The image blocks of three different scales are respectively sent to the corresponding pathological large model to obtain the image features of three dimensions. The image features of the three dimensions are obtained by the following calculation formula: ； in, A function representing a large model of pathology, The values ​​of are large, mid and small, indicating different scales; For the Image features of different scales; For the Image patches of various scales; The image features of the three dimensions are connected to form a feature vector, and the calculation formula used for the connection is: ； in, is the feature vector; is the connection function; The feature vector is input into a fully connected neural network with a nonlinear activation function to obtain the classification results of each image block. The calculation formula used by the fully connected neural network is: ； ； in, is the calculation result of the first fully connected layer, is the random inactivation function of the neural network, used to avoid overfitting, is a nonlinear activation function, is the weight matrix of the first fully connected layer, is the bias vector of the first fully connected layer, is the weight matrix of the second fully connected layer, is the bias vector of the second fully connected layer, is the classification result.

8. The multi-scale pathological full-slice image classification method according to claim 7, characterized in that: The classification result of the pathological full-slice image is generated based on the classification result of each image block, and the calculation formula used is: ； in, The pathological full-slice image is predicted to be The probability of the class, Represents the total number of image blocks, Represents an image block The corresponding network output; where The function converts the input value into a probability distribution so that the predicted probability of each class is between 0 and 1 and the sum of all class probabilities is 1.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method for classifying multi-scale pathological full-slice images described in any one of claims 1 to 8 is implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the method for classifying multi-scale pathological full-slice images described in any one of claims 1 to 8 is implemented.

Images