Prostate cancer pathological image classification method and device, equipment and medium

Abstract

The embodiment of the invention provides a prostate cancer pathological image classification method and device, equipment and a medium. The method comprises the following steps: acquiring a prostate cancer pathological image; performing feature extraction on the prostatic cancer pathological image through a feature extractor to obtain a plurality of instance features; feature reconstruction is conducted on the multiple instance features through an RLMI L model, global features and local features are obtained, the RLMI L model comprises a reordering Mama module and a linear deformable convolution module, the reordering Mama module is used for extracting the global features among the multiple instance features, and the linear deformable convolution module is used for capturing the local features among the multiple instance features; performing feature aggregation on the global features and the local features to obtain packet-level features; and calculating prediction probability distribution of each category of the package-level features to obtain a classification result of the prostatic cancer pathological image. Based on the method, the accuracy of classifying the prostatic cancer pathological images can be improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of medical image processing, and in particular, to a method and device, equipment, and medium for classifying prostate cancer pathological images. Background Art

[0002] Prostate cancer is one of the most common cancers in men, and its early diagnosis and accurate classification are crucial for formulating effective treatment plans. Whole-slide imaging examination is regarded as the gold standard for cancer diagnosis. However, the high resolution of its images and the lack of pixel-level label characteristics make the examination complex.

[0003] Currently, weakly supervised multi-instance learning has become the mainstream method for whole-slide imaging examination. However, existing methods do not consider the context information around a single region and the correlation between different regions, making it impossible for the model to fully learn the potential feature changes when processing complex pathological images, thus affecting the accuracy of prostate cancer pathological image classification. Therefore, how to improve the accuracy of prostate cancer pathological image classification has become an urgent technical problem to be solved. Summary of the Invention

[0004] The embodiments of the present invention provide a method and device, equipment, and medium for classifying prostate cancer pathological images, which can improve the accuracy of prostate cancer pathological image classification.

[0005] In a first aspect, the embodiments of the present invention provide a method for classifying prostate cancer pathological images, including:

[0006] Obtain a prostate cancer pathological image, where the prostate cancer pathological image is a whole-slide digital slice image;

[0007] Extract features from the prostate cancer pathological image through a feature extractor to obtain multiple instance features;

[0008] Reconstruct the features of multiple instance features through a pre-trained multi-instance learning RLMIL model based on re-ranked Mamba and linearly deformable convolution to obtain global features and local features, where the RLMIL model includes a re-ranked Mamba module and a linearly deformable convolution module, the re-ranked Mamba module is used to extract the global features between multiple instance features, and the linearly deformable convolution module is used to capture the local features between multiple instance features;

[0009] Aggregate the global features and the local features to obtain bag-level features;

[0010] Calculate the predicted probability distribution of each category of the bag-level features to obtain the classification result of the prostate cancer pathological image.

[0011] In some embodiments, the prostate cancer pathological image is subjected to feature extraction by a feature extractor to obtain a plurality of instance features, including:

[0012] Foreground extraction is performed on the prostate cancer pathological image, and a tissue region is screened out;

[0013] The tissue region is divided into a plurality of instance image patches;

[0014] The feature extractor is used to map the plurality of instance images to the plurality of instance features in different feature dimensions.

[0015] In some embodiments, the method further includes:

[0016] Preprocessing is performed on the prostate cancer pathological image, and the preprocessing process includes:

[0017] Using a threshold segmentation algorithm to perform non-overlapping slicing on the prostate cancer image at a set magnification to obtain a plurality of non-overlapping non-overlapping slices of a fixed size;

[0018] Using the ResNet50 pre-trained on ImageNet as a feature extractor to perform feature extraction on the non-overlapping slices to obtain a plurality of low-dimensional feature vectors.

[0019] In some embodiments, the plurality of instance features are subjected to feature reconstruction by a pre-trained multi-instance learning RLMIL model based on reordering Mamba and linearly deformable convolution to obtain global features and local features, including:

[0020] The reordering Mamba module is used to analyze the global relationship between the plurality of instance features to extract the global features;

[0021] The linearly deformable convolution module is used to analyze the local relationship between the plurality of instance features to capture the local features.

[0022] In some embodiments, the reordering Mamba module is used to analyze the global relationship between the plurality of instance features to extract the global features, including:

[0023] Combining a first sequence scanning method and a second sequence scanning method to determine the input sequence of the plurality of instance features, where the first sequence scanning method is a scanning method with a horizontal direction, and the second sequence scanning method is a scanning method with a vertical direction;

[0024] The input sequence is mapped through an implicit state to generate an output sequence to obtain continuous parameters;

[0025] The continuous parameters are converted into discrete parameters by using a zero-order hold discretization rule;

[0026] The reordering Mamba module calculates the discrete parameters in the global convolution mode and outputs the global features.

[0027] In some embodiments, the method of using the linear deformable convolution module to analyze the local relationships between multiple instance features and capture the local features includes:

[0028] Define the coordinates of multiple initial sampling points;

[0029] Based on the coordinates of multiple initial sampling points, the offset learned by the sub-network of the linear deformable convolution module, and the convolution parameters of the linear deformable convolution module, dynamically adjust the sampling position of the convolution kernel to determine multiple target regions;

[0030] Use the linear deformable convolution module to learn the local relationships between multiple instance features in different target regions to obtain the local features.

[0031] In some embodiments, the training method of the RLMIL model includes:

[0032] Construct a target loss function, which is determined according to the standard cross-entropy loss function, the weight factor of the bag-level feature classification loss function, and the instance-level loss function. Among them, the instance-level loss function is used to represent the total loss of instance clustering, and the instance-level loss function is composed of the binary SVM loss function;

[0033] Train the RLMIL model based on the target loss function to obtain the trained RLMIL model.

[0034] In a second aspect, an embodiment of the present invention further provides a prostate cancer pathological image classification device, and the device includes:

[0035] An acquisition module, configured to acquire a prostate cancer pathological image, and the prostate cancer pathological image is a whole-slide digital section image;

[0036] An extraction module, configured to extract features from the prostate cancer pathological image through a feature extractor to obtain multiple instance features;

[0037] A reconstruction module for reconstructing the features of multiple instance features through a pre-trained multi-instance learning RLMIL model based on reordered Mamba and linearly deformable convolution to obtain global features and local features, where the RLMIL model includes a reordered Mamba module and a linearly deformable convolution module, the reordered Mamba module is used to extract the global features between the multiple instance features, and the linearly deformable convolution module is used to capture the local features between the multiple instance features;

[0038] An aggregation module for aggregating the global features and the local features to obtain bag-level features;

[0039] A classification module for calculating the predicted probability distribution of each category of the bag-level features to obtain the classification result of the prostate cancer pathological image.

[0040] In a third aspect, an embodiment of the present invention further provides an electronic device, including: a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein when the processor executes the computer program, the prostate cancer pathological image classification method described in the first aspect is implemented.

[0041] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium storing computer-executable instructions for executing the prostate cancer pathological image classification method described in the first aspect.

[0042] According to the prostate cancer pathological image classification method, device, equipment and medium provided by the embodiments of the present invention, the prostate cancer pathological image classification method includes: obtaining a prostate cancer pathological image, where the prostate cancer pathological image is a whole-slide digital section image; extracting features from the prostate cancer pathological image through a feature extractor to obtain a plurality of instance features; reconstructing the features of the plurality of instance features through a pre-trained multi-instance learning RLMIL model based on re-ranked Mamba and linearly deformable convolution to obtain global features and local features, where the RLMIL model includes a re-ranked Mamba module and a linearly deformable convolution module, the re-ranked Mamba module is used to extract global features between the plurality of instance features, and the linearly deformable convolution module is used to capture local features between the plurality of instance features; aggregating the global features and local features to obtain bag-level features; calculating the predicted probability distribution of each category of the bag-level features to obtain the classification result of the prostate cancer pathological image. By introducing the re-ranked Mamba module and the linearly deformable convolution module, the present invention enables the RLMIL model to capture and integrate the global and local features between instances through a two-stream mechanism, thereby improving the understanding of tumor heterogeneity, significantly enhancing the analysis ability of complex pathological images, and improving the classification ability of the model for complex pathological images. Compared with the prior art, the present invention shows stronger capabilities in capturing key details and processing complex data. Based on this, the embodiments of the present invention can improve the accuracy of prostate cancer pathological image classification. Description of the Drawings

[0043] Figure 1 is a flowchart of the prostate cancer pathological image classification method provided by an embodiment of the present invention;

[0044] Figure 2 is a flowchart of steps S201 to S202 provided by an embodiment of the present invention;

[0045] Figure 3 is a diagram of the WSIs preprocessing process provided by an embodiment of the present invention;

[0046] Figure 4 is a sub-flowchart of step S102 provided by an embodiment of the present invention;

[0047] Figure 5 is an overall framework diagram of the RLMIL model provided by an embodiment of the present invention;

[0048] Figure 6 is a sub-flowchart of step S103 provided by an embodiment of the present invention;

[0049] Figure 7 is a sub-schematic diagram of step S601 provided by an embodiment of the present invention;

[0050] Figure 8 It is a diagram of a sequence scanning method provided by an embodiment of the present invention;

[0051] Figure 9 It is a sub - schematic diagram of step S602 provided by an embodiment of the present invention;

[0052] Figure 10 It is a flowchart of a training method for the RLMIL model provided by an embodiment of the present invention;

[0053] Figure 11 It is a schematic structural diagram of a prostate cancer pathological image classification device provided by an embodiment of the present invention;

[0054] Figure 12 It is a schematic diagram of an electronic device provided by an embodiment of the present invention. Detailed implementation manners

[0055] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0056] It should be noted that although functional module division is carried out in the device schematic diagram and the logical sequence is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order from the module division in the device or the sequence in the flowchart. Terms such as "first", "second", etc. in the specification, claims and the following drawings are used to distinguish similar objects, and do not necessarily need to describe a specific order or sequence.

[0057] In the embodiments of the present invention, words such as "furthermore", "exemplarily" or "optionally" are used to represent examples, illustrations or explanations, and should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Using words such as "furthermore", "exemplarily" or "optionally" aims to present relevant concepts in a specific manner.

[0058] In order to more conveniently describe the working principle of the embodiments of the present invention in the following, an introduction to the relevant technical scenarios is given first.

[0059] Prostate cancer is one of the most common cancers in men, and its early diagnosis and accurate classification are crucial for formulating effective treatment plans. Whole - slide pathology image examination is regarded as the gold standard for cancer diagnosis. However, the high resolution of its images and the lack of pixel - level label characteristics make the examination complex.

[0060] At present, weakly supervised multi-instance learning has become the mainstream method for whole-slide image examination. However, existing methods do not consider the context information around individual regions and the correlation between different regions, resulting in the model being unable to fully learn potential feature changes when processing complex pathological images, thus affecting the accuracy of prostate cancer pathological image classification. Therefore, how to improve the accuracy of prostate cancer pathological image classification has become a technical problem to be solved urgently.

[0061] Based on this, the present invention provides a method, device, equipment and medium for classifying prostate cancer pathological images. Among them, the method for classifying prostate cancer pathological images includes: obtaining a prostate cancer pathological image, where the prostate cancer pathological image is a whole-slide digital section image; extracting features from the prostate cancer pathological image through a feature extractor to obtain multiple instance features; performing feature reconstruction on the multiple instance features through a pre-trained multi-instance learning RLMIL model based on re-ranked Mamba and linearly deformable convolutions to obtain global features and local features, where the RLMIL model includes a re-ranked Mamba module and a linearly deformable convolution module, the re-ranked Mamba module is used to extract global features between multiple instance features, and the linearly deformable convolution module is used to capture local features between multiple instance features; aggregating the global features and local features to obtain bag-level features; calculating the predicted probability distribution of each category of the bag-level features to obtain the classification result of the prostate cancer pathological image. By introducing the re-ranked Mamba module and the linearly deformable convolution module, the present invention enables the RLMIL model to capture and integrate global and local features between instances through a two-stream mechanism, thereby improving the understanding of tumor heterogeneity, significantly enhancing the analysis ability for complex pathological images, and improving the classification ability of the model for complex pathological images. Compared with the prior art, the present invention shows stronger capabilities in capturing key details and processing complex data. Based on this, the embodiments of the present invention can improve the accuracy of prostate cancer pathological image classification.

[0062] The following further elaborates on the embodiments of the present invention in conjunction with the accompanying drawings.

[0063] As Figure 1 shown, Figure 1 is a flowchart of a method for classifying prostate cancer pathological images provided by an embodiment of the present invention. The method for classifying prostate cancer pathological images may include but is not limited to steps S101 to S105.

[0064] Step S101, obtaining a prostate cancer pathological image, where the prostate cancer pathological image is a whole-slide digital section image;

[0065] Step S102, extracting features from the prostate cancer pathological image through a feature extractor to obtain multiple instance features;

[0066] Step S103, feature reconstruction is performed on multiple instance features through a pre-trained multi-instance learning RLMIL model based on reordered Mamba and linear deformable convolution to obtain global features and local features. The RLMIL model includes a reordered Mamba module and a linear deformable convolution module. The reordered Mamba module is used to extract global features among multiple instance features, and the linear deformable convolution module is used to capture local features among multiple instance features;

[0067] Step S104, feature aggregation is performed on the global features and local features to obtain bag-level features;

[0068] Step S105, calculate the predicted probability distribution of each category of the bag-level features to obtain the classification result of the prostate cancer pathological image.

[0069] It can be understood that, in order to achieve a more accurate benign and malignant classification of prostate cancer pathological sections, the technical solution of the present invention constructs a multi-instance learning method based on reordered Mamba and linear variable convolution, and proposes a multi-instance learning model RLMIL (Multiple Instance Learning based on Reordered Mamba and Linear Deformable Convolution). The RLMIL model captures global and local features between instances through a two-stream mechanism, improving the model's classification ability for complex pathological images.

[0070] It can be understood that the RLMIL model uses the reordered Mamba module to extract global features between instances. The reordered Mamba module changes the input sequence order using the reordering method and then quickly analyzes the overall structure of the pathological image using the Mamba method. In addition, the RLMIL model also introduces a linear deformable convolution LDConv module to capture local features between instances. The LDConv module dynamically adjusts the sampling position of the convolution kernel, enabling the model to adapt to the features between instances in different regions. By combining these two modules, namely the reordered Mamba module and the linear deformable convolution module, the present invention significantly enhances the analysis ability of pathological images, providing more accurate support for the diagnosis of prostate cancer.

[0071] It is understandable that the present invention constructs a dual-stream multi-instance learning model RLMIL. First, the Reordered Mamba module is used to capture the global features between instances and quickly analyze the overall structure of the image. Secondly, the linear deformable convolution module is introduced to dynamically collect the feature distribution between instances to capture local features. In the absence of fine-grained labels, the model is trained and predicted by using slice-level labels, so that it can still maintain high accuracy when processing complex WSIs. The present invention shows good potential in realizing the accurate grading of prostate cancer and provides important support for clinical pathological analysis. This not only helps to improve the accuracy of tumor diagnosis, but also provides an important basis for the formulation of personalized treatment plans.

[0072] It is understandable that multi-instance learning is a common weakly supervised learning framework. In this framework, the training samples consist of bags containing a large number of instances, and the labels of these instances are unknown, only the bag-level labels are known. If the bag contains at least one positive instance, the bag-level label is positive, otherwise it is negative. For the binary classification task of WSIs, the input is N bags I = {B 1 ,B 2 ,…,B N}, where an instance is represented as B i = {x i,1 ,x i,2 ,…,x i,K}, and the corresponding hidden label of the instance is L i = {y i,1 ,y i,2 ,…,y i,K}. The relationship between the bag-level label Y i and the instance label y i,j is defined as:

[0073]

[0074] The process of multi-instance learning inferring the bag-level label can be expressed as:

[0075] Y i = g(f(x 1 ),f(x 2 ),…,f(x K )) (2)

[0076] In the formula, f(·) is an embedding function, which can be any type of embedding function, whether the function has parameters or is differentiable. For example, in common multi-instance methods, the output result of this function is a one-dimensional space, similar to a probability space. g(·) represents an aggregation function, which calculates the final bag-level feature by weighted summation of the instance features. Y i represents the prediction result of the bag-level feature.

[0077] In one embodiment, as Figure 2 shown, the prostate cancer pathological image classification method of the present invention may include but is not limited to the following steps:

[0078] Step S201, using a threshold segmentation algorithm to perform non-overlapping slicing on a prostate cancer image at a set magnification to obtain a plurality of non-overlapping non-overlapping slices of a fixed size;

[0079] Step S202, using the ResNet50 pre-trained on ImageNet as a feature extractor to extract features from the non-overlapping slices to obtain a plurality of low-dimensional feature vectors.

[0080] It can be understood that obtaining the prostate cancer pathological image as a whole-slide digital section image, the whole-slide digital section image WSIs can be preprocessed. Using a threshold segmentation algorithm (OTSU) to perform non-overlapping slicing on the ultra-high pixel-level WSIs at 20× magnification (discarding the background and retaining the part where the tissue area occupies more than 40% of the image area), n non-overlapping image slices of a fixed size of 512×512 pixels can be obtained. Then, using the ResNet50 pre-trained on ImageNet as a feature extractor, these patches are mapped into n 1024-dimensional low-dimensional feature vectors. The entire preprocessing process is as Figure 3 shown.

[0081] In one embodiment, as Figure 4 shown, step S102 includes but is not limited to the following steps:

[0082] Step S401, performing foreground extraction on the prostate cancer pathological image to screen and obtain the tissue area;

[0083] Step S402, dividing the tissue area into a plurality of instance image patches;

[0084] Step S403, using a feature extractor to map a plurality of instance images to a plurality of instance features of different feature dimensions.

[0085] It can be understood that the RLMIL model framework is as Figure 5 shown, and its specific implementation steps are as follows: When first given a WSI, the RLMIL model performs foreground extraction on the WSI, screens out the tissue area and divides the tissue area into M instance image patches {P 1 , P 2 , …, P M}, and secondly uses a feature extractor to map the instance images to instance features {x 1 , x 2 , …, x M}, where x ∈ R M×D , and D is the feature dimension.

[0086] In one embodiment, as Figure 6 shown, step S103 includes but is not limited to the following steps:

[0087] Step S601, using the reordered Mamba module to analyze the global relationship between multiple instance features and extracting the global features;

[0088] Step S602, using the linear deformable convolution module to analyze the local relationship between multiple instance features and capturing the local features.

[0089] It can be understood that in the feature reconstruction stage, the dual-stream architecture of the reordered Mamba module and the linear deformable convolution module is used to capture the global and local relationships between features, obtaining dual-stream features, which include global features and local features. Finally, the dual-stream features are superimposed and aggregated into packet-level features for classification tasks.

[0090] In one embodiment, as Figure 7 shown, step S601 includes but is not limited to the following steps:

[0091] Step S701, determining the input sequence of multiple instance features by combining the first sequence scanning method and the second sequence scanning method, where the first sequence scanning method is a scanning method with a horizontal direction, and the second sequence scanning method is a scanning method with a vertical direction;

[0092] Step S702, mapping the input sequence through the implicit state to generate the output sequence to obtain continuous parameters;

[0093] Step S703, converting the continuous parameters into discrete parameters by using the discretization rule of zero-order hold;

[0094] Step S704, the reordered Mamba module calculates the discrete parameters through the global convolution mode and outputs the global features.

[0095] It can be understood that the reordered Mamba module is a key component in the RLMIL model, which uses the SSM to quickly model the dependencies between instances, and the SSM is a classic continuous system, and the input sequence of this system is expressed as Subsequently, through the implicit state generates the mapping of the output sequence .

[0096] h i = Ah i-1 + Bx i , y i = Ch i (3)

[0097] To adapt this continuous system to deep learning applications, it is often necessary to discretize formula (3) using the time scale parameter Δ. Usually, like Mamba, it will use the discretization rule of zero-order hold to convert the continuous parameters A, B, and C into discrete parameters and

[0098]

[0099] After the parameter discretization is completed, the Mamba model calculates the output through the global convolution mode to achieve efficient parallel computing.

[0100]

[0101] It can be observed from formula (3) that the current input sequence is associated with the implicit state and affects the result of the output sequence, while the sequence scanning method affects the input sequence of Mamba. Therefore, an efficient scanning method helps to promote the global feature learning of Mamba. The traditional Bi-mamba (Bidirectional-Mamba) forms horizontal bidirectional modeling with the scanning methods in Figure 8 (a) and (b) therein. Inspired by ZigMa, the RLMIL model of the present invention combines the scanning methods in Figure 8 (a) and (c) therein, increasing the vertical component in the two-dimensional space.

[0102] In one embodiment, as shown in Figure 9 , step S602 includes but is not limited to the following steps:

[0103] Step S901, defining the coordinates of multiple initial sampling points;

[0104] Step S902, based on the coordinates of multiple initial sampling points, the sub-network of the linearly deformable convolution module learns the offset and the convolution parameters of the linearly deformable convolution module to dynamically adjust the sampling position of the convolution kernel, and determines multiple target regions;

[0105] Step S903, using the linearly deformable convolution module to learn the local relationship between multiple instance features in different target regions to obtain local features.

[0106] It can be understood that when processing instance features in a two-dimensional space, traditional convolution will learn within a fixed region. The limitation of the fixed region makes the model perform inadequately when dealing with instance features of pathological images with various shapes. To solve this problem, the present invention introduces a linearly deformable convolution module (LDConv module). The LDConv module adapts to the irregular structural distribution of instances in the image by dynamically adjusting the sampling position of the convolution kernel. The specific principle process of the LDConv module is as shown in Figure 5(c) As shown: (1) Define M initial sampling points; (2) Based on the sub-network, learn the offset, which is designed to offset the sampling points on the input feature map so as to focus on the regions or targets of interest in this article; (3) Finally, use the new convolutional network to learn the new regions to obtain the feature representation. When defining the coordinates P of the initial sampling points 0 After that, the corresponding variable convolution operation at this position is as follows:

[0107]

[0108] In the formula, R, W, and x(P 0 +P n +ΔP n ) represent the sampling grid, convolution parameters, and pixels corresponding to the offset position respectively. When the resampling in the LDConv module is completed, the instance feature dimension of the pathological image changes from (C, H, W) to (C, M×H, W), and then 2D convolution with a convolution kernel size of (M×1) is used for feature extraction to reduce the convolution operation to linear complexity, and finally the output feature with a dimension of (C, H, W) is obtained.

[0109] In one embodiment, as Figure 10 shown, the training method of the RLMIL model includes but is not limited to the following steps:

[0110] Step S1001, construct the target loss function, which is determined according to the standard cross-entropy loss function, the weight factor of the packet-level feature classification loss function, and the instance-level loss function. Among them, the instance-level loss function is used to characterize the total loss of instance clustering, and the instance-level loss function is composed of the SVM loss function of binary classification;

[0111] Step S1002, train the RLMIL model based on the target loss function to obtain the trained RLMIL model.

[0112] It can be understood that for the classification task, the features H = {h 1 , h 2 , …, h K} passing through the two-stream architecture need to be mapped to the category space, and this method includes a feature aggregator and a feature classifier. In the feature aggregator, N parallel and independent instance-level feature classifiers (W c,1 , W c,2 , …, W c,N ) are constructed to obtain the corresponding packet-level scores for each class in the feature H, and its specific expression is as follows:

[0113]

[0114] A i,K is the attention score of the Kth instance in category i, Hbag,i ∈R 1×D is a bag-level representation aggregated using the distribution of the i-th type of attention score. Then, through the bag-level classifier W c,i obtain the bag-level score S bag,i , and subsequently apply the Softmax function to S bag,i Calculate the predicted probability distribution for each category of the bag-level features.

[0115] S bag,i = W c,i H bag,i (8)

[0116] During the classification task, there are both instance-level classifiers and bag-level classifiers. Therefore, the total loss function during training is Loss Total , and its mathematical expression is as follows:

[0117] Loss Total = λLoss Bag + (1 - λ)Loss Patch (9)

[0118] Among them, the bag-level loss function Loss Bag is composed of the standard cross-entropy loss function, and λ is the weight factor of the bag-level feature classification loss function. The instance-level loss function Loss Patch is the sum of instance clustering losses and is composed of the binary classification SVM (Support Vector Machine) loss function. Based on the above total loss function, the RLMIL model is trained to obtain a trained RLMIL model.

[0119] Based on this, compared with the prior art, the prostate cancer pathological image classification method of the present invention has at least the following beneficial effects:

[0120] (1) The present invention adopts a weakly supervised multi-instance learning method to effectively achieve the benign and malignant classification task of prostate cancer under the condition of limited labeled data.

[0121] (2) The present invention constructs a re-ranked Mamba module, which adjusts the order of the input images and uses the Mamba method to quickly learn and extract the global features of the images, enhancing the model's understanding and capture ability of the overall structure of pathological images.

[0122] (3) The present invention constructs a linearly deformable convolution module, which enhances the adaptability to the features of different local regions by dynamically adjusting the position of the convolution kernel, especially in the fine-grained lesion recognition, improving the accuracy and robustness of the model.

[0123] It should be noted that the present invention proposes a multi-instance learning method based on re-ordered Mamba and linearly deformable attention, which is specifically used for the benign and malignant classification of prostate cancer pathological sections. In existing MIL methods, although some methods capture local features through attention mechanisms and clustering constraints, they often ignore the context information of images and the dynamic associations between instances. In contrast, the present invention can learn and integrate the global and local information of instance features simultaneously from two-stream directions by introducing re-ordered Mamba and linearly deformable attention modules, thereby improving the understanding of tumor heterogeneity. Compared with the prior art, the present invention shows stronger capabilities in capturing key details and processing complex data, and has higher clinical application value and promotion potential.

[0124] In addition, as Figure 11 shown, an embodiment of the present invention also discloses a prostate cancer pathological image classification device, which includes:

[0125] An acquisition module 110, configured to acquire a prostate cancer pathological image, where the prostate cancer pathological image is a whole-slide digital section image;

[0126] An extraction module 120, configured to extract features from the prostate cancer pathological image through a feature extractor to obtain a plurality of instance features;

[0127] A reconstruction module 130, configured to perform feature reconstruction on the plurality of instance features through a pre-trained multi-instance learning RLMIL model based on re-ordered Mamba and linearly deformable convolution to obtain global features and local features, where the RLMIL model includes a re-ordered Mamba module and a linearly deformable convolution module, the re-ordered Mamba module is used to extract global features between the plurality of instance features, and the linearly deformable convolution module is used to capture local features between the plurality of instance features;

[0128] An aggregation module 140, configured to perform feature aggregation on the global features and local features to obtain bag-level features;

[0129] A classification module 150, configured to calculate the predicted probability distribution of each category of the bag-level features to obtain the classification result of the prostate cancer pathological image.

[0130] The prostate cancer pathological image classification device according to the embodiment of the present invention is used to execute the prostate cancer pathological image classification method in the above embodiment, and its specific processing process is the same as that of the prostate cancer pathological image classification method in the above embodiment, and will not be elaborated here one by one.

[0131] In addition, as Figure 12As shown in the figure, an embodiment of the present invention also discloses an electronic device, including: at least one processor 210; at least one memory 220 for storing at least one program; when the at least one program is executed by the at least one processor 210, it implements the prostate cancer pathological image classification method in any of the previous embodiments.

[0132] In addition, an embodiment of the present invention also discloses a computer-readable storage medium, in which computer-executable instructions are stored, and the computer-executable instructions are used to execute the prostate cancer pathological image classification method in any of the previous embodiments.

[0133] The system architecture and application scenarios described in the embodiments of the present invention are for more clearly illustrating the technical solutions of the embodiments of the present invention, and do not constitute a limitation on the technical solutions provided by the embodiments of the present invention. Those skilled in the art know that with the evolution of the system architecture and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present invention are equally applicable to similar technical problems.

[0134] Those of ordinary skill in the art can understand that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, and their appropriate combinations.

[0135] In the hardware implementation, the division between the functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component can have multiple functions, or a function or step can be executed by several physical components in cooperation. Some physical components or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include a computer storage medium (or non-transitory medium) and a communication medium (or transitory medium). As is well known to those of ordinary skill in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory, or other memory technologies, CD-ROM, digital versatile disc (DVD), or other optical disc storage, magnetic cassette, tape, magnetic disk storage, or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, as is well known to those of ordinary skill in the art, a communication medium typically includes computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and can include any information delivery medium.

[0136] As used in this specification, the terms "component", "module", "system", etc. are used to denote a computer-related entity, hardware, firmware, a combination of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable, an execution thread, a program, or a computer. By way of illustration, both an application running on a computing device and the computing device can be components. One or more components can reside within a process or execution thread, and a component can be located on one computer or distributed between two or more computers. Further, these components can execute from various computer-readable media having various data structures stored thereon. A component can, for example, communicate by way of signals among one or more data packets (e.g., data from two components interacting with one another from a local system, a distributed system, or a network, such as the Internet interacting with other systems via signals).

Claims

1. A prostate cancer pathology image classification method, comprising: Acquiring a prostate cancer pathology image, wherein the prostate cancer pathology image is a full-field digital slice image; Extracting features from the prostate cancer pathology image using a feature extractor to obtain multiple instance features; Reconstructing the features of the plurality of instance features through a pre-trained multi-instance learning RLMIL model based on reordering Mamba and linear deformable convolution to obtain global features and local features, wherein the RLMIL model includes a reordering Mamba module and a linear deformable convolution module, the reordering Mamba module is used to extract the global features between the plurality of instance features, and the linear deformable convolution module is used to capture the local features between the plurality of instance features; Performing feature aggregation on the global features and the local features to obtain package-level features; The predicted probability distribution of each category of the packet-level features is calculated to obtain the classification result of the prostate cancer pathology image.

2. The method according to claim 1, characterized in that The feature extractor is used to extract features from the prostate cancer pathology image to obtain multiple instance features, including: Performing foreground extraction on the prostate cancer pathology image to screen and obtain tissue regions; dividing the tissue region into a plurality of instance image blocks; The feature extractor is used to map the plurality of instance images to the plurality of instance features of different feature dimensions.

3. The method according to claim 1, characterized in that The method further comprises: The prostate cancer pathology image is preprocessed, and the preprocessing process includes: Using a threshold segmentation algorithm to perform non-overlapping slicing on the prostate cancer image at a set magnification to obtain a plurality of non-overlapping non-overlapping slices of a fixed size; ResNet50 pre-trained on ImageNet is used as a feature extractor to extract features from the non-overlapping slices to obtain multiple low-dimensional feature vectors.

4. The method according to claim 1, characterized in that: The feature reconstruction of multiple instance features is performed through the pre-trained multi-instance learning RLMIL model based on reordering Mamba and linear deformable convolution to obtain global features and local features, including: The reordering Mamba module is used to analyze the global relationship between the multiple instance features to extract the global feature; The linear deformable convolution module is used to analyze the local relationship between the multiple instance features to capture the local features.

5. The method according to claim 4, characterized in that The adopting the reordering Mamba module to analyze the global relationship between the multiple instance features and extracting the global features includes: Determine an input sequence of the plurality of instance features by combining a first sequence scanning mode and a second sequence scanning mode, wherein the first sequence scanning mode is a scanning mode with a horizontal direction, and the second sequence scanning mode is a scanning mode with a vertical direction; Mapping the input sequence to generate an output sequence through an implicit state to obtain a continuous parameter; The continuous parameter is converted into a discrete parameter using a zero-order hold discretization rule; The reordering Mamba module calculates the discrete parameters through a global convolution mode and outputs the global features.

6. The method according to claim 4, characterized in that The adopting the linear deformable convolution module to analyze the local relationship between the multiple instance features to capture the local features includes: Define multiple initial sampling point coordinates; Dynamically adjust the sampling position of the convolution kernel based on the coordinates of the multiple initial sampling points, the sub-network learning offset of the linear deformable convolution module and the convolution parameters of the linear deformable convolution module to determine multiple target areas; The linear deformable convolution module is used to learn the local relationship between the multiple instance features in different target areas to obtain the local features.

7. The method according to claim 1, characterized in that The training method of the RLMIL model includes: Constructing a target loss function, wherein the target loss function is determined according to a standard cross entropy loss function, a weight factor of a packet-level feature classification loss function, and an instance-level loss function, wherein the instance-level loss function is used to characterize the sum of losses of instance clustering, and the instance-level loss function is composed of a binary SVM loss function; The RLMIL model is trained based on the target loss function to obtain the trained RLMIL model.

8. A prostate cancer pathology image classification device, characterized in that: The device comprises: An acquisition module, used for acquiring a prostate cancer pathology image, wherein the prostate cancer pathology image is a full-field digital slice image; An extraction module, used for extracting features from the prostate cancer pathology image through a feature extractor to obtain multiple instance features; A reconstruction module, used to reconstruct the features of the plurality of instance features through a pre-trained multi-instance learning RLMIL model based on reordering Mamba and linear deformable convolution, so as to obtain global features and local features, wherein the RLMIL model includes a reordering Mamba module and a linear deformable convolution module, the reordering Mamba module is used to extract the global features between the plurality of instance features, and the linear deformable convolution module is used to capture the local features between the plurality of instance features; An aggregation module, used for performing feature aggregation on the global features and the local features to obtain a package-level feature; The classification module is used to calculate the predicted probability distribution of each category of the package-level features to obtain the classification result of the prostate cancer pathology image.

9. An electronic device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the prostate cancer pathology image classification method as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute the prostate cancer pathology image classification method according to any one of claims 1 to 7.

Images