Gleason grading method and device for prostatic cancer pathological image, equipment and medium

Abstract

The embodiment of the invention provides a Grignson grading method and device for a prostate cancer pathological image, equipment and a medium. The method comprises the following steps: acquiring a prostate cancer pathological image which is a full-view digital slice image; performing feature extraction on the prostate cancer image to obtain pathological image features; the pathological image features are input into a multi-instance learning model based on time-frequency interaction to obtain fusion features, the multi-instance learning model based on time-frequency interaction comprises a feature reconstruction module and a time-frequency feature interaction module, the feature reconstruction module is used for extracting high-frequency feature information and low-frequency feature information from the pathological image features, and the time-frequency feature interaction module is used for interacting with the high-frequency feature information and the low-frequency feature information; the time-frequency feature interaction module is used for extracting time-domain feature information and frequency-domain feature information from pathological image features; and inputting the fusion features into a classifier to obtain a Grignson classification result of the prostatic cancer pathological image. Based on this, the embodiment of the invention can improve the accuracy of Gleason grading of the prostatic cancer pathological image.

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of medical image processing, and in particular to a Gleason grading method and apparatus, device and medium for prostate cancer pathological images. Background Art

[0002] Prostate cancer is one of the most common cancers in men, and its early diagnosis and accurate grading are crucial to developing effective treatment plans. The Gleason Grading System is a standard scoring tool for prostate cancer. By evaluating the degree of glandular differentiation of tumor tissue, it can help doctors determine the degree of malignancy and prognosis. As the basis and gold standard for tumor diagnosis, the accuracy of pathological imaging grading is crucial to developing treatment plans and evaluating prognosis.

[0003] With the development of digital pathology, it has become possible to analyze the digitization of high-resolution tissue sections, which has promoted the advancement of computer-aided diagnosis (CAD) technology. However, whole-view digital slice images (WSIs) have high pixel density (usually up to 40,000×40,000 pixels), and it is difficult to obtain labeled data. At the same time, there is significant heterogeneity between tumors of different grades, which makes pathological image analysis based on deep learning complicated. The ability to capture key details of pathological images and process complex data is not high enough, thus affecting the accuracy of Gleason grading. Therefore, how to improve the accuracy of Gleason grading of prostate cancer pathological images has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] The embodiments of the present invention provide a Gleason grading method and apparatus, a device and a medium for prostate cancer pathology images, which can improve the accuracy of Gleason grading of prostate cancer pathology images.

[0005] In a first aspect, an embodiment of the present invention provides a Gleason grading method for prostate cancer pathology images, comprising:

[0006] Acquiring a prostate cancer pathology image, wherein the prostate cancer pathology image is a full-field digital slice image;

[0007] Extracting features from the prostate cancer image to obtain pathological image features;

[0008] Inputting the pathological image features into a pre-trained multi-instance learning model based on time-frequency interaction to obtain fusion features, wherein the multi-instance learning model based on time-frequency interaction includes a feature reconstruction module based on wavelet transform and a time-frequency feature interaction module based on Mamba, the feature reconstruction module based on wavelet transform is used to extract high-frequency feature information and low-frequency feature information from the pathological image features, and the time-frequency feature interaction module based on Mamba is used to extract time-domain feature information and frequency-domain feature information from the pathological image features;

[0009] The fusion feature is input into a classifier to obtain the Gleason grading result of the prostate cancer pathology image.

[0010] In some embodiments, extracting features from the prostate cancer image to obtain pathological image features includes:

[0011] Using a threshold segmentation algorithm to perform non-overlapping slicing on the prostate cancer image at a set magnification to obtain a plurality of non-overlapping non-overlapping slices of a fixed size;

[0012] ResNet50 pre-trained on ImageNet is used as a feature extractor to extract features from the non-overlapping slices to obtain the pathological image features.

[0013] In some embodiments, the step of inputting the pathological image features into a pre-trained multi-instance learning model based on time-frequency interaction to obtain fusion features includes:

[0014] Inputting the pathological image features into the feature reconstruction module based on wavelet transform to obtain the high-frequency feature information and the low-frequency feature information;

[0015] Performing standard normalization processing on the pathological image features to obtain normalized features;

[0016] Inputting the high-frequency feature information, the low-frequency feature information and the normalized feature into the Mamba-based time-frequency feature interaction module to obtain the time-domain feature information and the frequency-domain feature information;

[0017] The fusion feature is generated according to the time domain feature information and the frequency domain feature information.

[0018] In some embodiments, the step of inputting the pathological image features into the wavelet transform-based feature reconstruction module to obtain the high-frequency feature information and the low-frequency feature information includes:

[0019] Reshape the one-dimensional feature tensor of the pathological image feature into a two-dimensional spatial feature;

[0020] Down-sampling the two-dimensional spatial features by using discrete wavelet transform through the wavelet transform-based feature reconstruction module to obtain a first high-frequency component and a first low-frequency component in the row direction, and a second high-frequency component and a second low-frequency component in the column direction;

[0021] Connecting the first high-frequency component, the first low-frequency component, the second high-frequency component and the second low-frequency component along a channel dimension to obtain a feature matrix;

[0022] Dynamically reconstructing the feature matrix using a feature modulation matrix to obtain the high-frequency feature information and the low-frequency feature information;

[0023] The feature tensors of the high-frequency feature information and the low-frequency feature information are flattened into a one-dimensional feature vector by using an inverse discrete wavelet transform.

[0024] In some embodiments, the inputting the high-frequency feature information, the low-frequency feature information and the normalized feature into the Mamba-based time-frequency feature interaction module to obtain the time domain feature information and the frequency domain feature information includes:

[0025] Performing feature analysis and feature extraction on the high-frequency feature information, the low-frequency feature information and the normalized feature through the Mamba-based time-frequency feature interaction module to generate a feature sequence;

[0026] The feature sequence is converted into the time domain feature information and the frequency domain feature information.

[0027] In some embodiments, after converting the feature sequence into the time domain feature information and the frequency domain feature information, the method further includes:

[0028] Performing information interaction on the time domain feature information and the frequency domain feature information through the dual-branch channel of the Mamba-based time-frequency feature interaction module;

[0029] The time domain feature information and the frequency domain feature information are fused to obtain the fused feature.

[0030] In some embodiments, the training method of the multi-instance learning model based on time-frequency interaction includes:

[0031] Determine a cross entropy loss function, wherein the cross entropy loss function is determined based on the true label, the linear projection matrix, and the probability distribution predicted by the model;

[0032] The multi-instance learning model based on time-frequency interaction is trained based on the cross entropy loss function to obtain the trained multi-instance learning model based on time-frequency interaction.

[0033] In a second aspect, an embodiment of the present invention further provides a Gleason grading device for prostate cancer pathological images, the device comprising:

[0034] An acquisition module, used for acquiring a prostate cancer pathology image, wherein the prostate cancer pathology image is a full-field digital slice image;

[0035] An extraction module, used for extracting features from the prostate cancer image to obtain pathological image features;

[0036] A learning module, used for inputting the pathological image features into a pre-trained multi-instance learning model based on time-frequency interaction to obtain fusion features, wherein the multi-instance learning model based on time-frequency interaction includes a feature reconstruction module based on wavelet transform and a time-frequency feature interaction module based on Mamba, the feature reconstruction module based on wavelet transform is used for extracting high-frequency feature information and low-frequency feature information from the pathological image features, and the time-frequency feature interaction module based on Mamba is used for extracting time-domain feature information and frequency-domain feature information from the pathological image features;

[0037] The grading module is used to input the fusion features into a classifier to obtain the Gleason grading result of the prostate cancer pathology image.

[0038] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the Gleason grading method for prostate cancer pathology images as described in the first aspect is implemented.

[0039] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium storing computer-executable instructions for executing the Gleason grading method for prostate cancer pathology images as described in the first aspect.

[0040] According to the Gleason grading method and apparatus, device and medium of the pathological image of prostate cancer provided by the embodiment of the present invention, the Gleason grading method of the pathological image of prostate cancer includes: obtaining a pathological image of prostate cancer, which is a full-field digital slice image; extracting features of the prostate cancer image to obtain pathological image features; inputting the pathological image features into a pre-trained multi-instance learning model based on time-frequency interaction to obtain fusion features, wherein the multi-instance learning model based on time-frequency interaction includes a feature reconstruction module based on wavelet transform and a time-frequency feature interaction module based on Mamba, the feature reconstruction module based on wavelet transform is used to extract high-frequency feature information and low-frequency feature information from pathological image features, and the time-frequency feature interaction module based on Mamba is used to extract time-domain feature information and frequency-domain feature information from pathological image features; the fusion features are input into a classifier to obtain the Gleason grading result of the pathological image of prostate cancer. The present invention can effectively integrate high-frequency and low-frequency features and their spatial information by introducing a feature reconstruction module based on wavelet transform and a time-frequency feature interaction module based on Mamba, thereby improving the understanding of tumor heterogeneity. Compared with the prior art, the present invention has a stronger ability in capturing key details and processing complex data. Based on this, the embodiment of the present invention can improve the accuracy of Gleason grading of prostate cancer pathological images. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 is a flow chart of a Gleason grading method for prostate cancer pathology images provided by one embodiment of the present invention;

[0042] Figure 2 is a sub-flow chart of step S102 provided by an embodiment of the present invention;

[0043] Figure 3 is a diagram of a WSIs preprocessing process provided by an embodiment of the present invention;

[0044] Figure 4 is a sub-flow chart of step S103 provided by an embodiment of the present invention;

[0045] Figure 5 is an overall framework diagram of TFI-MIL provided by an embodiment of the present invention;

[0046] Figure 6 is a sub-flow chart of step S401 provided by an embodiment of the present invention;

[0047] Figure 7 It is a schematic diagram of the structure of a feature reconstruction module based on wavelet transform provided by an embodiment of the present invention;

[0048] Figure 8is a sub-flow chart of step S403 provided by an embodiment of the present invention;

[0049] Fig. 9 is a flow chart of a training method for a multi-instance learning model based on time-frequency interaction provided by an embodiment of the present invention;

[0050] Fig.10 is a schematic structural diagram of a Gleason grading device for prostate cancer pathological images provided by an embodiment of the present invention;

[0051] Fig.11 is a schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0053] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification and claims and the following drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0054] In the embodiments of the present invention, the words "further", "exemplarily" or "optionally" are used to indicate examples, illustrations or descriptions, and should not be interpreted as being more preferred or more advantageous than other embodiments or designs. The use of the words "further", "exemplarily" or "optionally" is intended to present related concepts in a specific way.

[0055] In order to more conveniently describe the working principle of the embodiment of the present invention later, an introduction to relevant technical scenarios is first given below.

[0056] Prostate cancer is one of the most common cancers in men, and its early diagnosis and accurate grading are crucial to developing effective treatment plans. The Gleason Grading System is a standard scoring tool for prostate cancer. By evaluating the degree of glandular differentiation of tumor tissue, it can help doctors determine the degree of malignancy and prognosis. As the basis and gold standard for tumor diagnosis, the accuracy of pathological imaging grading is crucial to developing treatment plans and evaluating prognosis.

[0057] With the development of digital pathology, it has become possible to analyze the digitization of high-resolution tissue sections, which has promoted the advancement of computer-aided diagnosis (CAD) technology. However, whole-view digital slice images (WSIs) have high pixel density (usually up to 40,000×40,000 pixels), and it is difficult to obtain labeled data. At the same time, there is significant heterogeneity between tumors of different grades, which makes pathological image analysis based on deep learning complicated. The ability to capture key details of pathological images and process complex data is not high enough, thus affecting the accuracy of Gleason grading. Therefore, how to improve the accuracy of Gleason grading of prostate cancer pathological images has become a technical problem that needs to be solved urgently.

[0058] Based on this, the present invention provides a Gleason grading method and device, equipment and medium for prostate cancer pathology images. Among them, the Gleason grading method for prostate cancer pathology images includes: obtaining a prostate cancer pathology image, which is a full-field digital slice image; extracting features from the prostate cancer image to obtain pathology image features; inputting the pathology image features into a pre-trained multi-instance learning model based on time-frequency interaction to obtain fusion features, wherein the multi-instance learning model based on time-frequency interaction includes a feature reconstruction module based on wavelet transform and a time-frequency feature interaction module based on Mamba, the feature reconstruction module based on wavelet transform is used to extract high-frequency feature information and low-frequency feature information from pathology image features, and the time-frequency feature interaction module based on Mamba is used to extract time domain feature information and frequency domain feature information from pathology image features; inputting the fusion features into a classifier to obtain the Gleason grading result of the prostate cancer pathology image. By introducing a feature reconstruction module based on wavelet transform and a time-frequency feature interaction module based on Mamba, the present invention can effectively integrate high-frequency and low-frequency features and their spatial information, and establish an effective interaction between time domain features and frequency domain features, thereby improving the understanding of tumor heterogeneity. Compared with the prior art, the present invention has a stronger ability in capturing key details and processing complex data. Based on this, the embodiment of the present invention can improve the accuracy of Gleason grading of prostate cancer pathological images.

[0059] The embodiments of the present invention are further described below in conjunction with the accompanying drawings.

[0060] like Figure 1 As shown, Figure 1 It is a flowchart of a Gleason grading method for prostate cancer pathology images provided by an embodiment of the present invention. The Gleason grading method for prostate cancer pathology images may include but is not limited to steps S101 to S104.

[0061] Step S101, obtaining a prostate cancer pathology image, which is a full-field digital slice image;

[0062] Step S102, extracting features from the prostate cancer image to obtain pathological image features;

[0063] Step S103, inputting the pathological image features into a pre-trained multi-instance learning model based on time-frequency interaction to obtain fusion features, wherein the multi-instance learning model based on time-frequency interaction includes a feature reconstruction module based on wavelet transform and a time-frequency feature interaction module based on Mamba, the feature reconstruction module based on wavelet transform is used to extract high-frequency feature information and low-frequency feature information from pathological image features, and the time-frequency feature interaction module based on Mamba is used to extract time-domain feature information and frequency-domain feature information from pathological image features;

[0064] Step S104, inputting the fusion features into a classifier to obtain the Gleason grading result of the prostate cancer pathology image.

[0065] It can be understood that the present invention proposes a multi-instance learning method based on time-frequency interaction for Gleason grading of prostate cancer pathological sections. Specifically, the multi-instance learning model based on time-frequency interaction of the present invention includes:

[0066] (1) Wavelet Transform-based Reconstruction Module (WTRM), which is used to learn high-frequency and low-frequency feature information in pathological images. This module uses frequency domain processing to enable the model to more accurately capture the edge, texture, and shape information of pathological slice features that are difficult to detect in the time domain, thereby improving the ability to recognize key details and establish spatial relationships between instances within a package.

[0067] (2) Time-Frequency Feature Interaction Module Based on Mamba (TFIM-BM), which aims to enhance the dynamic interaction and information fusion between different features. This module effectively promotes the interaction between features, ensuring that the spatial structure and detail information can be better preserved when processing complex pathological images, thereby improving the accuracy of classification and grading.

[0068] By combining these two modules, the present invention significantly enhances the analysis capability of pathological images and provides more accurate support for the diagnosis of prostate cancer.

[0069] It is understandable that the present invention constructs a time-frequency interaction based multiple instance learning (TFI-MIL) method for Gleason grading of prostate cancer pathological sections. First, the features are spatially reconstructed through a feature reconstruction module based on wavelet transform to learn high- and low-frequency feature information in pathological images and establish spatial information between instances. Secondly, a Mamba-based time-frequency feature interaction module is designed to enhance the dynamic interaction and information fusion between different features. In the absence of fine-grained labels, the model is trained and predicted using slice-level labels, so that it can still maintain a high accuracy when processing complex WSIs. The present invention shows good potential in achieving accurate grading of prostate cancer and provides important support for clinical pathological analysis. This not only helps to improve the accuracy of tumor diagnosis, but also provides an important basis for the formulation of personalized treatment plans.

[0070] In one embodiment, if Figure 2 As shown, step S102 includes but is not limited to the following steps:

[0071] Step S201, using a threshold segmentation algorithm to perform non-overlapping slicing on the prostate cancer image at a set magnification to obtain a plurality of non-overlapping non-overlapping slices of a fixed size;

[0072] Step S202, using ResNet50 pre-trained by ImageNet as a feature extractor to extract features from non-overlapping slices to obtain pathological image features.

[0073] It can be understood that the full-view digital slice images WSIs are preprocessed and the ultra-high pixel-level WSIs are processed at 20× magnification using a threshold segmentation algorithm (OTSU) for non-overlapping slices (discarding the background and retaining the tissue area that occupies more than 40% of the image area). Then, n non-overlapping image slices of a fixed size of 512×512 pixels can be obtained. Then, the ImageNet pre-trained ResNet50 is used as a feature extractor to map these patches into n 1024-dimensional low-dimensional feature vectors. The entire preprocessing process is as follows Figure 3 shown.

[0074] In one embodiment, if Figure 4 As shown, step S103 includes but is not limited to the following steps:

[0075] Step S401, inputting the pathological image features into a feature reconstruction module based on wavelet transform to obtain high-frequency feature information and low-frequency feature information;

[0076] Step S402, performing standard normalization processing on the pathological image features to obtain normalized features;

[0077] Step S403, inputting the high-frequency feature information, the low-frequency feature information and the normalized feature into a Mamba-based time-frequency feature interaction module to obtain time-domain feature information and frequency-domain feature information;

[0078] Step S404: generating fusion features according to the time domain feature information and the frequency domain feature information.

[0079] It can be understood that the overall framework of TFI-MIL constructed by the present invention is as follows Figure 5 As shown in the figure, the specific implementation process of TFI-MIL is as follows: First, the input features are passed to the dual-branch shared deep linear layer module, which is designed to preliminarily extract time domain features. Through the weighted combination of multiple neurons, this layer deeply explores the potential relationship between features, thereby enhancing the representativeness of the features. Next, in branch one, the features are input into the WTRM module for spatial reconstruction. In this module, the original features are first converted into a two-dimensional space for subsequent processing. Then, the wavelet transform is applied to decompose and reconstruct the features, effectively separating the high-frequency and low-frequency information in the features, so as to better capture the detailed key information in the data. At the same time, the feature modulation matrix dynamically adjusts the reconstruction parameters to further enhance the overall expression ability of the features. In branch two, the features are subjected to standard normalization. Then, the features of the two branches are jointly input into the Mamba-based time-frequency feature interaction module TFIM-BM to achieve interactive learning and effective fusion between time-frequency features to generate the final representation of WSIs. Finally, the final representation of WSIs is processed by a linear classification layer to output the Gleason grade prediction results of prostate cancer WSIs.

[0080] In one embodiment, if Figure 6 As shown, step S401 includes but is not limited to the following steps:

[0081] Step S601, reshaping the one-dimensional feature tensor of the pathological image feature into a two-dimensional spatial feature;

[0082] Step S602, down-sampling the two-dimensional spatial features by using a discrete wavelet transform through a feature reconstruction module based on wavelet transform to obtain a first high-frequency component and a first low-frequency component in the row direction, and a second high-frequency component and a second low-frequency component in the column direction;

[0083] Step S603, connecting the first high-frequency component, the first low-frequency component, the second high-frequency component, and the second low-frequency component along the channel dimension to obtain a feature matrix;

[0084] Step S604, dynamically reconstructing the feature matrix using a feature modulation matrix to obtain high-frequency feature information and low-frequency feature information;

[0085] Step S605: Re-flatten the feature tensors of the high-frequency feature information and the low-frequency feature information into a one-dimensional feature vector by using inverse discrete wavelet transform.

[0086] It can be understood that the feature reconstruction module based on wavelet transform of the present invention is as follows Figure 7 As shown. By integrating wavelet transform into the Transformer module, it is possible to downsample high-resolution image features while retaining important high-frequency, low-frequency details and edge information. In addition, a learnable feature modulation matrix is ​​used instead of the traditional self-attention layer to more effectively mix markers at different spatial locations. The specific implementation process of WTRM for feature reconstruction is as follows:

[0087] First, for each input image, the one-dimensional feature Tokens tensor X∈R C×N×D , reshape it into a two-dimensional space feature Among them, C represents the number of channels, N represents the number of tokens, and D represents the feature dimension.

[0088] Then, WTRM uses discrete wavelet transform (DWT) to downsample the input feature tensor. To simplify the calculation, we use the classic Haar wavelet as the wavelet basis in discrete wavelet transform. After DWT processing, four subbands can be obtained:

[0089]

[0090] Specifically, the DWT uses a low-pass filter High Pass Filter The input features are decomposed in the row direction to obtain two sub-waves: low-frequency component (X L ) and high frequency components (X H ), similarly, using the same low-pass filter and high-pass filter in the column direction to decompose the input features, we can get two other sub-bands: low-frequency component (X L ) and high frequency components (X H ). Finally, four wavelet subbands can be obtained: Among them, X LL Represents the low-frequency component, reflecting the basic information of the image at a coarse-grained level. LH , X HL and X HHRepresents high-frequency components and retains image texture detail information at a fine-grained level. Each subband can be regarded as a downsampling result of the input feature, covering all the details of the input feature information without losing any detail information.

[0091] Next, the four wavelet subbands are connected along the channel dimension to obtain a new feature matrix:

[0092]

[0093] Then, the convolution operation is performed between the characteristic modulation matrix and the coefficients of the last level of decomposition. The characteristic modulation matrix is ​​defined as D M ∈R 4C×h×w , which is equivalent to a deep global recurrent convolution of size h / 2×w / 2. By learning the parameters in the feature modulation matrix, the model can automatically adjust the frequency weights to achieve noise adjustment and feature enhancement of the image. Its formulation is inspired by the frequency filter in digital image processing and can be regarded as a set of learnable frequency weights applicable to different hidden dimensions. The specific operation is as follows:

[0094]

[0095] Finally, the feature tensor is updated and restored using the inverse discrete wavelet transform (IDWT) and flattened back into a one-dimensional feature vector:

[0096]

[0097] This method makes full use of the advantages of wavelet transform, allowing the model to fully utilize the information of each frequency component and better capture the details and texture information in the input data. Through the decomposition and reconstruction of features, the model's ability to process high-resolution features is enhanced, and the spatial structure of the image is captured more effectively.

[0098] In one embodiment, if Figure 8 As shown, step S403 includes but is not limited to the following steps:

[0099] Step S801, performing feature analysis and feature extraction on high-frequency feature information, low-frequency feature information and normalized features through a Mamba-based time-frequency feature interaction module to generate a feature sequence;

[0100] Step S802: converting the feature sequence into time domain feature information and frequency domain feature information.

[0101] Step S803, performing information interaction on the time domain feature information and the frequency domain feature information through a dual-branch channel of a Mamba-based time-frequency feature interaction module;

[0102] Step S804: fuse the time domain feature information and the frequency domain feature information to obtain fused features.

[0103] It can be understood that in order to more effectively realize the contextual connection of features and establish the interaction between time-frequency features, the present invention introduces a time-frequency feature interaction module based on Mamba, such as Figure 5 This module aims to enhance the model's ability to understand feature information by processing complex time-frequency features.

[0104] Specifically, this module receives two different types of feature inputs: X t , X f By designing a specific convolutional layer and activation layer structure, the module is able to deeply analyze and extract these input features. After being processed by these layers, the generated feature sequence will provide richer and more detailed temporal features for the state space model (SSM) block.

[0105] SSM is a mathematical framework that uses a set of linear equations to describe the evolution of the state of a system over time. SSM modeling defines how the state changes over time and how inputs are mapped to outputs through the following equations:

[0106] h t =Ah t-1 +Bx t

[0107] y t =C T h t

[0108] SSM describes the evolution of state over time through the state transfer matrix A, the input matrix B maps the input x to the state space, and the output matrix C maps the state space to the output space.

[0109] The attention mechanism enables the model to focus on the important parts of sequence data when processing it. In the Transformer model, attention is achieved by calculating the relationship between the query (Q), key (K), and value (V). The attention mechanism is usually defined by the following formula:

[0110]

[0111] Among them, d k Indicates the dimension of the key.

[0112] The state-space dual framework combines the state-space model with the attention mechanism to enhance the model's ability to process sequence data. SSD introduces a structured mask matrix L to control state transfer, replacing the traditional softmax operation, thereby achieving functions similar to the attention mechanism and making the attention calculation have linear time complexity. Its formula is expressed as:

[0113] SSD=(LoQK T )·V

[0114] At the beginning of each module, A, K, Q, and V are mapped in parallel. A data-based position information mask matrix L is used to control the propagation of information in time, thereby replacing the traditional heuristic position embedding method.

[0115] To further enhance the learning ability and robustness of the model, PhysMamba introduces a dual-channel model. To avoid the redundancy that may occur between independent channels, the present invention designs TFIM-BM, such as Figure 5 As shown in Figure 2. This module promotes information sharing and complementary advantages by sharing feature information between the two branches. One branch provides time domain feature information, and the other branch provides frequency domain feature information. This enables the two branches to share feature information and achieve mutual constraint and promotion between the two branches. The attention output of the two channels is:

[0116]

[0117] After passing through the TFIM-BM module, the dual-branch feature X cf , X ct Connect along the channel dimension to obtain the fused feature X fusion .

[0118] X fusion =Concat(X cf , X ct )

[0119] Next, these fused features will be input into the classifier to achieve Gleason grading for prostate cancer WSI.

[0120] In one embodiment, if Fig. 9 As shown, the training method of the multi-instance learning model based on time-frequency interaction includes but is not limited to the following steps:

[0121] Step S901, determining a cross entropy loss function, where the cross entropy loss function is determined based on the true label, the linear projection matrix, and the probability distribution predicted by the model;

[0122] Step S902 , training the multi-instance learning model based on time-frequency interaction based on a cross entropy loss function to obtain a trained multi-instance learning model based on time-frequency interaction.

[0123] It is understandable that the present invention uses a cross entropy loss function to optimize the model. The loss function can be expressed as:

[0124]

[0125] Among them, y i is the true label, M is a linear projection matrix, P is the probability distribution predicted by the model, and is the total number of categories. The multi-instance learning model based on time-frequency interaction is trained based on the cross entropy loss function to obtain a trained multi-instance learning model based on time-frequency interaction.

[0126] Based on this, compared with the prior art, the Gleason grading method for prostate cancer pathology images of the present invention has at least the following advantages:

[0127] Beneficial effects:

[0128] (1) The present invention adopts a weakly supervised multi-instance learning method to effectively implement the Gleason grading task of prostate cancer under the condition of limited labeled data.

[0129] (2) The present invention constructs a feature reconstruction module based on wavelet transform, aiming to efficiently extract and learn high-frequency and low-frequency feature information in pathological images, establish spatial relationships between instances, and thus enhance the ability to recognize key details.

[0130] (3) The present invention designs a time-frequency feature interaction module based on Mamba, aiming to strengthen the dynamic interaction and information fusion between different features and improve the overall performance and accuracy of the model.

[0131] It should be noted that the prior art often ignores the contextual information of the image and the dynamic association between instances. In contrast, the embodiment of the present invention can effectively integrate high- and low-frequency features and their spatial information by introducing a feature reconstruction module based on wavelet transform and a time-frequency feature interaction module based on Mamba, and establish an effective interaction between time domain features and frequency domain features, thereby improving the understanding of tumor heterogeneity. Compared with the prior art, the present invention shows a stronger ability in capturing key details and processing complex data, and has higher clinical application value and promotion potential.

[0132] In addition, if Fig.10 As shown, one embodiment of the present invention also discloses a Gleason grading device for prostate cancer pathology images, the device comprising:

[0133] An acquisition module 110 is used to acquire a prostate cancer pathology image, which is a full-field digital slice image;

[0134] An extraction module 120 is used to extract features from the prostate cancer image to obtain pathological image features;

[0135] A learning module 130 is used to input the pathological image features into a pre-trained multi-instance learning model based on time-frequency interaction to obtain fusion features, wherein the multi-instance learning model based on time-frequency interaction includes a feature reconstruction module based on wavelet transform and a time-frequency feature interaction module based on Mamba, the feature reconstruction module based on wavelet transform is used to extract high-frequency feature information and low-frequency feature information from the pathological image features, and the time-frequency feature interaction module based on Mamba is used to extract time-domain feature information and frequency-domain feature information from the pathological image features;

[0136] The grading module 140 is used to input the fusion features into the classifier to obtain the Gleason grading result of the prostate cancer pathology image.

[0137] The Gleason grading device for prostate cancer pathology images in the embodiment of the present invention is used to execute the Gleason grading method for prostate cancer pathology images in the above embodiment. Its specific processing process is the same as the Gleason grading method for prostate cancer pathology images in the above embodiment, and will not be repeated here.

[0138] In addition, if Fig.11 As shown, an embodiment of the present invention also discloses an electronic device, including: at least one processor 210; at least one memory 220, for storing at least one program; when the at least one program is executed by the at least one processor 210, the Gleason grading method of prostate cancer pathology images as in any of the previous embodiments is implemented.

[0139] In addition, an embodiment of the present invention further discloses a computer-readable storage medium, in which computer-executable instructions are stored. The computer-executable instructions are used to execute the Gleason grading method for prostate cancer pathology images as in any of the previous embodiments.

[0140] The system architecture and application scenarios described in the embodiments of the present invention are intended to more clearly illustrate the technical solutions of the embodiments of the present invention, and do not constitute a limitation on the technical solutions provided by the embodiments of the present invention. Those skilled in the art will appreciate that with the evolution of the system architecture and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present invention are equally applicable to similar technical problems.

[0141] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0142] In hardware implementation, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some physical components or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those of ordinary skill in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0143] The terms "component", "module", "system", etc. used in this specification are used to represent computer-related entities, hardware, firmware, a combination of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, or a computer. By way of illustration, both applications running on a computing device and a computing device can be components. One or more components may reside in a process or an execution thread, and a component may be located on a computer or distributed between two or more computers. In addition, these components may be executed from various computer-readable media having various data structures stored thereon. Components may communicate, for example, through local or remote processes based on signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system, or a network, such as the Internet interacting with other systems through signals).

Claims

1. A Gleason grading method for prostate cancer pathology images, comprising: Acquiring a prostate cancer pathology image, wherein the prostate cancer pathology image is a full-field digital slice image; Extracting features from the prostate cancer image to obtain pathological image features; Inputting the pathological image features into a pre-trained multi-instance learning model based on time-frequency interaction to obtain fusion features, wherein the multi-instance learning model based on time-frequency interaction includes a feature reconstruction module based on wavelet transform and a time-frequency feature interaction module based on Mamba, the feature reconstruction module based on wavelet transform is used to extract high-frequency feature information and low-frequency feature information from the pathological image features, and the time-frequency feature interaction module based on Mamba is used to extract time-domain feature information and frequency-domain feature information from the pathological image features; The fusion feature is input into a classifier to obtain the Gleason grading result of the prostate cancer pathology image.

2. The method according to claim 1, characterized in that The step of extracting features from the prostate cancer image to obtain pathological image features includes: Using a threshold segmentation algorithm to perform non-overlapping slicing on the prostate cancer image at a set magnification to obtain a plurality of non-overlapping non-overlapping slices of a fixed size; ResNet50 pre-trained on ImageNet is used as a feature extractor to extract features from the non-overlapping slices to obtain the pathological image features.

3. The method according to claim 1, characterized in that The pathological image features are input into a pre-trained multi-instance learning model based on time-frequency interaction to obtain fusion features, including: Inputting the pathological image features into the feature reconstruction module based on wavelet transform to obtain the high-frequency feature information and the low-frequency feature information; Performing standard normalization processing on the pathological image features to obtain normalized features; Inputting the high-frequency feature information, the low-frequency feature information and the normalized feature into the Mamba-based time-frequency feature interaction module to obtain the time-domain feature information and the frequency-domain feature information; The fusion feature is generated according to the time domain feature information and the frequency domain feature information.

4. The method according to claim 3, characterized in that The step of inputting the pathological image features into the feature reconstruction module based on wavelet transform to obtain the high-frequency feature information and the low-frequency feature information includes: Reshape the one-dimensional feature tensor of the pathological image feature into a two-dimensional spatial feature; Down-sampling the two-dimensional spatial features by using discrete wavelet transform through the wavelet transform-based feature reconstruction module to obtain a first high-frequency component and a first low-frequency component in the row direction, and a second high-frequency component and a second low-frequency component in the column direction; Connecting the first high-frequency component, the first low-frequency component, the second high-frequency component and the second low-frequency component along a channel dimension to obtain a feature matrix; Dynamically reconstructing the feature matrix using a feature modulation matrix to obtain the high-frequency feature information and the low-frequency feature information; The feature tensors of the high-frequency feature information and the low-frequency feature information are flattened into a one-dimensional feature vector by using an inverse discrete wavelet transform.

5. The method according to claim 3, characterized in that: The step of inputting the high-frequency feature information, the low-frequency feature information and the normalized feature into the Mamba-based time-frequency feature interaction module to obtain the time-domain feature information and the frequency-domain feature information includes: Performing feature analysis and feature extraction on the high-frequency feature information, the low-frequency feature information and the normalized feature through the Mamba-based time-frequency feature interaction module to generate a feature sequence; The feature sequence is converted into the time domain feature information and the frequency domain feature information.

6. The method according to claim 5, characterized in that After converting the feature sequence into the time domain feature information and the frequency domain feature information, the method further includes: Performing information interaction on the time domain feature information and the frequency domain feature information through the dual-branch channel of the Mamba-based time-frequency feature interaction module; The time domain feature information and the frequency domain feature information are fused to obtain the fused feature.

7. The method according to claim 1, characterized in that The training method of the multi-instance learning model based on time-frequency interaction includes: Determine a cross entropy loss function, wherein the cross entropy loss function is determined based on the true label, the linear projection matrix, and the probability distribution predicted by the model; The multi-instance learning model based on time-frequency interaction is trained based on the cross entropy loss function to obtain the trained multi-instance learning model based on time-frequency interaction.

8. A Gleason grading device for prostate cancer pathology images, characterized in that: The device comprises: An acquisition module, used for acquiring a prostate cancer pathology image, wherein the prostate cancer pathology image is a full-field digital slice image; An extraction module, used for extracting features from the prostate cancer image to obtain pathological image features; A learning module, used for inputting the pathological image features into a pre-trained multi-instance learning model based on time-frequency interaction to obtain fusion features, wherein the multi-instance learning model based on time-frequency interaction includes a feature reconstruction module based on wavelet transform and a time-frequency feature interaction module based on Mamba, the feature reconstruction module based on wavelet transform is used for extracting high-frequency feature information and low-frequency feature information from the pathological image features, and the time-frequency feature interaction module based on Mamba is used for extracting time-domain feature information and frequency-domain feature information from the pathological image features; The grading module is used to input the fusion features into a classifier to obtain the Gleason grading result of the prostate cancer pathology image.

9. An electronic device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the Gleason grading method for prostate cancer pathology images as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing computer-executable instructions for executing the Gleason grading method for prostate cancer pathology images according to any one of claims 1 to 7.

Images