A multi-task method for full-volume histopathology images based on fuzzy kernel

Abstract

The present invention provides a multi-task method for full-volume histopathology images based on fuzzy kernels, which belongs to the field of medical image intelligent diagnosis technology and solves the technical problem that the current full-volume image model cannot simultaneously perform high-accuracy classification and prediction tasks. Its technical solution is: first, the full-volume image is preprocessed into a microenvironment image I TEM and low-resolution image I LR Then, the features are blurred and defuzzified using membership functions, and the results are stored in the kernel. Next, the features are fused using cross-attention. Furthermore, a fusion diffusion model is designed that can be guided by the dynamic fuzzy kernel during the diffusion process. Finally, the diffused features are fed into the classification and survival prediction neural network to obtain the final results. The beneficial effects of the present invention are: multi-tasking processing of multimodal medical full-scale images can accurately classify diseases and predict survival, helping doctors to more comprehensively and accurately understand tumor aggressiveness and metastasis tendencies.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image intelligent diagnosis, and in particular to a multi-task method for full-volume histopathology imaging based on a fuzzy kernel. Background Art

[0002] Multimodal medical full-volume imaging is a widely used technology for high-resolution digitization of tissue sections, playing a particularly important role in the diagnosis, classification, and prognostic assessment of tumors. High-resolution tissue section images enable pathologists to more accurately observe tumor morphology, tissue structure, and microenvironment, providing a more comprehensive basis for survival prediction. However, in actual clinical diagnosis, due to the high resolution and complexity of full-volume imaging, manual analysis is time-consuming and expensive, and interpretation relies heavily on individual experience and expertise, which can affect the consistency and accuracy of diagnosis.

[0003] In this context, the integration of artificial intelligence and multimodal medical imaging technology is particularly important. It can greatly assist pathologists in their diagnosis, reveal the molecular and genetic characteristics of diseases, and provide more valuable information for personalized treatment. Therefore, the importance of multimodal medical imaging in clinical practice is becoming increasingly prominent. Its application not only improves the accuracy of diagnosis and treatment, but also promotes the development and innovation of medical imaging technology, bringing greater health protection and hope to patients.

[0004] In a research paper titled "Joint region and nucleus segmentation for characterization of tumor infiltrating lymphocytes in breast cancer," a deep learning model was used to classify the entire image set. However, their method has significant limitations, as it can only analyze a single modality and has low accuracy, and still requires manual intervention from doctors.

[0005] In a research paper titled "Spatial Organization and Molecular Correlation of Tumor-Infiltrating Lymphocytes Using Deep Learning on Pathology Images," two convolutional neural networks were used to extract features from different regions of the full image by evaluating image resolution. However, their method required significant computational resources and did not properly address the dimensionality differences between modalities when fusion was performed.

[0006] Traditional multi-task methods typically use generative models to generate images by gradually training the model. These methods require manual selection and design of feature extractors, have difficulty processing high-dimensional complex data, and are susceptible to noise and interference.

[0007] How to solve the above technical problems is the subject faced by the present invention. Summary of the Invention

[0008] The purpose of the present invention is to provide a multi-task method for full-volume histopathology imaging based on fuzzy kernels, which solves the technical problems of cascade classification and survival prediction of full-volume images, aims to improve the accuracy and generalization ability of multimodal full-volume images in classification and survival prediction tasks, and has broad application prospects in pathology, genetic research and clinical medicine.

[0009] In order to achieve the above-mentioned object of the invention, the present invention adopts a technical solution specifically as follows: a multi-task method for full-volume histopathology images based on a fuzzy kernel, comprising the following steps:

[0010] S1: For each patient, full-length images and genomic data are collected and matched one to one. The full-length images are then divided into 224×224 image blocks, and the Resnet classifier is used to classify these blocks into microenvironment images I TEM and low-resolution image I LR At the same time, the genomic data G is expanded and spliced ​​to unify the dimension to form a sample set, which includes the microenvironment image I TEM , low-resolution image I LR and genomics data G, and finally divide the sample set into a training sample set and a test sample set;

[0011] S2: Define the dynamic fuzzy kernel, given the input set {I LR ,G}, first define the fuzzy membership function to fuzzify the input to obtain the fuzzy feature, then calculate the distance between the fuzzy feature and the fuzzy kernel, and finally defuzzify the fuzzy kernel response and continuously update the dynamic fuzzy kernel through the gating mechanism;

[0012] S3: Construct a fuzzy guidance block. This module takes the dynamic blur kernel and the output of the previous neural network layer as input. The dynamic blur kernel is expanded to the appropriate dimension through dense projection and used as the index matrix and content matrix of the cross attention. The output of the previous neural network layer passes through a residual module and a self-attention module and serves as the query matrix of the cross attention. Finally, the three matrices are subjected to cross attention.

[0013] S4: Construct a diffusion model that integrates dynamic fuzzy kernels and fuses multimodal features into the diffusion model by embedding fuzzy guidance blocks in the neural network during the diffusion process;

[0014] S5: The feature matrix output by the diffusion model is fed into the classification neural network and the survival prediction neural network respectively to obtain the final result.

[0015] Furthermore, step S2 includes the following steps:

[0016] Step S2.1: Define two membership functions μ∈{μ B (x;a1,b1,c1),μ S (x; a2, b2)}, where μ B (x; a1, b1, c1) is a bell-shaped membership function, x∈{I LR ,G} is the input data, a1 controls the width of the curve, b1 controls the slope of the curve, c1 is the center of the curve, μ S (x; a2, b2) is the S-shaped membership function, a2 and b2 are the starting point and end point of the curve respectively, and the final output is the fuzzy feature. The ~ symbol above the variable indicates that the variable has been subjected to the membership function, that is, the fuzzified binary variable, where

[0017]

[0018] Step S2.2: Randomly initialize the learnable weights W q and W k , and define the double fuzzy distance

[0019]

[0020] in (·) T represents the transpose of the matrix, and d represents the dynamic blur kernel F t The kernel vector f i The dimension, F t ={f1,f2,...,f i ,...,f N}, is the i-th kernel vector, t∈{1,2,...,t,...,T} is the number of steps of the current diffusion of the model, T is the total number of diffusion steps, N is the number of vectors, is a low-resolution image I LR The double fuzzy distance, is the double fuzzy distance of genomics data G;

[0021] Step S2.3: Double fuzzy distance Normalize and get the kernel vector f i Fuzzy importance weight

[0022]

[0023] Where exp(·) is the exponential function;

[0024] Step S2.4: Randomly initialize multiple learnable weights is the learnable weight, for the t-th model diffusion step, the dynamic fuzzy kernel vector f i Perform a linear transformation to obtain the fuzzy memory response

[0025]

[0026] Step S2.5: Fuzzy memory response Defuzzification is r t

[0027]

[0028] Among them, r t is the memory response after defuzzification, w B and w S are the learnable weights of different membership functions;

[0029] Step S2.6: Dynamically update the dynamic blur kernel F using a gating mechanism t , dynamically controlling F by retaining or forgetting the information in the sequence t Information flow in

[0030]

[0031] in, is the intermediate result, MLP(·) is a multi-layer perceptron, where W fo and W in is r in each gate t The trainable weights, V fo and V in Is the F in each door t+1 The trainable weights, superscripts fo and in represent the forget gate and input gate respectively, tanh(·) is the hyperbolic tangent function, F t+1 is the dynamic blur kernel of the previous step in the model diffusion, and finally the updated dynamic blur kernel F is obtained t for

[0032]

[0033] where ⊙ is the Hadamard product and σ(·) is the Sigmoid function.

[0034] Furthermore, step S3 includes the following steps:

[0035] Step S3.1: Construct a fuzzy guidance module FMG(·) based on the unet model, the input of which is the updated dynamic fuzzy kernel F t and the output L from the previous neural network layer, where L∈{D k ,E k}, D k is the output of the neural network in the unet model encoder, E k is the output of the neural network in the unet model decoder, and k represents the kth layer in the unet model;

[0036] Step S3.2: For the updated motion blur kernel F t , first will pass through W k1 , W ν1 Dense projection, then used as the index matrix K and content matrix V of the cross attention mechanism, for the output L of the previous layer of neural network, it will first pass through a residual module Res(·) and a self-attention module SA(·), and then pass through W q1 Project to a unified dimension and use it as the query matrix Q, and finally pass it through FMG(Q,K,V) for cross attention;

[0037] Q={L|L∈D k ‖L∈E k}W q1 ,K=F t W k1 ,V=F t W ν1 (11)

[0038]

[0039] Step S3.3: In the encoder, the fuzzy guidance module FMG(·) operates as follows:

[0040] E k+1 =FMG(SA(Res(E k ,t)),F t ) (13)

[0041] Among them, E k+1 represents the output of the next layer of neural network. In the decoder, the fuzzy guidance module FMG(·) operates as follows:

[0042] D k =FMG(Res(D k+1 ,E k ,t),F t ) (14)

[0043] Among them, D k+1 Represents the output of the next layer of neural network.

[0044] Furthermore, the step S4 includes the following steps:

[0045] Step S4.1: Construct a diffusion model based on the fuzzy guidance module. TEM , first pass through an image encoder ε(·), and get the encoded feature I=ε(I TEM ), a total of T steps are gradually added to the Gaussian noise and it becomes pure Gaussian noise I T , denoted as q(I t |I t-1 ,I0), the process of adding noise is as follows

[0046]

[0047] where α is used to gradually mix the data distribution into the noise distribution at each time step, t is the tth step, To express the cumulative product of the form α, describing the cumulative noise scaling effect from time step t = 0 to T;

[0048] Step S4.2: Constructing an inverse model using the Bayesian formula

[0049]

[0050] Among them, q(I t-1 |I t ,I0,F T ) represents the image I given the current time step t t , initial image I0 and dynamic blur kernel F T , calculate the image I at time step t-1 t-1 The posterior distribution of q(I t |I t-1 ,I0,F T ) indicates that given the previous step image I t-1 and the initial image I0, the image I at time step t t The probability distribution of q(I t-1 |I0,F T ) represents the image I directly from the initial image I0 to the time step t-1 t-1 The probability distribution of q(I t |I0,F T ) represents the image I from the initial image I0 to the time step t t The probability distribution of , q(·|·) represents the probability distribution function;

[0051] Step S4.3: Substitute the known quantity into each function of equation (15) to obtain equations (16)(17)(18)

[0052]

[0053] Among them Indicates normal distribution, the first · indicates the mean, and the second · indicates the variance;

[0054] Step S4.4: Obtain probability distribution by calculation

[0055]

[0056] Here, ∝ indicates proportionality, and C(·) indicates a constant.

[0057] Compared with the prior art, the present invention has the following beneficial effects:

[0058] 1. Processing Multimodal Data: A multi-task approach for full-scale histopathology imaging based on fuzzy kernels can simultaneously process multimodal input data, integrating multiple information to improve the accuracy of classification and survival prediction. For multi-task models, this invention can simultaneously utilize full-scale imaging and genomic data, integrating multiple information to improve multi-task accuracy.

[0059] 2. Reduced Computational Complexity: The core concept of the dynamic fuzzy kernel is to employ an intelligent fuzzification strategy that dynamically adjusts and compresses the features of different modal data to reduce the storage requirements of redundant information. This fuzzy kernel mechanism flexibly selects and aggregates relevant features based on the needs of the task at hand, significantly reducing computational complexity while maintaining information integrity.

[0060] 3. Improved Prediction Accuracy: A multi-task approach for full-scale histopathology imaging based on a fuzzy kernel utilizes a dynamic fuzzy kernel, enabling information sharing between multiple tasks and enabling knowledge transfer and interaction. The dynamic fuzzy kernel's real-time update and optimization mechanism enables the model to adapt to new data and changing environments. The model establishes close connections between different tasks, forming a beneficial knowledge sharing mechanism that improves prediction accuracy across tasks.

[0061] 4. Strong generalization capability: The multi-task method for full-volume histopathology imaging based on fuzzy kernels has good generalization performance, can handle high-dimensional complex data, and is robust to noise and interference. The dynamic fuzzy kernel can adaptively adapt to different patient backgrounds and clinical data by adjusting the storage and use of features in real time, thereby enhancing the model's adaptability to new data. Secondly, this method adopts a multi-task learning framework, which enables the synergy between different tasks to further enhance generalization capabilities. In medical image analysis, there is an intrinsic connection between multiple related tasks such as tumor classification, pathology scoring, and survival prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention but do not constitute a limitation of the present invention.

[0063] Figure 1 This is the overall framework diagram of the multi-task method for full-volume histopathology imaging based on fuzzy kernels of the present invention;

[0064] Figure 2 This is a dynamic fuzzy kernel module diagram of the multi-task method for full-volume histopathology imaging based on fuzzy kernel of the present invention;

[0065] Figure 3 This is an overview of the multimodal diffusion model of the present invention's fuzzy kernel-based multi-task method for full-volume histopathology imaging;

[0066] Figure 4 This is a fuzzy-guided module diagram of the fuzzy kernel-based multi-task method for full-volume tissue pathology images of the present invention.

[0067] Figure 5 for Figure 1 Comparison chart of evaluation indicators of the present invention and other models. DETAILED DESCRIPTION

[0068] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. Of course, the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0069] Example 1

[0070] See also Figures 1 to 4 This embodiment provides a technical solution for a multi-task method for full-volume histopathology images based on a fuzzy kernel. Taking a full-volume sarcoma image I1 and the corresponding genomic data G1 as an example, the method includes the following steps from loading the data to obtaining classification and survival prediction results:

[0071] S1: For each patient, full-length images and genomic data are collected and matched one to one. The full-length images are then divided into 224×224 image blocks, and the Resnet classifier is used to classify these blocks into microenvironment images I TEM and low-resolution image I LR At the same time, the genomic data G is expanded and spliced ​​to unify the dimensions to form a sample set, which includes microenvironment images Low-resolution images and genomics data Finally, the sample set is divided into a training sample set and a test sample set;

[0072] S2: Define the dynamic fuzzy kernel, given the input set {I LR ,G}, first define the fuzzy membership function to fuzzify the input to obtain the fuzzy feature, then calculate the distance between the fuzzy feature and the fuzzy kernel, and finally defuzzify the fuzzy kernel response and continuously update the dynamic fuzzy kernel through the gating mechanism;

[0073] S3: Construct a fuzzy guidance block. This module takes the dynamic blur kernel and the output of the previous neural network layer as input. The dynamic blur kernel is expanded to the appropriate dimension through dense projection and used as the index matrix and content matrix of the cross attention. The output of the previous neural network layer passes through a residual module and a self-attention module and serves as the query matrix of the cross attention. Finally, the three matrices are subjected to cross attention.

[0074] S4: Construct a diffusion model that integrates dynamic fuzzy kernels and fuses multimodal features into the diffusion model by embedding fuzzy guidance blocks in the neural network during the diffusion process;

[0075] S5: The feature matrix output by the diffusion model is fed into the classification neural network and the survival prediction neural network respectively to obtain the final result.

[0076] The multi-task method for full-volume histopathology images based on a fuzzy kernel according to claim 1 is characterized in that the specific steps of step S2 are as follows:

[0077] Step S2.1: Define two membership functions μ∈{μ B (x;a1,b1,c1),μ S (x; a2, b2)}, where μ B (x; a1, b1, c1) is a bell-shaped membership function, x∈{I LR ,G} is the input data, a1 controls the width of the curve, b1 controls the slope of the curve, c1 is the center of the curve, μ S (x; a2, b2) is the S-shaped membership function, a2 and b2 are the starting point and end point of the curve respectively, and the final output is the fuzzy feature. The ~ symbol above the variable indicates that the variable has been subjected to the membership function, that is, the fuzzified binary variable, where

[0078]

[0079] μ B (G; 0.2, 1, 0.5), μ S (G; 0.3, 0.7) Similarly;

[0080] Step S2.2: Randomly initialize the learnable weights W q and W q , and define the double fuzzy distance The learnable weights are randomly initialized, and the results are as follows:

[0081]

[0082]

[0083] Then calculate the double fuzzy distance for:

[0084]

[0085] in (·) T represents the transpose of the matrix, d=128 represents the dynamic blur kernel F t The kernel vector f i The dimension, F t ={f1,f2,...,f i ,...,f N}, is the i-th kernel vector, t∈{1,2,...,t,...,T} is the number of model diffusion steps, T=1000, N=16 is the number of vectors, is a low-resolution image I LR The double fuzzy distance, is the double fuzzy distance of genomics data G, where They are as follows:

[0086]

[0087] Substitute the formula to calculate the double fuzzy distance for:

[0088]

[0089] Step S2.3: Double fuzzy distance Normalize and get the kernel vector f iFuzzy importance weight for:

[0090]

[0091] Where exp(·) is the exponential function;

[0092] Step S2.4: Randomly initialize multiple learnable weights is the learnable weight, for the t-th model diffusion step, the dynamic fuzzy kernel vector f i Perform a linear transformation to obtain the fuzzy memory response for:

[0093]

[0094] Step S2.5: Fuzzy memory response Defuzzification is r t :

[0095]

[0096] Among them, r t is the memory response after defuzzification, w B and w S are the learnable weights of different membership functions;

[0097] Step S2.6: Dynamically update the dynamic blur kernel F using a gating mechanism t , dynamically controlling F by retaining or forgetting the information in the sequence t Information flow in

[0098]

[0099]

[0100] in, is the intermediate result, MLP(·) is a multi-layer perceptron, where W fo and W in is r in each gate t The trainable weights, V fo and V in Is the F in each door t+1 The trainable weights, superscripts fo and in represent the forget gate and input gate respectively, tanh(·) is the hyperbolic tangent function, F t+1 is the dynamic blur kernel of the previous step in the model diffusion, and finally the updated dynamic blur kernel F is obtained t for

[0101]

[0102] where ⊙ is the Hadamard product and σ(·) is the Sigmoid function.

[0103] The multi-task method for full-volume histopathology images based on a fuzzy kernel according to claim 1, wherein the specific steps of step S3 are as follows:

[0104] Step S3.1: Construct a fuzzy guidance module FMG(·) based on the unet model, the input of which is the updated dynamic fuzzy kernel F t and the output L from the previous neural network layer, where L∈{D k ,E k}, D k is the output of the neural network in the unet model encoder, E k is the output of the neural network in the unet model decoder, and k represents the kth layer in the unet model;

[0105] Step S3.2: For the updated motion blur kernel F t , first will pass through W k1 , W ν1 Dense projection, then used as the index matrix K and content matrix V of the cross attention mechanism, for the output L of the previous layer of neural network, it will first pass through a residual module Res(·) and a self-attention module SA(·), and then pass through W q1 Project to a unified dimension and use it as the query matrix Q, and finally pass it through FMG(Q,K,V) for cross attention;

[0106] Q={L|L∈D k ‖L∈E k}W q1 ,K=F t W k1 ,V=F t W ν1 (16)

[0107]

[0108] Step S3.3: In the encoder, the fuzzy guidance module FMG(·) operates as follows, where we take k = 1:

[0109]

[0110] Where E2 represents the output of the next layer of neural network. In the decoder, the fuzzy guidance module FMG(·) operates as follows:

[0111]

[0112] Among them, D2 represents the output of the next layer of neural network.

[0113] The multi-task method for full-volume histopathology images based on a fuzzy kernel according to claim 1 is characterized in that the specific steps of step S4 are as follows:

[0114] Step S4.1: Construct a diffusion model based on the fuzzy guidance module. TEM , first pass through an image encoder ε(·), and get the encoded feature I=ε(I TEM ), a total of T steps are gradually added to the Gaussian noise and it becomes pure Gaussian noise I T , denoted as q(I t |I t-1 ,I0), the process of adding noise is as follows

[0115]

[0116] where α is used to gradually mix the data distribution into the noise distribution at each time step, t is the tth step, To express the cumulative product of the form α, describing the cumulative noise scaling effect from time step t = 0 to T, take t = 200, the obtained features are as follows:

[0117]

[0118] Step S4.2: Constructing an inverse model using the Bayesian formula

[0119]

[0120] Among them, q(I t-1 |I t ,I0,F T ) represents the image I given the current time step t t , initial image I0 and dynamic blur kernel F T , calculate the image I at time step t-1 t-1 The posterior distribution of q(I t |I t-1 ,I0,F T ) indicates that given the previous step image I t-1 and the initial image I0, the image I at time step t t The probability distribution of q(I t-1 |I0,F T ) represents the image I directly from the initial image I0 to the time step t-1 t-1 The probability distribution of q(I t |I0,F T ) represents the image I from the initial image I0 to the time step t tThe probability distribution of , q(·|·) represents the probability distribution function. This step is derived from mathematical theory. The specific calculation results are shown in step S4.4;

[0121] Step S4.3: Substitute the known quantity into each function of equation (23) to obtain equations (24)(25)(26)

[0122]

[0123] Among them represents the normal distribution, the first · represents the mean, and the second · represents the variance. This step is derived from mathematical theory. The specific calculation results are shown in step S4.4;

[0124] Step S4.4: Obtain probability distribution by calculation

[0125]

[0126] Where ∝ indicates proportionality, and C(·) indicates a constant. Substituting these into the equation yields the following:

[0127]

[0128] Example 2

[0129] Referring to Example 1, this example uses the parameters and results calculated in Example 1 to compare with traditional methods to demonstrate the superiority of the present invention. In this specific comparison, we used different models and compared different indicators. The final results show that the present invention is superior to the traditional method.

[0130] 1. Traditional Model

[0131] With reference to relevant research at home and abroad, this embodiment selected the following comparison models:

[0132] 1) ViT Model: The ViT (Vision Transformer) model divides images into fixed-size patches (typically 16x16 pixels) and flattens these patches as input sequences, similar to how text is processed. These patches are then processed using a standard Transformer architecture, employing a multi-head self-attention mechanism to capture global image features. Disadvantages: The Transformer architecture has high computational complexity, and for large-scale WSI images, the computational and storage requirements can be substantial.

[0133] 2) GiMP Model: The GiMP model combines image and genomic data for classification through multimodal pre-training. The model first pre-processes and encodes the genomic data, then fuses it with image features. Through joint training, it improves classification performance. Disadvantages: The need to process and fuse multiple types of data complicates model training and parameter adjustment, increasing implementation difficulty. Furthermore, the GiMP model is primarily targeted at applications that combine genomic data and cannot fully leverage its advantages in WSI classification using pure image data.

[0134] 3) SNN Model: The SNN (Self-normalizing Neural Network) model uses the SELU (Scaled Exponential Linear Unit) activation function to maintain network normalization during training, thereby reducing reliance on batch normalization. Disadvantages: While SNNs offer advantages in training stability, they may not perform as well as specially designed CNN or Transformer architectures when processing complex and detail-rich WSI images.

[0135] 4) A-MIL Model: The A-MIL (Attention-based Multiple Instance Learning) model introduces an attention mechanism to weight important instances within the multiple instance learning (MIL) framework. This model is suitable for processing situations where each sample consists of multiple instances. Through the attention mechanism, the model automatically selects the instances that contribute most to the classification, thereby improving classification performance. Disadvantages: For very large-scale WSI images with a large number of instances, the A-MIL model requires high computing resources and storage, which may affect its application.

[0136] 2. Comparative indicators

[0137] This example is used for classification and survival prediction tasks. In classification, we use accuracy (ACC), true positive rate (TPR), true negative rate (TNR), positive predictive value (PPV), and F1 score (F1) as evaluation metrics. In survival prediction, we use the consistency index (c-Index) as the evaluation metric.

[0138] ACC: The ratio of correctly classified samples to the total number of samples. It reflects the overall accuracy of the model's classification. Benefit: Simple and intuitive, suitable for cases with a balanced class distribution. Disadvantage: When the classes are unbalanced, it may mask the model's performance on the minority class.

[0139] TPR: This is the proportion of samples that are correctly predicted as positive among samples that are actually positive. Purpose: This measures the model's ability to identify positive samples and is suitable for scenarios that focus on positive detection. Disadvantage: This only focuses on positive samples and ignores negative samples.

[0140] TNR: This is the ratio of samples that are correctly predicted as negative among samples that are actually negative. Purpose: This metric measures the model's ability to identify negative samples and is suitable for scenarios focused on negative detection. Disadvantage: This metric focuses solely on negative samples and ignores positive recognition.

[0141] PPV: The proportion of samples predicted as positive that are actually positive. Purpose: This measures the proportion of positive samples predicted by the model that are actually positive. It is suitable for scenarios where prediction accuracy is of primary concern. Disadvantage: It focuses solely on the accuracy of positive predictions.

[0142] F1: Comprehensively considers the performance of the model. Purpose: Applicable to cases with class imbalance and comprehensively evaluates model performance.

[0143] Disadvantage: It cannot reflect the recognition of negative classes.

[0144] c-Index: The consistency index is primarily used to assess the predictive power of survival analysis models, indicating the degree of consistency between the model's predicted ranking and the actual ranking. Purpose: This index measures the model's ability to predict survival time or risk ranking and is commonly used in survival analysis and risk prediction.

[0145] Each indicator has its unique role and limitations in different application scenarios. This embodiment uses a comprehensive evaluation method to objectively compare the present invention with other models, aiming to highlight the superiority of the present invention.

[0146] 3. Comparison results

[0147] As can be seen from the charts and data, the present invention performs well across all evaluation metrics, significantly outperforming other models. First, in terms of ACC, the present invention achieves 86.44%, significantly higher than the 71.99% of the ViT model, 75.99% of the GiMP model, 61.45% of the SNN model, and 64.18% of the A-MIL model. This demonstrates that the present invention achieves higher overall classification accuracy and is better able to distinguish between positive and negative samples.

[0148] The proposed method also performed exceptionally well in terms of TPR, reaching 85.03%. In comparison, the GiMP model and ViT model achieved 76.61% and 68.38%, respectively, while the SNN model and A-MIL model achieved 63.06% and 61.88%, respectively. This demonstrates that the proposed method is more sensitive in detecting positive samples and can more efficiently identify them.

[0149] The present invention also performed well in terms of TNR, reaching 87.55%. This metric was compared to 75.42% and 70.48% for the GiMP and ViT models, respectively, and 59.52% and 61.26% for the SNN and A-MIL models, respectively. The present invention also demonstrated strong capabilities in identifying negative samples, accurately identifying more negative samples.

[0150] In terms of PPV, the present invention achieved a high score of 85.34%, significantly higher than other models. The PPVs of the ViT model and GiMP model were 73.61% and 73.55%, respectively, while those of the SNN model and A-MIL model were 57.35% and 66.70%, respectively. This demonstrates that the present invention has a greater advantage in predicting positive samples and can more accurately predict positive samples.

[0151] The proposed method also demonstrated excellent performance in F1, achieving 87.83%. In comparison, the ViT and GiMP models achieved F1 scores of 71.59% and 73.91%, respectively, while the SNN and A-MIL models achieved scores of 60.22% and 62.56%, respectively. The F1 score, which combines precision and recall, demonstrates the proposed method's strong overall ability to identify and predict positive samples.

[0152] Although the SNN model is slightly higher than other models in terms of c-Index with a score of 80.70%, the 84.82% of the present invention still shows its strong ability in ranking prediction.

[0153] Overall, the present invention significantly outperforms other models in multiple key evaluation indicators, demonstrating its superior performance and stability in classification tasks.

[0154] Table 1 Comparison table of the present invention and other models

[0155]

[0156] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multi-task method for full-volume histopathology images based on fuzzy kernels, characterized by: The following steps are involved: S1: For each patient, full-length images and genomic data are collected and matched one to one. The full-length images are then divided into 224×224 image blocks, and the Resnet classifier is used to classify these blocks into microenvironment images I TEM and low-resolution image I LR At the same time, the genomic data G is expanded and spliced ​​to unify the dimension to form a sample set, which includes the microenvironment image I TEM , low-resolution image I LR and genomics data G, and finally divide the sample set into a training sample set and a test sample set; S2: Define the dynamic fuzzy kernel, given the input set {I LR ,G}, first define the fuzzy membership function to fuzzify the input to obtain the fuzzy feature, then calculate the distance between the fuzzy feature and the fuzzy kernel, and finally defuzzify the fuzzy kernel response and continuously update the dynamic fuzzy kernel through the gating mechanism; S3: Construct a fuzzy guidance block. This module takes the dynamic blur kernel and the output of the previous neural network layer as input. The dynamic blur kernel is expanded to the appropriate dimension through dense projection and used as the index matrix and content matrix of the cross attention. The output of the previous neural network layer passes through a residual module and a self-attention module and serves as the query matrix of the cross attention. Finally, the three matrices are subjected to cross attention. S4: Construct a diffusion model that integrates dynamic fuzzy kernels and fuses multimodal features into the diffusion model by embedding fuzzy guidance blocks in the neural network during the diffusion process; S5: The feature matrix output by the diffusion model is fed into the classification neural network and the survival prediction neural network respectively to obtain the final result.

2. The multi-task method for full-volume histopathology images based on fuzzy kernel according to claim 1, characterized in that: The step S2 includes the following steps: Step S2.1: Define two membership functions μ∈{μ B (x;a1,b1,c1),μ S (x; a2, b2)}, where μ B (x; a1, b1, c1) is a bell-shaped membership function, x∈{I LR ,G} is the input data, a1 controls the width of the curve, b1 controls the slope of the curve, c1 is the center of the curve, μ S (x; a2, b2) is the S-shaped membership function, a2 and b2 are the starting point and end point of the curve respectively, and the final output is the fuzzy feature. The ~ symbol above the variable indicates that the variable has been subjected to the membership function, that is, the fuzzified binary variable, where Step S2.2: Randomly initialize the learnable weights W q and W k , and define the double fuzzy distance D Ω in (·) T represents the transpose of the matrix, and d represents the dynamic blur kernel F t The kernel vector f i The dimension, F t ={f1,f2,...,f i ,...,f N }, is the i-th kernel vector, t∈{1,2,...,t,...,T} is the number of steps of the current diffusion of the model, T is the total number of diffusion steps, N is the number of vectors, For low-resolution image I LR The double fuzzy distance, is the double fuzzy distance of genomics data G; Step S2.3: Double fuzzy distance Normalize and get the kernel vector f i Fuzzy importance weight Where exp(·) is the exponential function; Step S2.4: Randomly initialize multiple learnable weights is the learnable weight, for the t-th model diffusion step, the dynamic fuzzy kernel vector f i Perform a linear transformation to obtain the fuzzy memory response Step S2.5: Fuzzy memory response Defuzzification is r t Among them, r t is the memory response after defuzzification, w B and w S are the learnable weights of different membership functions; Step S2.6: Dynamically update the dynamic blur kernel F using a gating mechanism t , dynamically controlling F by retaining or forgetting the information in the sequence t Information flow in in, is the intermediate result, MLP(·) is a multi-layer perceptron, where W fo and W in is r in each gate t The trainable weights, V fo and V in Is the F in each door t+1 The trainable weights, superscripts fo and in represent the forget gate and input gate respectively, tanh(·) is the hyperbolic tangent function, F t+1 is the dynamic blur kernel of the previous step in the model diffusion, and finally the updated dynamic blur kernel F is obtained t for where σ(·) is the Hadamard product and σ(·) is the sigmoid function.

3. The multi-task method for full-volume histopathology images based on fuzzy kernel according to claim 1, characterized in that: The step S3 includes the following steps: Step S3.1: Construct a fuzzy guidance module FMG(·) based on the unet model, the input of which is the updated dynamic fuzzy kernel F t and the output L from the previous neural network layer, where L∈{D k ,E k }, D k is the output of the neural network in the unet model encoder, E k is the output of the neural network in the unet model decoder, and k represents the kth layer in the unet model; Step S3.2: For the updated motion blur kernel F t , will first pass through W k1 , W ν1 Dense projection, then used as the index matrix K and content matrix V of the cross attention mechanism, for the output L of the previous layer of neural network, it will first pass through a residual module Res(·) and a self-attention module SA(·), and then pass through W q1 Project to a unified dimension and use it as the query matrix Q, and finally pass FMG(Q,K,V) for cross attention; Q={L|L∈D k ‖L∈E k }W q1 ,K=F t W k1 ,V=F t W ν1 (11) Step S3.3: In the encoder, the fuzzy guidance module FMG(·) operates as follows: Yes k+1 =FMG(SA(Res(E k ,t)),F t ) (13) Among them, E k+1 represents the output of the next layer of neural network. In the decoder, the fuzzy guidance module FMG(·) operates as follows: D k =FMG(Res(D k+1 ,E k ,t),F t ) (14) Among them, D k+1 Represents the output of the next layer of neural network.

4. The multi-task method for full-volume histopathology images based on fuzzy kernel according to claim 1, characterized in that: The step S4 includes the following steps: Step S4.1: Construct a diffusion model based on the fuzzy guidance module. TEM , first pass through an image encoder ε(·), and get the encoded feature I=ε(I TEM ), a total of T steps are gradually added to the Gaussian noise and it becomes pure Gaussian noise I T , denoted as q(I t |I t-1 ,I0), the process of adding noise is as follows where α is used to gradually mix the data distribution into the noise distribution at each time step, t is the tth step, To express the cumulative product of the form α, describing the cumulative noise scaling effect from time step t = 0 to T; Step S4.2: Constructing an inverse model using the Bayesian formula Among them, q(I t-1 |I t ,I0,F T ) represents the image I given the current time step t t , initial image I0 and dynamic blur kernel F T , calculate the image I at time step t-1 t-1 The posterior distribution of q(I t |I t-1 ,I0,F T ) indicates that given the previous step image I t-1 and the initial image I0, the image I at time step t t The probability distribution of q(I t-1 |I0,F T ) represents the image I directly from the initial image I0 to the time step t-1 t-1 The probability distribution of q(I t |I0,F T ) represents the image I from the initial image I0 to the time step t t The probability distribution of , q(·|·) represents the probability distribution function; Step S4.3: Substitute the known quantity into each function of equation (15) to obtain equations (16)(17)(18) Among them Indicates normal distribution, the first · indicates the mean, and the second · indicates the variance; Step S4.4: Obtain probability distribution by calculation Here, ∝ indicates proportionality, and C(·) indicates a constant.

Images