Histopathology full-amount image multi-task method based on fuzzy kernel

Abstract

The invention provides a histopathology full-amount image multi-task method based on a fuzzy kernel, belongs to the technical field of medical image intelligent diagnosis, and solves the technical problem that a current full-amount image model cannot perform high-accuracy classification and prediction tasks at the same time. According to the technical scheme, the method comprises the following steps: firstly, preprocessing a total image into a microenvironment image ITEM and a low-resolution image ILR; then, fuzzification and defuzzification are carried out through a membership function, and a result is stored in a kernel; thirdly, fusing the features through cross attention; thirdly, designing a fusion diffusion model which can be guided by the dynamic fuzzy kernel in the diffusion process; and finally, putting the diffused features into a classification and survival prediction neural network to obtain a final result. The method has the beneficial effects that multi-task processing is carried out on the multi-modal medical full-dose image, classification and survival prediction can be accurately carried out on diseases, and doctors are helped to more comprehensively and accurately understand tumor invasiveness and metastasis tendency.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image intelligent diagnosis, and in particular to a fuzzy kernel-based multi-task method for full-volume histopathology images. Background Art

[0002] Multimodal medical full-volume imaging is a widely used high-resolution digital technology for tissue sections, playing an important role in the diagnosis, classification, and prognosis assessment of tumors. High-resolution tissue section images allow pathologists to more accurately observe the morphology, tissue structure, and microenvironment of tumors, providing a more comprehensive basis for survival prediction. However, in actual clinical diagnosis, due to the high resolution and complexity of full-volume imaging, manual analysis time costs are high, and interpretation relies heavily on one's own experience and expertise, which may affect the consistency and accuracy of diagnosis.

[0003] In this context, the integration of artificial intelligence and multimodal medical imaging technology is particularly important, which can assist pathologists in their diagnosis to a greater extent, reveal the molecular and genetic characteristics of the disease, and provide more valuable information for individualized treatment. Therefore, the importance of multimodal medical imaging in clinical practice is becoming increasingly prominent. Its application not only improves the accuracy of diagnosis and treatment, but also promotes the development and innovation of medical imaging technology, bringing more health protection and hope to patients.

[0004] In the research paper titled "Joint region and nucleus segmentation for characterization of tumor infiltrating lymphocytes in breast cancer", a deep learning model was used to classify all images. However, their method has great limitations. It can only analyze a single modality and has a low accuracy rate, and still requires manual intervention by doctors.

[0005] In the research paper titled "Spatial organization and molecular correlation of tumor infiltrating lymphocytes using deep learning on pathology images", two convolutional neural networks were used to extract features of different regions in the full image by evaluating the image resolution. However, their method consumes a lot of computing resources and does not handle the dimensional differences between different modalities when fusing features.

[0006] Traditional multi-task methods usually use generative models to generate images by gradually training the model. These methods require manual selection and design of feature extractors, have difficulty processing high-dimensional complex data, and are susceptible to noise and interference.

[0007] How to solve the above technical problems is the subject faced by the present invention. Summary of the invention

[0008] The purpose of the present invention is to provide a multi-task method for full-volume tissue pathology images based on fuzzy kernels, which solves the technical problems of cascade classification and survival prediction of full-volume images, aims to improve the accuracy and generalization ability of multimodal full-volume images in classification and survival prediction tasks, and has broad application prospects in pathology, genetics research and clinical medicine.

[0009] In order to achieve the above-mentioned invention object, the technical solution adopted by the present invention is specifically: a multi-task method for full-volume histopathology images based on fuzzy kernel, comprising the following steps:

[0010] S1: For each patient, we collected full-volume images and genomic data and matched them one by one. We then divided the full-volume images into 224×224 blocks and used the Resnet classifier to classify these blocks into microenvironment images I TEM and low-resolution image I LR At the same time, the genomic data G is expanded and spliced ​​to unify the dimension to form a sample set, which includes the microenvironment image I TEM , low resolution image I LR and genomics data G, and finally divide the sample set into a training sample set and a test sample set;

[0011] S2: Define the dynamic fuzzy kernel, given the input set {I LR ,G}, firstly define the fuzzy membership function to fuzzify the input to obtain the fuzzy feature, then calculate the distance between the fuzzy feature and the fuzzy kernel, and finally defuzzify the fuzzy kernel response, and continuously update the dynamic fuzzy kernel through the gating mechanism;

[0012] S3: Build a fuzzy guidance block. This module inputs the dynamic fuzzy kernel and the output of the previous layer of the neural network. For the dynamic fuzzy kernel, it will be expanded to a suitable dimension through dense projection and used as the index matrix and content matrix of the cross attention. For the output of the previous layer of the neural network, it will pass through a residual module and a self-attention module and serve as the query matrix of the cross attention. Finally, the three matrices are cross-attended.

[0013] S4: Construct a diffusion model fused with dynamic fuzzy kernel, and fuse multimodal features into the diffusion model by embedding fuzzy guidance blocks in the neural network during the diffusion process;

[0014] S5: The feature matrix output by the diffusion model is put into the classification neural network and the survival prediction neural network respectively to obtain the final result.

[0015] Furthermore, the step S2 includes the following steps:

[0016] Step S2.1: Define two membership functions μ∈{μ B (x;a1,b1,c1),μ S (x; a2, b2)}, where μ B (x; a1, b1, c1) is a bell-shaped membership function, x∈{I LR , G} is the input data, a1 controls the width of the curve, b1 controls the slope of the curve, c1 is the center of the curve, μ S (x; a2, b2) is an S-shaped membership function, a2 and b2 are the starting point and end point of the curve respectively, and the final output is the fuzzy feature. The ~ symbol above the variable indicates that the variable has been subjected to the membership function, that is, the fuzzified dual variable, where

[0017]

[0018] Step S2.2: Randomly initialize the learnable weights W q and W k , and define the double fuzzification distance

[0019]

[0020] in (·) T represents the transpose of the matrix, and d represents the dynamic blur kernel F t The kernel vector f i The dimension, F t ={f1,f2,...,f i ,...,f N}, is the i-th kernel vector, t∈{1,2,...,t,...,T} is the number of steps of the current diffusion of the model, T is the total number of diffusion steps, N is the number of vectors, For low-resolution image I LR The double fuzzy distance, is the double fuzzy distance of genomics data G;

[0021] Step S2.3: Double fuzzy distance Normalize and get the kernel vector f i The fuzzy importance weight of

[0022]

[0023] Where exp(·) is an exponential function;

[0024] Step S2.4: Randomly initialize multiple learnable weights is the learnable weight, and for the t-th model diffusion step, the dynamic fuzzy kernel vector f i Perform a linear transformation to obtain the fuzzy memory response

[0025]

[0026] Step S2.5: Fuzzy memory response Defuzzification is r t

[0027]

[0028] Among them, r t is the memory response after defuzzification, w B and w S are the learnable weights of different membership functions;

[0029] Step S2.6: Dynamically update the dynamic blur kernel F using a gating mechanism t , dynamically controlling F by retaining or forgetting the information in the sequence t Information flow in

[0030]

[0031] in, is the intermediate result, MLP(·) is a multi-layer perceptron, where W fo and W in is the number of gates in each t The trainable weights, V fo and V in Is the F in each door t+1 The trainable weights of F are as follows. The superscripts fo and in represent the forget gate and input gate, respectively. Tanh(·) is the hyperbolic tangent function. t+1 is the dynamic blur kernel of the previous step in the model diffusion, and finally the updated dynamic blur kernel F is obtained t for

[0032]

[0033] where ⊙ is the Hadamard product and σ(·) is the Sigmoid function.

[0034] Furthermore, step S3 includes the following steps:

[0035] Step S3.1: Construct a fuzzy guidance module FMG(·) based on the unet model, the input of which is the updated dynamic fuzzy kernel F t and the output L from the previous neural network layer, where L∈{D k ,E k}, D k is the output of the neural network in the unet model encoder, E k is the output of the neural network in the unet model decoder, and k represents the kth layer in the unet model;

[0036] Step S3.2: For the updated dynamic blur kernel F t , first through W k1 , W ν1 Dense projection, then used as the index matrix K and content matrix V of the cross attention mechanism, for the output L of the previous neural network, it will first pass through a residual module Res(·) and a self-attention module SA(·), and then pass through W q1 Project to a unified dimension and use it as the query matrix Q, and finally pass FMG(Q,K,V) for cross attention;

[0037] Q={L|L∈D k ‖L∈E k}W q1 ,K=F t W k1 ,V=F t W ν1 (11)

[0038]

[0039] Step S3.3: In the encoder, the fuzzy guidance module FMG(·) operates as follows:

[0040] E k+1 =FMG(SA(Res(E k ,t)),F t ) (13)

[0041] Among them, E k+1 represents the output of the next layer of neural network. In the decoder, the fuzzy guidance module FMG(·) operates as follows:

[0042] D k =FMG(Res(D k+1 ,E k ,t),F t ) (14)

[0043] Among them, D k+1 Represents the output of the next layer of neural network.

[0044] Further, the step S4 includes the following steps:

[0045] Step S4.1: Construct a diffusion model based on the fuzzy guidance module. TEM , firstly passes through an image encoder ε(·) to obtain the encoded feature I = ε(I TEM ), a total of T steps are gradually added to the Gaussian noise and it becomes pure Gaussian noise I T , denoted as q(I t |I t-1 ,I0), the process of adding noise is as follows

[0046]

[0047] where α is used to gradually mix the data distribution into the noise distribution at each time step, t is the tth step, To represent the cumulative product of the form α, describing the cumulative noise scaling effect from time step t = 0 to T;

[0048] Step S4.2: Constructing the inverse model through the Bayesian formula

[0049]

[0050] Among them, q(I t-1 |I t ,I0,F T ) represents the image I given the current time step t t , initial image I0 and dynamic blur kernel F T , calculate the image I at time step t-1 t-1 The posterior distribution of q(I t |I t-1 ,I0,F T ) means that given the previous step image I t-1 and the initial image I0, the image I at time step t t The probability distribution of q(I t-1 |I0,F T ) represents the image I directly from the initial image I0 to the image I at time step t-1 t-1 The probability distribution of q(I t |I0,F T ) represents the image I from the initial image I0 to the time step t t The probability distribution of , q(·|·) represents the probability distribution function;

[0051] Step S4.3: Substitute the known quantity into each function of equation (15) to obtain equations (16), (17), (18)

[0052]

[0053] Among them~ represents normal distribution, the first · represents the mean, and the second · represents the variance;

[0054] Step S4.4: Obtain probability distribution by calculation

[0055]

[0056] Among them, ∝ indicates proportionality, and C(·) indicates a constant.

[0057] Compared with the prior art, the present invention has the following beneficial effects:

[0058] 1. Processing multimodal data: The multi-task method of histopathology full-volume imaging based on fuzzy kernel can simultaneously process multimodal input data and fuse multiple information to improve the accuracy of classification and survival prediction. For multi-task models, the present invention can use full-volume imaging and genomics data at the same time, and fuse multiple information to improve the accuracy of multiple tasks.

[0059] 2. Reduce computational complexity: The core idea of ​​the dynamic fuzzy kernel is to adopt an intelligent fuzzification strategy to dynamically adjust and compress the features of different modal data to reduce the storage requirements of redundant information. This fuzzy kernel mechanism can flexibly select and aggregate relevant features according to the needs of the current task, thereby significantly reducing computational complexity while maintaining information integrity.

[0060] 3. Improve prediction accuracy: Based on the multi-task method of tissue pathology full-volume imaging based on fuzzy kernel, the model can share information between multiple tasks through dynamic fuzzy kernel, realizing knowledge transfer and interaction. The real-time update and optimization mechanism of dynamic fuzzy kernel enables the model to adapt to new data and changing environment. The model establishes a close connection between different tasks, forming a benign knowledge sharing mechanism, which improves the prediction accuracy of each task.

[0061] 4. Strong generalization ability: The multi-task method of tissue pathology full-volume images based on fuzzy kernel has good generalization performance, can handle high-dimensional complex data, and has good robustness to noise and interference. The dynamic fuzzy kernel can adaptively adapt to different patient backgrounds and clinical data by adjusting the storage and use of features in real time, thereby enhancing the model's adaptability to new data. Secondly, this method adopts a multi-task learning framework, so that the synergy between different tasks further enhances the generalization ability. In medical image analysis, there is an intrinsic connection between multiple related tasks such as tumor classification, pathology scoring, and survival prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention but do not constitute a limitation of the present invention.

[0063] Figure 1 It is the overall framework diagram of the multi-task method of the tissue pathology full-volume image based on fuzzy kernel of the present invention;

[0064] Figure 2 It is a dynamic fuzzy kernel module diagram of the multi-task method of tissue pathology full-volume imaging based on fuzzy kernel of the present invention;

[0065] Figure 3 An overview of the multimodal diffusion model of the fuzzy kernel-based multi-task method for full-volume histopathology images of the present invention;

[0066] Figure 4 This is a fuzzy-guided module diagram of the fuzzy kernel-based tissue pathology full-volume image multi-task method of the present invention.

[0067] Figure 5 for Figure 1 Comparison chart of evaluation indicators of the present invention and other models. DETAILED DESCRIPTION

[0068] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. Of course, the specific embodiments described here are only used to explain the present invention and are not used to limit the present invention.

[0069] Example 1

[0070] See also Figures 1 to 4 The technical solution provided in this embodiment is a multi-task method for full-volume histopathology images based on fuzzy kernels. A full-volume image of a sarcoma I1 and the corresponding genomics data G1 are selected from the data set as an example. From loading the data to obtaining the classification and survival prediction results, the following steps are included:

[0071] S1: For each patient, we collected full-volume images and genomic data and matched them one by one. We then divided the full-volume images into 224×224 blocks and used the Resnet classifier to classify these blocks into microenvironment images I TEM and low-resolution image I LR At the same time, the genomic data G is expanded and spliced ​​to unify the dimension to form a sample set, which includes microenvironment images Low-resolution images and genomics data Finally, the sample set is divided into a training sample set and a test sample set;

[0072] S2: Define the dynamic fuzzy kernel, given the input set {I LR ,G}, firstly define the fuzzy membership function to fuzzify the input to obtain the fuzzy feature, then calculate the distance between the fuzzy feature and the fuzzy kernel, and finally defuzzify the fuzzy kernel response, and continuously update the dynamic fuzzy kernel through the gating mechanism;

[0073] S3: Build a fuzzy guidance block. This module inputs the dynamic fuzzy kernel and the output of the previous layer of the neural network. For the dynamic fuzzy kernel, it will be expanded to a suitable dimension through dense projection and used as the index matrix and content matrix of the cross attention. For the output of the previous layer of the neural network, it will pass through a residual module and a self-attention module and serve as the query matrix of the cross attention. Finally, the three matrices are cross-attended.

[0074] S4: Construct a diffusion model fused with dynamic fuzzy kernel, and fuse multimodal features into the diffusion model by embedding fuzzy guidance blocks in the neural network during the diffusion process;

[0075] S5: The feature matrix output by the diffusion model is put into the classification neural network and the survival prediction neural network respectively to obtain the final result.

[0076] The multi-task method for full-volume histopathology images based on fuzzy kernel according to claim 1 is characterized in that the specific steps of step S2 are as follows:

[0077] Step S2.1: Define two membership functions μ∈{μ B (x;a1,b1,c1),μ S (x; a2, b2)}, where μ B (x; a1, b1, c1) is a bell-shaped membership function, x∈{I LR , G} is the input data, a1 controls the width of the curve, b1 controls the slope of the curve, c1 is the center of the curve, μ S (x; a2, b2) is an S-shaped membership function, a2 and b2 are the starting point and end point of the curve respectively, and the final output is the fuzzy feature. The ~ symbol above the variable indicates that the variable has been subjected to the membership function, that is, the fuzzified dual variable, where

[0078]

[0079] μ B (G; 0.2, 1, 0.5), μ S (G; 0.3, 0.7) Similarly;

[0080] Step S2.2: Randomly initialize the learnable weights W q and W q , and define the double fuzzification distance The learnable weights are randomly initialized, and the results are as follows:

[0081]

[0082]

[0083] Then calculate the double fuzzy distance for:

[0084]

[0085] in (·) T represents the transpose of the matrix, d=128 represents the dynamic blur kernel F t The kernel vector f i The dimension, F t ={f1,f2,...,f i ,...,f N}, is the i-th kernel vector, t∈{1,2,...,t,...,T} is the number of steps of model diffusion, T=1000, N=16 is the number of vectors, For low-resolution image I LR The double fuzzy distance, is the double fuzzy distance of genomics data G, where They are as follows:

[0086]

[0087] Substitute the formula to calculate the double fuzzy distance for:

[0088]

[0089] Step S2.3: Double fuzzy distance Normalize and get the kernel vector f iThe fuzzy importance weight of for:

[0090]

[0091] Where exp(·) is an exponential function;

[0092] Step S2.4: Randomly initialize multiple learnable weights is the learnable weight, and for the t-th model diffusion step, the dynamic fuzzy kernel vector f i Perform a linear transformation to obtain the fuzzy memory response for:

[0093]

[0094] Step S2.5: Fuzzy memory response Defuzzification is r t :

[0095]

[0096] Among them, r t is the memory response after defuzzification, w B and w S are the learnable weights of different membership functions;

[0097] Step S2.6: Dynamically update the dynamic blur kernel F using a gating mechanism t , dynamically controlling F by retaining or forgetting the information in the sequence t Information flow in

[0098]

[0099]

[0100] in, is the intermediate result, MLP(·) is a multi-layer perceptron, where W fo and W in is the number of gates in each t The trainable weights, V fo and V in Is the F in each door t+1 The trainable weights of F are as follows. The superscripts fo and in represent the forget gate and input gate, respectively. Tanh(·) is the hyperbolic tangent function. t+1 is the dynamic blur kernel of the previous step in the model diffusion, and finally the updated dynamic blur kernel F is obtained t for

[0101]

[0102] where ⊙ is the Hadamard product and σ(·) is the Sigmoid function.

[0103] The multi-task method for full-volume histopathology images based on fuzzy kernel according to claim 1 is characterized in that the specific steps of step S3 are as follows:

[0104] Step S3.1: Construct a fuzzy guidance module FMG(·) based on the unet model, the input of which is the updated dynamic fuzzy kernel F t and the output L from the previous neural network layer, where L∈{D k ,E k}, D k is the output of the neural network in the unet model encoder, E k is the output of the neural network in the unet model decoder, and k represents the kth layer in the unet model;

[0105] Step S3.2: For the updated dynamic blur kernel F t , first through W k1 , W ν1 Dense projection, then used as the index matrix K and content matrix V of the cross attention mechanism, for the output L of the previous neural network, it will first pass through a residual module Res(·) and a self-attention module SA(·), and then pass through W q1 Project to a unified dimension and use it as the query matrix Q, and finally pass FMG(Q,K,V) for cross attention;

[0106] Q={L|L∈D k ‖L∈E k}W q1 ,K=F t W k1 ,V=F t W ν1 (16)

[0107]

[0108] Step S3.3: In the encoder, the fuzzy guidance module FMG(·) operates as follows, where we take k = 1:

[0109]

[0110] Where E2 represents the output of the next layer of neural network. In the decoder, the fuzzy guidance module FMG(·) operates as follows:

[0111]

[0112] Among them, D2 represents the output of the next layer of neural network.

[0113] The multi-task method for full-volume histopathology images based on fuzzy kernel according to claim 1 is characterized in that the specific steps of step S4 are as follows:

[0114] Step S4.1: Construct a diffusion model based on the fuzzy guidance module. TEM , firstly passes through an image encoder ε(·) to obtain the encoded feature I = ε(I TEM ), a total of T steps are gradually added to the Gaussian noise and it becomes pure Gaussian noise I T , denoted as q(I t |I t-1 ,I0), the process of adding noise is as follows

[0115]

[0116] where α is used to gradually mix the data distribution into the noise distribution at each time step, t is the tth step, To represent the cumulative product of the form α, describing the cumulative noise scaling effect from time step t = 0 to T, take t = 200, and the obtained features are as follows:

[0117]

[0118] Step S4.2: Constructing the inverse model through the Bayesian formula

[0119]

[0120] Among them, q(I t-1 |I t ,I0,F T ) represents the image I given the current time step t t , initial image I0 and dynamic blur kernel F T , calculate the image I at time step t-1 t-1 The posterior distribution of q(I t |I t-1 ,I0,F T ) means that given the previous step image I t-1 and the initial image I0, the image I at time step t t The probability distribution of q(I t-1 |I0,F T ) represents the image I directly from the initial image I0 to the image I at time step t-1 t-1 The probability distribution of q(I t |I0,F T ) represents the image I from the initial image I0 to the time step t tThe probability distribution of , q(·|·) represents the probability distribution function. This step is derived from mathematical theory. The specific calculation results are shown in step S4.4;

[0121] Step S4.3: Substitute the known quantity into each function of equation (23) to obtain equations (24), (25), (26)

[0122]

[0123] Among them~ represents normal distribution, the first · represents mean, and the second · represents variance. This step is derived from mathematical theory. The specific calculation results are shown in step S4.4;

[0124] Step S4.4: Obtain probability distribution by calculation

[0125]

[0126] Among them, ∝ means proportional, C(·) means a constant, and the result is:

[0127]

[0128] Example 2

[0129] Referring to Example 1, this example will use the parameters and results calculated in Example 1 to compare with the traditional method to prove the superiority of the present invention. In the specific comparison, we used different models and compared in different indicators. The final results show that the present invention is preferred over the traditional method.

[0130] 1. Traditional Model

[0131] With reference to relevant research at home and abroad, this embodiment selects the following comparison models:

[0132] 1) ViT model: The ViT (Vision Transformer) model divides the image into fixed-size patches (usually 16x16 pixels) and flattens these patches as input sequences, similar to how text is processed. These patches are then processed through a standard Transformer architecture, using a multi-head self-attention mechanism to capture global image features. Disadvantages: The Transformer architecture has a high computational complexity, and for large-scale WSI images, the computational and storage requirements may be very large.

[0133] 2) GiMP model: The GiMP model combines image data and genomic data for classification through multimodal pre-training. The model first pre-processes and encodes the genetic data, and then fuses it with image features to improve the classification performance through joint training. Disadvantages: Since it is necessary to process and fuse multiple types of data, the training and parameter adjustment process of the model becomes more complicated, which increases the difficulty of implementation. In addition, the GiMP model is mainly aimed at application scenarios combined with genetic data, and its advantages cannot be fully utilized in the WSI classification of pure image data.

[0134] 3) SNN model: The SNN (Self-normalizing Neural Networks) model uses the SELU (Scaled Exponential Linear Unit) activation function to keep the network normalized during training, thereby reducing the reliance on batch normalization. Disadvantages: Although SNN has advantages in training stability, it may not perform as well as specially designed CNN or Transformer architectures when processing complex and detailed WSI images.

[0135] 4) A-MIL model: The A-MIL (Attention-based Multiple Instance Learning) model introduces the attention mechanism to weight important instances in the multiple instance learning (MIL) framework. This model is suitable for processing situations where each sample consists of multiple instances. Through the attention mechanism, the model can automatically select the instance that contributes most to the classification, thereby improving the classification performance. Disadvantages: For ultra-large-scale WSI images, the number of instances is huge, and the A-MIL model has high requirements for computing resources and storage, which may affect its application.

[0136] 2. Comparison indicators

[0137] This embodiment is used for classification tasks and survival prediction tasks. In classification, we use accuracy (denoted as ACC), true positive rate (denoted as TPR), true negative rate (denoted as TNR), positive predictive value (denoted as PPV) and F1 score (denoted as F1) as evaluation indicators. In survival prediction, we use consistency index (denoted as c-Index) as evaluation indicator.

[0138] ACC: It is the ratio of the number of correctly classified samples to the total number of samples. It reflects the overall classification accuracy of the model. Function: It is simple and intuitive, and is suitable for cases where the category distribution is balanced. Disadvantage: When the categories are unbalanced, it may mask the performance of the model on the minority category.

[0139] TPR: refers to the proportion of samples that are actually positive that are correctly predicted as positive. Function: Measures the model's ability to identify positive samples, suitable for scenarios that focus on positive detection. Disadvantage: Only focuses on positive samples, not negative samples.

[0140] TNR: refers to the proportion of samples that are actually negative that are correctly predicted as negative. Function: Measures the model's ability to identify negative samples, suitable for scenarios that focus on negative detection. Disadvantage: Only focuses on negative classes, not positive class recognition.

[0141] PPV: refers to the proportion of samples predicted as positive that are actually positive. Function: Measures how many positive samples predicted by the model are actually positive. It is suitable for scenarios that focus on prediction accuracy. Disadvantage: Only focuses on the accuracy of positive predictions.

[0142] F1: Comprehensively considers the performance of the model. Function: Applicable to the case of imbalanced categories, and comprehensively evaluates the performance of the model.

[0143] Disadvantages: It cannot reflect the recognition status of negative classes.

[0144] c-Index: The consistency index is mainly used to evaluate the predictive ability of the survival analysis model, indicating the consistency between the model's predicted ranking and the actual ranking. Function: It measures the model's ability to predict survival time or risk ranking, and is often used in survival analysis and risk prediction.

[0145] Each indicator has its unique role and limitations in different application scenarios. This embodiment uses a comprehensive evaluation method to objectively compare the present invention with other models, aiming to highlight the superiority of the present invention.

[0146] 3. Comparison results

[0147] It can be seen from the charts and data that the present invention performs well in all evaluation indicators and is significantly better than other models. First, in terms of ACC, the present invention reaches 86.44%, which is much higher than 71.99% of the ViT model, 75.99% of the GiMP model, 61.45% of the SNN model, and 64.18% of the A-MIL model. This shows that the present invention has a higher accuracy in overall classification and can better distinguish between positive and negative samples.

[0148] In terms of TPR, the performance of the present invention is also very outstanding, reaching 85.03%. In comparison, the GiMP model and the ViT model are 76.61% and 68.38% respectively, while the SNN model and the A-MIL model are 63.06% and 61.88% respectively. This shows that the present invention is more sensitive in detecting positive samples and can identify positive samples more efficiently.

[0149] For TNR, the present invention also performs well, reaching 87.55%. This index is 75.42% and 70.48% in the GiMP model and ViT model, respectively, and 59.52% and 61.26% in the SNN model and A-MIL model, respectively. The present invention also shows a strong ability in identifying negative samples, and can accurately identify more negative samples.

[0150] In terms of PPV, the present invention achieved a high score of 85.34%, which is significantly higher than other models. The PPVs of the ViT model and the GiMP model are 73.61% and 73.55% respectively, while those of the SNN model and the A-MIL model are 57.35% and 66.70% respectively. This shows that the present invention has an advantage in predicting the accuracy of positive samples and can predict positive samples more accurately.

[0151] The present invention also shows its excellent performance in F1, reaching 87.83%. In comparison, the F1 scores of the ViT model and the GiMP model are 71.59% and 73.91% respectively, and the SNN model and the A-MIL model are 60.22% and 62.56% respectively. The F1 score combines the precision and recall rate. The high score of the present invention in this regard shows that it has a strong comprehensive ability in the recognition and prediction of positive samples.

[0152] Although the SNN model is slightly higher than other models in terms of c-Index with a score of 80.70%, the 84.82% of the present invention still shows its strong ability in ranking prediction.

[0153] Overall, the present invention is significantly better than other models in multiple key evaluation indicators, demonstrating its superior performance and stability in classification tasks.

[0154] Table 1 Comparison table of the present invention and other models

[0155]

[0156] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A multi-task method for full-volume histopathology images based on fuzzy kernels, characterized in that: The following steps are involved: S1: For each patient, we collected full-volume images and genomic data and matched them one by one. We then divided the full-volume images into 224×224 blocks and used the Resnet classifier to classify these blocks into microenvironment images I TEM and low-resolution image I LR At the same time, the genomic data G is expanded and spliced ​​to unify the dimension to form a sample set, which includes the microenvironment image I TEM , low resolution image I LR and genomics data G, and finally divide the sample set into a training sample set and a test sample set; S2: Define the dynamic fuzzy kernel, given the input set {I LR ,G}, firstly define the fuzzy membership function to fuzzify the input to obtain the fuzzy feature, then calculate the distance between the fuzzy feature and the fuzzy kernel, and finally defuzzify the fuzzy kernel response, and continuously update the dynamic fuzzy kernel through the gating mechanism; S3: Build a fuzzy guidance block. This module inputs the dynamic fuzzy kernel and the output of the previous layer of the neural network. For the dynamic fuzzy kernel, it will be expanded to a suitable dimension through dense projection and used as the index matrix and content matrix of the cross attention. For the output of the previous layer of the neural network, it will pass through a residual module and a self-attention module and serve as the query matrix of the cross attention. Finally, the three matrices are cross-attended. S4: Construct a diffusion model fused with dynamic fuzzy kernel, and fuse multimodal features into the diffusion model by embedding fuzzy guidance blocks in the neural network during the diffusion process; S5: The feature matrix output by the diffusion model is put into the classification neural network and the survival prediction neural network respectively to obtain the final result.

2. The multi-task method for full-volume histopathology images based on fuzzy kernel according to claim 1, characterized in that: The step S2 comprises the following steps: Step S2.1: Define two membership functions μ∈{μ B (x;a1,b1,c1),μ S (x; a2, b2)}, where μ B (x; a1, b1, c1) is a bell-shaped membership function, x∈{I LR , G} is the input data, a1 controls the width of the curve, b1 controls the slope of the curve, c1 is the center of the curve, μ S (x; a2, b2) is an S-shaped membership function, a2 and b2 are the starting point and end point of the curve respectively, and the final output is the fuzzy feature. The ~ symbol above the variable indicates that the variable has been subjected to the membership function, that is, the fuzzified dual variable, where Step S2.2: Randomly initialize the learnable weights W q and W k , and define the double fuzzification distance D Ω in (·) T represents the transpose of the matrix, and d represents the dynamic blur kernel F t The kernel vector f i The dimension, F t ={f1,f2,...,f i ,...,f N }, is the i-th kernel vector, t∈{1,2,...,t,...,T} is the number of steps of the current diffusion of the model, T is the total number of diffusion steps, N is the number of vectors, For low-resolution image I LR The double fuzzy distance, is the double fuzzy distance of genomics data G; Step S2.3: Double fuzzy distance Normalize and get the kernel vector f i The fuzzy importance weight of Where exp(·) is an exponential function; Step S2.4: Randomly initialize multiple learnable weights is the learnable weight, and for the t-th model diffusion step, the dynamic fuzzy kernel vector f i Perform a linear transformation to obtain the fuzzy memory response Step S2.5: Fuzzy memory response Defuzzification is r t Among them, r t is the memory response after defuzzification, w B and w S are the learnable weights of different membership functions; Step S2.6: Dynamically update the dynamic blur kernel F using a gating mechanism t , dynamically controlling F by retaining or forgetting the information in the sequence t Information flow in in, is the intermediate result, MLP(·) is a multi-layer perceptron, where W fo and W in is the number of gates in each t The trainable weights, V fo and V in Is the F in each door t+1 The trainable weights of F are as follows. The superscripts fo and in represent the forget gate and input gate, respectively. Tanh(·) is the hyperbolic tangent function. t+1 is the dynamic blur kernel of the previous step in the model diffusion, and finally the updated dynamic blur kernel F is obtained t for where σ(·) is the Hadamard product and σ(·) is the sigmoid function.

3. The multi-task method for full-volume histopathology images based on fuzzy kernel according to claim 1, characterized in that: The step S3 comprises the following steps: Step S3.1: Construct a fuzzy guidance module FMG(·) based on the unet model, the input of which is the updated dynamic fuzzy kernel F t and the output L from the previous neural network layer, where L∈{D k ,E k }, D k is the output of the neural network in the unet model encoder, E k is the output of the neural network in the unet model decoder, and k represents the kth layer in the unet model; Step S3.2: For the updated dynamic blur kernel F t , first through W k1 , W ν1 Dense projection, then used as the index matrix K and content matrix V of the cross attention mechanism, for the output L of the previous neural network, it will first pass through a residual module Res(·) and a self-attention module SA(·), and then pass through W q1 Project to a unified dimension and use it as the query matrix Q, and finally pass FMG(Q,K,V) for cross attention; Q={L|L∈D k ‖L∈E k }W q1 ,K=F t W k1 ,V=F t W ν1 (11) Step S3.3: In the encoder, the fuzzy guidance module FMG(·) operates as follows: Yes k+1 =FMG(SA(Res(E k ,t)),F t ) (13) Among them, E k+1 represents the output of the next layer of neural network. In the decoder, the fuzzy guidance module FMG(·) operates as follows: D k =FMG(Res(D k+1 ,E k ,t),F t ) (14) Among them, D k+1 Represents the output of the next layer of neural network.

4. The multi-task method for full-volume histopathology images based on fuzzy kernel according to claim 1, characterized in that: The step S4 includes the following steps: Step S4.1: Construct a diffusion model based on the fuzzy guidance module. TEM , firstly passes through an image encoder ε(·) to obtain the encoded feature I = ε(I TEM ), a total of T steps are gradually added to the Gaussian noise and it becomes pure Gaussian noise I T , denoted as q(I t |I t-1 ,I0), the process of adding noise is as follows where α is used to gradually mix the data distribution into the noise distribution at each time step, t is the tth step, To represent the cumulative product of the form α, describing the cumulative noise scaling effect from time step t = 0 to T; Step S4.2: Constructing the inverse model through the Bayesian formula Among them, q(I t-1 |I t ,I0,F T ) represents the image I given the current time step t t , initial image I0 and dynamic blur kernel F T , calculate the image I at time step t-1 t-1 The posterior distribution of q(I t |I t-1 ,I0,F T ) means that given the previous step image I t-1 and the initial image I0, the image I at time step t t The probability distribution of q(I t-1 |I0,F T ) represents the image I directly from the initial image I0 to the image I at time step t-1 t-1 The probability distribution of q(I t |I0,F T ) represents the image I from the initial image I0 to the time step t t The probability distribution of , q(·|·) represents the probability distribution function; Step S4.3: Substitute the known quantity into each function of equation (15) to obtain equations (16), (17), (18) Among them~ represents normal distribution, the first · represents the mean, and the second · represents the variance; Step S4.4: Obtain probability distribution by calculation Among them, ∝ indicates proportionality, and C(·) indicates a constant.

Images