A medical image classification method based on prior knowledge guided dual-stream fusion network

Abstract

This invention discloses a medical image classification method based on a prior knowledge-guided dual-stream fusion network, belonging to the field of intelligent medical image analysis technology. The method includes: performing branch-specific preprocessing on the input raw medical image to generate semantic input images and structural input images adapted to different image branches; constructing a semantic alignment branch by extracting the global image semantic embedding of the semantic input image and calculating the classification output of the semantic alignment branch; constructing a structure-aware branch by extracting local structural features of the structural input image, constructing a category structural feature prototype matrix, and calculating the classification output of the structure-aware branch; and adaptively weighting and fusing the classification outputs of the semantic alignment branch and the structure-aware branch to obtain the final fused classification result. This invention overcomes the technical bias of existing technologies that only fuse at the decision level and cannot correct errors in the feature learning stage, significantly improving the pathological orientation and discriminative ability of model features in low-sample scenarios.

Description

Technical Field

[0001] This invention relates to the field of intelligent medical image analysis technology, and in particular to a medical image classification method based on a prior knowledge-guided dual-stream fusion network. Background Technology

[0002] Few-shot medical image classification (FSL) refers to the accurate identification of disease categories under the condition that each class of medical images has only a small number of labeled samples. It is one of the core research directions in the field of intelligent medical image analysis, and can effectively solve the industry pain points of scarce labeled data and high labeling costs in clinical scenarios. Existing few-shot medical image classification technologies are mainly divided into three categories: First, methods based on transfer learning, which use natural images or large-scale medical images to complete the pre-trained basic network for fine-tuning, but suffer from the problem of large differences between medical images and pre-training data domains and the tendency to cause negative transfer; Second, methods based on meta-learning, which learn transferable adaptation strategies across tasks, but suffer from high training costs, sensitivity to task construction, and insufficient stability across clinical scenarios; Third, methods based on vision-language pre-trained models, which use the large-scale semantic prior of CLIP vision-language pre-trained models to improve the generalization ability in few-shot scenarios, and are the current mainstream research direction.

[0003] In recent years, few-shot medical image classification schemes based on semantic-structural dual-stream architectures have been widely proposed. These schemes extract global disease semantic features through the CLIP semantic alignment branch and extract local texture and boundary features through the convolutional neural network structure perception branch. Finally, the outputs of the two branches are fused to achieve classification, effectively taking into account both global semantic priors and local structural details, and improving classification performance in few-shot scenarios to a certain extent.

[0004] However, existing dual-stream architecture solutions suffer from a deep-seated and unresolved core technical flaw: the semantic alignment branch and the structure-aware branch operate completely independently during the feature learning phase, fusing results only at the final decision layer, lacking effective training process guidance. Under extreme conditions of few-sample annotation, the structure-aware branch can only learn discriminative features from a very small number of samples, making it highly susceptible to misclassifying accidental patterns in samples that have no causal relationship with the disease, such as patient-specific anatomical variations, inherent noise from imaging equipment, and non-pathological background textures co-occurring with the disease, as discriminative features of the disease, thus learning a large number of spurious correlations.

[0005] This blindness in the feature learning stage leads to extremely poor generalization ability of features extracted by the structure-aware branch. Even with a weighted fusion mechanism at the decision layer, it is impossible to correct the erroneous results output by the structure-aware branch based on false features. Ultimately, this severely impairs the cross-domain robustness and hard-case discrimination ability of the dual-stream fusion network guided by the original prior knowledge. Currently, no existing technology has proposed an effective solution to this deep-seated technical problem, failing to fundamentally suppress the false feature learning of the structure-aware branch under few-shot conditions, making it difficult to achieve stable and reliable operation of few-shot medical image classification networks in cross-clinical applications. Summary of the Invention

[0006] Therefore, it is necessary to provide a medical image classification method based on a prior knowledge-guided dual-stream fusion network to address the aforementioned technical problems.

[0007] In a first aspect, the present invention provides a medical image classification method based on a prior knowledge-guided dual-stream fusion network, comprising: S1. Construct a medical image classification task for the support set and query set, and perform branch-specific preprocessing on the input raw medical image to generate semantic input image and structural input image adapted to different image branches respectively. S2. Construct a semantic alignment branch. By extracting the global image semantic embedding of the semantic input image, calculate the classification output of the semantic alignment branch. Based on the global image semantic embedding, generate a semantic saliency map that matches the spatial size of the semantic input image, as a semantic attention guide. S3. Construct a structure-aware branch by extracting local structural features from the input image, constructing a class structure feature prototype matrix, and calculating the classification output of the structure-aware branch; and apply purification constraints to the local structural features based on semantic attention guidance to construct a region feature separation loss. S4. Based on the uncertainty of the semantic alignment branch, the classification outputs of the semantic alignment branch and the structure-aware branch are adaptively weighted and fused to obtain the final fused classification result. S5. Based on the region feature separation loss and the fusion classification results, a joint total loss function is constructed to train the prior knowledge-guided dual-stream fusion network end-to-end, and the trained dual-stream fusion network is used for medical image classification tasks.

[0008] Furthermore, a medical image classification task is constructed for the support set and query set, and branch-specific preprocessing is performed on the input raw medical images to generate semantic input images and structural input images adapted to different image branches, including: S11. A medical image classification task is constructed using a dual-constraint paradigm of class sample size, and the medical image dataset is divided into a support set for model training and a query set for performance evaluation. S12. Normalize the size of the input raw medical images and scale all raw medical images to a preset uniform resolution. S13. Perform branch-specific decoupling normalization processing on the original medical image after size normalization; wherein, the mean and variance of the semantic input image are consistent with the pre-trained visual language alignment model and are normalized, and the mean and variance of the structural input image are normalized using the mean and variance consistent with the pre-trained structural perception branch.

[0009] Furthermore, a semantic alignment branch is constructed. By extracting the global image semantic embedding of the semantic input image, the classification output of the semantic alignment branch is calculated, including: S21. Construct a category hint text for each disease category, consisting of a learnable context token and a category name embedding, and encode the category hint text for all categories to obtain a category text embedding matrix; S22. Encode the semantic input image to obtain the global image semantic embedding, and normalize the global image semantic embedding and the category text embedding matrix. S23. Calculate the cosine similarity between the normalized global image semantic embedding and the category text embedding matrix, and calculate the classification output of the semantic alignment branch based on the similarity result. S24. Construct a symmetric contrast learning objective consisting of image-to-text contrast loss and text-to-image contrast loss, and achieve bidirectional alignment of global image semantic embedding and category text embedding matrices in the shared embedding space by optimizing the symmetric contrast learning objective.

[0010] Furthermore, based on the global image semantic embedding, a semantic saliency map matching the spatial size of the semantic input image is generated, serving as a semantic attention guide, including: S25. A lightweight, learnable attention derivation module is constructed using global image semantic embedding as the sole input. Upsampling and dimensionality transformation are completed through two-level cascaded transposed convolutional units, and the output is an intermediate feature map that perfectly matches the spatial size of the intermediate layer feature map of the structure-aware branch. S26. By using an activation function, each pixel value of the intermediate feature map is mapped to a numerical range of 0-1 to generate a semantic saliency map, which serves as a semantic attention guide; wherein, each spatial location of the semantic saliency map is mapped one-to-one with the corresponding spatial region of the semantic input image.

[0011] Furthermore, a structure-aware branch is constructed. By extracting local structural features from the structural input image, a class structure feature prototype matrix is ​​built. The classification output of the structure-aware branch is then calculated, including: S31. Construct an enhanced residual network that includes a deep convolutional stem, an anti-aliasing stride convolutional module, and an attention pooling module. Input the structural input image into the enhanced residual network to extract multi-scale local structural features and global structural feature vectors. S32. Based on the labeled samples in the support set, the global structural feature vectors of the corresponding samples are aggregated by mean according to category to obtain the structural feature prototype vector of each category, and the structural feature prototype vectors of all categories are combined to form the category structural feature prototype matrix. S33. Calculate the cosine similarity between the global structural feature vector of the sample to be classified and the category structural feature prototype matrix, and obtain the classification output of the structure-aware branch based on the similarity calculation result.

[0012] Furthermore, an enhanced residual network is constructed, comprising a deep convolutional stem, an anti-aliasing stride convolutional module, and an attention pooling module. The structural input image is then input into the enhanced residual network to extract multi-scale local structural features and global structural feature vectors, including: S311. A deep convolutional stem is constructed using three convolutional layers. Each convolutional layer is followed by a batch normalization layer and a ReLU activation layer. An average pooling layer is set at the end of the convolutional stem to complete downsampling. S312. Before all convolutional downsampling operations with a stride of 2, an average pooling layer is inserted for low-pass filtering, and a convolutional layer with a stride of 1 is used to complete the feature transformation, thus constructing an anti-aliasing cross-row stride convolutional module. S313. An attention pooling module is constructed using a multi-head attention module. The spatial feature map corresponding to the multi-scale local feature structure output by the enhanced residual network is flattened into a sequence, and then the global label and position embedding are concatenated and input into the multi-head attention module to aggregate and obtain the global structure feature vector.

[0013] Furthermore, based on the semantic attention guide, purification constraints are imposed on local structural features, and the region feature separation loss is constructed as follows: S34. Based on the numerical values ​​of the semantic saliency map, the spatial feature map corresponding to the multi-scale local structural features extracted by the structure perception branch is divided into regions according to semantic relevance to obtain high-relevance regions that are highly correlated with the semantic concept of the disease and low-relevance regions that are not highly correlated with the semantic concept of the disease. S35. For highly correlated regions, the structure perception branch is constrained by the first regularization term to enhance the inter-channel correlation, feature response intensity and expression diversity of multi-scale local structural features in highly correlated regions, so as to preserve pathologically relevant fine-grained structural information. S36. For low-relevance regions, the structure perception branch is constrained by the second regularization term to reduce the overall response intensity of multi-scale local structural features in low-relevance regions, and the inter-channel differences of multi-scale local structural features in low-relevance regions are constrained by the third regularization term to reduce the sensitivity of the structure perception branch to accidental patterns in semantically irrelevant regions. S37. The first regularization term, the second regularization term, and the third regularization term are weighted and summed according to preset weights to construct the regional feature separation loss as the feature purification loss.

[0014] Furthermore, based on the uncertainty of the semantic alignment branch, the classification outputs of the semantic alignment branch and the structure-aware branch are adaptively weighted and fused to obtain the final fused classification results, including: S41. The classification output of the semantic alignment branch is converted into a category probability distribution through a probability mapping function, and Shannon entropy is calculated based on the category probability distribution. Shannon entropy is used to quantify the prediction uncertainty of the semantic alignment branch. S42. Based on the calculated Shannon entropy, generate adaptive fusion weights. When the prediction uncertainty of the semantic alignment branch increases, increase the contribution ratio of the classification output of the structure-aware branch; when the prediction uncertainty of the semantic alignment branch decreases, increase the contribution ratio of the classification output of the semantic alignment branch. S43. Based on adaptive fusion weights, the classification outputs of the semantic alignment branch and the structure-aware branch are weighted and fused to obtain the final fusion classification result.

[0015] Furthermore, based on the region feature separation loss and the fusion classification results, a joint total loss function is constructed. This function is then used to train the prior knowledge-guided dual-stream fusion network end-to-end. The trained dual-stream fusion network is then used for medical image classification tasks, including: S51. The fusion classification loss is calculated based on the fusion classification result and the real label. The structure-aware branch auxiliary loss is calculated based on the classification output of the structure-aware branch and the real label. A joint total loss function consisting of the fusion classification loss, the structure-aware branch auxiliary loss and the feature purification loss is constructed. S52. Based on the joint total loss function, a pre-defined optimizer and learning rate scheduling strategy are used to perform end-to-end training on a dual-stream fusion network with feature purification guided by prior knowledge. S53. With the parameters of the feature-purified dual-stream fusion network guided by the prior knowledge after fixed training, the medical image to be classified is input into the dual-stream fusion network, and the final classification result is output to complete the few-sample medical image classification task.

[0016] Secondly, the present invention also provides a medical image classification system based on a prior knowledge-guided dual-stream fusion network, the system comprising: The medical image processing module is used to construct medical image classification tasks for support sets and query sets, and to perform branch-specific preprocessing on the input raw medical images to generate semantic input images and structural input images adapted to different image branches. The semantic branch construction module is used to construct semantic alignment branches. It calculates the classification output of semantic alignment branches by extracting the global image semantic embedding of the semantic input image; and generates a semantic saliency map that matches the spatial size of the semantic input image based on the global image semantic embedding, which serves as a semantic attention guide. The structure branch construction module is used to construct the structure-aware branch. It extracts local structural features from the structural input image, constructs a class structure feature prototype matrix, calculates the classification output of the structure-aware branch, and applies purification constraints to the local structural features according to the semantic attention guide to construct the region feature separation loss. The weighted fusion classification module is used to adaptively weight and fuse the classification outputs of the semantic alignment branch and the structure-aware branch based on the uncertainty of the semantic alignment branch, so as to obtain the final fusion classification result. The model training module is used to construct a joint total loss function based on the region feature separation loss and the fusion classification results, to perform end-to-end training of the prior knowledge-guided dual-stream fusion network, and to use the trained dual-stream fusion network for medical image classification tasks.

[0017] The beneficial effects of this invention are as follows: 1. This invention, while retaining the complementary advantages of the original dual-stream architecture's global semantic prior and local structural details, adds a semantically guided structural feature purification mechanism. This mechanism upgrades the semantic alignment branch from a participant in decision fusion to a guide in structural feature learning, suppressing the structural perception branch's learning of false and irrelevant patterns from the source of feature learning. This solves the technical bias of existing technologies that only fuse at the decision level and cannot correct errors in the feature learning stage, and significantly improves the pathological orientation and discriminative ability of model features in low-sample scenarios.

[0018] 2. By using feature purification constraints, structural features are precisely focused on pathology-related regions. Combined with the uncertainty-aware adaptive fusion mechanism of the original scheme, the decision contribution of the two stream branches can be dynamically balanced, effectively alleviating the misjudgment problem caused by the overconfidence of the semantic branch, and significantly improving the cross-domain robustness of the model under different clinical scenarios and different imaging devices. At the same time, the purification mechanism only takes effect during the training phase and has no additional computational overhead during the inference phase, which is fully adapted to the real-time deployment needs of the clinical end.

[0019] 3. It fully inherits the efficient prompting learning strategy of the original scheme parameters, without the need for large-scale fine-tuning of the pre-trained model backbone, and is perfectly adapted to the clinical scenario where medical image annotation data is scarce; at the same time, the generated semantic saliency map and purified structural features can be visualized, clearly showing the core areas of interest for model decision-making, providing clinicians with interpretable model decision-making basis, effectively making up for the lack of interpretability of existing schemes, and better meeting the clinical compliance and implementation requirements of medical AI. Attached Figure Description

[0020] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings: Figure 1 This is a flowchart of a medical image classification method based on a prior knowledge-guided dual-stream fusion network according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the enhanced residual network structure branches and attention pooling module in a medical image classification method based on prior knowledge-guided dual-stream fusion network according to an embodiment of the present invention. Figure 3 This is a schematic diagram of the overall framework of a prior knowledge-guided dual-stream fusion network in a medical image classification method based on prior knowledge-guided dual-stream fusion network according to an embodiment of the present invention. Figure 4 This is a block diagram illustrating the principle of a medical image classification system based on a prior knowledge-guided dual-stream fusion network according to an embodiment of the present invention.

[0021] The icons are labeled as follows: 1. Medical image processing module; 2. Semantic branch construction module; 3. Structural branch construction module; 4. Weighted fusion classification module; 5. Model training processing module. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0023] Please see Figure 1 This paper presents a medical image classification method based on a prior knowledge-guided dual-stream fusion network, including: S1. Construct a medical image classification task for the support set and query set, and perform branch-specific preprocessing on the input raw medical images to generate semantic input images and structural input images adapted to different image branches.

[0024] In the description of this invention, a medical image classification task is constructed for support sets and query sets, and branch-specific preprocessing is performed on the input raw medical images to generate semantic input images and structural input images adapted to different image branches, including: S11. A medical image classification task is constructed using a dual-constraint paradigm of class sample size, and the medical image dataset is divided into a support set for model training and a query set for performance evaluation.

[0025] Specifically, the dual-constraint paradigm for category sample size is the N-way K-shot paradigm, a standard in the few-shot learning domain, and is suitable for medical image classification scenarios where labeled samples are scarce. Specific implementation methods include: 1. Task Construction Object: The medical image dataset used includes X-ray image dataset, gastrointestinal endoscopy image dataset, and pathological tissue slide image dataset. This embodiment uses the MedFMC public dataset, which includes three types of medical image data: chest X-ray screening (ChestDR), colon pathology (Colon), and endoscopic lesions (Endo), and is pre-divided into training subset, validation subset, and test subset.

[0026] 2. Task Sampling Rules: Sampling is performed on a task-by-task basis. Each task contains N disease categories. K labeled samples are randomly selected from the training subset for each category to form the support set. Then, several samples are randomly selected from the corresponding categories to form the query set. The value of K can be set to 1, 5, or 10, which correspond to the three scenarios of very low supervision, low supervision, and medium supervision, respectively. K=1 means that only 1 labeled support sample is used for each category. 3. Training and Evaluation Rules: During the training phase, the model parameters are updated using the support set and query set. The model learns by repeatedly sampling multiple tasks to improve its ability to quickly adapt to unseen categories. During the evaluation phase, the trained model parameters are fixed, and multiple tasks are sampled from the test subset according to the same paradigm. The average classification performance after multiple samplings is used as the final evaluation result. For single-label classification tasks, classification accuracy (Acc) is used as the evaluation metric, and for multi-label classification tasks, mean average precision (mAP) is used as the evaluation metric.

[0027] It should be noted that during the training phase, each episode adopts a fixed 5-way 5-shot task construction rule, that is, each task contains 5 randomly sampled disease categories, 5 randomly selected labeled samples from each category to form the support set, and an additional 10 randomly selected samples from each category to form the query set; during the training process, 200 episodes are randomly sampled in each training round to complete the parameter update.

[0028] During the testing phase, the same 5-way K-shot task construction rule as the training phase is adopted, with K values ​​set to 1, 5, and 10 to correspond to different few-shot scenarios. During test inference, the class structure feature prototype matrix is ​​updated based on the support set samples of the current task, and the structure branch classification output is calculated using the updated prototype matrix. There is no need to update the model backbone parameters, thus achieving rapid adaptation.

[0029] S12. Normalize the size of the input raw medical images and scale all raw medical images to a preset uniform resolution.

[0030] Specifically, the original medical images are first scaled and cropped to a preset resolution of 224×224. For medical images that do not match the original size, zero-value padding and center cropping are used to ensure that all input images are of the same size and avoid distortion of the pathological content of the images, thus fully adapting to the input size requirements of the subsequent semantic alignment branch and structure-aware branch.

[0031] S13. Perform branch-specific decoupling normalization processing on the original medical image after size normalization; wherein, the mean and variance of the semantic input image are consistent with the pre-trained visual language alignment model and are normalized, and the mean and variance of the structural input image are normalized using the mean and variance consistent with the pre-trained structural perception branch.

[0032] Specifically, this invention employs a branch-decoupling normalization strategy, performing two independent normalization processes on the same normalized medical image, ensuring that the input data of each branch matches the feature distribution of its respective pre-training stage. This reduces feature drift caused by cross-domain offset and improves the numerical stability of the training process, as detailed below: 1. Semantic Input Image Normalization: Using the CLIP vision-language pre-training model (Contrastive Language-Image Pre-training, a multimodal pre-training model based on contrastive learning) and its corresponding ImageNet dataset, the mean and variance are fixed. The images after size normalization are then standardized using the following formula: ; In the formula, x represents the original medical image after size normalization. The mean of the images corresponding to CLIP pre-training. The standard deviation of the images corresponding to CLIP pre-training. The input image is the semantic image used to generate the final product.

[0033] 2. Input Image Normalization: Using the ImageNet dataset corresponding to the ResNet pre-training system with fixed mean and variance, the size-normalized image is standardized using the following formula: ; In the formula, The mean of the images corresponding to the ResNet pre-training. The standard deviation of the images corresponding to ResNet pre-training. Input the image to the final generated structure.

[0034] S2. Construct a semantic alignment branch. By extracting the global image semantic embedding of the semantic input image, calculate the classification output of the semantic alignment branch. Based on the global image semantic embedding, generate a semantic saliency map that matches the spatial size of the semantic input image, which serves as a semantic attention guide.

[0035] In the description of this invention, constructing a semantic alignment branch involves extracting the global image semantic embedding of the semantic input image and calculating the classification output of the semantic alignment branch, including: S21. Construct a category hint text for each disease category, consisting of a learnable context token and a category name embedding, and encode the category hint text for all categories to obtain a category text embedding matrix.

[0036] Specifically, this invention constructs a semantic alignment branch based on the open-source CLIPViT-B / 32 vision-language pre-trained model. During training, the backbone parameters of the CLIP model are frozen, and only the learnable context tokens and the parameters of subsequent newly added modules are updated, thereby achieving efficient parameter adaptation in the medical field.

[0037] During the construction of category hint text, corresponding category hint text is constructed for each disease category k. ,in to For a learnable context token, the value of m ranges from 4 to 16, and in this embodiment, m=8 is preferred; Embed the category name for this disease category as a fixed text vector.

[0038] The selection criteria for parameter m are as follows: when m < 4, the expressive power of the learnable context is insufficient, and it is impossible to adapt the general semantics to the semantics of the medical field; when m > 16, there are too many learnable parameters, which are prone to overfitting in scenarios with few samples; m = 8 is the optimal value that balances expressive power and the risk of overfitting, and can be adjusted in the range of 4-16 according to the number of task categories. The more categories there are, the greater the value of m can be.

[0039] The category hint texts for all categories are input into the text encoder of the CLIP model to obtain the text embedding vector for each category. After L2 normalization of all category text embedding vectors, they are stacked row by row to obtain the category text embedding matrix.

[0040] S22. Encode the semantic input image to obtain the global image semantic embedding, and normalize the global image semantic embedding and the category text embedding matrix.

[0041] Specifically, the semantic input image is input into the image encoder of the CLIP model to extract a fixed-dimensional global image semantic embedding. Then, L2 normalization is performed on the global image semantic embedding. The normalization formula is as follows: ; In the formula, This represents the image feature vector after L2 normalization; This represents the L2 norm (Euclidean norm), used to normalize eigenvectors; θ represents the CLIP image encoder, and θ is its learnable parameter.

[0042] The processed global image semantic embedding and the normalized category text embedding matrix are in the same shared embedding space, providing a unified metric for subsequent similarity calculations.

[0043] S23. Calculate the cosine similarity between the normalized global image semantic embedding and the category text embedding matrix, and calculate the classification output of the semantic alignment branch based on the similarity result.

[0044] Specifically, the cosine similarity between the normalized global image semantic embedding and the cosine similarity between the cosine similarity ... ; In the formula, The classification output (classification logits) for the semantic alignment branch. is the normalized category text embedding matrix, 100 is the preset scaling factor used to adjust the distribution range of similarity values ​​to adapt to the requirements of subsequent probability mapping and loss calculation; T is the transpose matrix symbol.

[0045] S24. Construct a symmetric contrast learning objective consisting of image-to-text contrast loss and text-to-image contrast loss, and achieve bidirectional alignment of global image semantic embedding and category text embedding matrices in the shared embedding space by optimizing the symmetric contrast learning objective.

[0046] Specifically, considering the characteristics of few-sample medical image classification tasks, both image-to-text contrast loss and text-to-image contrast loss are calculated using a class-level alignment method. The calculation logic and parameter definitions for the two losses are as follows: The image-to-text comparison loss uses the semantic embedding of a single medical image as the query object and the text embeddings of all disease categories as the matching target. The total number of samples in the training batch is used as the calculation benchmark. For each sample, the similarity between the sample's image semantic embedding and its true category text embedding is calculated. Then, the sum of the similarities between the sample's image semantic embedding and all category text embeddings is calculated. The single-sample loss is obtained by calculating the logarithmic probability. Finally, the average of the losses of all samples in the batch is taken to obtain the complete image-to-text comparison loss. Cosine similarity is used for similarity calculation. All similarity values ​​are scaled using a fixed temperature coefficient, ranging from 0.01 to 0.2. In this embodiment, a temperature coefficient of 0.07 is preferred to adjust the sharpness of the similarity distribution and enhance the distinguishability between categories.

[0047] The text-to-image contrast loss is the symmetric term of the image-to-text contrast loss. It takes the text embedding of a single disease category as the query object, the image semantic embedding of all samples in the training batch as the matching target, and the total number of disease categories in the task as the calculation benchmark. For each category, the similarity between the text embedding of the category and the image semantic embedding of the corresponding real sample is calculated. Then, the sum of the similarities between the text embedding of the category and the image semantic embedding of all samples is calculated. The single-class loss is obtained by calculating the log probability. Finally, the average of the losses of all categories in the task is taken to obtain the complete text-to-image contrast loss.

[0048] The image-to-text contrast loss and the text-to-image contrast loss are directly added together to obtain the overall objective of symmetric contrast learning. By optimizing this objective, two constraints can be achieved simultaneously: first, image embeddings of the same category and corresponding text embeddings of the same category are closest in the shared embedding space; second, text embeddings of the same category and corresponding image embeddings of the same category are closest in the shared embedding space. Ultimately, this achieves bidirectional alignment between the global image semantic embedding and the category text embedding matrix, enhancing the adaptability of the pre-trained semantic prior of general visual language to medical disease categories.

[0049] In the description of this invention, a semantic saliency map matching the spatial size of the semantic input image is generated based on the global image semantic embedding, serving as a semantic attention guide, including: S25. A lightweight, learnable attention derivation module is constructed using global image semantic embedding as the sole input. Upsampling and dimensionality transformation are completed through two-level cascaded transposed convolutional units, and the output is an intermediate feature map that perfectly matches the spatial size of the intermediate layer feature map of the structure-aware branch.

[0050] Specifically, the attention derivation module is a lightweight fully convolutional network with no fully connected layers. It consists of only two levels of cascaded transposed convolutional units. During training, only the learnable parameters of this module are updated. The only input is the global image semantic embedding obtained in S22. The input dimension is fixed at 512, which is consistent with the image embedding dimension output by the CLIPViT-B / 32 model. When inputting, the one-dimensional vector is first transformed into a 1×1×512 feature tensor through a reshape operation.

[0051] Each level of transposed convolutional unit consists of a transposed convolutional layer, a batch normalization layer, and a ReLU activation layer connected in series. 3. The first level of transposed convolutional unit uses a 4×4 transposed convolutional kernel with a stride of 2 and padding of 1. It has 512 input channels, 256 output channels, and an output feature map size of 2×2×256, completing the first dimensionality upsampling. The second level of transposed convolutional unit uses a 4×4 transposed convolutional kernel with a stride of 4 and padding of 0. It has 256 input channels, 1 output channel, and an output feature map size of 7×7×1, completing the second upsampling and channel adjustment.

[0052] The final output 7×7×1 intermediate feature map has a spatial size that perfectly matches the 7×7 spatial feature map output by the fourth layer of the ResNet50 backbone in the structure-aware branch, with each spatial position corresponding one-to-one; 5. Through learnable transpose convolution operation, the high-dimensional abstract global semantic concept is mapped to the region correlation features in the image space, providing a foundation for the subsequent semantic saliency map generation.

[0053] S26. By using an activation function, each pixel value of the intermediate feature map is mapped to a numerical range of 0-1 to generate a semantic saliency map, which serves as a semantic attention guide; wherein, each spatial location of the semantic saliency map is mapped one-to-one with the corresponding spatial region of the semantic input image.

[0054] Specifically, the Sigmoid function is used as the activation function to numerically map each pixel value of the intermediate feature map output by S25, generating a semantic saliency map of fixed size. After mapping, the value of each position is fixed in the range of 0-1.

[0055] Each spatial location in the semantic saliency map has a fixed-ratio downsampling-to-one mapping relationship with the corresponding spatial region of the original semantic input image. The value of each location in the semantic saliency map represents the degree of correlation between its corresponding input image region and the target disease semantic concept: the closer the value is to 1, the higher the correlation between the corresponding region and the disease semantic, and it belongs to the key region of suspected lesions; the closer the value is to 0, the lower the correlation between the corresponding region and the disease semantic, and it belongs to the irrelevant region such as normal tissue or background. Finally, the semantic saliency map is used as a semantic attention guide to provide the semantic guidance basis of spatial regions for the feature purification constraints of the subsequent structure perception branch.

[0056] S3. Construct a structure-aware branch by extracting local structural features from the input image, construct a class structure feature prototype matrix, calculate the classification output of the structure-aware branch, and apply purification constraints to the local structural features according to the semantic attention guide to construct the region feature separation loss.

[0057] In the description of this invention, constructing a structure-aware branch involves extracting local structural features from the structural input image, constructing a class structure feature prototype matrix, and calculating the classification output of the structure-aware branch, including: S31. Construct an enhanced residual network that includes a deep convolutional core, an anti-aliasing stride convolutional module, and an attention pooling module. Input the structural input image into the enhanced residual network to extract multi-scale local structural features and global structural feature vectors.

[0058] Specifically, such as Figures 2-3 As shown, the enhanced residual network is built on the ResNet50 backbone, fully preserving the four-level residual feature extraction structure of ResNet50, while being specifically enhanced through three core modules to adapt to the fine-grained structural feature extraction requirements of medical images. The structural input image generated by S13 is sequentially input into the deep convolutional backbone, the four-level residual block with anti-aliasing stride convolution module, and the attention pooling module to complete the feature extraction process; the spatial feature map output by the fourth layer in the four-level residual block is a multi-scale local structural feature, containing local texture, edge, and morphological details of the input image, used for subsequent feature purification constraints; the fixed-dimensional vector finally output by the attention pooling module is a global structural feature vector, used for subsequent category prototype construction and classification calculation.

[0059] Furthermore, it should be noted that the enhanced residual network is built on the ResNet50 backbone, fully retaining the four-level residual feature extraction structure of ResNet50. The specific configuration of the four-level residual blocks is as follows: layer 1 contains 3 Bottleneck residual blocks, with 256 output channels and an output feature map size of 56×56; layer 2 contains 4 Bottleneck residual blocks, with 512 output channels and an output feature map size of 28×28; layer 3 contains 6 Bottleneck residual blocks, with 1024 output channels and an output feature map size of 14×14; and layer 4 contains 3 Bottleneck residual blocks, with 2048 output channels and an output feature map size of 7×7.

[0060] The anti-aliasing stride convolution module is inserted in layers 1 to 4, before the first convolutional layer of all Bottleneck residual blocks that need to perform downsampling operations with a stride of 2. There are a total of 4 insertion positions, which correspond to the downsampling stages from layer 1 to layer 2, from layer 2 to layer 3, from layer 3 to layer 4, and the downsampling stage of the first residual block of layer 1.

[0061] In the description of this invention, an enhanced residual network is constructed, comprising a deep convolutional stem, an anti-aliasing stride convolutional module, and an attention pooling module. The structural input image is then input into the enhanced residual network to extract multi-scale local structural features and global structural feature vectors, including: S311. A deep convolutional stem is constructed using three convolutional layers. Each convolutional layer is followed by a batch normalization layer and a ReLU activation layer. An average pooling layer is set at the end of the convolutional stem to complete downsampling.

[0062] Specifically, the deep convolutional stem replaces the original ResNet50's single-layer 7×7 convolutional start structure, enhancing the ability to extract low-level texture and edge features from medical images. The convolutional layer configuration uses three 3×3 convolutional layers to extract features step-by-step, with output channels of 32, 32, and 64 respectively. Each convolutional layer is followed by a batch normalization layer and a ReLU activation layer. A 2×2 average pooling layer is placed at the end of the convolutional stem to complete the initial downsampling. The final output feature map is then input into the subsequent four-level residual blocks.

[0063] S312. Before all convolutional downsampling operations with a stride of 2, an average pooling layer is inserted for low-pass filtering, and a convolutional layer with a stride of 1 is used to complete the feature transformation, thus constructing an anti-aliasing cross-row stride convolutional module.

[0064] Specifically, the anti-aliasing stride convolution module is used to reduce aliasing distortion introduced during downsampling and improve the spatial consistency of structural features. The specific implementation is as follows: In the fourth-level residual block of the enhanced residual network, at all positions where a stride of 2 convolution downsampling operation needs to be performed, an average pooling layer with a stride of 2 is first inserted for low-pass filtering, and then a convolutional layer with a stride of 1 is used to complete feature transformation and channel adjustment. By using low-pass filtering first and then feature transformation, the spectral aliasing problem caused by direct downsampling of traditional stride convolution is eliminated, and fine-grained structural information such as lesion boundaries and textures in medical images are preserved.

[0065] S313. An attention pooling module is constructed using a multi-head attention module. The spatial feature map corresponding to the multi-scale local feature structure output by the enhanced residual network is flattened into a sequence, and then the global label and position embedding are concatenated and input into the multi-head attention module to aggregate and obtain the global structure feature vector.

[0066] Specifically, the attention pooling module replaces the traditional global average pooling layer of ResNet, dynamically aggregating features of diagnostically relevant regions, which better aligns with the clinical interpretation logic of "focusing on key lesion areas." The specific implementation method is as follows: 1. Flatten the multi-scale local structure feature map output from the fourth layer of the enhanced residual network into a two-dimensional feature sequence according to its spatial location; 2. A learnable global label is concatenated at the beginning of the feature sequence, and then a learnable positional embedding is superimposed to obtain the enhanced feature sequence X; 3. The enhanced sequence is input into a multi-head self-attention module for feature aggregation. The number of heads in the multi-head self-attention module is a positive integer, which can be set to 4, 8, 12, or 16. In this embodiment, the preferred number of heads is 8. The query, key, and value in the multi-head self-attention module are all obtained from the input sequence through independent linear projection. The calculation formula is as follows: Q=XW q K=XW k V=XW v ; Among them, W q W k W v The projection matrix is ​​a learnable matrix, with fixed projection dimensions for Q, K, and V. It is divided into multiple heads for parallel attention computation, and the dimension of a single head is the total dimension divided by the number of heads.

[0067] 4. Among the features output by the multi-head self-attention module, the feature vector corresponding to the global label is taken, and after linear mapping, a fixed-dimensional global structural feature vector is obtained, which is used for subsequent prototype construction and structure discrimination.

[0068] S32. Based on the labeled samples in the support set, the global structural feature vectors of the corresponding samples are aggregated by mean according to category to obtain the structural feature prototype vector of each category, and the structural feature prototype vectors of all categories are combined to form the category structural feature prototype matrix.

[0069] Specifically, based on the few-shot task support set of labeled samples, the global structural feature vectors are aggregated by category to construct a category structural feature prototype matrix. The formula for calculating the category prototype mean is as follows: ; in, For the set of supporting samples of category c, The number of supporting samples for category c. To support the global structural feature vector corresponding to sample i, Let be the structural feature prototype vector of category c.

[0070] Stack the structural feature prototype vectors of all categories in category order to obtain the category structural feature prototype matrix.

[0071] S33. Calculate the cosine similarity between the global structural feature vector of the sample to be classified and the category structural feature prototype matrix, and obtain the classification output of the structure-aware branch based on the similarity calculation result.

[0072] Specifically, firstly, the global structural feature vector of the sample to be classified and the prototype matrix of the category structural feature are subjected to L2 normalization. Then, the cosine similarity between the feature of the sample to be classified and the prototype vector of each category is calculated. All similarity results are multiplied by a fixed scaling factor consistent with the semantic alignment branch for numerical adjustment, and finally the classification output of the structure-aware branch is obtained.

[0073] In the description of this invention, based on the semantic attention guide, purification constraints are applied to local structural features, and the region feature separation loss is constructed as follows: S34. Based on the numerical values ​​of the semantic saliency map, the spatial feature map corresponding to the multi-scale local structural features extracted by the structure-aware branch is divided into regions according to semantic relevance to obtain high-relevance regions that are highly correlated with the semantic concept of the disease and low-relevance regions that are not highly correlated with the semantic concept of the disease.

[0074] Specifically, the region segmentation operation targets the spatial feature map corresponding to the multi-scale local structural features output by the fourth layer of the structure perception branch. Its spatial size is completely consistent with the spatial size of the semantic saliency map generated by S26, with each spatial location corresponding one-to-one. The specific implementation includes: setting a preset region segmentation threshold of 0.5, dividing spatial locations in the semantic saliency map with values ​​greater than 0.5 into high-relevance regions, corresponding to suspected lesion areas in the input image that are highly related to the disease semantic concept; dividing spatial locations in the semantic saliency map with values ​​less than or equal to 0.5 into low-relevance regions, corresponding to normal tissue, background, and other regions in the input image that are unrelated to the disease semantic; generating a binary mask matrix based on the segmentation results, and extracting the structural feature blocks corresponding to the high-relevance and low-relevance regions respectively for subsequent regularization constraints.

[0075] S35. For highly correlated regions, the structure perception branch is constrained by the first regularization term to enhance the inter-channel correlation, feature response intensity and expression diversity of multi-scale local structural features in highly correlated regions, so as to preserve pathologically relevant fine-grained structural information.

[0076] Specifically, the first regularization term is the feature channel covariance regularization term, which is used to encourage rich associations between feature channels in highly correlated regions, improve the diversity of feature expression, and avoid the loss of fine-grained pathological information in lesion regions. Specifically, it includes: calculating the covariance matrix of the structural feature blocks corresponding to highly correlated regions in the feature channel dimension; encouraging the off-diagonal elements of the covariance matrix to be non-zero values ​​through regularization constraints, thereby strengthening the correlation between feature channels and improving the diversity of feature expression in highly correlated regions; and simultaneously improving the overall feature response intensity of highly correlated regions through L2 regularization constraints, fully preserving the fine-grained pathological structural information of lesion regions.

[0077] S36. For low-relevance regions, the structure perception branch is constrained by the second regularization term to reduce the overall response intensity of multi-scale local structural features in low-relevance regions, and the inter-channel differences of multi-scale local structural features in low-relevance regions are constrained by the third regularization term to reduce the sensitivity of the structure perception branch to accidental patterns in semantically irrelevant regions.

[0078] Specifically, the second regularization term is an L1 sparsity regularization term, and the third regularization term is a feature channel decorrelation regularization term. Both work together on low-correlation regions, suppressing the structure-aware branch's learning of features from irrelevant regions from two dimensions: response intensity and feature diversity, thus preventing it from learning spurious correlations. Specifically, the second regularization term (sparse constraint) forces the feature values ​​of low-correlation regions to approach 0 as much as possible through L1 sparsity regularization, reducing the overall feature response intensity of the region, suppressing the structure-aware branch's activation of features in irrelevant regions, and reducing the interference of irrelevant regions on the final classification result. The third regularization term (decorrelation constraint) calculates the covariance matrix of the structural feature blocks corresponding to low-correlation regions in the feature channel dimension, and forces this covariance matrix to approach the identity matrix through regularization constraints, making the feature channels in low-correlation regions independent of each other and their feature expressions converge and become flat, eliminating spurious correlations between channels, completely blunting the sensitivity of the structure-aware branch to accidental patterns in irrelevant regions, and preventing the learning of spurious features unrelated to the disease.

[0079] S37. The first regularization term, the second regularization term, and the third regularization term are weighted and summed according to preset weights to construct the regional feature separation loss as the feature purification loss.

[0080] Specifically, the region feature separation loss (i.e. feature purification loss) is obtained by weighting and summing the first regularization term, the second regularization term, and the third regularization term according to preset fixed weight coefficients. The three weight coefficients can be adjusted according to the task scenario. In this embodiment, the weight of the first regularization term is preferably 1.0, the weight of the second regularization term is 0.5, and the weight of the third regularization term is 1.0. The region feature separation loss function only takes effect during the model training phase and does not need to be calculated during the inference phase, thus not increasing the computational cost of the model's inference. During training, the feature purification loss is incorporated into the model's joint total loss function for end-to-end optimization, achieving semantically guided structural feature purification and fundamentally preventing the structure-aware branch from learning false patterns unrelated to the disease.

[0081] S4. Based on the uncertainty of the semantic alignment branch, the classification outputs of the semantic alignment branch and the structure-aware branch are adaptively weighted and fused to obtain the final fused classification result.

[0082] In the description of this invention, based on the uncertainty of the semantic alignment branch, the classification outputs of the semantic alignment branch and the structure-aware branch are adaptively weighted and fused to obtain the final fused classification result, including: S41. The classification output of the semantic alignment branch is converted into a category probability distribution through a probability mapping function, and Shannon entropy is calculated based on the category probability distribution. Shannon entropy is used to quantify the prediction uncertainty of the semantic alignment branch.

[0083] Specifically, the classification output of the semantic alignment branch is first probabilistically mapped, and then the uncertainty of the prediction is quantified by Shannon entropy. The specific implementation method is as follows: 1. For single-label multi-class classification tasks, the softmax function is used as the probability mapping function, while for binary or multi-label classification tasks, the sigmoid function is used as the probability mapping function to convert the classification output of the semantic alignment branch into a class probability distribution. 2. Calculate the Shannon entropy based on the obtained category probability distribution. The calculation formula is as follows: ; in, Let be the predicted probability of the i-th category, and N be the total number of disease categories. To ensure that the entropy values ​​of tasks with different number of categories are comparable, the Shannon entropy is normalized to log2N.

[0084] The larger the Shannon entropy value, the more dispersed the prediction probability distribution of the semantic alignment branch, and the higher the prediction uncertainty; the smaller the Shannon entropy value, the more concentrated the prediction probability distribution of the semantic alignment branch, and the lower the prediction uncertainty.

[0085] S42. Based on the calculated Shannon entropy, generate adaptive fusion weights. When the prediction uncertainty of the semantic alignment branch increases, increase the contribution ratio of the classification output of the structure-aware branch; when the prediction uncertainty of the semantic alignment branch decreases, increase the contribution ratio of the classification output of the semantic alignment branch.

[0086] Specifically, the uncertainty weights generated based on the normalized Shannon entropy are calculated using the following formula: ; Where H is the Shannon entropy corresponding to the semantic branch prediction probability distribution, C is the total number of disease categories in the current classification task, and β is the uncertainty sensitivity parameter used to control the response strength of the weights to semantic uncertainty. In this embodiment, β=1.0 is preferred.

[0087] The weight generation logic is as follows: when the predicted probability distribution of the semantic alignment branch is more "flat", the Shannon entropy increases, and the uncertainty weight U increases accordingly, which means that the prediction reliability of the semantic alignment branch is lower. During fusion, the contribution ratio of the classification output of the structure perception branch is increased. When the predicted probability distribution of the semantic alignment branch is more concentrated, the Shannon entropy decreases, and the uncertainty weight U decreases accordingly, which means that the prediction reliability of the semantic alignment branch is higher. During fusion, the contribution ratio of the classification output of the semantic alignment branch is increased, thus achieving adaptive dynamic adjustment of the contribution of the two branches.

[0088] S43. Based on adaptive fusion weights, the classification outputs of the semantic alignment branch and the structure-aware branch are weighted and fused to obtain the final fusion classification result.

[0089] Specifically, an uncertainty-aware fusion strategy is adopted. After probability mapping of the classification outputs of the two branches, weighted fusion is completed according to adaptive fusion weights. The calculation formula is as follows: ; Where σ(⋅) is the probability mapping function, which is consistent with the probability mapping function in S41. For single-label multi-class tasks, it is softmax, and for binary / multi-label tasks, it is sigmoid; α is the structural branch contribution adjustment parameter, and in this embodiment, α=1.0 is preferred. This is the final fusion classification result; This fusion strategy can achieve the following: when the semantic alignment branch prediction uncertainty is high, the contribution of the structure perception branch is automatically amplified to reduce the risk of misjudgment caused by semantic overconfidence; when the semantic alignment branch prediction certainty is high, the cross-domain generalization advantage brought by visual language prior is fully preserved to achieve adaptive complementarity between the two branches.

[0090] S5. Based on the region feature separation loss and the fusion classification results, a joint total loss function is constructed to train the prior knowledge-guided dual-stream fusion network end-to-end, and the trained dual-stream fusion network is used for medical image classification tasks.

[0091] In the description of this invention, a joint total loss function is constructed based on the region feature separation loss and the fusion classification results. An end-to-end training process is then performed on the prior knowledge-guided dual-stream fusion network. The trained dual-stream fusion network is then used to perform a medical image classification task, including: S51. The fusion classification loss is calculated based on the fusion classification result and the real label. The structure-aware branch auxiliary loss is calculated based on the classification output of the structure-aware branch and the real label. A joint total loss function consisting of the fusion classification loss, the structure-aware branch auxiliary loss and the feature purification loss is constructed.

[0092] Specifically, the four components of the joint total loss function, as well as the complete end-to-end training rules, are explained in detail below: The first part is the fusion classification loss, which is the main loss during model training. It is calculated based on the final fusion classification result and the true labels of the samples, and its core function is to constrain the accuracy of the model's final classification result. For different classification task types, appropriate loss calculation methods are adopted: For single-label multi-class classification tasks, cross-entropy loss is used to calculate the fusion classification loss. Using the total number of samples in the training batch as a benchmark, for each sample and all categories, the logarithmic loss between the true label and the corresponding predicted probability is calculated, and the average loss of all samples is taken. For binary or multi-label classification tasks, binary cross-entropy loss is used to calculate the fusion classification loss. Using the product of the total number of samples and the total number of categories in the training batch as a benchmark, for each category of each sample, the logarithmic loss corresponding to the positive and negative labels is calculated separately, and the average of all loss terms is taken.

[0093] The second part is the structure-aware branch auxiliary loss, which is calculated based on the independent classification output of the structure-aware branch and the true label of the sample. The loss calculation method is completely matched with the fusion classification loss. Its core function is to separately constrain the independent discrimination ability of the structure-aware branch, avoid the problem of feature expression collapse caused by the structure-aware branch relying solely on the fusion path to update parameters during training, and ensure that the complementary characteristics of the semantic alignment branch and the structure-aware branch are always effective.

[0094] The third part is the feature purification loss, which is the region feature separation loss constructed in step S37. Its core function is to apply semantic guidance constraints to the feature learning process of the structure-aware branch during training, suppress the learning of false patterns in semantically irrelevant regions by the structure branch, realize the pathological orientation purification of structural features, and ensure that structural features always focus on key regions related to lesions.

[0095] The fourth part is the symmetric contrastive learning loss, which is the bidirectional symmetric contrastive learning objective constructed in step S24. Its core function is to constrain the cross-modal semantic alignment effect of the semantic alignment branch, ensuring that the general semantic prior of CLIP pre-training can be effectively adapted to the specific scenario of medical disease classification, and avoiding the degradation of semantic prior during fine-tuning.

[0096] Based on the above four losses, a joint total loss function is constructed for end-to-end training of the model. The total loss is calculated as follows: ; In the formula, λ, γ, and δ are the weight coefficients corresponding to the three auxiliary losses, used to balance the contribution of the four losses in the training process. The value range and preferred value of each parameter are as follows: λ is the weight coefficient of the structure-aware branch auxiliary loss, with a value range of 0.2 to 0.8, and the preferred value in this embodiment is 0.5; γ is the weight coefficient of the feature purification loss, with a value range of 0.1 to 0.5, and the preferred value in this embodiment is 0.3; δ is the weight coefficient of the symmetric contrastive learning loss, with a value range of 0.1 to 0.4, and the preferred value in this embodiment is 0.2.

[0097] Using the fusion classification loss as the base term, structure-aware branch auxiliary loss, feature purification loss, and symmetric contrastive learning loss are superimposed respectively. The three superimposed loss terms correspond to preset weight coefficients, and the contribution of the four losses in the training process is balanced by the weight coefficients.

[0098] During the complete training process of the model, the above four loss terms participate in gradient backpropagation simultaneously, jointly completing the end-to-end update of all learnable parameters of the model. There is no need for staged training, ensuring that the training logic of the four core links of semantic alignment, structural feature extraction, feature purification, and classification decision is completely closed-loop, without any problems of training objective conflict or logical gap.

[0099] S52. Based on the joint total loss function, a pre-defined optimizer and learning rate scheduling strategy are used to perform end-to-end training on a dual-stream fusion network with feature purification guided by prior knowledge.

[0100] Specifically, the model training implementation method is as follows: 1. The AdamW optimizer is used for parameter optimization, with a fixed weight decay coefficient. The core hyperparameters of the optimizer are set as follows: first moment coefficient β1=0.9, second moment coefficient β2=0.999, weight decay coefficient is 0.01, and numerical stability term ε=1e-8. Gradient clipping is set during training, and the maximum gradient norm is set to 1.0 to avoid gradient explosion.

[0101] 2. A learning rate preheating and cosine decay scheduling strategy is adopted. First, through a 5-round preheating phase, the learning rate is linearly increased from 0 to the preset base learning rate of 2×10−4. After the preheating is completed, the learning rate is gradually reduced through the cosine decay strategy. 3. The training batch size is set to 32, and the total number of training rounds is set to 50. During training, the backbone parameters of the CLIP vision-language pre-trained model are frozen, and only the trainable parameters of the learnable prompt text, attention inference module, and augmented residual network are updated to achieve efficient parameter training. 4. During training, monitor the classification performance of the model in real time on the validation set, and save the model weights with the best performance on the validation set as the final deployment weights.

[0102] S53. With the parameters of the feature-purified dual-stream fusion network guided by the prior knowledge after fixed training, the medical image to be classified is input into the dual-stream fusion network, and the final classification result is output to complete the few-sample medical image classification task.

[0103] Specifically, the implementation of model inference includes: fixing the trained model parameters, inputting the medical image to be classified into the model, and sequentially performing branch-specific preprocessing, semantic alignment branch feature extraction, structure-aware branch feature extraction, and uncertainty-aware fusion steps, ultimately outputting the disease category prediction result; during the model inference stage, there is no need to perform semantic saliency map generation and feature purification loss calculation steps, and the computational load of the inference process is completely consistent with the original two-stream fusion network, with no additional computational overhead, adapting to the real-time requirements of clinical deployment; during the inference process, the visualization results of semantic saliency map and structure-aware branch feature interest regions can be output simultaneously, clearly showing the interest regions of the model's decision, providing interpretable decision-making basis for clinical diagnosis, and meeting the clinical compliance requirements of medical AI.

[0104] Please see Figure 4 Furthermore, a medical image classification system based on a prior knowledge-guided dual-stream fusion network is provided, the system comprising: The medical image processing module 1 is used to construct a medical image classification task for support sets and query sets, and to perform branch-specific preprocessing on the input raw medical images to generate semantic input images and structural input images adapted to different image branches. Semantic branch construction module 2 is used to construct semantic alignment branches. It calculates the classification output of semantic alignment branches by extracting the global image semantic embedding of the semantic input image; and generates a semantic saliency map that matches the spatial size of the semantic input image based on the global image semantic embedding, which serves as a semantic attention guide. The structure branch construction module 3 is used to construct the structure-aware branch. It extracts local structural features from the structural input image, constructs a class structure feature prototype matrix, calculates the classification output of the structure-aware branch, and applies purification constraints to the local structural features according to the semantic attention guide to construct the region feature separation loss. The weighted fusion classification module 4 is used to adaptively weight and fuse the classification outputs of the semantic alignment branch and the structure-aware branch based on the uncertainty of the semantic alignment branch, so as to obtain the final fusion classification result. Model training and processing module 5 is used to construct a joint total loss function based on the region feature separation loss and the fusion classification results, to perform end-to-end training of the prior knowledge-guided dual-stream fusion network, and to use the trained dual-stream fusion network for medical image classification tasks.

[0105] Table 1. Performance of different methods on the few-shot classification dataset of MedFMC.

[0106] As shown in Table 1, the method of the present invention has been validated on three types of medical image tasks in the MedFMC public dataset. The three types of tasks are ChestDR (chest X-ray disease screening), Endo (gastrointestinal endoscopy image lesion classification), and Colon (colon pathology tissue section tumor classification), covering three typical few-shot supervised scenarios: 1-shot, 5-shot, and 10-shot.

[0107] The comparative methods selected in this experiment include: Full (unfreezing the entire backbone network for end-to-end parameter tuning), Linear (freezing the backbone network and training only the top classifier head as an efficient baseline for parameters), MLP-3 (an adaptation method that inserts 3 lightweight MLP modules before the classifier head), VP (a cue learning method that directly adds learnable cue vectors to the input image channels), VPT (a traditional visual cue tuning method that freezes the backbone and tunes only the learnable cue for the input token sequence), and EPT (an advanced cue tuning method that embeds learnable cue within the extended dimension of the attention mechanism channel).

[0108] Experimental results show that the method of this invention achieves excellent classification performance in all few-shot scenarios of the three datasets, with an average performance of 45.16%, which is 20.49 percentage points higher than the Linear baseline and 5.34 percentage points higher than the current state-of-the-art EPT method. In particular, in the endoscopic lesion classification task, the method of this invention achieves a doubling of performance in 1-shot, 5-shot, and 10-shot scenarios. The mAP reaches 33.36% in the 1-shot scenario and 89.09% in the 10-shot scenario for colon pathology classification, far exceeding various comparative methods.

[0109] Figure 2Input[N,3,224,224] represents the batch of medical images input, where N is the batch size, 3 is the number of channels, and 224×224 is the input resolution. `stem` is a deep convolutional stem module, consisting of three 3×3 convolutional layers, a batch normalization layer, a ReLU activation layer, and an average pooling layer connected in series. Layers 1 through 4 are four-level residual block structures, each composed of Bottleneck units connected in series. Each Bottleneck unit consists of 1×1, 3×3, and 1×1 convolutions connected in series. The main branch and shortcut branch are connected by adding residuals. Before all downsampling operations with a stride of 2, an anti-aliasing cross-stride convolutional module is inserted to reduce downsampling aliasing distortion by first performing average pooling low-pass filtering and then stride 1 convolution. `AttentionPool2d` is an attention pooling module used to aggregate spatial feature maps into global structural feature vectors. Feature extraction is completed through feature flattening, concatenating global labels, overlaying position embedding, and multi-head self-attention aggregation.

[0110] Figure 3 This is a schematic diagram of the overall framework of the prior knowledge-guided dual-stream fusion network of this invention. The left branch is the semantic alignment branch (Few-shot CLIP), built based on the CLIP vision-language pre-trained model. It obtains the category text embedding by encoding the category prompt text through a text encoder, and obtains the global image semantic embedding by encoding the input medical image through an image encoder, ultimately outputting the semantic branch classification logits Z0. The right branch is the structure-aware branch (Modified ResNet), also known as the enhanced residual network in the specification. It extracts local structural features through deep convolutional stems, four-level residual blocks, and attention pooling modules, ultimately outputting the structure branch classification logits. Z1; The global image semantic embedding output of the semantic alignment branch is used to generate a semantic saliency map through the attention inference module. This map serves as a semantic attention guide, applying purification constraints to the local structural features of the structure-aware branch to achieve semantically guided structural feature purification. The fusion module generates adaptive fusion weights based on the prediction uncertainty of the semantic alignment branch, and performs weighted fusion of the classification outputs of the semantic and structural branches to output the final classification result. During training, the model is optimized end-to-end by fusing the classification loss, the structure-aware branch auxiliary loss, the feature purification loss, and the symmetric contrastive learning loss into a joint total loss function.

[0111] This invention, through the collaborative design of learnable semantic prompting, structure-enhanced residual networks, uncertainty-aware adaptive fusion, and a newly added semantically guided structural feature purification mechanism, solves the core problem of easy learning of false relevance in structure-aware branches in low-sample scenarios where labeled samples are extremely scarce. It also fully leverages the complementary advantages of global semantic priors and local structural details, significantly improving the model's classification accuracy, cross-domain generalization ability, and training stability, thus verifying the effectiveness and superiority of the technical solution of this invention.

[0112] In summary, by leveraging the above-mentioned technical solution of this invention, while retaining the complementary advantages of the original dual-stream architecture's global semantic prior and local structural details, this invention adds a semantically guided structural feature purification mechanism. This upgrades the semantic alignment branch from a participant in decision fusion to a guide in structural feature learning, suppressing the learning of spurious and irrelevant patterns by the structural awareness branch from the source of feature learning. This solves the technical bias of existing technologies that only fuse at the decision level and cannot correct errors in the feature learning stage, significantly improving the pathological orientation and discriminative ability of model features in low-sample scenarios. Through feature purification constraints, structural features accurately focus on pathologically relevant regions. Combined with the uncertainty-aware adaptive fusion mechanism of the original scheme, the decision contributions of the dual-stream branches can be dynamically balanced, effectively mitigating the misjudgment problem caused by the overconfidence of the semantic branch, and greatly improving the model's cross-domain robustness under different clinical scenarios and imaging devices. Furthermore, the purification mechanism only takes effect during the training phase, with no additional computational overhead during the inference phase, fully adapting to the real-time deployment needs of clinical settings. It fully inherits the efficient prompting learning strategy of the original scheme parameters, without the need for large-scale fine-tuning of the pre-trained model backbone, and is perfectly adapted to clinical scenarios where medical image annotation data is scarce; at the same time, the generated semantic saliency map and purified structural features can be visualized, clearly showing the core areas of interest for model decision-making, providing clinicians with interpretable model decision-making basis, effectively making up for the lack of interpretability of existing schemes, and is more in line with the clinical compliance and implementation requirements of medical AI.

[0113] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

Claims

1. A medical image classification method based on a prior knowledge-guided dual-stream fusion network, characterized in that, include: S1. Construct a medical image classification task for the support set and query set, and perform branch-specific preprocessing on the input raw medical image to generate semantic input image and structural input image adapted to different image branches respectively. S2. Construct a semantic alignment branch. By extracting the global image semantic embedding of the semantic input image, calculate the classification output of the semantic alignment branch. Based on the global image semantic embedding, generate a semantic saliency map that matches the spatial size of the semantic input image, as a semantic attention guide. S3. Construct a structure-aware branch by extracting local structural features from the input image, constructing a class structure feature prototype matrix, and calculating the classification output of the structure-aware branch; and apply purification constraints to the local structural features based on semantic attention guidance to construct a region feature separation loss. S4. Based on the uncertainty of the semantic alignment branch, the classification outputs of the semantic alignment branch and the structure-aware branch are adaptively weighted and fused to obtain the final fused classification result. S5. Based on the region feature separation loss and the fusion classification results, a joint total loss function is constructed to train the prior knowledge-guided dual-stream fusion network end-to-end, and the trained dual-stream fusion network is used for medical image classification tasks.

2. The medical image classification method based on a prior knowledge-guided dual-stream fusion network according to claim 1, characterized in that, The construction of a medical image classification task for supporting sets and query sets, and the branch-specific preprocessing of the input raw medical images to generate semantic input images and structural input images adapted to different image branches, includes: S11. A medical image classification task is constructed using a dual-constraint paradigm of class sample size, and the medical image dataset is divided into a support set for model training and a query set for performance evaluation. S12. Normalize the size of the input raw medical images and scale all raw medical images to a preset uniform resolution. S13. Perform branch-specific decoupling normalization processing on the original medical image after size normalization; wherein, the mean and variance of the semantic input image are consistent with the pre-trained visual language alignment model and are normalized, and the mean and variance of the structural input image are normalized using the mean and variance consistent with the pre-trained structural perception branch.

3. The medical image classification method based on a prior knowledge-guided dual-stream fusion network according to claim 1, characterized in that, The construction of the semantic alignment branch, by extracting the global image semantic embedding of the semantic input image and calculating the classification output of the semantic alignment branch, includes: S21. Construct a category hint text for each disease category, consisting of a learnable context token and a category name embedding, and encode the category hint text for all categories to obtain a category text embedding matrix; S22. Encode the semantic input image to obtain the global image semantic embedding, and normalize the global image semantic embedding and the category text embedding matrix. S23. Calculate the cosine similarity between the normalized global image semantic embedding and the category text embedding matrix, and calculate the classification output of the semantic alignment branch based on the similarity result. S24. Construct a symmetric contrast learning objective consisting of image-to-text contrast loss and text-to-image contrast loss, and achieve bidirectional alignment of global image semantic embedding and category text embedding matrices in the shared embedding space by optimizing the symmetric contrast learning objective.

4. The medical image classification method based on a prior knowledge-guided dual-stream fusion network according to claim 1, characterized in that, The step of generating a semantic saliency map that matches the spatial size of the semantic input image based on the global image semantic embedding, as a semantic attention guide, includes: S25. A lightweight, learnable attention derivation module is constructed using global image semantic embedding as the sole input. Upsampling and dimensionality transformation are completed through two-level cascaded transposed convolutional units, and the output is an intermediate feature map that perfectly matches the spatial size of the intermediate layer feature map of the structure-aware branch. S26. By using an activation function, each pixel value of the intermediate feature map is mapped to a numerical range of 0-1 to generate a semantic saliency map, which serves as a semantic attention guide; wherein, each spatial location of the semantic saliency map is mapped one-to-one with the corresponding spatial region of the semantic input image.

5. The medical image classification method based on a prior knowledge-guided dual-stream fusion network according to claim 1, characterized in that, The construction of the structure-aware branch involves extracting local structural features from the input image, constructing a class structure feature prototype matrix, and calculating the classification output of the structure-aware branch, including: S31. Construct an enhanced residual network that includes a deep convolutional stem, an anti-aliasing stride convolutional module, and an attention pooling module. Input the structural input image into the enhanced residual network to extract multi-scale local structural features and global structural feature vectors. S32. Based on the labeled samples in the support set, the global structural feature vectors of the corresponding samples are aggregated by mean according to category to obtain the structural feature prototype vector of each category, and the structural feature prototype vectors of all categories are combined to form the category structural feature prototype matrix. S33. Calculate the cosine similarity between the global structural feature vector of the sample to be classified and the category structural feature prototype matrix, and obtain the classification output of the structure-aware branch based on the similarity calculation result.

6. The medical image classification method based on a prior knowledge-guided dual-stream fusion network according to claim 1, characterized in that, The construction includes an enhanced residual network comprising a deep convolutional stem, an anti-aliasing stride convolutional module, and an attention pooling module. The structural input image is then fed into the enhanced residual network to extract multi-scale local structural features and global structural feature vectors, including: S311. A deep convolutional stem is constructed using three convolutional layers. Each convolutional layer is followed by a batch normalization layer and a ReLU activation layer. An average pooling layer is set at the end of the convolutional stem to complete downsampling. S312. Before all convolutional downsampling operations with a stride of 2, an average pooling layer is inserted for low-pass filtering, and a convolutional layer with a stride of 1 is used to complete the feature transformation, thus constructing an anti-aliasing cross-row stride convolutional module. S313. An attention pooling module is constructed using a multi-head attention module. The spatial feature map corresponding to the multi-scale local feature structure output by the enhanced residual network is flattened into a sequence, and then the global label and position embedding are concatenated and input into the multi-head attention module to aggregate and obtain the global structure feature vector.

7. The medical image classification method based on a prior knowledge-guided dual-stream fusion network according to claim 1, characterized in that, The method of applying purification constraints to local structural features based on semantic attention guidance to construct a region feature separation loss includes: S34. Based on the numerical values ​​of the semantic saliency map, the spatial feature map corresponding to the multi-scale local structural features extracted by the structure perception branch is divided into regions according to semantic relevance to obtain high-relevance regions that are highly correlated with the semantic concept of the disease and low-relevance regions that are not highly correlated with the semantic concept of the disease. S35. For highly correlated regions, the structure perception branch is constrained by the first regularization term to enhance the inter-channel correlation, feature response intensity and expression diversity of multi-scale local structural features in highly correlated regions, so as to preserve pathologically relevant fine-grained structural information. S36. For low-relevance regions, the structure perception branch is constrained by the second regularization term to reduce the overall response intensity of multi-scale local structural features in low-relevance regions, and the inter-channel differences of multi-scale local structural features in low-relevance regions are constrained by the third regularization term to reduce the sensitivity of the structure perception branch to accidental patterns in semantically irrelevant regions. S37. The first regularization term, the second regularization term, and the third regularization term are weighted and summed according to preset weights to construct the regional feature separation loss as the feature purification loss.

8. The medical image classification method based on a prior knowledge-guided dual-stream fusion network according to claim 1, characterized in that, The uncertainty of the semantic alignment branch is addressed by adaptively weighting and fusing the classification outputs of the semantic alignment branch and the structure-aware branch to obtain the final fused classification result, which includes: S41. The classification output of the semantic alignment branch is converted into a category probability distribution through a probability mapping function, and Shannon entropy is calculated based on the category probability distribution. Shannon entropy is used to quantify the prediction uncertainty of the semantic alignment branch. S42. Based on the calculated Shannon entropy, generate adaptive fusion weights. When the prediction uncertainty of the semantic alignment branch increases, increase the contribution ratio of the classification output of the structure-aware branch; when the prediction uncertainty of the semantic alignment branch decreases, increase the contribution ratio of the classification output of the semantic alignment branch. S43. Based on adaptive fusion weights, the classification outputs of the semantic alignment branch and the structure-aware branch are weighted and fused to obtain the final fusion classification result.

9. The medical image classification method based on a prior knowledge-guided dual-stream fusion network according to claim 1, characterized in that, The method involves constructing a joint total loss function based on region feature separation loss and fusion classification results, performing end-to-end training on a prior knowledge-guided dual-stream fusion network, and then using the trained dual-stream fusion network for medical image classification tasks, including: S51. The fusion classification loss is calculated based on the fusion classification result and the real label. The structure-aware branch auxiliary loss is calculated based on the classification output of the structure-aware branch and the real label. A joint total loss function consisting of the fusion classification loss, the structure-aware branch auxiliary loss and the feature purification loss is constructed. S52. Based on the joint total loss function, a pre-defined optimizer and learning rate scheduling strategy are used to perform end-to-end training on a dual-stream fusion network with feature purification guided by prior knowledge. S53. With the parameters of the feature-purified dual-stream fusion network guided by the prior knowledge after fixed training, the medical image to be classified is input into the dual-stream fusion network, and the final classification result is output to complete the few-sample medical image classification task.

10. A medical image classification system based on a prior knowledge-guided dual-stream fusion network, used to implement the medical image classification method based on a prior knowledge-guided dual-stream fusion network as described in any one of claims 1-9, characterized in that, The system includes: The medical image processing module is used to construct medical image classification tasks for support sets and query sets, and to perform branch-specific preprocessing on the input raw medical images to generate semantic input images and structural input images adapted to different image branches. The semantic branch construction module is used to construct semantic alignment branches. It calculates the classification output of semantic alignment branches by extracting the global image semantic embedding of the semantic input image; and generates a semantic saliency map that matches the spatial size of the semantic input image based on the global image semantic embedding, which serves as a semantic attention guide. The structure branch construction module is used to construct the structure-aware branch. It extracts local structural features from the structural input image, constructs a class structure feature prototype matrix, calculates the classification output of the structure-aware branch, and applies purification constraints to the local structural features according to the semantic attention guide to construct the region feature separation loss. The weighted fusion classification module is used to adaptively weight and fuse the classification outputs of the semantic alignment branch and the structure-aware branch based on the uncertainty of the semantic alignment branch, so as to obtain the final fusion classification result. The model training module is used to construct a joint total loss function based on the region feature separation loss and the fusion classification results, to perform end-to-end training of the prior knowledge-guided dual-stream fusion network, and to use the trained dual-stream fusion network for medical image classification tasks.

Images