Medical image classification method and system based on cross-modal knowledge distillation and medium

Abstract

The application discloses a medical image classification method and system based on cross-modal knowledge distillation and a medium, relates to the technical field of medical image processing, and comprises the following steps: obtaining a training sample set; a first branch extracts global semantic features of high-quality modality images and classifies the global semantic features; a second branch extracts slice-level features of low-quality modality images; the global semantic features are used as queries, the slice-level features are used as keys, a teacher distribution is generated through cross-modal attention, fusion features are obtained by weighted aggregation of the slice-level features, and the fusion features are classified; low-quality modality images are used as input, multi-scale features are extracted and fused into spatial fusion feature representations, a student distribution is generated through a self-attention module, aggregated features are obtained by weighted aggregation of the spatial fusion feature representations, and the aggregated features are classified; the method effectively improves the classification performance of low-quality modality by using distillation, effectively transfers knowledge from high-quality modality, and is free from strict spatial registration in mechanism and overcomes interference caused by unregistered data.

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, and in particular to a medical image classification method, system, and medium based on cross-modal knowledge distillation. Background Technology

[0002] In medical research and clinical diagnosis, medical images of different modalities often contain specific anatomical structures or lesion information, such as tumor location, edema area, and lesion details. Among them, high-quality modalities such as magnetic resonance imaging (MRI), with its excellent soft tissue resolution and rich anatomical information, are important bases for auxiliary diagnosis. However, in real and resource-constrained clinical environments, low-quality medical imaging modalities such as computed tomography (CT) often become the most commonly used medical imaging modalities due to their advantages of fast acquisition and high accessibility. Because of their inherent limited soft tissue contrast and resolution, relying solely on these low-quality modalities for complex medical image classification (such as pediatric brain tumor classification) can easily lead to the loss of some detailed structural edge details, and the classification accuracy often fails to meet clinical diagnostic requirements. Therefore, combining different sequence modalities to extract complementary information or transfer knowledge has become an urgent problem to be solved.

[0003] In recent years, with the development of deep learning, Cross-Modal Knowledge Distillation (CMKD) has provided an effective end-to-end solution to the problem of poor image classification accuracy in resource-constrained scenarios. This approach aims to transfer diagnostic prior knowledge from high-quality modalities (such as MRI) to low-quality modalities during the training phase, thereby achieving higher accuracy using only the low-quality modalities in actual deployment. However, this transfer faces two major obstacles in real-world clinical applications.

[0004] One challenge is spatial misregistration. Due to differences in patient poses and variations in slice thickness and spacing, it is difficult to achieve voxel-level geometric correspondence (i.e., spatial misregistration) between images of different modalities. Many existing distillation methods that rely on registration force alignment on weakly corresponding or completely misregistered data, which can easily lead to model optimization failures and result in information loss and accuracy degradation when transferring features.

[0005] Second, there is the negative transfer crisis caused by differences in cross-modal representation. Due to the different physical imaging principles, rigid matching of heterogeneous feature spaces will not only amplify modality-specific noise, but also lose the true intermediate-level structural prior indications that are valuable for diagnosis and discrimination, leading to performance degradation. Summary of the Invention

[0006] Based on the technical problems existing in the background technology, this invention proposes a medical image classification method, system and medium based on cross-modal knowledge distillation. It effectively improves the classification performance of low-quality modalities by distillation, effectively transfers knowledge from high-quality modalities, and can also break away from the constraints of strict spatial registration from the mechanism, overcoming the interference caused by unregistered data.

[0007] The medical image classification method based on cross-modal knowledge distillation proposed in this invention includes: Obtain a training sample set, where each sample contains a high-quality modal image, a low-quality modal image, and a category label; A two-branch teacher network: the first branch extracts global semantic features from high-quality modal images and classifies them; the second branch extracts slice-level features from low-quality modal images; using the global semantic features as queries and slice-level features as keys, a teacher distribution is generated through cross-modal attention, and the slice-level features are weighted and aggregated to obtain fused features and then classified. Student Network: Taking low-quality modal images as input, multi-scale features are extracted and fused into a spatial fusion feature representation. The student distribution is generated through a self-attention module, and the spatial fusion feature representation is weighted and aggregated to obtain aggregated features and classified. The first stage trains the teacher network to ensure that the bi-branch classification results are consistent with the classification labels. The second stage freezes the teacher network, constrains the student distribution to be consistent with the teacher distribution, and trains the student network using the combined classification loss.

[0008] Furthermore, the high-quality and low-quality modes in each sample are not spatially consistent.

[0009] Furthermore, the extraction and fusion of multi-scale features into a spatial fusion feature representation, the generation of student distributions via a self-attention module, and the weighted aggregation of the spatial fusion feature representations to obtain aggregated features and classification are specifically as follows: The extracted intermediate features at different scales are pooled in spatial dimensions H and W respectively. After concatenating the two pooled features, linear projection is performed to obtain the pooled fusion dimensionality reduction features, where H and W are the height and width of each slice in the low-quality modality image. All pooled and dimensionality-reduced features are concatenated and fused along the channel dimension, and a learnable spatial location code is superimposed to obtain a spatial fusion feature representation. Linear mapping is performed on the spatial fusion feature representation to obtain the query and key, the dot product is calculated to obtain the autocorrelation matrix, the one-dimensional self-attention pooling score is compressed along the column direction, and the student distribution is obtained by Softmax normalization. The spatial fusion feature representation is weighted and aggregated using the student distribution to obtain the aggregated features of the student network; The aggregated features are mapped through a linear classifier to obtain the classification prediction values ​​of the student network.

[0010] Furthermore, the teacher distribution generated via cross-modal attention is specifically as follows: The query vector is obtained by linearly mapping the global semantic features, and the key vector is obtained by linearly mapping the slice-level features. After calculating the scaling dot product correlation score, the teacher distribution is obtained by Softmax normalization; The teacher distribution is weighted and aggregated with the slice-level features to obtain the fused features of the second branch; The global semantic features are input into the first branch classification head to obtain the first classification logical value. ; The fused features are input into the second branch classification head to obtain the second classification logical value. .

[0011] Furthermore, the first stage trains the teacher network to ensure that the bi-branch classification results are consistent with the classification labels, specifically as follows: Calculate the average classification loss in the first stage. : ; ; ; in, The classification loss for the first branch, The classification loss for the second branch, Let cross-entropy be the loss function. For category labels.

[0012] Furthermore, in the second stage, the specific way to ensure that the student distribution is consistent with the teacher distribution is as follows: a temperature parameter is introduced to soften the teacher and student distributions, and the KL divergence of temperature scaling is calculated as the cross-modal distillation loss.

[0013] Furthermore, the cross-modal distillation loss The formula is as follows: ; in, For temperature parameters, The number of slices for low-quality modal images. For slice index, The first in the softened teacher distribution The weights of each slice, The softened student distribution The weights of each slice.

[0014] Furthermore, the student network execution process is as follows: low-quality modal images are used as input to the trained student network, and the output is the classification prediction result.

[0015] A computer system includes a memory, a processor, and a computer program stored in the memory, characterized in that the processor executes the computer program to implement the method described above.

[0016] A computer-readable storage medium storing a plurality of computer programs for being invoked by a processor and executing the method described above.

[0017] The advantages of the medical image classification method, system, and medium based on cross-modal knowledge distillation provided by this invention are: (1) Breaking through spatial registration limitations and adapting to real clinical scenarios: Using one-dimensional cross-modal slice attention, only the higher-order knowledge of slice importance distribution is transferred, without the need for hard alignment in three-dimensional space, effectively solving the knowledge transfer problem under unregistered or weakly registered data; (2) Two-stage distillation framework: The first stage jointly trains the teacher network (high-quality modality + low-quality modality), enabling the teacher network to learn to use the prior of high-quality modality to guide the slice importance assessment of low-quality modality; The second stage uses temperature-scaled KL Divergence distills the teacher distribution to the student network, and the student network can obtain classification performance close to multimodal fusion by relying only on low-quality modalities; (3) It alleviates the negative transfer caused by cross-modal differences: the distillation is the slice-level attention distribution, which belongs to the structural prior that is more valuable for diagnosis, rather than the low-level features, thereby reducing the risk of negative transfer and improving the stability and generalization ability of distillation; (4) After training, only a lightweight student network is deployed. When reasoning, only low-quality modalities (such as CT) need to be input, without the need to store or preprocess high-quality modalities, which reduces hardware costs and computational burden, and is particularly suitable for resource-constrained scenarios such as primary hospitals and emergency departments. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the structural process of the present invention; Figure 2 A schematic diagram of the cross-modal attention mechanism structure of a two-branch teacher network; Figure 3 A schematic diagram of the multi-scale self-attention mechanism structure of a student network. Detailed Implementation

[0019] The technical solution of the present invention will now be described in detail through specific embodiments. Many specific details are set forth in the following description to provide a thorough understanding of the invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0020] like Figures 1 to 3As shown, the medical image classification method based on cross-modal knowledge distillation proposed in this invention includes: Obtain a training sample set, where each sample contains a high-quality modal image, a low-quality modal image, and a category label; A two-branch teacher network: the first branch extracts global semantic features from high-quality modal images and classifies them; the second branch extracts slice-level features from low-quality modal images; using the global semantic features as queries and slice-level features as keys, a teacher distribution is generated through cross-modal attention, and the slice-level features are weighted and aggregated to obtain fused features and then classified. Student Network: Taking low-quality modal images as input, multi-scale features are extracted and fused into a spatial fusion feature representation. The student distribution is generated through a self-attention module, and the spatial fusion feature representation is weighted and aggregated to obtain aggregated features and classified. The first stage trains the teacher network to ensure that the bi-branch classification results are consistent with the classification labels. The second stage freezes the teacher network, constrains the student distribution to be consistent with the teacher distribution, and trains the student network using the combined classification loss.

[0021] In actual medical image acquisition (such as MRI and CT), due to variations in patient position, scanning parameters, and slice thickness, achieving precise voxel-level alignment is difficult. This invention addresses this by introducing spatial inconsistencies between high-quality and low-quality modalities in each sample, allowing direct use of unregistered data. This eliminates the need for extensive manual and algorithmic registration preprocessing, making it easier to generalize to clinical applications. By introducing spatial inconsistency during training, the learning process is forced to prioritize slice importance distributions based on semantic content rather than fixed spatial locations. Even during testing, when image misalignment exists (e.g., missing slices or slice thickness variations), key slices can still be stably evaluated, demonstrating stronger generalization ability. The teacher network utilizes global semantic features from the high-quality modality and slice-level features from the low-quality modality for cross-modal attention computation, generating distributional knowledge of which slices are more important for classification. This knowledge is a location-independent, higher-order diagnostic prior, independent of voxel-level correspondences. Spatial inconsistency precisely demonstrates the effectiveness of this distillation method, avoiding overfitting to pseudo-correspondences after registration.

[0022] This embodiment aims to address the problems of poor performance in low-quality modal image classification under resource-constrained scenarios and the difficulty of adapting existing technologies to real, non-strictly aligned clinical data, providing a more practical and efficient solution for medical image classification.

[0023] This embodiment introduces a two-stage cross-modal knowledge distillation framework: unlike traditional methods that rely solely on the monotonic representation network of low-quality modal images (such as CT scans) for direct classification, this embodiment innovatively designs a two-stage cross-modal knowledge distillation framework. This framework distills the prior structural information and slice sensitivity weights of high-quality modal images (such as MRI scans) into the student network through a model mechanism. During the deployment phase, only the low-quality modal images need to be input into the student network, improving the classification accuracy and robustness of the student network when performing tasks independently, making it more suitable for resource-constrained or emergency clinical scenarios.

[0024] This embodiment utilizes cross-modal attention to mitigate cross-modal discrepancies and improve distillation stability: Compared to existing cross-modal distillation methods for medical images that require voxel-by-voxel geometric alignment and feature-dependent matching, this embodiment employs one-dimensional cross-modal attention, transferring only higher-order one-dimensional knowledge of the importance distribution of different slices to the final tumor identification, without requiring hard alignment of different sequences in three-dimensional space. This effectively circumvents the spatial mismatch problem caused by non-strictly aligned clinical data, reducing the interaction resistance of heterogeneous cross-modal information; furthermore, it distills more diagnostically useful structural knowledge, achieving stable and reliable gains in weakly paired scenarios.

[0025] In this embodiment, pediatric brain tumor (PBT) classification data integrated from Beijing Tiantan Hospital and the Radiopaedia website is used as an example to verify the algorithm's advantages in unregistered scenarios. The input data includes MRI images as the high-quality modality and CT images as the low-quality modality. The dataset is labeled with classification tags for four types of tumors: glioma, ependymoma, medulloblastoma, and choroid plexus tumor. The number of slices in the MRI images is [not specified in the original text]. The number of slices in a CT image is 8. The value is 32. There is spatial misalignment between the two in terms of layer thickness, layer level, and position; they have not undergone rigorous three-dimensional registration. In the experimental parameter hyperparameter example, the network backbone uses 2D ResNet-18 (two-dimensional residual network-18) for feature extraction, and is preprocessed with uniform-size cropping and scaling into a two-dimensional slice sequence with height H=224 (configurable) and width W=224 (configurable), where H and W are the height and width of the slice. This method mainly includes the following steps: Step 1: Obtain the training sample set; Obtain a training sample set of 3D medical images The training samples include high-quality modal images. (e.g., MRI), corresponding low-quality modal images (e.g., CT scan), and the corresponding brain tumor pathology category label Y; in this embodiment, the number of slices of the high-quality modality image. Number of slices compared to low-quality modal images There are differences ( ),in, The total number of training samples, For sample index.

[0026] Step 2: Construct a two-branch teacher classification network; Intermediate feature representations are extracted using 2D ResNet-18. The first branch is used to extract intermediate representations from high-quality modality images (MRI images), and global semantic features are generated through cross-slice dimension average pooling output. The second branch is used to extract low-quality modal images (CT images) to obtain slice-level features. In this embodiment, the feature dimension is... Using the global semantic features obtained above Performing cross-modal attention computation with slice-level features yields teacher distributions. ,like Figure 2 As shown, it specifically includes steps S2.1 to S2.4.

[0027] Step 2.1: Global semantic features Perform a linear mapping to obtain the query vector For slice-level features The key vector is obtained by performing a linear mapping. , To find the learnable linear transformation matrix of the query vector, Let be the learnable linear transformation matrix of the key vector. This is the attention projection dimension. In this embodiment... .

[0028] Query vector Global diagnostic priors carrying high-quality modalities, key vectors It carries local features of each slice of the low-quality modality. By calculating the dot product correlation between the two, the degree of matching between each slice and the global semantics can be obtained, thereby generating the teacher distribution.

[0029] Step 2.2: Calculate the scaling dot product correlation score based on the scaling dot product attention mechanism and bimodal information. In this embodiment, this step incorporates high signal-to-noise ratio global semantic features (i.e., global MRI information) to guide the assessment of the importance of slice-level features (i.e., different CT slices).

[0030] Step 2.3: Scale the dot product correlation score After normalization using the Softmax function, a teacher distribution with clear structural constraints at the slice level is obtained. .

[0031] Step 2.4: Obtain the teacher distribution slice-level features Weighted aggregation is performed to obtain the fusion features of the second branch. The specific calculation is as follows: ,in and They represent the first Attention weights and feature representations for slice-level features (i.e., CT slices). Simultaneously, global semantic features... Input the first branch classification header (i.e., the MRI classification header) to obtain the first classification logical value. ; to integrate features Input the second branch classification header (i.e., the CT classification header) to obtain the second classification logical value. .

[0032] Step 3: Build a student network; By removing teacher associations, a student self-attention diagnostic pathway is constructed entirely driven by low-quality modal images (CT modality in this embodiment). A multi-scale self-attention module is used to extract and fit student distributions, aiming to enable independent inference based on low-quality modal images (CT modality) to meet the requirements of clinical assessment and diagnostic deduction. Figure 3 As shown, it specifically includes steps S3.1 to S3.4.

[0033] Step 3.1: Extract intermediate features at different scales Global average pooling and max pooling are performed on the spatial dimensions H (height) and W (width) respectively. After concatenating the two pooling features, linear projection is performed to obtain the pooling fusion dimensionality reduction features. ,in, For scale indexing, For the first in the student network Intermediate features at each scale Indicates student network, The first The spatial height, width, and number of channels of each scale feature map. In this embodiment, Taking values ​​1 and 4, the number of channels after projection is: Subsequently, the projection fusion features at different scales are uniformly stitched together along the channel dimension to obtain the combined features; to ensure the complete alignment of the spatial layer sequence, the combined features are encoded with learnable spatial locations. Addition and fusion produce spatial fusion feature representation. In this embodiment, the number of feature channels after splicing is . .

[0034] Step 3.2: Representing Spatial Fusion Features Using a Self-Attention Mechanism Feature processing is performed, specifically, through linear mapping... Mapped to queries respectively s and keys ,in For learnable parameter matrix, For the self-attention projection dimension. Calculate the query. AND key The dot product between the segments yields the autocorrelation matrix between the slices of this mode. Then, average pooling compression is performed along the column direction to obtain the one-dimensional self-attention pooling score. , representing the self-attention relevance of each slice. For the first The autocorrelation matrix of each slice. In this embodiment, the self-attention projection dimension... 256 is acceptable.

[0035] Step 3.3: Similarly, for one-dimensional self-attention pooling fractions... After normalization using the Softmax function, the student distribution of the corresponding student network is generated. , of which Student distribution in slices Calculated as ,in, For slice index, For the first in the student network The original one-dimensional self-attention pooling scores of each slice, For the first in the student network The original one-dimensional self-attention pooling scores of each slice. The distribution of the slice weight components, as the student distribution, will subsequently be used together with the features to generate the network output decision of the student network.

[0036] Step 3.4: Utilizing student distribution Spatial fusion feature representation Weighted aggregation is performed to obtain the aggregation characteristics of the student network. The specific aggregation calculation is as follows: ,in and They represent the first The student attention weights and spatial fusion features of each CT slice are used to generate the classification predictions of the student network through a linear classifier mapping. .

[0037] Step 4: The training is divided into two stages. The first stage trains the dual-branch teacher network. The second stage starts distillation transfer, freezes the weight values ​​of the teacher network, and performs slice attention fitting and classification training on the student network.

[0038] Step 4.1: The first stage trains a dual-branch teacher network, inputting two modalities of images (high-quality modality images and low-quality modality images) to achieve joint classification optimization. The first branch's global semantic features... Fusion characteristics of the second branch The first classification logical value is generated by inputting two independent classification headers (i.e., the first branch classification header and the second branch classification header). Second category logical value .

[0039] The overall optimization objective of the teacher network is to simultaneously optimize the average classification loss of both branches, based on... , Given the category label Y, calculate the average classification loss for the first stage: ; ; ; in, The classification loss for the first branch, The classification loss for the second branch, This is the cross-entropy loss function.

[0040] Wherein, the classification probability corresponds to a certain predicted logistic value z. The formula for cross-entropy loss at time is: C represents the total number of categories. For category indexing, This is the one-hot encoded truth value of the real label.

[0041] Step 4.2: In the second stage, distillation transfer is initiated, the weights of the teacher network are frozen and no longer updated, and the student network is trained. The specific implementation method is as follows: Step 4.2.1: Calculate the correlation score of the scaled dot product. and one-dimensional self-attention pooling scores Introducing temperature parameters The target teacher component is obtained by performing a softening and scaling process. And predicting student volume Subsequently, the temperature-scaled KL divergence of both was calculated as the cross-modal distillation loss: .in, The first in the softened teacher distribution The weights of each slice, The softened student distribution The weight of each slice. In this embodiment, a temperature parameter can be set. Perform knowledge smoothing.

[0042] Step 4.2.2: Calculate the final classification prediction values ​​generated by the student network. Combine category tags Calculate the supervision loss for the second-stage classification task: ,in The first prediction for student networks The probability of a class.

[0043] Step 4.2.3: The overall optimization objective of the second phase of student network training combines the constraints of task supervision and knowledge distillation: .in To balance task monitoring and distillation accuracy, a constant weighting coefficient is used. In this embodiment, a weight can be set. This is to ensure the correct direction of the classification of student network subjects.

[0044] Experiments were conducted on the aforementioned integrated dataset, using the 2D ResNet-18 model without any distillation methods as the baseline model, and combining it with mainstream distillation methods (FitNets, KD, CRD, ASKD, SKD, SHAKE, DKD, MLD). The detailed combination results are shown in Table 1 below.

[0045] Table 1

[0046] The mainstream distillation methods are FitNets, KD, CRD, ASKD, SKD, SHAKE, DKD, and MLD. FitNets is an adaptation network, and its literature source is Romero A, Ballas N, Kahou SE, et al. FitNets: Hints for Thin Deep Nets[C] / / ICLR. 2015. KD is knowledge distillation, and its literature source is Hinton G, Vinyals O, Dean J. Distilling the Knowledge in a Neural Network[PP / OL]. arXiv (2015-03-09) [2026-03-27]. http: / / arxiv.org / abs / 1503.02531. CRD stands for Contrastive Representation Distillation, and its source is Tian Y, Krishnan D, Isola P. Contrastive representation distillation[C] / / 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. ASKD stands for Attention Similarity Knowledge Distillation, and its source is Shin S, Lee J, Lee J, et al. Teaching Where to Look: Attention Similarity Knowledge Distillation for Low Resolution Face Recognition[C] / / Avidan S, Brostow G, Cissé M, et al. Computer Vision – ECCV 2022: v.13672. Cham: Springer Nature Switzerland, 2022: 631-647. SKD stands for Structured Knowledge Distillation. Its source is Zhang L, Ma K. Structured Knowledge Distillation for Accurate and Efficient Object Detection[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, 45(12): 15706-15724.SHAKE stands for Shadow Knowledge Distillation, and its reference is Li L, Zhe J. Shadow knowledge distillation: bridging of fine and online knowledge transfer [C] / / Proceedings of the 36th International Conference on Neural Information Processing Systems. Red Hook, NY, USA: Curran Associates Inc., 2022: 635-649. DKD stands for Decoupled Knowledge Distillation, and its reference is Zhao B, Cui Q, Song R, et al. Decoupled Knowledge Distillation [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2022: 11953-11962. MLD stands for Multi-Level Logit Distillation. Its reference is Jin Y, Wang J, Lin D. Multi-Level Logit Distillation[C] / / Proceedings of the IEEE / CVF Conferenceon Computer Vision and Pattern Recognition, 2023: 24276-24285.

[0047] Compared with the baseline model, this embodiment can produce a classification improvement of approximately 3.5% in F1 score and approximately 3.1% in accuracy. This shows that, based on a simple 2D ResNet-18 without introducing any distillation, the cross-modal knowledge distillation method proposed in this embodiment significantly improves the classification performance of low-quality modality (CT) images.

[0048] Among the eight mainstream distillation methods compared, the method in this embodiment achieved the highest values ​​in both indicators (F1 score and accuracy). Specifically, in terms of F1, the method in this embodiment (0.625) was 0.036 higher than CRD (0.589); and in terms of accuracy, the method in this embodiment (0.750) was 0.013 higher than CRD (0.737).

[0049] In addition, the F1 scores of some mainstream distillation methods are lower than those of the baseline models. For example, FitNets has an F1 score of 0.570, which is 0.020 lower than the baseline model; ASKD has an F1 score of 0.563, which is 0.027 lower than the baseline model; SHAKE has an F1 score of 0.575, which is 0.015 lower than the baseline model; and DKD has an F1 score of 0.572, which is 0.018 lower than the baseline model.

[0050] In terms of accuracy, some mainstream distillation methods also fall short of the benchmark model. For example, ASKD (0.692) and SHAKE (0.705) are significantly lower than the benchmark model.

[0051] The above analysis shows that in unregistered cross-modal medical image classification tasks, existing distillation methods are prone to negative transfer or unstable performance. However, the method in this embodiment provides an effective solution for multimodal medical image classification in weak registration scenarios, and can stably obtain positive gain.

[0052] Based on the above description of the embodiments, those skilled in the art will understand that the medical image classification method, system, and medium based on cross-modal knowledge distillation described in this embodiment can be implemented in pure software or deployed and run on a general-purpose or dedicated computing hardware platform. Based on this essence, the technical solution of this embodiment can be specifically implemented in the form of a software product containing program instructions. This software product can be stored on various non-volatile storage media or directly deployed as a local or cloud service. The program instructions are used to cause computer devices with processing capabilities—including but not limited to personal computers, server clusters, mobile terminals, or other network devices—to execute the steps described in this embodiment.

[0053] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A medical image classification method based on cross-modal knowledge distillation, characterized in that, include: Obtain a training sample set, where each sample contains a high-quality modal image, a low-quality modal image, and a category label; A two-branch teacher network: the first branch extracts global semantic features from high-quality modal images and classifies them; the second branch extracts slice-level features from low-quality modal images; using the global semantic features as queries and slice-level features as keys, a teacher distribution is generated through cross-modal attention, and the slice-level features are weighted and aggregated to obtain fused features and then classified. Student Network: Taking low-quality modal images as input, multi-scale features are extracted and fused into a spatial fusion feature representation. The student distribution is generated through a self-attention module, and the spatial fusion feature representation is weighted and aggregated to obtain aggregated features and classified. The first stage trains the teacher network to ensure that the bi-branch classification results are consistent with the classification labels. The second stage freezes the teacher network, constrains the student distribution to be consistent with the teacher distribution, and trains the student network using the combined classification loss.

2. The method according to claim 1, characterized in that, In each sample, the high-quality and low-quality modes are not spatially consistent.

3. The method according to claim 1, characterized in that, The process involves extracting multi-scale features and fusing them into a spatial fusion feature representation. A student distribution is then generated via a self-attention module. The spatial fusion feature representation is weighted and aggregated to obtain aggregated features, which are then classified. Specifically: The extracted intermediate features at different scales are pooled in spatial dimensions H and W respectively. After concatenating the two pooled features, linear projection is performed to obtain the pooled fusion dimensionality reduction features, where H and W are the height and width of each slice in the low-quality modality image. All pooled and dimensionality-reduced features are concatenated and fused along the channel dimension, and a learnable spatial location code is superimposed to obtain a spatial fusion feature representation. Linear mapping is performed on the spatial fusion feature representation to obtain the query and key, the dot product is calculated to obtain the autocorrelation matrix, the one-dimensional self-attention pooling score is compressed along the column direction, and the student distribution is obtained by Softmax normalization. The spatial fusion feature representation is weighted and aggregated using the student distribution to obtain the aggregated features of the student network; The aggregated features are mapped through a linear classifier to obtain the classification prediction values ​​of the student network.

4. The method according to claim 1, characterized in that, The teacher distribution generated via cross-modal attention is specifically as follows: The query vector is obtained by linearly mapping the global semantic features, and the key vector is obtained by linearly mapping the slice-level features. After calculating the scaling dot product correlation score, the teacher distribution is obtained by Softmax normalization; The teacher distribution is weighted and aggregated with the slice-level features to obtain the fused features of the second branch; The global semantic features are input into the first branch classification head to obtain the first classification logical value. ; The fused features are input into the second branch classification head to obtain the second classification logical value. .

5. The method according to claim 4, characterized in that, The first stage of training the teacher network aims to ensure that the bi-branch classification results are consistent with the classification labels. Specifically: Calculate the average classification loss in the first stage. : ； ； ； in, The classification loss for the first branch, The classification loss for the second branch, Let cross-entropy be the loss function. For category labels.

6. The method according to claim 1, characterized in that, In the second stage, the specific way to ensure that the student distribution is consistent with the teacher distribution is as follows: a temperature parameter is introduced to soften the teacher and student distributions, and the KL divergence of temperature scaling is calculated as the cross-modal distillation loss.

7. The method according to claim 6, characterized in that, The cross-modal distillation loss The formula is as follows: ； in, For temperature parameters, The number of slices for low-quality modal images. For slice index, The first in the softened teacher distribution The weights of each slice, The softened student distribution The weights of each slice.

8. The method according to claim 1, characterized in that, The student network execution process is as follows: low-quality modal images are used as input to the trained student network, and the output is the classification prediction result.

9. A computer system comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the method according to any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of computer programs, which are used to be invoked by a processor and to execute the method as described in any one of claims 1-8.

Images