Improved multi-modal three-dimensional medical image classification method based on fusion assistance

Abstract

The invention discloses an improved multi-modal three-dimensional medical image classification method based on fusion assistance. The method comprises the following steps: 1, inputting multi-modal medical image data into ResNet to extract modal features and fuse the modal features; 2, constructing a multi-branch network, inputting a fusion feature into a main classification branch, and extracting a cross-plane global context and a fine-grained feature in combination with cross-plane key slice selection and Transform; 3, introducing discriminant prior knowledge generated by the main classification branch into a fusion auxiliary branch, and extracting enhanced features; 4, fusing the main branch fine features and the auxiliary branch enhanced features to obtain semantic level fusion features; 5, inputting the fused features into a classifier to output category probabilities, and taking a category corresponding to the maximum value as a diagnosis label; and 6, joint loss is constructed based on prediction and real labels, and the multi-branch multi-task network is trained and optimized. According to the method, complementary information of the multi-modal medical image is fully mined, focus perception is enhanced by combining judgment prior guidance and multi-task collaborative optimization, and the diagnosis accuracy and stability are improved.

Description

Technical Field

[0001] This invention relates to the fields of 3D medical image classification and deep learning technology, specifically to an improved multimodal 3D medical image classification method based on fusion assistance. Background Technology

[0002] With the development of medical informatics and intelligent algorithms, medical imaging is increasingly widely used in disease screening, assisted diagnosis, and prognostic analysis. Neurodegenerative diseases, represented by Alzheimer's disease and its early stages of mild cognitive impairment, are characterized by slow disease progression and atypical early clinical manifestations. Clinically, imaging evidence is usually relied upon in conjunction with quantitative analysis to improve diagnostic consistency and reduce the impact of human interpretation differences. In current technologies, medical image classification has evolved from traditional machine learning to deep learning. Traditional methods rely on manually designed features, which are susceptible to insufficient feature representation capabilities and cross-center data differences. Although deep learning methods can automatically learn representations, they generally face challenges in medical imaging scenarios such as high sample annotation costs, limited scale, uneven category distribution, and differences in device protocols, which restrict the generalization performance of the model under different data sources.

[0003] Furthermore, single-modal imaging has limitations in its coverage of disease characteristics. Magnetic resonance imaging focuses on reflecting changes in anatomical structures, while positron emission tomography (PET) can provide functional information at the metabolic level. Although multimodal joint analysis can utilize complementary information, the fusion process is often affected by factors such as spatial alignment errors, differences in modal distribution, and information redundancy. If the fusion strategy is unreasonable, it can easily lead to insufficient utilization of complementary information or insufficient response of key lesions, affecting classification stability.

[0004] Three-dimensional medical images are characterized by high dimensionality, redundant slices, and significant differences in anatomical orientation. Existing three-dimensional classification methods mostly extract features at the voxel or local block level, which is insufficient for selecting key slices between orthogonal planes and modeling cross-plane correlations. This can easily lead to insufficient capture of key clues and high background interference, thus limiting the ability to characterize early fine-grained changes.

[0005] Different modalities of imagery exhibit significant complementary characteristics in disease representation, providing a more comprehensive diagnostic dimension for clinicians. However, efficiently mining and integrating heterogeneous information from multimodal data remains a significant challenge. Due to insufficient utilization of multimodal correlation information in existing solutions, the rich features inherent in image data fail to be effectively translated into discriminative advantages for the model. This not only weakens the system's accuracy in identifying complex lesions but also easily leads to the omission of key diagnostic elements, thus adversely affecting the reliability of clinical decision-making. Summary of the Invention

[0006] To overcome the problems existing in medical image classification technology, this invention provides an improved multimodal 3D medical image classification method based on fusion assistance, aiming to more comprehensively understand the information in multimodal medical images and improve the accuracy and efficiency of image classification.

[0007] To solve the above problems, the present invention adopts the following technical solution: The present invention provides an improved multimodal three-dimensional medical image classification method based on fusion assistance, characterized by the following steps: S1. Obtain the preprocessed multimodal 3D medical image dataset. ,in, Indicates the first The first subject A modal 3D image volume, The number of channels in a 3D image volume. , , These represent the width, height, and depth of the three-dimensional image volume, respectively. For the first Category labels for each subject The total number of categories, This represents the total number of modes. Indicates the total number of subjects; S2. Construct a 3D ResNet feature extraction network and perform... Intramodal feature extraction is performed to obtain the first... The first subject Multiscale jump connection features and the The first subject Low-resolution feature body of each modality ;in, The number of channels for the aligned feature body. , and The length, width, and depth of the low-resolution feature volume; S3. Construct a cross-modal fusion module and perform the following steps: Low-resolution feature bodies of each subject across different modalities To perform intermodal interaction and fusion, the first... Fusion features of individual subjects ; S4. Construct a key slice selection module for the main classification branch, and calculate along the three orthogonal directions of coronal, sagittal, and axial. Coronal slice importance score vector Sagittal slice importance score vector Axial slice importance score vector ; and thus by , and Construct the first The three-dimensional spatial attention map of the first subject and as the first Discriminant prior for each subject Used for After weighted aggregation, we obtain the first... Global representation of individual subjects ; Select the highest score in each orthogonal direction After piecing together the features of the first slice, we get the second slice. Orthogonal slice embedding feature sequence of each subject ;Will The input embedding layer performs classification tag embedding, planar tag embedding, and position encoding respectively to obtain the first... Embedded sequences of individual subjects Then, the input is used for long-range dependency modeling in the Transformer encoder to obtain the discriminative feature vector of the main classification branch. ;in, The dimension of the embedded feature sequence for orthogonal slices. This represents the total number of slice features selected; S5. Construct a fusion-assisted reconstruction branch and utilize the reconstruction module to... and Perform decoding and reconstruction, and output the first... Reconstructed images of each subject And perform three-dimensional convolutional semantic encoding to obtain the first... Auxiliary semantic feature representation of each subject ;right After scaling transformation, we obtain the first... Scale-aligned attention maps of individual subjects Thus As a spatial gating prior guidance auxiliary semantic feature extraction network Feature extraction is performed to obtain the first... High-level semantic vector representation of each subject And further output auxiliary classification prediction probability. ;in, The dimension representing the high-level semantic vector representation; S6. Construct a classifier and perform... , and The process is performed to obtain the main classification prediction probability. and further output the first The predicted category for each subject; S7, based on , , With real labels Constructing a joint loss function The network, consisting of a main classification branch, a cross-modal fusion module, a cross-planar slice selection module, and a fusion-assisted reconstruction branch, is optimized end-to-end. During the inference phase, the trained 3D medical image classification model is used to output the final classification result for the input multimodal 3D images.

[0008] The improved multimodal three-dimensional medical image classification method of the present invention is also characterized in that step S4 is obtained using equation (1). : (1) In equation (1), Softmax represents the outer product construction, where Softmax represents the outer product construction. , and Dimensional normalization operation; Using equation (2) to obtain : (2) In equation (2), express In spatial location The channel vector at that location, express In spatial location The channel vector at a given position, where ":" indicates the index range of the channel dimension. .

[0009] Furthermore, in step S5, equation (3) is used to... Perform channel attention modulation and gating calibration to obtain the first Calibrated high-level semantic vector representation of each subject : (3) In equation (3), This represents the channel attention operator. This indicates element-wise multiplication.

[0010] Furthermore, step S7 involves the joint loss function. The process is as follows: Step S7.1: Construct the main classification loss using equation (4). : (4) In equation (4), For real labels The resulting labels smooth the target distribution; for The predicted probability in the q-th class; Step S7.2: Construct the reconstruction loss using equation (5) : (5) In equation (5), , , There are 3 weighting coefficients; Indicates the mean absolute error; This represents a 3D structural similarity measure. Let represent the differential gradient operator along any dimension d in D, H, W; Step S7.3: Construct distillation loss using equation (6) : (6) In equation (6), Let KL divergence be denoted as KL divergence.

[0011] Step S7.4: Construct the joint loss function using equation (7). : (7) In equation (7), , and There are 3 hyperparameters.

[0012] The present invention provides an electronic device, including a memory and a processor, characterized in that the memory is used to store a program supporting the processor in executing the method, and the processor is configured to execute the program stored in the memory.

[0013] The present invention provides a computer-readable storage medium on which a computer program is stored, characterized in that the computer program is executed by a processor to perform the steps of the method.

[0014] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention proposes a fusion-assisted classification framework. Based on feature-level fusion, it introduces end-to-end joint optimization of the main classification path and the auxiliary learning path, and implements synergistic constraints on the fusion representation at the semantic level, making the coupling between fusion features and classification decisions tighter. Compared to existing methods that rely solely on a single path or simple fusion, this invention can more fully exploit the complementary information of multimodal data, reduce redundancy and noise interference on discrimination, and improve the stability and generalization ability of classification results.

[0015] 2. This invention proposes an attention-guided consistency constraint mechanism. It injects the spatial attention prior generated by the main classification path into the auxiliary learning process and achieves embedded calibration of the discriminative prior through consistency constraints, making the learning process of the auxiliary branches more focused on classification-related regional details. Compared to existing methods that only employ auxiliary objectives such as reconstruction or consistency without discriminative constraints, this invention can alleviate optimization inconsistencies and gradient interference between auxiliary and classification objectives, improve the spatial accuracy and structural rationality of the fused representation, and enhance the interpretability of the model output.

[0016] 3. This invention designs a cross-plane key slice selection module CP-KSS, which comprehensively considers the differences in information distribution in the axial, coronal, and sagittal triorthogonal directions. It generates slice importance scores through cross-plane correlation modeling and filters key slices. Simultaneously, it constructs a spatial attention prior to highlight diagnostically relevant regions and suppress invalid background responses. Compared to existing technologies that do not perform key slice screening or only perform local modeling, this invention can effectively reduce information interference caused by slice redundancy, improve the efficiency of extracting key discriminative clues, and enhance diagnostic robustness. Attached Figure Description

[0017] Figure 1 The flowchart of the improved multimodal three-dimensional medical image classification method based on fusion assistance of the present invention is shown below; Figure 2 This is a schematic diagram of the specific network framework of the present invention; Figure 3 This is a schematic diagram of the specific structure of the cross-plane key slice selection module (CP-KSS) of the present invention. Detailed Implementation

[0018] In this embodiment, an improved multimodal three-dimensional medical image classification method based on fusion assistance is illustrated using two different modalities of medical images, MRI and PET, as examples. The general process is as follows: Figure 1 As shown, the specific network framework of the improved multimodal 3D medical image classification method based on fusion assistance is as follows: Figure 2As shown, the addition of auxiliary branches, consistency constraint mechanisms, and cross-plane slice selection modules significantly improves the model's image classification efficiency and accuracy, providing valuable reference for future applications in assisted medical treatment. Specifically, this classification method includes the following steps: S1. Obtain the preprocessed multimodal 3D medical image dataset. ,in, Indicates the first The first subject A modal 3D image volume, The number of channels in a 3D image volume. , , These represent the width, height, and depth of the three-dimensional image volume, respectively. For the first Category labels for each subject The total number of categories, This represents the total number of modes. This represents the total number of subjects.

[0019] In practice, P=2 was used. The experimental dataset was obtained from the Alzheimer's Disease Neuroimaging Project (ADNI) database and the Australian Imaging and Biomarker Lifestyle Study (AIBL) database. For the same subject, MRI and PET images acquired no more than one month apart were selected to form multimodal sample pairs. All multimodal data were provided in NIfTI file format (.nii.gz).

[0020] This embodiment used a total of 456 sample pairs, including 456 MRI images and 456 PET images. The ADNI database provided 300 sample pairs, including 101 groups of Alzheimer's disease (AD) patients, 118 groups of subjects with mild cognitive impairment (MCI), and 81 groups of normal controls (CN); the AIBL database provided 156 sample pairs, including 77 groups of Alzheimer's disease (AD) patients, 46 groups of subjects with mild cognitive impairment (MCI), and 33 groups of normal controls (CN).

[0021] Both MRI and PET images are grayscale medical images. The voxel size of each medical image is 182×218×182, that is, D=182, H=218, W=182.

[0022] All image data were processed using a standardized preprocessing workflow to eliminate acquisition artifacts, correct spatial differences, and achieve consistency across modal numerical scales, thus providing a reliable data foundation for subsequent model training and quantitative analysis. Specifically, firstly, the brain extraction tool BET from the FSL toolbox was used to perform skull stripping on MRI and PET images, removing non-brain tissue regions from the original images to reduce interference from irrelevant structures such as the skull and scalp on feature learning and fusion analysis. Secondly, the N4BiasFieldCorrection function from the ANTs advanced normalization package was used to perform bias field correction on the images, eliminating low-frequency intensity artifacts and grayscale inconsistencies caused by magnetic field inhomogeneities, thereby improving the uniformity of image intensity distribution and tissue contrast.

[0023] Subsequently, to ensure accurate voxel-level correspondence between MRI and PET images within a unified space, a multi-level spatial registration strategy was employed to normalize the multimodal images of each subject to a unified anatomical reference space. This registration process included a coarse-to-fine multi-scale registration procedure, combined with nonlinear deformation correction, to minimize global translation, rotation, scale changes, and local deformation deviations caused by differences in acquisition conditions and individual anatomy, thereby ensuring the consistency and comparability of cross-modal images in spatial location. Finally, to further eliminate the order-of-magnitude differences in grayscale dynamic range between different modalities, maximum-minimum normalization was performed on both registered MRI and PET images, mapping the intensity values ​​of each modality to a unified range to improve the stability of cross-modal feature fusion and joint modeling.

[0024] The above preprocessing steps can effectively improve image quality and establish a stable basis for spatial alignment and numerical uniformity, thereby meeting the requirements of data consistency and repeatability for subsequent algorithm implementation.

[0025] S2. Construct a 3D ResNet feature extraction network and perform... Intramodal feature extraction is performed to obtain the first... The first subject Multiscale jump connection features and the The first subject Low-resolution feature body of each modality Among them, the multi-scale skip connection feature The low-resolution feature volume of this modality is composed of intermediate features output by the 3D ResNet in the first three downsampling stages, and is used for detail compensation in subsequent reconstruction through skip connections; This represents the deep features output by the last downsampling layer of the 3D ResNet. The number of channels for the aligned feature body. , and The length, width, and depth of the low-resolution feature volume are defined. In specific implementations, the 3D feature extraction network preferably adopts a 3D ResNet-18 backbone structure based on MedicalNet pre-trained weight initialization, and its first layer of 3D convolution is channel-adapted to enable it to receive single-channel grayscale data input.

[0026] Specifically, input body data The process proceeds sequentially through: a first 3D convolutional layer (kernel size 7×7×7, stride 2, padding 3), a normalization layer, and a non-linear activation layer, followed by a 3D max-pooling layer for initial downsampling. Subsequently, deep semantic features are extracted through multi-level residual blocks (layer1, layer2, layer3, etc.) of a residual network. To improve feature robustness, a 3D Dropout (preferably with a dropout rate p=0.2) can be introduced during the deep feature extraction stage to suppress overfitting.

[0027] S3. Construct a cross-modal fusion module and perform the following steps: Low-resolution feature bodies of each subject across different modalities To perform intermodal interaction and fusion, the first... Fusion features of individual subjects ; Preferably, in this embodiment, P=2 is set, corresponding to the MRI and PET modalities respectively. To ensure channel consistency during fusion, the two encoded features are first aligned (e.g., by projecting them to a uniform number of channels using a 1×1×1 three-dimensional convolution). Then it is sent to the cross-modal fusion module. In this embodiment, the number of channels is taken as... The value is 64, to balance expressive power and computational efficiency.

[0028] Furthermore, the cross-modal fusion module preferably adopts a three-dimensional dual-input fusion structure, which utilizes a channel compression ratio... (Preferred) =4) Adaptively select cross-modal complementary information and implement interactive enhancement and redundancy suppression in the spatial and channel dimensions to output a fused feature tensor. .

[0029] S4, such as Figure 3 As shown, a key slice selection module for the main classification branch is constructed, and the slices are selected along three orthogonal directions: coronal, sagittal, and axial. The slices are expanded, and spatial average pooling is performed on each slice to obtain a slice-level feature sequence. Subsequently, a cross-plane attention mechanism is used to interactively model the slice sequences in three directions: the slice sequence in the current direction is used as the query. After splicing the slice sequences from the other two directions, the bonds are constructed through spatial downsampling and linear transformation. Sum Calculate the attention weights and output the coronal slice importance score vector corresponding to the current direction. Sagittal slice importance score vector Axial slice importance score vector ; and thus by , and Construct the first using equation (1) The three-dimensional spatial attention map of the first subject and as the first Discriminant prior for each subject Used for After weighted aggregation, the result is obtained using equation (2). Global representation of individual subjects ; (1) In equation (1), Softmax represents the outer product construction, where Softmax represents the outer product construction. Dimensional normalization operation; (2) In equation (2), express In spatial location The channel vector at that location, express In spatial location The channel vector at a given position, where ":" indicates the index range of the channel dimension. .

[0030] Select the highest score in each orthogonal direction After piecing together the features of the first slice, we get the second slice. Orthogonal slice embedding feature sequence of each subject ;Will The input embedding layer is used to perform classification label embedding, planar label embedding, and positional encoding respectively, to obtain the first... Embedded sequences of individual subjects Preferably, in Learnable classification tags are inserted before the data is processed, and learnable separator tags are inserted between different planar segments. Simultaneously, learnable planar embeddings are superimposed in the coronal, sagittal, and axial directions, along with learnable positional codes, to explicitly distinguish between the data. The spatial source and order information are used to obtain the embedded sequence. .in, The dimension of the embedded feature sequence for orthogonal slices. This represents the total number of selected slice features. .

[0031] Then, Input the Transformer encoder to perform long-range dependency modeling and obtain the discriminative feature vector of the main classification branch. Preferably, the Transformer encoder consists of stacked multi-layer coding blocks, each layer containing a multi-head self-attention and feedforward network (MLP) and employing residual connections and layer normalization for stable training; wherein the number of attention heads can be 8 and the number of coding layers can be 8, to achieve global interactive modeling of cross-plane and cross-slice sequence representations. Finally, the planar embedding features in the output sequence are taken as the discriminant features of the main classification branch. .

[0032] S5. Construct a fusion-assisted reconstruction branch to decode and reconstruct fusion features, thereby providing structural constraints and improving the model's interpretability and generalization ability.

[0033] Specifically, the 3D decoding and reconstruction module is used to... and Perform step-by-step upsampling decoding and reconstruction, and output the first... Reconstructed images of each subject Preferably, the 3D decoding and reconstruction module adopts a U-Net-style 3D transposed convolutional decoding structure, which includes multiple levels of upsampling units; in each level of upsampling, the multi-scale skip connection features output from step S2 are introduced. As a detail compensation, this information is fused with the current decoded features through splicing to recover more refined spatial structure information. Furthermore, to enhance channel selection capability, a channel attention module (e.g., an SE module) is introduced after each level of feature splicing. This module adaptively modulates the channel weights using a global pooling-fully connected-gating approach, thereby highlighting the responses of key channels and suppressing redundant channels. Finally, the reconstructed image volume is obtained through an output convolutional layer. Furthermore, activation functions such as Sigmoid can be used to normalize the reconstruction results to a consistent numerical range, facilitating reconstruction constraint calculations with the input mode.

[0034] right Perform 3D convolutional semantic encoding to obtain the first... Auxiliary semantic feature representation of each subject ; To ensure that auxiliary branches extract semantic information consistent with the semantic concerns identified by the main branch, a consistency constraint mechanism is introduced. After scaling, the first feature space with the same scale as the intermediate feature space of the auxiliary branch is obtained. Scale-aligned attention maps of individual subjects Specifically, interpolation, transposed convolution upsampling, or convolution downsampling can be used to... Make scale adjustments so that The spatial dimensions of features at different levels of the auxiliary branches are matched to achieve voxel-level gating guidance.

[0035] Calibration Prior As a spatially gated prior injection auxiliary semantic feature extraction network, preferably, the auxiliary semantic feature extraction network adopts a three-dimensional ResNet backbone structure, and inserts guided attention modules into multiple intermediate layers: the guided attention modules can first perform channel attention enhancement on intermediate features (e.g., adaptive average pooling combined with gated weights), and then utilize... The features are spatially weighted and modulated, and the original features are fused using a residual method to ensure stability. Through the above guidance mechanism, the auxiliary branch can focus on consistent key spatial regions under the constraint of the prior judgment of the main branch. After completing the multi-layer guided feature extraction, the final semantic features are globally pooled and flattened, thereby utilizing Equation (3) to... Perform channel attention modulation and gating calibration to obtain the first Calibrated high-level semantic vector representation of each subject Will Input an auxiliary classifier (e.g., a two-layer fully connected network), output the auxiliary classification prediction probability. ;in, The dimension of the auxiliary semantic features is represented; in this embodiment, F is 512 (consistent with the final channel of the 3D ResNet). (These are Alzheimer's disease patients (AD), patients with mild cognitive impairment (MCI), and patients with normal cognition (CN), respectively).

[0036] (3) In equation (3), This represents the channel attention operator. This indicates element-wise multiplication.

[0037] S6. Construct a classifier and use feature concatenation to... , and A fusion process is performed, and random deactivation (Dropout=0.3) is applied to the fused joint representation to improve generalization ability. The joint representation is then input into the main classifier (linear classifier head) to obtain the main classification prediction probability. and further output the first The predicted category for each subject; in this embodiment, The dimension is , The dimension is , The dimension is Then the joint representation dimension is This is consistent with the classification header input in the implementation.

[0038] S7, based on , , With real labels The joint loss function is constructed using equation (4). The network, consisting of a main classification branch, a cross-modal fusion module, a cross-planar slice selection module, and a fusion-assisted reconstruction branch, is optimized end-to-end. During the inference phase, the trained 3D medical image classification model is used to output the final classification result for the input multimodal 3D images.

[0039] (4) In equation (4), , and There are 3 hyperparameters. The main classification loss is obtained from equation (5). To incorporate the auxiliary classification loss, during the training phase... Apply and The loss is a smooth cross-entropy loss for homomorphic labels (consistent with Equation (5)). The difference is that independent predictions for fused auxiliary classification are not output during the inference phase. It is used only during the training period as regularization supervision for representation learning. For reconstruction loss, the reconstructed image volume used to constrain the nth subject Its corresponding reference image volume The consistency in voxel strength, structural similarity, and edge gradient is obtained from equation (6). The distillation loss is given by equation (8); (5) In equation (5), For real labels The resulting labels smooth the target distribution; for The predicted probability on the q-th class.

[0040] (6) In equation (6), This is a reconstruction difference metric function used to measure the volumetric size of the reconstructed image. With reference image volume Differences exist in voxel strength, structural similarity, and edge gradient. Furthermore: (7) In equation (7), , , These are the weighting coefficients; Indicates the mean absolute error; This represents a 3D structural similarity measure. Let represent the differential gradient operator along any dimension d in D, H, W.

[0041] (8) In equation (8), The KL divergence is used to constrain the probability of fusion auxiliary classification prediction. With the main classification prediction probability Consistency.

[0042] In this embodiment, an electronic device includes a memory and a processor. The memory stores a program that supports the processor in executing the above-described method, and the processor is configured to execute the program stored in the memory.

[0043] In this embodiment, a computer-readable storage medium stores a computer program, which is executed by a processor to perform the steps of the above method.

Claims

1. An improved multimodal three-dimensional medical image classification method based on fusion assistance, characterized in that, Includes the following steps: S1. Obtain the preprocessed multimodal 3D medical image dataset. ,in, Indicates the first The first subject A modal 3D image volume, The number of channels in a 3D image volume. , , These represent the width, height, and depth of the three-dimensional image volume, respectively. For the first Category labels for each subject The total number of categories, This represents the total number of modes. Indicates the total number of subjects; S2. Construct a 3D ResNet feature extraction network and perform... Intramodal feature extraction is performed to obtain the first... The first subject Multiscale jump connection features and the The first subject Low-resolution feature body of each modality ;in, The number of channels for the aligned feature body. , and The length, width, and depth of the low-resolution feature volume; S3. Construct a cross-modal fusion module and perform the following steps: Low-resolution feature bodies of each subject in each modality To perform intermodal interaction and fusion, the first... Fusion features of individual subjects ; S4. Construct a key slice selection module for the main classification branch, and calculate along the three orthogonal directions of coronal, sagittal, and axial. Coronal slice importance score vector Sagittal slice importance score vector Axial slice importance score vector ; and thus by , and Construct the first The three-dimensional spatial attention map of the first subject and as the first Discriminant prior for each subject Used for After weighted aggregation, we obtain the first... Global representation of individual subjects ; Select the highest score in each orthogonal direction After piecing together the features of the first slice, we get the second slice. Orthogonal slice embedding feature sequence of each subject ;Will The input embedding layer performs classification tag embedding, planar tag embedding, and position encoding respectively to obtain the first... Embedded sequences of individual subjects Then, the input is used for long-range dependency modeling in the Transformer encoder to obtain the discriminative feature vector of the main classification branch. ;in, The dimension of the embedded feature sequence for orthogonal slices. This represents the total number of slice features selected; S5. Construct a fusion-assisted reconstruction branch and utilize the reconstruction module to... and Perform decoding and reconstruction, outputting the first... Reconstructed images of the subjects And perform three-dimensional convolutional semantic encoding to obtain the first... Auxiliary semantic feature representation of each subject ;right After scaling transformation, we obtain the first... Scale-aligned attention maps of individual subjects Thus As a spatial gating prior guidance auxiliary semantic feature extraction network Feature extraction is performed to obtain the first... High-level semantic vector representation of each subject And further output auxiliary classification prediction probability. ;in, The dimension representing the high-level semantic vector representation; S6. Construct a classifier and perform... , and The process is performed to obtain the main classification prediction probability. and further output the first The predicted category for each subject; S7, based on , , With real labels Constructing a joint loss function The network, consisting of a main classification branch, a cross-modal fusion module, a cross-planar slice selection module, and a fusion-assisted reconstruction branch, is optimized end-to-end. During the inference phase, the trained 3D medical image classification model is used to output the final classification result for the input multimodal 3D images.

2. The improved multimodal three-dimensional medical image classification method according to claim 1, characterized in that, In step S4, equation (1) is used to obtain... : (1) In equation (1), Softmax represents the outer product construction, where Softmax represents the outer product construction. , and Dimensional normalization operation; Using equation (2) to obtain : (2) In equation (2), express In spatial location The channel vector at that location, express In spatial location The channel vector at the specified position, where ":" indicates the index range of the channel dimension. .

3. The improved multimodal three-dimensional medical image classification method according to claim 1, characterized in that, Step S5 uses equation (3) to... Perform channel attention modulation and gating calibration to obtain the first Calibrated high-level semantic vector representation of each subject : (3) In equation (3), This represents the channel attention operator. This indicates element-wise multiplication.

4. The improved multimodal three-dimensional medical image classification method according to claim 1, characterized in that, Step S7 is the joint loss function. The following steps are followed to build it: Step S7.1: Construct the main classification loss using equation (4). : (4) In equation (4), For real labels The resulting labels smooth the target distribution; for The predicted probability in the q-th class; Step S7.2: Construct the reconstruction loss using equation (5) : (5) In equation (5), , , There are 3 weighting coefficients; Indicates the mean absolute error; This represents a 3D structural similarity measure. Let represent the differential gradient operator along any dimension d in D, H, W; Step S7.3: Construct distillation loss using equation (6) : (6) In equation (6), Let KL divergence be denoted as KL divergence. Step S7.4: Construct the joint loss function using equation (7). : (7) In equation (7), , and There are 3 hyperparameters.

5. An electronic device, comprising a memory and a processor, characterized in that, The memory is used to store a program that supports the processor in executing the method of any one of claims 1-4, the processor being configured to execute the program stored in the memory.

6. A computer-readable storage medium storing a computer program thereon, characterized in that, The computer program is executed by the processor to perform the steps of any of the methods described in claims 1-4.

Images