A medical data processing method and product based on multi-stage transfer learning and multi-modal data collaborative fusion

Abstract

The application discloses a medical data processing method and product based on multi-stage transfer learning and multi-modal data collaborative fusion, relates to the technical field of medical data processing and artificial intelligence, and adopts MedicalNet medical special pre-training weights to initialize a classification model backbone network; a three-stage progressive fine-tuning framework is constructed on the basis, field adaptive coarse classification fine-tuning and target task fine classification are performed, and clinical structured data is introduced in the third stage; after high-dimensional image features are reduced in dimension and low-dimensional clinical features are increased in dimension through an adaptive multi-branch multi-level MLP architecture, mid-term deep fusion is performed, the application can effectively mine the complex relationships such as complementation, correlation and cooperation of multi-modal heterogeneous data, improve the lung disease classification precision and model generalization capability, and significantly inhibit the small sample overfitting phenomenon.

Description

Technical Field

[0001] This invention relates to the fields of medical data processing and artificial intelligence technology, specifically a medical data processing method and product based on multi-stage transfer learning and multi-modal data collaborative fusion. Background Technology

[0002] Currently, AI-based medical data classification mainly faces three technical solutions: First, it relies on labeled or unlabeled image data combined with category labels to directly build a classification model, but it is prone to overfitting under small sample conditions; second, it uses models pre-trained on natural image datasets (such as ImageNet) for single-stage transfer learning, but due to the significant differences between natural images and lung 3DCT images in imaging principles, spatial dimensions, and numerical distribution, the cross-domain bias problem is prominent, resulting in slow model convergence and weak generalization ability; third, it uses self-built medical image pre-trained models for transfer learning, but it mostly adopts simple fine-tuning strategies, with insufficient feature mining depth, and when fusing image and clinical data, it mostly only performs simple feature splicing, failing to form synergistic optimization and making it difficult to fully leverage the complementary advantages of multimodal data. Summary of the Invention

[0003] To address the technical problems in existing medical image classification technologies, such as significant cross-domain offset between natural and medical images, insufficient depth of single-stage fine-tuning feature mining, poor synergy due to simplistic and crude multimodal data fusion methods, and the tendency for models to overfit in small sample scenarios, this invention provides a medical data processing method and product based on multi-stage transfer learning and multimodal data fusion. This invention aims to achieve high-precision and high-generalization classification of lung diseases through medical-specific pre-trained weight initialization, a three-stage progressive feature mining architecture, and an adaptive multi-branch fusion mechanism.

[0004] The specific technical solution is as follows:

[0005] A medical data processing method based on multi-stage transfer learning and multi-modal data fusion includes the following steps:

[0006] S1. Data Preprocessing: Acquire 3D CT images of the lungs of the target disease group and matching clinical structured data, as well as 3D CT images of the lungs of non-target disease groups. Perform validity verification, consistency verification with labeled data, lung parenchyma segmentation and standardization preprocessing on the CT image data; perform validity verification and standardization preprocessing on the clinical data.

[0007] S2. Initialization of Medical-Specific Pre-trained Weights: The 3D ResNet backbone network is initialized using MedicalNet weights pre-trained on various 3D medical image datasets to mitigate cross-domain offset between natural images and medical images.

[0008] S3, Multi-stage progressive transfer learning fine-tuning:

[0009] S31. First stage: Domain adaptive fine-tuning. Using images of the target disease group and non-target disease group after lung parenchyma segmentation, a binary classification model is trained to distinguish between the target disease group and the non-target disease group. All backbone network parameters are frozen, and only the classification head is trained. In this stage, the model outputs the corresponding category probability distribution vector of the target disease group and the non-target disease group based on the macroscopic image representation of the input sample.

[0010] S32. Second stage: Fine-tuning of the target task. Load the optimal model from the first stage, freeze the shallow network parameters, unfreeze the high-level network parameters, and fine-tune the classification using fine-annotation of images and lesions within the target disease group. In this stage, the model outputs the probability distribution vector of different categories within the corresponding target disease group based on the fine-grained image representation of the input sample.

[0011] S33. Third stage: Multimodal collaborative fusion fine-tuning. Based on the model in the second stage, an adaptive multi-branch multi-level MLP fusion architecture is adopted to deeply fuse deep image features with clinical features. In this stage, the model outputs the probability distribution vector of different categories within the corresponding target disease group according to the multimodal fusion features of the input sample.

[0012] S4. Model Evaluation and Deployment: Evaluate model performance using an independent test set and deploy the model on a medical data processing terminal or server.

[0013] The validity verification of CT image data in S1 includes the following verification items in a preset order: non-DICOM slice verification and exclusion, DICOM slice validity verification, DICOM slice naming validity verification and correction, duplicate DICOM slice verification and exclusion, DICOM slice layer thickness correlation minimum slice number verification, DICOM slice continuity verification, and DICOM slice layer spacing consistency verification.

[0014] The consistency check between CT images and annotation data in S1 includes: comparing the overall size, coordinate origin, spatial direction, and voxel spacing of the images and annotations. If the overall size is inconsistent, the sample is directly excluded. If the overall size is consistent, the coordinate origin, spatial direction, and voxel spacing are compared again. If there is a deviation, the coordinates, direction, and voxel spacing of the annotation data are aligned and reset based on the image data.

[0015] The standardization preprocessing of CT image data in S1 includes, in a preset order:

[0016] The process involves setting the batch size, loading the training and validation sets, adding channel dimensions, cropping the foreground based on the region of interest (lung parenchyma), unifying the spatial coordinate system and voxel spacing, geometrically augmenting the training set, unifying the batch processing scale, performing grayscale augmentation on the training set, and truncating and minimax normalizing based on the range of lung window CT values. In the second and third stages, after adding channel dimensions, the spatial coordinate system and voxel spacing are first unified, and then foreground cropping is performed based on the region of interest (lesion), while the rest remains unchanged.

[0017] The validity verification of clinical data in S1 includes the following verification items in a preset order: continuous value feature verification, category or discrete value feature verification, missing value format verification, and consistency verification with image data.

[0018] The preprocessing of clinical data in S1 includes the following pre-defined processing items: filling missing values ​​with the mean or median for continuous value features and adding a missing value marker column; filling missing values ​​with the mode or independent class for discrete value features with missing values; normalizing continuous value features and one-hot encoding discrete value features.

[0019] In S31, all samples of different disease types within the target disease group are uniformly labeled as the first category, and all samples outside the target disease group are uniformly labeled as the second category. The training set, validation set, and independent test set are divided in a fixed ratio and remain unchanged throughout the process. The independent test set does not participate in model training at any stage.

[0020] The hierarchical fine-tuning strategy in S32 is as follows: load the optimal model parameters of the first stage, replace the first stage classification head with a new randomly initialized classification head; freeze the head module and the first three residual stages in the backbone network, and only unfreeze the fourth residual stage and the new classification head, wherein the fourth residual stage adopts a lower learning rate and the classification head adopts a higher learning rate.

[0021] The adaptive multi-branch multi-level MLP fusion architecture in S33 includes:

[0022] Image Feature Branch MLP: The high-dimensional image features output by the backbone network are sequentially subjected to global average pooling, flattened, and then reduced in dimensionality by two fully connected layers.

[0023] Clinical Feature Branch MLP: The input clinical feature vector is non-linearly upsized through a fully connected layer so that the output feature dimension is on the same order of magnitude as the output feature dimension of the image feature branch MLP, so as to unify the dual-modal feature magnitude.

[0024] Multimodal fusion branch MLP: After concatenating the output features of the above two branches, a higher-level joint feature representation is generated through mapping and nonlinear activation interaction in a fully connected layer, and then the class probability distribution vector is generated through the output layer.

[0025] The multimodal fusion fine-tuning in S33 adopts a staged joint optimization strategy:

[0026] Initially, the entire image backbone network is frozen, and only the multimodal fusion branch (MLP) is trained.

[0027] Mid-term unfreezing and joint training of high-level parameters and all fusion branches of the image backbone network;

[0028] In the later stage, all parameters of the image backbone network were fully unfrozen, and end-to-end joint optimization was performed using a lower learning rate.

[0029] Furthermore, the method is executed collaboratively by the processor and graphics processor of the medical data processing device, and the specific hardware interaction process includes:

[0030] During the data preprocessing stage, the processor extracts the coordinate values ​​and slice thickness parameters from the DICOM file header of the CT image through the arithmetic logic unit, and performs floating-point operations and logical comparisons in the register to complete the verification, and caches the qualified image data in memory.

[0031] During the weight initialization phase, the processor parses the MedicalNet pre-trained weight tensor and loads it into the graphics processor. By modifying the model computation graph, the decoder submodule is removed at the underlying hardware level and replaced with an identity mapping to complete the physical structure reconstruction of the 3D ResNet backbone network.

[0032] In the multi-stage fine-tuning stage, the tensor computation core of the graphics processor performs three-dimensional convolution operations on the 3D ResNet backbone network. The gradient update mechanism at the hardware level truncates the shallow parameter gradients during backpropagation according to the hierarchical fine-tuning strategy. In the third stage, the storage loading unit of the graphics processor synchronously pulls image feature vectors and clinical feature vectors, completes the physical stitching of dual-modal features through the underlying bus, and the matrix multiplication acceleration array sequentially performs the dimensionality reduction mapping of the image feature branch, the dimensionality increase mapping of the clinical feature branch, and the nonlinear activation calculation of the multimodal fusion branch. Finally, the class probability distribution electrical signal is output through the fully connected layer.

[0033] The present invention also provides a medical data processing device, including a memory, a graphics processor, and a processor connected to the memory and the graphics processor; the memory is used to store a computer program; the graphics processor is configured to include a tensor calculation core, a storage loading unit, and a matrix multiplication acceleration array, for performing three-dimensional convolution operations, physical splicing of feature tensors, and nonlinear mapping calculations; the processor is used to call and run the computer program to execute the medical data processing method described above.

[0034] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the medical data processing method described above.

[0035] The advantages of this invention are:

[0036] 1. Mitigating domain bias and optimizing model convergence efficiency: The model is initialized using MedicalNet medical-specific pre-trained weights, which avoids the problem of insufficient adaptability between natural image pre-trained features and medical CT image features, reduces the model iteration cycle, optimizes model convergence efficiency, and allows the model performance to stabilize with a small number of iterations.

[0037] 2. Optimize model representation capabilities and improve classification accuracy: Through a multi-stage progressive fine-tuning framework, the common features of lung images and the differential features of target samples are explored layer by layer. Combined with the multimodal fusion strategy of images and clinical data, the feature difference representation of different samples is refined to meet the needs of intelligent classification and recognition of medical data.

[0038] 3. Suppress overfitting in small sample sizes and improve generalization performance: Expand the model training feature dimensions by relying on massive lung CT auxiliary data, and combine multi-stage learning rate decay and progressive fine-tuning strategies to improve the model overfitting problem in small sample training scenarios and enhance the model's general representation ability.

[0039] 4. Standardized training process and excellent model reproducibility: This invention adopts a fixed dataset partitioning method and a unified data preprocessing standard throughout the entire process. The training logic is clear and the parameter configuration is standardized, ensuring the consistency and reproducibility of experimental results, which facilitates the implementation and promotion of the technology.

[0040] 5. Aligns with the logic of comprehensive medical assessment and has strong application adaptability: It integrates 3D CT image features with clinical structured data features and matches multi-dimensional medical data analysis paradigms, enabling multi-dimensional feature analysis and quantitative assessment of lung lesions, effectively improving the analysis and processing efficiency of massive medical data, and possessing high engineering implementation value and industrial application potential. Attached Figure Description

[0041] Figure 1 This is the overall flowchart of the present invention;

[0042] Figure 2 This is a diagram showing the slices sorted based on the Z-coordinate of the top-left corner of a DICOM slice.

[0043] Figure 3 Diagram of the pre-trained network structure of MedicalNet (3D ResNet-50);

[0044] Figure 4 The process of initializing the pre-trained weights for MedicalNet;

[0045] Figure 5 Graph for the first-stage model;

[0046] Figure 6 Graph for the second-stage model;

[0047] Figure 7 Graph for the third-stage model;

[0048] Figure 8 This is a schematic diagram of the processing method of the present invention. Detailed Implementation

[0049] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] 1.1 Data Preparation

[0051] 1.1.1 Input Data

[0052] The data required for modeling in this invention is divided into target disease group datasets and non-target disease group datasets, with the specific requirements as follows:

[0053] 1. Target disease group dataset

[0054] (1) Data categories: CT imaging data and structured clinical data of archived lung infection samples. Various types of imaging data are classified and collected in folders according to infection category labels; various types of clinical data can be stored separately in different spreadsheet files or integrated into the same spreadsheet file, and distinguished by the infection category field.

[0055] (2) Data volume: The data volume of a single sample should preferably be no less than 300 cases, and all of them should be independent scanned samples, and duplicate scanned data should be eliminated.

[0056] (3) Image data annotation: The infectious lesion areas in the image data are manually annotated pixel by pixel and the compliance review is carried out. The annotation labels of samples of the same category are kept in a uniform standard.

[0057] (4) Clinical data structuring: Clinical data integrates multi-dimensional structured features, including demographic features, imaging parameters, lesion location information, physical signs data, medical history information, smoking characteristics, pathological information, laboratory test indicators, etc., enriching the feature dimensions of model building.

[0058] (5) Image and clinical data association: Image data and corresponding clinical data can be associated and matched with unique indexes or combined indexes.

[0059] 2. Non-target disease group dataset

[0060] (1) Data categories: CT image data of the remaining lung infection and non-infection samples that have been archived, as well as normal lung CT image sample data. All data are archived into one category.

[0061] (2) Data volume: The total data volume is set to 1 to 2 times the total data volume of the target disease group sample, and the number of samples should be expanded as much as possible. Since there is a sufficient reserve of lung-related sample imaging data resources, it can provide sufficient data support for subsequent multimodal modeling analysis.

[0062] (3) Data annotation: Select about 100 samples to complete lung parenchyma annotation, which will be used for the construction of lung parenchyma segmentation model and lung parenchyma segmentation.

[0063] 1.1.2 Verification of image data availability or validity

[0064] Input: Candidate CT sequences (CT image data corresponding to one sample)

[0065] Step 1: Determine if there are non-DICOM format file objects in the CT sequence. The determination and processing methods are as follows:

[0066] (1) If there are directories (not files) or metadata files generated by common CT devices (such as DICOMDIR, LOCKFILE, VERSION), delete them.

[0067] (2) Read the header of one of the slices:

[0068] (a) The first 128 bytes are preamble, which can be skipped.

[0069] (b) The next 4 bytes should contain “DICM”, otherwise it is not a DICOM file. Delete the file and then jump to (2) to continue processing the next slice;

[0070] (c) If the tag "SeriesInstanceUID" is not in the header, delete the file and then jump to (2) to continue processing the next slice;

[0071] (d) If the tag "InstanceNumber" is not in the header, delete the file and then jump to (2) to continue processing the next slice;

[0072] (e) If the tag "ImageOrientationPatient" is not in the header or the length of the tag value is less than 6, delete the file and then jump to (2) to continue processing the next slice;

[0073] (f) If the tag "ImagePositionPatient" is not in the header or the length of the tag value is less than 3, delete the file and then jump to (2) to continue processing the next slice.

[0074] Step 2: Check if there are any invalid filenames. For example, a filename starting with "." such as .CT..0019.0001.2018.01.22.12.34.40.0.8260049.IMA will cause an error when the program tries to read it. If such a filename exists, remove the "." before the filename.

[0075] Step 3: Determine if there are duplicate slices. The determination and handling methods are as follows:

[0076] (1) Create an empty dictionary object to store key-value pairs, where each key is unique.

[0077] (2) Read the value of the tag “Image Position(Patient)” representing the XYZ coordinates of the top left corner of one of the slices in the header of one of the slices, for example [-176.14550781 -272.14550781 -11.3]. Usually, the XY coordinates of each slice in the same sample are the same, only the Z coordinates are different.

[0078] Using the coordinates as the key, determine if the dictionary already exists:

[0079] (a) If it does not exist, read the pixel data portion of the slice as the value and add the key-value pair to the dictionary object;

[0080] (b) If it already exists, compare the pixel data part of the current slice with the value of the element with the same key value already stored in the dictionary. If they are equal, delete the current slice. Otherwise, jump to (2) to continue reading the next slice.

[0081] Step 4: Determine whether the number of slices meets the specified threshold.

[0082] If the slice thickness (usually equal to the interslice spacing) is large, it will result in a smaller number of slices, which is not conducive to the feature identification of some small or continuous lesions. The following thresholds should be used for judgment:

[0083] Layer thickness (unit: mm) Number of slices 1 >= 200 2 >=100 3 >=80 5 >=50

[0084] Read the "Slice Thickness" tag value representing the slice thickness in any slice and compare it with the slice number threshold corresponding to the slice thickness set in the table above. If it is less than the specified threshold or the slice thickness is not in the table, exclude the current sample, jump to the input, and continue reading the CT image slice sequence of the next sample.

[0085] Step 5: Determine if the slices are continuous. The method for determining this is as follows:

[0086] (1) Create a list to store the XYZ coordinates of the top left corner of the slice.

[0087] (2) Read the value of the tag "Image Position(Patient)" representing the XYZ coordinates of the top-left corner of the slice in the header of all slices, and sort the slices in ascending order based on the Z coordinate value, as shown in the attached figure. Figure 2 As shown in the image, the sorted XYZ coordinates are saved to the list above.

[0088] (2) For the sorted slices, read the value of the tag "InstanceNumber" representing the slice order in the header of the adjacent slices in turn. Assume , These represent the ordinal values ​​of the first two slices, respectively, and then the absolute value of their difference is calculated. If the absolute value equals 1, it means the two slices are adjacent, and then continue reading and calculating. The system continues to evaluate the data; if the absolute value is not equal to 1, the current sample is excluded, and the system jumps to the input to continue reading the CT image slice sequence of the next sample.

[0089] Step 6: Determine if the spacing between slice layers is consistent.

[0090] Even if the tag "Instance Number" values ​​of slices are consecutive, the layer spacing between adjacent slices may still be inconsistent. Therefore, it is necessary to further determine whether the layer spacing between adjacent slices is consistent. The determination method is as follows:

[0091] Based on the XYZ coordinates of the top left corner of each slice saved in step 5, the Z coordinate values ​​of adjacent slices are read sequentially, assuming... , , These represent the Z coordinates of the first three adjacent slices, and then each is calculated... and ,if This indicates that the interlayer spacing between these three slices is equal, and then the reading and comparison continue. and ;if If the current sample is excluded, the process jumps to the input and continues reading the slice sequence of the next sample.

[0092] Output: CT image samples that have completed all verification processes are determined to be valid samples and can proceed to subsequent data processing.

[0093] 1.1.3 Consistency verification between image data and corresponding annotation data

[0094] The consistency of lesion annotation data from samples in the target disease group and lung parenchyma annotation data from some samples in the non-target disease group with the corresponding CT image data was verified. The verification method is as follows:

[0095] Input: Image data (image file) and corresponding annotation data (mask file) of one sample.

[0096] Step 1: Read the size of the image and mask respectively. For example, the size of the image is (512, 512, 300), which represents the number of voxels in the X, Y, and Z directions, respectively. If the size of the image and mask are the same, proceed to Step 2; otherwise, exclude the current sample, jump to the input, and continue to validate the next sample.

[0097] Step 2: Read the metadata from the headers of the image data and the annotation data respectively: origin, direction, and spacing, all of which are triples. Determine whether the values ​​of each metadata corresponding to the image and mask are equal or whether the absolute value of the difference is less than a specified threshold. The determination method is as follows:

[0098] Assuming the metadata values ​​corresponding to the image and mask are a and b respectively, determine whether they meet the following tolerance:

[0099] absolute(`a` - `b`) <= (`atol` + `rtol` * absolute(`b`))

[0100] Where absolute(`a` - `b`) represents the absolute values ​​of a and b, rtol = 1.e-5, atol = 1.e-8, and absolute(`b`) represents the absolute value of b. If all conditions are met, it is a match, and the process proceeds to step 3; otherwise, it is a mismatch, and step 2 is executed. The values ​​of rtol and atol can be adjusted according to the strictness of the tolerance.

[0101] Step 3: Based on the values ​​of the image's metadata (origin, direction, and spacings), reset the corresponding metadata of the mask.

[0102] Output: This sample is valid and can proceed to the next processing step. Jump to input to continue processing the next sample.

[0103] 1.1.4 Construction of Lung Parenchyma Segmentation Model and Lung Parenchyma Segmentation

[0104] Because the image data of samples in the non-target disease group does not label lesion areas, the first stage requires constructing an unlabeled classification model for the target disease group and the non-target disease group. To avoid the model learning irrelevant region features, lung parenchyma region segmentation needs to be performed on all image samples in the target disease group and the non-target disease group (except for samples with lung parenchyma annotation). The purpose is to remove non-lung background regions such as the bed board, chest wall, heart, and major blood vessels in the images and generate a mask matrix that only contains the lung parenchyma region.

[0105] 1. Construction of lung parenchyma segmentation model

[0106] A lung parenchyma segmentation model can be constructed using 3D U-Net or nnU-Net convolutional neural networks and labeled lung parenchyma data. Here, we will use nnU-Net as an example to illustrate the construction steps, assuming that nnU-Net and its dependent software packages and environment variables are already available:

[0107] Step 1: Create the relevant directories according to the requirements of nnU-Net for the training dataset directory format and location:

[0108] ├── nnUNet_raw│ ├── Dataset001_Feishizhi│ │ ├── imagesTr│ │ ├── imagesTs│ │└── labelsTr

[0109] The `nnUNet_raw` directory is used to store the raw training data for each dataset. The directory name format for each dataset is "Datasetxxx_name", where xxx represents a unique three-digit number. The numbers for different datasets do not need to be consecutive, and the name does not need to be unique. `imagesTr` is used to store the training set image data, `imagesTs` is used to store the test set image data, and `labelsTr` is used to store the labeled data corresponding to the training set image data.

[0110] Step 2: Rename the training set image data, labeled data, and test set image data to be segmented according to the requirements of nnU-Net, and place them in the corresponding directories. The naming format for image data should be name_xxx_xxxx, for example:

[0111] feishizhi_000_0000.nii.gz feishizhi_001_0000.nii.gz feishizhi_002_0000.nii.gz...

[0112] The name can be any name, xxx represents a unique number, xxxx represents the channel number, and CT image data has only one channel, which is represented by 0000.

[0113] The test set image files to be segmented also need to be named in the same way as the training set image files.

[0114] The naming format for the labeled data corresponding to the training set images should be name_xxx, for example:

[0115] feishizhi_000.nii.gz feishizhi_001.nii.gz feishizhi_002.nii.gz...

[0116] The name of the annotation file must match the name of the image file. "xxx" represents a unique identifier that must match the corresponding image file. The annotation file does not need channel numbers.

[0117] Step 3: Generate the configuration file dataset.json in the Dataset001_Feishizhi directory, for example:

[0118] { "channel_names": { "0": "CT"}, "labels": { "background": 0, "feishizhi": 1}, "numTraining": 100, "file_ending": ".nii.gz", "overwrite_image_reader_writer": "SimpleITKIO"}

[0119] This file is used to describe: the modality of the image (here, CT); the class numbers of the background and foreground; the number of training sets (here, 100); the file extension representing the file format, here indicating NIFTIS format; and the file read / write interface, which is optional. nnU-Net will automatically call the corresponding file read / write interface according to the file format.

[0120] Step 4: Data Preprocessing. The main purposes of preprocessing are: data integrity checking, voxel spacing standardization, intensity value normalization, automatic localization of the volume of interest (VOI) (here, the lung parenchyma), image patch strategy optimization, and data augmentation scheme customization. The preprocessing commands are as follows:

[0121] nohup nnUNetv2_plan_and_preprocess -d 001 --verify_dataset_integrity&

[0122] Step 5: Training the segmentation model. Command format:

[0123] nohup nnUNetv2_train 001 3d_lowres all&

[0124] The model parameter files from the training results are automatically saved to the following path:

[0125] nnUNet_results / Dataset001_Feishizhi / nnUNetTrainer__nnUNetPlans__3d_lowres / fold_all

[0126] 2. Automatically split the test set.

[0127] The constructed lung parenchyma segmentation model was used to segment the lung parenchyma in the image data of all samples in the target disease group and non-target disease group that were not labeled with lung parenchyma, and lung parenchyma segmentation mask files were obtained.

[0128] The command format is:

[0129] nnUNetv2_predict -i input_path -o output_path -d 001 -c 3d_lowres -f all

[0130] You need to replace input_path (here it is / nnUNet_raw / Dataset001_Feishizhi / imagesTs) and output_path (that is, the path where the segmentation result file is stored).

[0131] 1.1.5 Clinical Data Validation

[0132] For the collected structured clinical data, stratified validation was performed by distinguishing between continuous and discrete value features. Simultaneously, the format of missing values ​​and sample association consistency were validated, as detailed below:

[0133] 1. Continuous value feature verification

[0134] For continuous numerical features such as age, height, weight, BMI, fasting blood glucose, and length of hospital stay, reasonable numerical thresholds are preset to eliminate or correct abnormal data that exceed the normal range.

[0135] 2. Category or discrete value feature verification

[0136] For categorical or discrete features such as gender, ethnicity, presence of specific imaging characteristics, smoking status, fever status, and blood CEA (elevated 1, normal 0), the system verifies whether the feature values ​​are within the preset category range and removes or corrects samples containing abnormal category items.

[0137] 3. Missing value format validation

[0138] Standardize the representation of missing values ​​by leaving all missing values ​​blank and removing non-standardized missing value markers such as text (e.g., "None", "No Result", "No Value", "Missing", etc.), symbols (e.g., "-", " / ", "?", etc.), and garbled characters (e.g., BMI is calculated based on height and weight using a formula; if one or both are missing, the BMI item may display garbled characters). This avoids problems such as abnormal data reading and incorrect feature type identification (e.g., if "None" appears in a numerical feature column, the program will identify this feature column as an object type instead of a numerical type).

[0139] 4. Consistency check

[0140] Verify the index matching relationship between imaging data and clinical data, and remove samples that cannot be matched one by one (including those with blank index values) to ensure the accuracy of the correspondence between multimodal data.

[0141] 1.2 Initialize the model with pre-trained weights (Phase 0)

[0142] The open MedicalNet pre-trained weights are based on 3D ResNet (supporting ResNet-10 / 18 / 34 / 50 / 101 / 152 / 200) and pre-trained on 23 different publicly available 3D medical image datasets. These datasets include multiple modalities (e.g., MRI, CT), multiple scan sites (e.g., head, chest, abdomen), pathological features, and multicenter mapping, resulting in a large amount of data features. The MedicalNet pre-trained model employs a segmentation network design, the structure of which is shown in the attached figure. Figure 3 As shown (using 3D ResNet-50 as an example).

[0143] The MedicalNet pre-trained network includes:

[0144] An encoder module is responsible for extracting multi-level features from the input image. The encoder module directly uses the 3DResNet-50 network backbone (excluding the classification head). For CT images, it has 1 input channel and 2048 output channels.

[0145] A decoder module is responsible for upsampling the high-dimensional feature map output by the encoder, outputting a segmentation map with the same spatial resolution scale as the input, and mapping it to the number of target segmentation categories. The decoder module contains a lightweight deconvolution layer (3D deconvolution ConvTranspose3d + batch normalization BN + non-linear activation ReLU) and two convolutional layers: 3D convolution Conv3d + BN + ReLU and 3D convolution Conv3d.

[0146] In the diagram, the numbers at the bottom represent the number of input channels, and the numbers at the top represent the number of output channels. In the MedicalNet segmentation network, the decoder ultimately outputs the number of segmentation classes, `num_seg_classes`. If the number of segmentation classes is 3, the input for each pixel is a probability distribution vector of length 3. In the 3D ResNet-50 classification network, the classification head ultimately outputs the number of classification classes, `num_classes`. If the number of classes is 3, the output for each sample is a probability distribution vector of length 3.

[0147] MedicalNet's "heavy encoder, light decoder" design is very effective in transfer learning. The pre-trained encoder provides powerful feature extraction capabilities, while we only need to train a small decoder from scratch to adapt to the new segmentation task, which greatly reduces the risk of overfitting.

[0148] By loading pre-trained weights from the MedicalNet encoder, various downstream tasks (such as segmentation, classification, and detection) can be adapted. MedicalNet has conducted transfer learning experiments in lung segmentation, lung nodule classification, and the LiTS challenge (which involves segmenting the liver and liver tumors). Results show that features extracted based on MedicalNet pre-trained weights provide lower initial loss values, significantly accelerate model convergence, and can be used as pre-training features for other tasks with small training datasets. Compared to networks trained using natural images or from scratch, those based on MedicalNet pre-trained weights achieve better results. MedicalNet pre-trained weights possess a natural ability to perceive medical image features, effectively mitigating domain shift issues.

[0149] This invention selects a 3D ResNet-50 network as the backbone classification network and uses MedicalNet pre-trained weights to initialize the backbone network. The model initialization process is shown in the attached figure. Figure 4 As shown, the steps are as follows:

[0150] Step 1: Generate the MedicalNet (3D ResNet-50) network structure based on the MedicalNet open source implementation, setting the parameters --model to 'resnet', --model_depth to 50, and --resnet_shortcut to 'B'.

[0151] Step 2: Load the pre-trained weights resnet_50_23dataset.pth. These weights are implemented for a 3DResNet-50-based segmentation network and have been trained on 23 different publicly available 3D medical image datasets.

[0152] Step 3: Remove the decoder layer. First, obtain the number of input channels of the decoder submodule. When migrating to the classification network, the classification head design needs to know this number of input channels. For the 3D ResNet-50 implementation, it is fixed at 2028, so it is not necessary to obtain the number of input channels of the decoder submodule. There are two removal schemes:

[0153] Solution 1: Obtain a list of all direct submodules of the model, and then remove the last decoder submodule conv_seg by the slice index [:-1].

[0154] Option 2: Directly replace the decoder submodule of the model with an identity mapping, for example, base_model.conv_seg= torch.nn.Identity(). This constant mapping does not process the input and returns directly.

[0155] Step 4: Use the model after removing the decoder as the backbone of the 3DResNet-50 classification network for the subsequent first-stage fine-tuning, and add a classification head.

[0156] 1.3 First Stage: Domain-Adaptive Fine-Tuning (Coarse Classification)

[0157] This stage aims to achieve adaptive adaptation of the model to the specific context of the lungs, enabling the model to fully learn common anatomical features of lung imaging (such as left and right lungs, mediastinum, lobes, lung segments and subsegments, trachea and bronchi, pulmonary arteries and veins, etc.) and basic texture features (such as vascular texture, bronchial texture, and lymphatic texture, etc.), identify common abnormal representation patterns in lung imaging (common features of lesions), and distinguish the macroscopic feature differences (such as lesion distribution, density, morphology, etc.) between target disease group and non-target disease group imaging data, laying the foundation for subsequent refined feature mining. Based on the lung parenchyma segmentation results, this stage builds a binary classification model for two sets of imaging data, with the output being the probability distribution vector of the target disease group and non-target disease group corresponding to the image features of the input sample.

[0158] 1.3.1 Establishing a Data Dictionary

[0159] Data mapping dictionaries were constructed for the target disease group and non-target disease group respectively, establishing a one-to-one correspondence between image files, lung parenchyma mask files, and category labels. All samples from the target disease group were uniformly labeled as the first category, and all samples from the non-target disease group were uniformly labeled as the second category, completing the structured collection of the dataset.

[0160] 1. Establish data dictionaries for sample datasets of different disease categories within the target disease group, specifically establishing the mapping relationship between image data (including loading path), lung parenchyma segmentation data (including loading path), and category labels. Here, the category of all samples in the target disease group is set to 0.

[0161] Assuming the target disease group contains only two categories of diseases (multiple categories are supported), the data dictionary example for the sample dataset containing disease A is as follows:

[0162] dataset_dict_a = [{'image': / path_to_image / a / image001.nii.gz, 'mask': / path_to_feishizhi_mask / a / image001.nii.gz, 'label': 0},{'image': / path_to_image / a / image002.nii.gz, 'mask': / path_to_feishizhi_mask / a / image002.nii.gz, 'label': 0},...]

[0163] The following is an example of the data dictionary for the sample dataset included in disease B:

[0164] dataset_dict_b = [{'image': / path_to_image / b / img001.nii.gz, 'mask': / path_to_feishizhi_mask / b / img001.nii.gz,'label': 0},{'image': / path_to_image / b / img002.nii.gz, 'mask': / path_to_feishizhi_mask / b / img002.nii.gz,'label': 0},...]

[0165] 2. Establish a data dictionary for all sample data in the non-target disease group, that is, establish the mapping relationship between image data (image, including loading path), lung parenchyma segmentation data (mask, including loading path), and class labels. Here, the class of all samples in the non-target disease group is set to 1.

[0166] The following is an example of a data dictionary for a sample dataset from a non-target disease group:

[0167] dataset_dict_c = [{'image': / path_to_image / c / ima001.nii.gz, 'mask': / path_to_feishizhi_mask / c / ima001.nii.gz,'label': 1},{'image': / path_to_image / c / ima002.nii.gz, 'mask': / path_to_feishizhi_mask / c / ima002.nii.gz,'label': 1},...]

[0168] 1.3.2 Splitting the Dataset

[0169] 1. Randomly shuffle the order of various sample data.

[0170] The order of data in the sample data dictionary for each category is randomly shuffled to ensure the randomness and diversity of data distribution in the training and validation sets after subsequent dataset splitting.

[0171] 2. Split the dataset

[0172] The training set, validation set, and independent test set are uniformly divided according to a ratio of 70%:20%:10%. The independent test set does not participate in the model training process and is only used for the final model performance evaluation.

[0173] The datasets in the data dictionaries dataset_dict_a, dataset_dict_b, and dataset_dict_c are divided into training set: validation set: test set ratios of 70%:20%:10%, respectively.

[0174] dataset_dict_a=dataset_dict_a_train(0.7)+dataset_dict_a_val(0.2)+dataset_dict_a_test(0.1)

[0175] dataset_dict_b=dataset_dict_b_train(0.7)+dataset_dict_b_val(0.2)+dataset_dict_b_test(0.1)

[0176] dataset_dict_c=dataset_dict_c_train(0.7)+dataset_dict_c_val(0.2)+dataset_dict_c_test(0.1)

[0177] 3. Save the partitioning results

[0178] Generate a standardized dataset splitting configuration file to solidify the data splitting rules and ensure experimental reproducibility. Save the above splitting results to dataset_split_stage1.json, in the following format:

[0179] {"a": {"train": ["image005.nii.gz", ...],"val': ["image205.nii.gz", ...]"test': ["image102.nii.gz",...]},"b": {"train": ["img002.nii.gz", ...],"val': ["img100.nii.gz", ...]"test': ["img210.nii.gz",...]},"c": {"train": ["ima009.nii.gz", ...],"val': ["ima550.nii.gz", ...]"test': ["ima320.nii.gz",...]}}

[0180] Only the segmentation results of the image data need to be recorded, because the corresponding lung parenchyma segmentation data can be found based on the file naming rules.

[0181] 4. Merge the training and validation sets.

[0182] Collect training and validation set data for various types of samples:

[0183] dataset_dict_train = dataset_dict_a_train + dataset_dict_b_train +dataset_dict_c_train

[0184] dataset_dict_val = dataset_dict_a_val + dataset_dict_b_val + dataset_dict_c_val

[0185] The test set is not needed for now.

[0186] 1.3.3 Image Data Preprocessing

[0187] To avoid feature distortion caused by improper preprocessing order, image data standardization preprocessing should be performed according to the following fixed priority order. For example, performing normalization first may lead to "mean shift" and "contrast dilution." This is because medical images (especially CT and MRI) typically contain a large amount of background regions, whose pixel values ​​are usually 0 (or extremely low / high fixed values) and occupy a large proportion of the image. The global mean will be lowered (or raised) by a large number of background pixels, and the global standard deviation will also be significantly affected by background noise. Subsequently, when cropping out lesion regions, the pixel values ​​of these regions have been "distorted" by incorrect global statistics, resulting in the loss of gray-level distribution features within the lesion (such as the medical significance of HU values), making it difficult for the model to learn effective pathological features.

[0188] The specific process is as follows:

[0189] 1. Configure batch processing parameters to adapt to the hardware devices.

[0190] It's necessary to consider the spatial resolution of the preprocessed images, the GPU's memory size, and the number of GPUs to set a reasonable batch size for parallel processing (batch_size) to prevent GPU memory overflow from causing training interruptions. Typically, the training set batch_size is a power of 2, such as 2, 4, 6, 8, ..., while the validation set batch_size is 1. You can initially set a large training batch_size (e.g., 16), and if GPU memory overflows, reduce it to 8, continuing this process until a reasonable batch_size is found.

[0191] 2. Load the training and validation sets.

[0192] Based on the set batch size for the training and validation sets, iterators for loading the training and validation sets are encapsulated respectively. During iterative training, before each reload of the training set, the data in the training set is shuffled and then loaded to implicitly augment the training dataset.

[0193] 3. Add channel dimension

[0194] Add channel dimensions to single-channel CT images. CT image data is single-channel data, and its channel dimension num_channels is 1. For example, if the size of a sample CT image data is (512, 512, 224), after adding the channel dimension, its size will be (1, 512, 512, 224).

[0195] 4. Foreground cropping

[0196] Image cropping is performed based on the region of interest (VOI) in the lung parenchyma (i.e., the foreground) to remove invalid background. The image is cropped based on the VOI regions (i.e., the lung parenchyma regions) marked in the lung parenchyma mask to obtain the smallest block (called the bounding box or patch) containing the VOI region. This can eliminate irrelevant learning regions, reduce noise, eliminate class imbalance, and save memory and computation time.

[0197] 5. Unify spatial orientation and voxel spacing

[0198] Unify the spatial coordinate system and voxel spacing of all images. Image data acquired from different centers exhibit differences in image coordinate orientation and voxel spacing due to variations in scanning parameters. Resampling is necessary to transform all data to the same standardized coordinate system and spatial scale. This prevents inconsistencies in coordinate system or scale from being misinterpreted by the model as differences in anatomical structures, thus interfering with feature extraction. The goal is to allow the model to focus more on learning realistic anatomical patterns, enabling it to accurately compare and learn spatial features between different images, thereby improving the model's learning efficiency and generalization ability.

[0199] 6. Data augmentation (geometric augmentation)

[0200] Geometric augmentation is performed only on the training set data to increase the diversity of sample features, improve model accuracy, and prevent overfitting. Geometric augmentation includes:

[0201] (1) Random flip: Randomly flip around the three coordinate axes respectively, namely left and right, front and back, and up and down mirror.

[0202] (2) Random rotation: Rotate randomly around the three coordinate axes by a small angle (±15°) to allow the network to see lesions from different angles.

[0203] 7. Standardize batch processing metrics

[0204] Unifying the spatial scale of batch data. Based on the aforementioned preprocessing, the spatial dimensions (depth, height, width) of images and labeled data from different samples are almost entirely different. During batch processing, it's necessary to combine the images and labels of all samples in a small batch into a single multidimensional tensor. The tensor's shape is (batch_size, num_channels, depth, height, width), representing the batch size, number of channels, depth, height, and width, respectively. Therefore, the spatial scale of each sample in the preprocessed small batch needs to be unified to a fixed size through background padding or cropping. Typically, the batch processing scale needs to be considered from two aspects:

[0205] (1) If the model has requirements on the spatial dimension of the input, the spatial dimension of the model must be satisfied;

[0206] (2) If the model does not have requirements on the size of the spatial dimension of the input, the aforementioned preprocessing can be performed on all samples, and the size of each spatial dimension can be counted. The 95th or 90th percentile of each dimension can be taken as the size of the spatial dimension of the batch processing. This can filter out 5% or 10% of the extreme samples with very large sizes, and the final fixed size will not be too large, and the GPU memory usage can be controlled.

[0207] 8. Data augmentation (grayscale augmentation)

[0208] Gray-scale augmentation is performed only on the training set data to expand the diversity of sample features, improve model accuracy, and prevent overfitting. Gray-scale augmentation includes:

[0209] (1) Randomly change the image brightness and contrast;

[0210] (2) Add a little bit of Gaussian noise to the image randomly.

[0211] 9. CT value truncation and normalization

[0212] Pixel value truncation and normalization are performed based on the standard HU value range for lung images. CT values ​​in the image are truncated and normalized based on the HU value range of the lung window (e.g., -1350 ~ 150). CT values ​​between -1350 and 150 are normalized to [0, 1], values ​​below -1350 are normalized to 0, and values ​​above 150 are normalized to 1. This helps accelerate gradient descent convergence, avoid gradient explosion, and improve the model's generalization ability. In medical image processing, normalization can also reduce bias caused by differences in contrast or brightness in lesion areas. Here, minimum-maximum normalization is used, calculated using the following formula:

[0213]

[0214] The CT value before normalization. The normalized value. .

[0215] 1.3.4 Model Construction

[0216] A binary classification model is built based on the initialized 3D ResNet-50 backbone network. The model consists of a backbone feature extraction network and an adaptive classification head.

[0217] 1. Model Structure

[0218] The classification network structure using 3D ResNet-50 as the backbone is shown in the attached figure. Figure 5 As shown, the network structure includes:

[0219] (1) Backbone Network (Feature Extraction Part): Pre-trained weights of MedicalNet have been loaded. It consists of a head (Stem, corresponding to the first convolutional layer Conv3d + Batch Normalization BN + non-linear activation ReLU and MaxPool3d) and four stages (Stage 1~4, corresponding to Layer 1~Layer 4). Layer 1~Layer 4 are composed of 3, 4, 6 and 3 residual blocks ResNetBottlenecks, respectively, with the number of output channels increasing progressively. Layer 1 does not change the spatial dimension scale of the input, while the first residual block in Layer 2~Layer 4 progressively reduces the spatial dimension of the input.

[0220] (2) Classification Head (Feature Classification Part): Composed of global average pooling (AdaptiveAvgPool3d), feature flattening layer, and fully connected layer (Linear). The global average pooling layer compresses the input space dimension [depth, height, width] from the backbone network to [1, 1, 1], and its output tensor dimension is [batch_size, num_channels, 1,1, 1]. It is then processed by the flattening layer to [batch_size, num_channels], and finally fed into the fully connected layer. Here, the output dimension of the classification head is 2 to match the binary labels of the target disease group and the non-target disease group.

[0221] In the diagram, the numbers at the top represent the number of input (for the input image) or output channels. Layers without labeled output channels indicate that the number of input channels remains unchanged. The numbers at the bottom represent the change in spatial dimension from input to output. For simplicity, we use one dimension (width w) of the input image for illustration; the rules for other dimensions are the same. In the flattening layer of the classification head, the features extracted from the input image are flattened into one-dimensional features, eliminating the spatial dimension.

[0222] 2. Parameter Freezing and Initialization

[0223] (1) The newly added classification head weights are initialized using Xavier or Kaiming to ensure gradient stability in the early stages of training.

[0224] (2) Freeze the backbone network parameters and train only the newly added classification head. This allows the model to learn how to map general features to the concept of "lung abnormalities" in the current task without destroying pre-trained knowledge. This process backpropagates gradients (although the weights are not updated), allowing the model to "perceive" the distribution characteristics of the new data, laying the foundation for subsequent fine-tuning. This stage is equivalent to a "buffered training" stage, allowing the model to converge on a semantically similar task (anomaly detection), significantly reducing training fluctuations during the second stage of fine-tuning and improving stability. During the training process in this stage, even if the backbone network parameters are frozen, its internal activation patterns will gradually adapt to the typical features of lung lesions (such as nodules, ground-glass opacities, etc.). This "implicit tuning" makes the backbone network more adept at responding to visual patterns related to lung diseases when entering the second stage, thereby accelerating the convergence of fine-classification tasks. The training results in this stage are not stored in the classification head, but are "precipitated" in the backbone network's response to the input data.

[0225] 3. Loss Function

[0226] We use the CrossEntropyLoss loss function. For a binary classification problem, the loss function is:

[0227]

[0228] For multi-class classification problems, the loss function is:

[0229]

[0230] in Represents the true category label, Represents the model's predicted value. This represents the number of categories. The smaller the cross-entropy value, the closer the model's predicted distribution is to the true distribution, and the better the model's performance.

[0231] 4. Optimizer

[0232] Using the Adam optimizer with a learning rate of 1e-4, we can quickly train newly added classification heads.

[0233] 1.3.5 Model Training

[0234] Use the merged training and validation sets for training and validation. Set the number of training iterations (epoches) to 300, and perform validation every two training iterations.

[0235] 1. Quantization model classification performance

[0236] To quantify the classification performance of the model, the continuous probability distribution values ​​output by the model need to be mapped to discrete category labels (0 for the target disease group and 1 for the non-target disease group), and compared with the true labels of the samples in the validation set to calculate relevant evaluation indicators.

[0237] For a binary classification model, where the model output is a two-dimensional probability distribution vector corresponding to two sample categories (category labels 0 and 1), let P0 be the probability for category 0 and P1 be the probability for category 1. Then, P0 + P1 = 1. There are two mapping schemes in this case:

[0238] Option 1: Set a threshold of 0.5 as the boundary for binary classification, and the determination rules are as follows:

[0239] (1) If The sample is predicted to belong to category 0.

[0240] (2) If The sample was determined to be of category 1.

[0241] Option 2: Select the category corresponding to the larger value as the final predicted category for the sample. Specific logic:

[0242] (1) If The sample was classified as a class 0 sample.

[0243] (2) Otherwise, the sample is classified as a Class 1 sample;

[0244] For three-class or multi-class classification models, the model output is a three-dimensional or multi-dimensional probability distribution vector, corresponding to three or more sample categories (category labels 0, 1, 2, ...). Let the probabilities corresponding to different categories be P0, P1, P2, ..., respectively. The mapping rule is: select the category corresponding to the maximum value as the final predicted category of the sample. Specific logic:

[0245] (1) If P0 is the largest, the sample is classified as class 0.

[0246] (2) If P1 is the largest, the sample is classified as a Class 1 sample;

[0247] (3) If P2 is the largest, the sample is classified as a class 2 sample;

[0248] ...

[0249] In the case of binary classification (label 1 is the positive class and label 0 is the negative class), the following confusion matrix is ​​used as an example to illustrate the calculation method of relevant evaluation indicators.

[0250] Predicted Category True Category Predict the negative class (label 0) Predict the positive class (label 1) Positive class (label 1) FN TP Negative class (label 0) TN FP

[0251] Accuracy:

[0252]

[0253] accuracy:

[0254]

[0255] Sensitivity or recall:

[0256]

[0257] Specificity:

[0258]

[0259] :

[0260]

[0261] Models are typically evaluated based on accuracy metrics. For binary classification, the accuracy of the positive class is called sensitivity, and the accuracy of the negative class is called specificity. For ternary or multi-class classification, the accuracy of each class is called recall, or one class of interest can be designated as the positive class and the remaining classes as the negative class, and sensitivity and specificity can be calculated.

[0262] 2. Adjust model parameters

[0263] During training, the optimizer adjusts the model parameters based on cross-entropy loss and iterates through the training process until the loss converges and the evaluation metrics on the validation set reach the expected usable results.

[0264] 3. Save model parameters

[0265] During the initial validation, the validation result metrics and model parameters are saved. In subsequent validations, if the result metrics are better than the previously saved ones, the previously saved model parameters are saved and replaced; otherwise, the model parameters are not saved.

[0266] 1.4 Second Phase: Fine-tuning of Goals and Tasks (Detailed Classification)

[0267] Based on the domain adaptive model trained in the first stage, the model in the first stage is fine-tuned using only image data of samples in the target disease group. The aim is to allow the model to focus on learning the subtle differences in features (such as structure, density, texture, cavity patterns, edge burrs, growth rate, etc.) between different categories of lung infection or non-infection in the target disease group, accurately capture fine-grained representation differences, achieve fine classification of multiple types of samples within the target disease group, and output the probability distribution vector of the image features of the input sample corresponding to the category of lung infection or non-infection in the target disease group.

[0268] 1.4.1 Establishing a Data Dictionary

[0269] For samples of different subcategories within the target disease group, an independent data dictionary is constructed, and a mapping relationship is established between image files, lesion annotation files, and subcategories, providing structured data support for refined classification modeling.

[0270] For different categories of sample datasets in the target disease group, a data dictionary is established, that is, a mapping relationship is established between image data (including loading path), lesion annotation data (including loading path), and category labels. Taking binary classification as an example (multi-class classification is supported), the category label of disease A in the target disease group is set to 0, and the category label of disease B is set to 1.

[0271] The following is an example of the data dictionary for the sample dataset included in disease A:

[0272] dataset_dict_a = [{'image': / path_to_image / a / image001.nii.gz,'mask': / path_to_mask / a / image001.nii.gz, 'label': 0}, {'image': / path_to_image / a / image002.nii.gz,'mask': / path_to_mask / a / image002.nii.gz, 'label': 0},...]

[0273] The following is an example of the data dictionary for the sample dataset included in disease B:

[0274] dataset_dict_b = [{'image': / path_to_image / b / img001.nii.gz,'mask': / path_to_mask / b / img001.nii.gz, 'label': 1}, {'image': / path_to_image / b / img002.nii.gz,'mask': / path_to_mask / b / img002.nii.gz, 'label': 1},...]

[0275] 1.4.2 Splitting the Dataset

[0276] The dataset partitioning rules established in the first phase are reused to split the target disease group samples into training, validation, and test sets. The training and validation data are aggregated, while the independent test set is retained for final performance evaluation.

[0277] 1. Read the partitioning results of disease A and disease B sample datasets from the first-stage partitioning result dataset_split_stage1.json, and partition the newly created data dictionaries dataset_dict_a and dataset_dict_b into training, validation, and test sets respectively:

[0278] dataset_dict_a = dataset_dict_a_train + dataset_dict_a_val + dataset_dict_a_test

[0279] dataset_dict_b = dataset_dict_b_train + dataset_dict_b_val + dataset_dict_b_test

[0280] The splitting results are then saved to dataset_split_stage2.json, containing only data for the target disease group.

[0281] 2. Training and validation sets of merged disease A and disease B sample datasets

[0282] dataset_dict_train = dataset_dict_a_train + dataset_dict_b_train

[0283] dataset_dict_val = dataset_dict_a_val + dataset_dict_b_val

[0284] The test set is used for the final model performance evaluation and is not needed at this time.

[0285] 1.4.3 Image Data Preprocessing

[0286] Compared to the first-stage preprocessing workflow, only the following adjustments are made:

[0287] 1. Replace the lung parenchyma region with manually labeled lesion regions in the foreground cropping and unified batch processing scale, focusing on the extraction of refined lesion features and filtering out interference from irrelevant regions.

[0288] 2. Swap the order of steps 4 and 5; that is, first unify the spatial orientation and resampling, then perform foreground cropping. The reason is as follows:

[0289] The first stage involves coarse cropping of a large area of ​​lung parenchyma, primarily removing non-lung background areas such as the surrounding thorax, chest wall, heart, and major blood vessels. This significantly reduces the image size, improves the speed of subsequent resampling and interpolation, and greatly reduces computational load. Furthermore, since lung parenchyma is an anatomical structure, cropping before resampling does not affect accuracy or the spatial structure of the lung parenchyma region.

[0290] The second stage involves fine-tuning based on the VOI of the lesion region. All cases must be cropped at the same physical scale (spatial orientation and voxel spacing) to ensure comparability of lesion size and contextual range, allowing the model to learn correctly. If lesion region cropping is performed first, followed by resampling, the lesions will be stretched / compressed and deformed, resulting in inconsistent scales across multiple cases and hindering the model's ability to learn fine features.

[0291] 1.4.4 Model Construction

[0292] 1. Model Structure

[0293] As attached Figure 6 As shown, the network structure includes:

[0294] (1) Backbone network (feature extraction part): reuse the 3D ResNet-50 backbone network and load the optimal model parameters of the first stage to complete the initialization.

[0295] (2) Classification Head (Feature Classification Part): A completely new sub-classification head replaces the classification head of the first stage to adapt to the fine classification task, with the same structure as the first stage. The first stage is "coarse-grained" classification (target disease group and non-target disease group), and the second stage is "fine-grained" classification (disease A and disease B in the target disease group). The decision boundary learned by the classification head in the first stage is to distinguish between "abnormal" and "normal / background" (in binary classification, one category is defined as the negative class and the other as the positive class), while the second stage needs to distinguish between two specific "abnormal" patterns. The second stage mainly fine-tunes the high-level features of the backbone network, but the classification head, as the final decision layer, needs to adapt to the new label space. Using a randomly initialized new classification head allows the model to explore the differential features between disease A and disease B more flexibly in the early stages of fine-tuning, without being constrained by the decision boundary of the first stage, thus avoiding erroneous prior bias.

[0296] 2. Parameter Freezing and Initialization

[0297] We still use Xavier or Kaiming to initialize the newly added class header weights to ensure gradient stability in the early stages of training.

[0298] A hierarchical parameter fine-tuning strategy is adopted, freezing the shallow parameters of the backbone network and only unfreezing the high-level feature layers and the new classification head: freezing the weight parameters of the head and Stages 1-3 in the backbone network, and training only Stage 4 and this newly added classification head in the backbone network.

[0299] 3. Loss Function

[0300] We will still use the cross-entropy loss function, CrossEntropyLoss.

[0301] 4. Optimizer

[0302] Using the Adam optimizer and a hierarchical learning rate strategy: the high-level feature layers are fine-tuned with a low learning rate to retain general feature extraction capabilities and prevent damage to existing representations. The learning rate for Stage 4 is set to 1e-5. The classification head uses a higher learning rate to quickly iterate and adapt to the subdivided tasks, avoiding model overfitting and feature loss problems. The learning rate for the new classification head is set to 1e-4.

[0303] 1.4.5 Model Training

[0304] The training process is the same as in the first stage. A maximum of 300 iterations is uniformly set, maintaining the same validation frequency and model saving rules as in the first stage. The model's fine-grained feature extraction capability is iteratively optimized, and the optimal model parameters for the subdivided classification tasks are saved.

[0305] 1.4.6 Visualization of Model Attention

[0306] The Grad-CAM (Gradient-weighted Class Activation Mapping) visualization algorithm is used to generate a heatmap of model attention, locate the core areas of interest for model feature extraction, verify whether the model focuses on the effective areas of lesions to complete feature learning, intuitively verify the effect of fine-tuning of the model, and identify problems such as underfitting, overfitting and feature learning bias.

[0307] 1.5 Phase 3: Multimodal Fusion Fine-tuning (Performance Enhancement)

[0308] Based on the second-stage refined image classification model, an adaptive multi-branch multi-level MLP fusion architecture is built by integrating 3D CT image features and structured clinical features. This architecture explores the inherent complementarity and synergy of heterogeneous multimodal data, makes up for the limitations of single image features, and improves the model's feature representation ability and generalization performance in complex sample scenarios. The output is the probability distribution vector of different categories of lung infection or non-infection in the target disease group corresponding to the multimodal features of the input sample.

[0309] The main advantages of multimodal information include:

[0310] (1) To make up for the limitations of single image information

[0311] CT images primarily characterize changes in lung anatomy (such as nodule size, density, and morphology), but cannot fully reflect physiological state-related features and individual background information. Some inflammatory samples and early abnormal lesion samples show high similarity in CT image characterization. By fusing information such as smoking history, age, physical signs, and blood test indicators corresponding to the samples, the accuracy of multimodal data modeling and feature classification can be effectively improved.

[0312] (2) Achieving the characteristic synergistic gain effect of "1+1>2"

[0313] The multimodal fusion approach simulates the multidimensional comprehensive judgment logic of professional physicians: it relies on imaging data to obtain anatomical structural features, and combines functional signals and clinical text information to supplement related features. By fusing imaging features with multidimensional information such as clinical text, synergistic feature gains can be formed, improving the accuracy of lung lesion feature grading, status assessment, and development trend prediction.

[0314] (3) Solving the problem of identifying long-tailed samples and fuzzy feature boundaries

[0315] For samples with atypical imaging features and ambiguous boundaries, clinical correlation data can provide key feature support. In the scenario of analyzing the intervention response trend of lung lesions, a multimodal model is constructed by integrating plain and enhanced CT imaging features with clinicopathological correlation data, which has better feature identification and quantitative analysis capabilities than a single imaging model.

[0316] 1.5.1 Clinical Data Preprocessing

[0317] Before establishing the data dictionary and partitioning the dataset, all clinical data underwent standardized preprocessing to unify the data format and feature dimensions. The main reasons are as follows:

[0318] (1) Filling missing values ​​in a feature column requires using all values ​​in that feature column. For example, fill with the median, mean, or mode of all values ​​in that feature column.

[0319] (2) For the normalization of continuous value characteristic columns, whether using minimum-maximum normalization or mean-standard deviation normalization, it is necessary to calculate the minimum, maximum, mean and standard deviation based on all values ​​of the characteristic column.

[0320] The specific preprocessing process is as follows:

[0321] 1. Load data

[0322] Read the clinical data of all samples in the target disease group.

[0323] 2. Split the associated index columns and feature columns

[0324] The system is divided into non-feature index columns and all feature columns. The non-feature index columns are used to associate with the image data of the samples; the feature columns contain continuous value features and discrete value features, which are used for collaborative fusion training with the image features of the samples.

[0325] 3. Handling missing values

[0326] (1) Identify all feature columns with missing values;

[0327] (2) Count the number of missing values ​​in each missing value feature column;

[0328] (3) Process each missing value feature column according to the following different situations:

[0329] Case 1: Handling missing values ​​for continuous features:

[0330] (a) Missing value imputation: For features with uniform distribution and no extreme values, the mean of all values ​​of the feature can be used to imput them; for features with extreme values, the median of all values ​​of the feature can be used to imput them, which is more robust.

[0331] (b) Add missing value marker columns: Add missing value marker columns to all missing value feature columns to preserve the implicit information inherent in missing data. For example, if the "height" feature column has missing values, after filling in the missing values ​​in this feature column, add a "height_missing value marker" feature column. If the original value of the corresponding "height" feature is true, the value of this missing value marker feature is 0; if the value of the corresponding "height" feature is a value filled in after missing values, the value of this missing value marker feature is 1. The purpose is to let the model know whether the value of the "height" feature column is a true value or a value filled in after missing values. Moreover, in some cases, missing values ​​are not random; the missing values ​​themselves may carry information, such as critically ill patients who were not tested.

[0332] Case 2: Handling missing values ​​for categorical or discrete features:

[0333] (a) Missing value imputation: If the missing value rate is 50% or higher, either treat the missing value as an independent category or delete the feature directly (because there is not enough effective information). Otherwise, imput the feature with the mode of all values.

[0334] (b) Add the corresponding missing marker feature column: the method is the same as above.

[0335] 4. Continuous value normalization

[0336] Each continuous feature is normalized separately to eliminate dimensional differences, accelerate model convergence, improve algorithm accuracy, and avoid feature weight imbalance. Method:

[0337] (a) If the feature values ​​are relatively uniformly distributed, use the min-max normalization method to scale all values ​​of the feature to... The calculation formula is as follows:

[0338]

[0339] This represents the feature vector composed of all the original values ​​of this feature. , These represent the maximum and minimum values ​​in the vector, respectively. This represents the normalized eigenvector.

[0340] (b) If the data contains outliers or the distribution is unknown, the mean-standard deviation normalization method is used, and the calculation formula is as follows:

[0341]

[0342] This represents the feature vector composed of all the original values ​​of this feature. This represents the arithmetic mean of all values ​​in the vector. This represents the standard deviation of all values ​​in the vector. This represents the normalized eigenvector.

[0343] If the distribution of features cannot be determined, it is recommended to use the mean-standard deviation normalization method.

[0344] 5. Discrete-valued vectorization

[0345] Convert all discrete value feature columns (including the missing label feature columns corresponding to each added missing value feature column) into one-hot vector format: the length of the vector is the number of class values ​​of the feature, and when a class value exists, the corresponding position is set to 1, and the other positions are set to 0.

[0346] 1.5.2 Establishing a Data Dictionary

[0347] A multimodal data dictionary is constructed to establish a four-dimensional mapping relationship between image files, lesion annotation files, standardized clinical feature vectors, and category labels, thereby achieving precise binding between image data and clinical data.

[0348] For different categories of sample datasets in the target disease group, a data dictionary is established, namely, a mapping relationship is established between image data (including loading path), lesion annotation data (including loading path), clinical data (already converted into feature vectors), and category labels. Taking binary classification as an example, the category label of disease A in the target disease group is set to 0, and the category label of disease B is set to 1.

[0349] The following is an example of the data dictionary for the sample dataset included in disease A:

[0350] dataset_dict_a = [{'image': / path_to_image / a / image001.nii.gz,'mask': / path_to_mask / a / image001.nii.gz, 'clinic': clinical_features[index_to_image001], 'label': 0}, {'image': / path_to_image / a / image002.nii.gz,'mask': / path_to_mask / a / image002.nii.gz, 'clinic': clinical_features[index_to_image002], 'label': 0},...]

[0351] The following is an example of the data dictionary for the sample dataset included in disease B:

[0352] dataset_dict_b = [{'image': / path_to_image / b / img001.nii.gz,'mask': / path_to_mask / b / img001.nii.gz, 'clinic': clinical_features[index_to_img001], 'label': 1}, {'image': / path_to_image / b / img002.nii.gz,'mask': / path_to_mask / b / img002.nii.gz, 'clinic': clinical_features[index_to_img002], 'label': 1},...]

[0353] 1.5.3 Splitting the Dataset

[0354] The dataset partitioning rules of the second stage are reused to collect multimodal training and validation data, while retaining an independent test set, ensuring that the dataset partitioning standards are consistent throughout the entire process.

[0355] 1. Read the partitioning results of disease A and disease B sample datasets from the second-stage partitioning result dataset_split_stage2.json, and partition the newly created data dictionaries dataset_dict_a and dataset_dict_b into training, validation, and test sets respectively:

[0356] dataset_dict_a = dataset_dict_a_train + dataset_dict_a_val + dataset_dict_a_test

[0357] dataset_dict_b = dataset_dict_b_train + dataset_dict_b_val + dataset_dict_b_test

[0358] 2. Then, merge the training and validation sets of the disease A and disease B sample datasets.

[0359] dataset_dict_train = dataset_dict_a_train + dataset_dict_b_train

[0360] dataset_dict_val = dataset_dict_a_val + dataset_dict_b_val

[0361] Note: Although clinical features are encapsulated in a dictionary and loaded into the data stream at the beginning of this stage, clinical features are only used after the model backbone network has extracted the image features.

[0362] 1.5.4 Image Data Preprocessing

[0363] The standardized preprocessing workflow for lesion-level images in the second stage is fully adopted to ensure uniformity in image feature extraction standards.

[0364] 1.5.5 Model Building

[0365] 1. Model Structure

[0366] As attached Figure 7 As shown, the network structure includes:

[0367] (1) Backbone network (feature extraction part): reuse the 3D ResNet-50 backbone network and load the optimal model parameters of the second stage to complete the initialization.

[0368] (2) Classification Head (Feature Classification Part): Replaced with a learnable, adaptive multi-branch, multi-level MLP (Multilayer Perceptron) feature fusion classification head to construct a multimodal model with dual-branch feature extraction and mid-term fusion. The specific architecture is as follows:

[0369] (a) Image Feature Branch MLP

[0370] By employing multi-layer fully connected layers, batch normalization, activation, and random discarding operations, the high-dimensional image features extracted by the model's backbone network are progressively reduced in dimensionality, redundant features are compressed, and the feature magnitude is unified.

[0371] Image features are numerous and high-dimensional; directly fusing them with clinical features can overwhelm the context of the clinical features. Adaptive dimensionality reduction using MLP is better than manual dimensionality reduction (such as PCA principal component analysis).

[0372] The first global average convergence layer of this branch compresses the input spatial dimension [depth, height, width] from the backbone to [1, 1, 1], and its output tensor dimension is [batch_size, num_channels, 1, 1, 1], which is then processed by the second flattening layer to [batch_size, num_channels] for output.

[0373] The third layer of this branch includes a fully connected layer followed by batch normalization (BN), nonlinear activation (ReLU), and dropout, which reduces the dimensionality of the 2048 features extracted from the backbone network to 256.

[0374] The fourth layer of this branch includes a fully connected layer, followed by batch normalization (BN) and nonlinear activation (ReLU), which further reduces the dimensionality of the 256 features output from the third layer to 128, so as to unify the "feature magnitude" (within the same order of magnitude, but not necessarily equal) with the number of features output from the clinical feature branch MLP.

[0375] (b) Clinical characteristic branch MLP

[0376] By employing fully connected, normalized, and activated operations, the low-dimensional linear clinical features are appropriately upscaled to uncover nonlinear correlation information among clinical features, balance the weights of bimodal features, and avoid single-modal features dominating the modeling process.

[0377] The input dimension of this branch comes from the length of the clinical feature vector in the input stream, obtained based on the key 'clinic' in the data dictionary. This branch is used to increase the dimensionality of clinical features. The number of output features for this branch needs to be adjusted based on the length of the clinical feature vector, typically twice the length of the input clinical feature vector (and also a multiple of 2). For example, if the length of the clinical feature vector is 63, the number of output features for this branch can be set to 128. The purpose of increasing the dimensionality of clinical features is:

[0378] (i) Unify the “feature magnitude” to avoid imaging features “overwhelming” clinical features. Only when the two are of similar magnitude and have equal status can the model truly learn the combined rules of “imaging + clinical”.

[0379] (ii) Enabling "non-linear expressive power" for clinical features. This branch includes a fully connected layer followed by batch normalization (BN) and non-linear activation (ReLU). Many of the original features are linear, low-order information. Through this branch, the model can learn the interactions between features (such as age × gender, tumor size × stage), mapping low-dimensional, simple features to a high-dimensional, abstract feature space. This is equivalent to performing automatic feature engineering on clinical features, giving them more "expressive space".

[0380] (iii) Slight dimensionality increase will not overfit. Clinical features are usually few. By increasing the dimensionality and combining it with batch normalization, overfitting is almost impossible. On the contrary, it can improve the generalization ability.

[0381] (c) Multimodal fusion branch MLP

[0382] Adaptive intermediate fusion learning is performed on the image features output by the image feature branch MLP and the clinical features output by the clinical feature branch MLP through a multi-layer MLP. This branch consists of two layers:

[0383] The first layer of this branch consists of a fully connected layer followed by batch normalization (BN), nonlinear activation (ReLU), and dropout. Its inputs are image features from the image feature branch MLP and clinical features from the clinical feature branch MLP. Through nonlinear interactions between modalities and mapping to a new joint representation space, a new, more compact, and more meaningful joint feature representation is generated. This new feature representation can capture more complex relationships between image and clinical features, such as complementarity, correlation, and synergy. This layer outputs 64 features.

[0384] The second layer of this branch includes a fully connected layer, which takes 64 features from the output of the first layer as input and outputs a probability distribution vector with a length equal to the number of categories (2 for binary classification).

[0385] 2. Parameter Freezing and Initialization

[0386] The newly added classification head weights are initialized using Xavier or Kaiming to ensure gradient stability during the early stages of training.

[0387] Backbone network parameter freezing strategy: selective freezing + joint fine-tuning.

[0388] (1) Initial stage: Freeze the image backbone network and train only the newly added multimodal fusion classification head. The purpose is to let the model learn "how to combine" the two modalities first.

[0389] (2) Mid-term: Unfreeze the high-level parameters (Stage 3~4) in the image backbone and fine-tune them together with the new classification head. Learn cross-modal interactions and adjust image features to better support multimodal feature fusion analysis.

[0390] (3) Later stage: Fully unfreeze the backbone network parameters, use an extremely low learning rate for end-to-end optimization, and gradually achieve deep collaborative adaptation of multimodal features.

[0391] 3. Loss Function

[0392] We will still use the cross-entropy loss function, CrossEntropyLoss.

[0393] 4. Optimizer

[0394] The Adam optimizer and hierarchical learning rate strategy are used, corresponding to the backbone network parameter freezing strategy: initially, the new classification head uses a learning rate of 1e-5; in the middle stage, the backbone network uses a learning rate of 1e-6 and the classification head uses a learning rate of 1e-5; in the later stage, a learning rate of 1e-6 is used for all.

[0395] 1.5.6 Model Training

[0396] The training process is divided into three stages, corresponding to the three stages of the parameter freezing strategy:

[0397] 1. Initial stage: Set the number of training iterations to 100, and set a verification to be performed every two training iterations.

[0398] 2. Mid-term: Set the number of training iterations to 50, and set a verification to be performed every two training iterations.

[0399] 3. Later stage: Set the number of training iterations to 30, and set a verification to be performed every two training iterations.

[0400] In the mid-term, the model is trained based on the final iteration of the initial stage, and in the late-term, the model is trained based on the final iteration of the mid-term stage. The training process for each stage is the same as that for the first and second stages mentioned above.

[0401] Maintain a consistent verification frequency and model saving rules, optimize the multimodal fusion strategy step by step, iteratively optimize the overall representation performance of the model, and save the globally optimal multimodal fusion model.

[0402] 1.6 Model Evaluation and Deployment

[0403] The final model after the third-stage fine-tuning was evaluated using an independent test set that was not involved in the model training process.

[0404] The evaluation process is as follows:

[0405] 1. Clinical data preprocessing: Same as Phase III

[0406] 2. Establish a data dictionary: Same as in the third stage.

[0407] 3. Split the dataset: Read the test set split results of disease A and disease B sample datasets from the split results dataset_split_stage2.json saved in the second stage, and split the test sets of disease A and disease B datasets respectively from the created data dictionary.

[0408] 4. Image data preprocessing, same as in stage three.

[0409] 5. Merge datasets: Merge the test sets of disease A and disease B sample datasets.

[0410] 6. Model Building: Create a model object and load the optimal model parameters from the third stage to complete the initialization.

[0411] 7. Classification Prediction: Outputs the probability distribution vector of the multimodal features of samples in the test set corresponding to their respective categories.

[0412] 8. Model Evaluation: Map the continuous class probability distribution output by the model to discrete class labels and compare them with the true labels of the samples in the test set to quantify the model performance.

[0413] The model trained by this invention can be deployed on medical image processing terminals, cloud servers, and medical workstations for automated verification, processing, analysis, and classification modeling of multimodal lung medical data, realizing the practical application of intelligent medical data processing.

[0414] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0415] The image processing apparatus of this invention has been described above from the perspective of functional modules in conjunction with the accompanying drawings. It should be understood that this functional module can be implemented in hardware, in software instructions, or in a combination of hardware and software modules. Specifically, in the medical data processing method of this invention, the combined hardware and software execution process of each step is as follows:

[0416] During the data preprocessing stage, when the software instructions corresponding to the processing module are called, the processor reads the DICOM file header information of the lung 3D CT image through the file I / O interface, extracts the coordinate values ​​of specific labels and parameters such as slice thickness through the arithmetic logic unit, and performs floating-point operations and logical comparisons in the register to complete the availability and consistency verification of the image data, and caches the qualified image data into memory.

[0417] In the medical-specific pre-trained weight initialization phase, the processor reads the MedicalNet pre-trained weight file from the storage medium, parses the weight tensor in memory and loads it into the graphics processor. Then, by modifying the model computation graph, the decoder submodule is removed at the underlying hardware level and replaced with an identity mapping, thereby completing the physical structure reconstruction and weight initialization of the 3D ResNet backbone network.

[0418] During the multi-stage progressive transfer learning fine-tuning phase, the tensor computation core of the graphics processing unit (GPU) is frequently invoked to perform 3D convolution operations on the 3D ResNet backbone network. In the first and second stages, the hardware-level gradient update mechanism, based on a hierarchical fine-tuning strategy, directly truncates the gradient calculation of shallow parameters during backpropagation, updating only high-level parameters and classification head parameters. In the third stage, during multimodal collaborative fusion fine-tuning, the GPU's storage loading unit simultaneously retrieves image feature vectors and clinical feature vectors from GPU memory. The dual-modal features are physically stitched together at the physical level via the underlying bus, then enter the matrix multiplication acceleration array of the multimodal fusion branch. Dimensionality reduction mapping of the image feature branch and dimension increase mapping of the clinical feature branch are performed sequentially. In the fusion branch, a joint feature representation is calculated using a nonlinear activation function. Finally, the class probability distribution electrical signal is output through a fully connected layer.

[0419] In summary, the medical data processing method steps in the embodiments of the present invention can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. This hardware-software collaborative implementation method, which combines specific medical image segmentation, multimodal feature fusion, and a phased parameter freezing strategy, should not be considered beyond the scope of protection of the present invention. Regarding the apparatus in the above embodiments, the specific methods by which each module performs its operations have been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0420] The key point of this invention is that, in order to overcome the technical problems of poor cross-domain adaptability, coarse multi-dimensional feature fusion, insufficient feature mining, and poor modeling performance with small samples in existing medical modeling, this invention builds a progressive multi-stage transfer learning framework, and optimizes model performance by combining medical-specific pre-trained weights and a multi-modal feature collaborative fusion strategy. The specific technical implementation points are as follows:

[0421] Key Point 1: The model is initialized using MedicalNet medical-specific pre-trained weights, replacing the traditional natural image pre-trained weights. This alleviates the cross-domain offset problem from the source, enabling the model to have the feature representation ability of lung CT images and improving the technical shortcomings of existing technologies where the adaptability of natural image pre-trained features and medical image features is mismatched.

[0422] Key Point 2: Design a multi-stage progressive fine-tuning process, sequentially executing domain adaptation, refined target category representation, and multimodal feature fusion, with each stage progressing step by step; relying on massive lung CT image data, the model learns the common features of lung images, then mines the differentiated representation features corresponding to target samples, and finally further improves the model's representation capabilities through multimodal feature fusion, realizing the step-by-step optimization and in-depth mining of image features, making up for the technical defects of insufficient transfer learning adaptation strategies and insufficient feature utilization.

[0423] Key Point 3: Adopting a learnable, adaptive, multi-branch, multi-level MLP feature fusion method to deeply integrate 3D CT image features with clinical structured data features, abandoning the traditional simple feature stitching mode; adaptively mining the intrinsic correlation and complementarity between the two types of features, giving full play to the representational advantages of multi-dimensional data, and improving the technical shortcomings of existing multimodal fusion methods such as coarse representation and insufficient feature synergy utilization.

[0424] Key Point 4: A fixed-division training set, validation set, and independent test set are adopted, with the independent test set not participating in model training throughout the process; at the same time, a stratified random sampling method is used to ensure a balanced distribution of the dataset, and a multi-stage learning rate decay strategy is used to effectively suppress model overfitting in small sample scenarios, improve model generalization performance and experimental reproducibility, and ensure model modeling stability.

Claims

1. A medical data processing method based on multi-stage transfer learning and multi-modal data collaborative fusion, characterized in that, Includes the following steps: S1. Data Preprocessing: Acquire 3D CT images of the lungs of the target disease group and matching clinical structured data, as well as 3D CT images of the lungs of non-target disease groups. Perform validity verification, lung parenchyma segmentation, consistency verification with labeled data, and standardized preprocessing on the CT images; perform validity verification and standardized preprocessing on the clinical data. S2. Initialization of Medical Pre-trained Weights: The 3D ResNet backbone network is initialized using MedicalNet weights pre-trained on various 3D medical image datasets to mitigate cross-domain offset between natural images and medical images. S3, Multi-stage progressive transfer learning fine-tuning: S31. First stage: Domain adaptive fine-tuning. Using images of the target disease group and non-target disease group after lung parenchyma segmentation, a binary classification model is trained to distinguish between the target disease group and non-target disease group. All backbone network parameters are frozen, and only the classification head is trained. Based on the extracted macroscopic image features, the probability distribution vector of the category of the input sample corresponding to the target disease group and non-target disease group is output. S32. Second stage: fine-tuning of the target task, loading the optimal model of the first stage, freezing the shallow network parameters, unfreezing the high-level network parameters, using fine-grained annotations of images and lesions within the target disease group for fine-tuning of classification, and outputting the probability distribution vector of different categories within the target disease group corresponding to the input sample based on the extracted fine-grained lesion features. S33. Third stage: Multimodal collaborative fusion fine-tuning. Based on the second stage model, an adaptive multi-branch multi-level MLP fusion architecture is adopted to deeply fuse deep image features with clinical features. Based on the extracted multimodal joint features, the probability distribution vector of different categories within the target disease group corresponding to the input sample is output. S4. Model Evaluation and Deployment: Evaluate model performance using an independent test set and deploy the model on a medical data processing terminal or server.

2. The method according to claim 1, characterized in that, The validity verification of CT images, consistency verification with labeled data, and standardized preprocessing include: Perform the following checks in a preset order: non-DICOM slice verification and exclusion, DICOM slice validity verification, DICOM slice naming validity verification and correction, duplicate DICOM slice verification and exclusion, DICOM slice layer thickness correlation minimum slice number verification, DICOM slice continuity verification, and DICOM slice layer spacing consistency verification. Compare the overall size, coordinate origin, spatial direction, and voxel spacing of the image and the annotation. If there is a deviation, the coordinates, spatial direction, and voxel spacing of the annotation data are aligned and reset based on the image data. The system performs standardized preprocessing on CT image data in a preset order. This standardized preprocessing includes adding channel dimensions, foreground cropping based on regions of interest (ROIs), resampling to unify voxel spacing and spatial coordinate system, truncation and normalization based on lung window CT value ranges, standardizing batch processing scales, and performing data augmentation on the training set. Specifically, in the first stage, the ROI-based foreground cropping is performed based on the lung parenchyma region, and this foreground cropping is performed before the resampling to unify voxel spacing and spatial coordinate system adjustment. In the second and third stages, the ROI-based foreground cropping is performed based on the lesion region, and the resampling to unify voxel spacing and spatial coordinate system adjustment is performed before the foreground cropping.

3. The method according to claim 1, characterized in that, The validity verification and standardized preprocessing of clinical data includes: Perform continuous value feature verification, category or discrete value feature verification, missing value format verification, and consistency verification with image data in a preset order; For continuous value features, missing values ​​are filled with the mean or median and a missing value label column is added. For discrete value features with missing values, mode filling or independent class processing is used. Continuous value features are normalized and discrete value features are one-hot encoded.

4. The method according to claim 1, characterized in that, In S31, all samples of different disease types within the target disease group are uniformly labeled as the first category, and all samples outside the target disease group are uniformly labeled as the second category; the training set, validation set, and independent test set are divided in a fixed ratio and remain unchanged throughout the process, and the independent test set does not participate in model training at any stage.

5. The method according to claim 1, characterized in that, The hierarchical fine-tuning strategy in S32 is as follows: load the optimal model parameters of the first stage, replace the first stage classification head with a new randomly initialized classification head; freeze the head module and the first three residual stages in the backbone network, and only unfreeze the fourth residual stage and the new classification head, wherein the learning rate of the fourth residual stage is less than the learning rate of the classification head.

6. The method according to claim 1, characterized in that, The adaptive multi-branch MLP fusion architecture in S33 includes: Image Feature Branch MLP: The high-dimensional image features output by the backbone network are sequentially subjected to full average pooling, flattening, and then dimensionality reduction through two fully connected layers; Clinical Feature Branch MLP: The input clinical feature vector is non-linearly upsized through a fully connected layer so that the output feature dimension is on the same order of magnitude as the output feature dimension of the image feature branch MLP, so as to unify the dual-modal feature magnitude. Multimodal fusion branch MLP: After concatenating the output features of the above two branches, a higher-level joint feature representation is generated through mapping and nonlinear activation interaction in a fully connected layer, and then the class probability distribution vector is generated through the output layer.

7. The method according to claim 6, characterized in that, The multimodal fusion fine-tuning in S33 adopts a staged joint optimization strategy: Initially, the entire image backbone network is frozen, and only the multimodal fusion branch (MLP) is trained. Mid-term unfreezing and joint training of high-level parameters and all fusion branches of the image backbone network; In the later stage, all parameters of the image backbone network were fully unfrozen, and end-to-end joint optimization was performed using a learning rate lower than that in the mid-term.

8. The method according to any one of claims 1 to 7, characterized in that, The method is executed collaboratively by the processor and graphics processor of the medical data processing device. The specific hardware interaction process includes: During the data preprocessing stage, the processor extracts the coordinate values ​​and slice thickness parameters from the DICOM file header of the CT image through the arithmetic logic unit, and performs floating-point operations and logical comparisons in the register to complete the verification, and caches the qualified image data in memory. During the weight initialization phase, the processor parses the MedicalNet pre-trained weight tensor and loads it into the graphics processor. By modifying the model computation graph, the decoder submodule is removed at the underlying hardware level and replaced with an identity mapping to complete the physical structure reconstruction of the 3D ResNet backbone network. In the multi-stage fine-tuning stage, the tensor computation core of the graphics processor performs three-dimensional convolution operations on the 3D ResNet backbone network. The gradient update mechanism at the hardware level truncates the shallow parameter gradients during backpropagation according to the hierarchical fine-tuning strategy. In the third stage, the storage loading unit of the graphics processor synchronously pulls image feature vectors and clinical feature vectors, completes the physical stitching of dual-modal features through the underlying bus, and the matrix multiplication acceleration array sequentially performs the dimensionality reduction mapping of the image feature branch, the dimensionality increase mapping of the clinical feature branch, and the nonlinear activation calculation of the multimodal fusion branch. Finally, the class probability distribution electrical signal is output through the fully connected layer.

9. A medical data processing device, characterized in that, Includes a memory, a graphics processor, and a processor connected to the memory and the graphics processor; The memory is used to store computer programs; The graphics processor is configured to include a tensor computation core, a storage loading unit, and a matrix multiplication acceleration array, for performing three-dimensional convolution operations, physical splicing of feature tensors, and nonlinear mapping calculations. The processor is used to invoke and run the computer program to perform the method as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 8.

Images