A medical image enhancement method and system based on multimodal fusion

Abstract

The present invention discloses a medical image enhancement method and system based on multimodal fusion, comprising: acquiring medical imaging data, including CT images, MRI images, PET images, and ultrasound images; calculating the mutual information value of each semantic label in a cross-modal feature vocabulary to generate an inter-modal feature correlation matrix; performing non-rigid spatial registration on a uniform dimensional image set to generate a geometrically consistent multimodal image set; extracting bone region features, soft tissue region features, and high metabolic region features from the multimodal image set to generate a core feature set; training the core feature set using a deep learning algorithm to generate a segmented image set containing a segmentation mask; and dynamically adjusting the fusion weights according to the regional features of the segmented image set to generate a fused enhanced image. The present invention achieves effective integration of medical imaging information from different modalities through feature extraction, spatial registration, and deep learning fusion of multimodal medical images.

Description

Technical Field

[0001] The present invention belongs to the field of image enhancement, and in particular relates to a medical image enhancement method and system based on multimodal fusion. Background Art

[0002] Medical image processing, by integrating information from multiple imaging techniques, can provide doctors with more comprehensive diagnostic evidence for subsequent clinical decisions. However, existing methods often struggle to fully exploit the unique information of each modality when implementing multimodal fusion. Loss of detail and noise interference are common in fused images, limiting the reliability of the information. A core challenge facing multimodal fusion stems from the differences in information characteristics between modalities. The imaging principles of different modalities result in distinct semantic representations of information. For example, features such as density, weighting, or metabolic activity are difficult to directly compare and map. This semantic disparity hinders the establishment of cross-modal feature sharing and complementarity, further limiting the information integrity of the fused image. Due to insufficient semantic mapping, the core advantage regions of each modality cannot be accurately identified and highlighted during the fusion process. For example, features of bone or soft tissue may be weakened or even lost. Furthermore, dimensional inconsistency between modalities exacerbates this problem. It is difficult to maintain information consistency during spatiotemporal alignment and fusion of image data of different dimensions, resulting in a reduction in the detail representation of the fused image. Furthermore, the lack of adaptive fusion strategies for different regions prevents the fusion process from dynamically adjusting modality weights based on regional characteristics, making it difficult to achieve refined image enhancement. Therefore, how to establish an effective semantic mapping mechanism in multimodal medical image fusion, unify the representation space of data of different dimensions, and design an adaptive fusion strategy to highlight the core advantages of each modality has become a key issue in achieving high-quality image enhancement. Summary of the Invention

[0003] To solve the above technical problems, the present invention provides a medical image enhancement method and system based on multimodal fusion. Among them, a medical image enhancement method based on multimodal fusion includes:

[0004] Acquiring medical imaging data, wherein the medical imaging data includes CT images, MRI images, PET images, and ultrasound images;

[0005] Based on the CT image, MRI image, and PET image, density feature vectors, relaxation time feature vectors, and metabolic feature vectors are extracted in sequence, and a dimensionality reduction algorithm is used to generate a modal feature vector set;

[0006] Based on the modal feature vector set, a clustering algorithm is used to generate a cross-modal feature vocabulary containing semantic labels; wherein the semantic labels include high-density clusters, soft tissue clusters, and high-metabolism clusters;

[0007] Calculating the mutual information value of each semantic tag in the cross-modal feature vocabulary to generate an inter-modal feature correlation matrix;

[0008] According to the inter-modality feature correlation matrix, a spatial transformation algorithm is used to map the ultrasound image, the CT image, and the MRI image to a unified three-dimensional space to generate a unified dimensional image set;

[0009] performing non-rigid spatial registration on the uniform-dimensional image set to generate a geometrically consistent multimodal image set;

[0010] Extracting bone region features, soft tissue region features, and high metabolic region features from the multimodal image set to generate a core feature set;

[0011] Using a deep learning algorithm to train the core feature set to generate a segmentation image set including a segmentation mask;

[0012] The fusion weights are dynamically adjusted according to the regional features of the segmented image set to generate a fused enhanced image.

[0013] Preferably, the process of acquiring medical imaging data includes:

[0014] Extract raw data sets from CT images, MRI images, PET images, and ultrasound images, parse them in DICOM format, and obtain standardized image data;

[0015] For the standardized image data, image preprocessing technology is used. If the image noise is higher than the preset threshold, the Gaussian filtering algorithm is applied to obtain the denoised image data;

[0016] Extracting features from the denoised image data, using a convolutional neural network algorithm to extract features based on pixel distribution of CT images, MRI images, PET images, and ultrasound images to obtain an image feature set;

[0017] According to the image feature set, a principal component analysis algorithm is used to reduce the feature dimension. If the feature redundancy exceeds a preset threshold, low-variance features are eliminated to obtain an optimized feature set.

[0018] A classification model is constructed for the optimized feature set. If the model input is a CT image or an MRI image, the image is classified according to the preset anatomical structure label to obtain an image classification result.

[0019] Extracting an abnormal area from the image classification result, and marking the abnormal area as a potential lesion if the pixel intensity exceeds a preset range, thereby obtaining lesion marking data;

[0020] According to the lesion marking data, the image information of the CT image, MRI image, PET image and ultrasound image is integrated, and a data fusion technology is used to obtain a comprehensive information data set.

[0021] Preferably, the process of extracting feature vectors and generating a modal feature vector set using a dimensionality reduction algorithm includes:

[0022] The CT images, MRI images, and PET images are subjected to secondary denoising and normalization processing using a preprocessing algorithm to obtain the first density image, the first relaxation image, and the first metabolic image respectively;

[0023] For the first density image, using a feature extraction algorithm to calculate pixel grayscale value distribution and texture features to generate a density feature vector;

[0024] For the first relaxation image, using a feature extraction algorithm to calculate T1 and T2 relaxation time distributions to generate a relaxation time feature vector;

[0025] For the first metabolic image, a feature extraction algorithm is used to calculate a standardized uptake value distribution to generate a metabolic feature vector;

[0026] If the dimension of the density feature vector is higher than a preset threshold, a principal component analysis algorithm is used to reduce the dimension of the density feature vector to obtain a second density feature vector;

[0027] If the dimension of the relaxation time feature vector is higher than a preset threshold, a principal component analysis algorithm is used to reduce the dimension of the relaxation time feature vector to obtain a second relaxation time feature vector;

[0028] If the dimension of the metabolic feature vector is higher than a preset threshold, a principal component analysis algorithm is used to reduce the dimension of the metabolic feature vector to obtain a second metabolic feature vector;

[0029] According to the second density feature vector, the second relaxation time feature vector, and the second metabolic feature vector, the feature vectors of each modality are integrated using a feature splicing method to generate a multimodal feature vector set;

[0030] Extracting a weight distribution of each modal feature vector from the multimodal feature vector set, and if the variance of the weight distribution is higher than a preset threshold, normalizing the multimodal feature vector set using a normalization algorithm to obtain a normalized feature vector set;

[0031] For the standardized feature vector set, a clustering algorithm is used to group the feature vectors to generate multimodal feature vector grouping results;

[0032] According to the multimodal feature vector grouping results, a feature mapping method is used to project the grouping results into a low-dimensional space to obtain a final set of modal feature vectors.

[0033] Preferably, the process of generating a cross-modal feature vocabulary containing semantic tags using a clustering algorithm based on the modal feature vector set includes:

[0034] The K-means clustering algorithm was used to cluster the modal feature vector set to determine the initial division of high-density clusters, soft tissue clusters, and high-metabolism clusters.

[0035] Based on the clustering results, a cross-modal feature vocabulary is generated, and semantic labels of high-density clusters, soft tissue clusters, and high-metabolism clusters are assigned to obtain a preliminary vocabulary.

[0036] If the intra-cluster variance in the preliminary vocabulary is greater than a preset threshold, the DBSCAN algorithm is used to perform secondary clustering on the outliers, update the cluster division, and obtain an optimized vocabulary;

[0037] Extracting a set of semantic labels for cross-modal features based on the optimized vocabulary and determining the accuracy of label assignment;

[0038] According to the label assignment results, the boundaries of each cluster in the feature vocabulary are adjusted to obtain the final cross-modal feature vocabulary;

[0039] According to the final feature vocabulary, the consistency between the feature vectors of each modality and the semantic labels is verified, and the cross-modal semantic mapping relationship is determined.

[0040] Preferably, the process of calculating the mutual information value of each semantic tag in the cross-modal feature vocabulary to generate an inter-modal feature correlation matrix includes:

[0041] Obtain a set of feature vectors corresponding to each semantic tag from a cross-modal feature vocabulary, and determine the modal distribution of the feature vectors;

[0042] If the intra-cluster variance of the modal distribution is greater than the preset threshold, the principal component analysis algorithm is used to reduce the dimension of the feature vector to obtain a reduced-dimensional feature set;

[0043] Calculating mutual information values ​​between semantic labels based on the dimension reduction feature set;

[0044] If the mutual information value is lower than a preset threshold, the corresponding label pair is eliminated to obtain a highly correlated label set;

[0045] Extracting label distribution features based on the highly correlated label set and generating a preliminary correlation matrix;

[0046] If the symmetry deviation of the preliminary correlation matrix is ​​greater than a preset threshold, the matrix elements are adjusted by symmetry processing to obtain an optimized correlation matrix;

[0047] Obtain inter-modal feature association weights from the optimized correlation matrix and determine the modal association structure;

[0048] If the connectivity of the modal association structure is lower than a preset threshold, virtual association edges are added to optimize the structure to obtain an enhanced association structure;

[0049] For the enhanced correlation structure, the feature extraction contribution of each modality is calculated, the information interaction strength between the modalities is determined, and the interaction strength distribution is obtained;

[0050] The interaction intensity distribution is used to adjust and optimize the weight distribution in the correlation matrix to generate a final inter-modality feature correlation matrix.

[0051] Preferably, the process of generating a uniform dimensional image set includes:

[0052] Based on the inter-modality feature correlation matrix, shared feature vectors of the ultrasound image, the CT image, and the MRI image are extracted by a matrix decomposition method to obtain a feature extraction result;

[0053] Based on the feature extraction results, a preset spatial transformation algorithm is used to calibrate the coordinate systems of the ultrasound image, the CT image, and the MRI image to generate a preliminarily aligned three-dimensional image set;

[0054] Determining the preliminarily aligned three-dimensional image set, if there is a local mismatch area, adjusting the spatial deviation between the images by an iterative closest point algorithm to obtain a spatially aligned image set;

[0055] Extract the boundary features of each modality image from the spatially aligned image set, generate a boundary description vector of unified dimension through the feature fusion algorithm, and determine the fusion feature set;

[0056] According to the fusion feature set, a stereoscopic microscopy algorithm is used to perform dimensional normalization processing on the three-dimensional image set to generate a uniform dimensional image set;

[0057] For a unified dimensional image set, the consistency between modalities within the image set is detected using a stereo depth analysis algorithm to determine the consistency verification results;

[0058] If the consistency verification result is lower than the preset threshold, the fusion feature set is optimized through the feature weighting adjustment algorithm to regenerate a unified dimensional image set.

[0059] Preferably, the process of performing non-rigid spatial registration on the uniform-dimensional image set to generate a geometrically consistent multimodal image set includes:

[0060] Based on the unified dimensional image set, a preprocessing method is used to unify the image dimensions to obtain a first image set;

[0061] If the resolution or size of the first image set is inconsistent, adjusting it through an interpolation algorithm to determine a second image set;

[0062] For the second image set, performing non-rigid spatial registration using a B-spline registration algorithm to obtain registration parameters;

[0063] Applying a geometric transformation to the second image set using the registration parameters to generate a third image set;

[0064] If the geometric consistency of the third image set does not reach a preset threshold, iteratively optimizing the registration parameters to obtain a fourth image set;

[0065] integrating multimodal data using image fusion technology based on the fourth image set to generate a fifth image set;

[0066] The spatial consistency of the fifth image set is evaluated to ultimately obtain a geometrically consistent multimodal image set.

[0067] Preferably, the process of generating the core feature set includes:

[0068] Using a multimodal image segmentation algorithm, the boundaries of the bone region, soft tissue region, and high metabolic region are obtained from the multimodal image set to obtain the regional segmentation results;

[0069] A convolutional neural network is used to extract features from the segmented bone area, soft tissue area, and high metabolic area to obtain the initial feature set of each area.

[0070] If the dimension of the initial feature set is higher than a preset threshold, the dimension is reduced by principal component analysis to obtain a compressed feature set;

[0071] Based on the compressed feature set, the features of the bone area, soft tissue area and high metabolic area are fused to generate a preliminary feature set;

[0072] If the feature correlation in the preliminary feature set is lower than a preset threshold, redundant features are eliminated through a feature selection algorithm to obtain an optimized feature set;

[0073] By using a data integration method, the optimized feature set is standardized to obtain a core feature set;

[0074] For the core feature set, a clustering algorithm is used to group the features to obtain the classified feature set.

[0075] Preferably, the process of generating a fused enhanced image includes:

[0076] Acquire segmented regions according to the segmented image set, determine region boundaries using a preset segmentation algorithm, and obtain a segmented region set;

[0077] For the segmented region set, extracting a feature vector of each region and determining the region features using a feature extraction algorithm;

[0078] According to the regional characteristics, a dynamic weight allocation method is adopted to determine the fusion weight of each region to obtain a weight set;

[0079] If the weight set meets the preset threshold, the segmented region set is fused and an initial fused image is generated by an image fusion algorithm;

[0080] Obtaining pixel distribution features from the initial fused image, determining enhancement requirements using a pixel analysis method, and obtaining enhancement parameters;

[0081] performing enhancement processing on the initial fused image according to the enhancement parameters, and generating an enhanced fused image by an image enhancement algorithm;

[0082] The final feature vector is extracted from the enhanced fusion image, and it is judged whether it meets the preset fusion quality standard to obtain the final enhanced fusion image.

[0083] The present invention also provides a medical image enhancement system based on multimodal fusion, comprising:

[0084] A medical image data acquisition module is used to acquire medical image data, wherein the medical image data includes CT images, MRI images, PET images and ultrasound images;

[0085] A feature vector extraction module is used to extract density feature vectors, relaxation time feature vectors, and metabolic feature vectors based on the CT image, MRI image, and PET image, and generate a modal feature vector set using a dimensionality reduction algorithm;

[0086] a cross-modal feature vocabulary generation module, configured to generate a cross-modal feature vocabulary containing semantic labels based on the modal feature vector set using a clustering algorithm; wherein the semantic labels include high-density clusters, soft tissue clusters, and high-metabolism clusters;

[0087] an inter-modal feature correlation matrix generation module, configured to calculate the mutual information value of each semantic tag in the cross-modal feature vocabulary to generate an inter-modal feature correlation matrix;

[0088] a unified dimensional image set generation module, configured to map the ultrasound image, CT image, and MRI image to a unified three-dimensional space using a spatial transformation algorithm based on the inter-modality feature correlation matrix to generate a unified dimensional image set;

[0089] a multimodal image set generation module, configured to perform non-rigid spatial registration on the uniform-dimensional image set to generate a geometrically consistent multimodal image set;

[0090] A core feature set extraction module is used to extract bone region features, soft tissue region features and high metabolic region features from the multimodal image set to generate a core feature set;

[0091] A segmentation image set generation module is used to train the core feature set using a deep learning algorithm to generate a segmentation image set including a segmentation mask;

[0092] The fusion enhanced image generation module is used to dynamically adjust the fusion weight according to the regional features of the segmented image set to generate a fusion enhanced image.

[0093] Compared with the prior art, the present invention has the following advantages and technical effects:

[0094] The present invention obtains multimodal medical imaging data such as CT, MRI, PET and ultrasound, extracts feature vectors of each modality and generates a cross-modal feature vocabulary, calculates the feature correlation matrix between modalities, and realizes spatial registration and unified dimensional mapping of multimodal images. Then, core features such as bones, soft tissues and high metabolic areas are extracted from the registered multimodal images, and a deep learning algorithm is used to train and generate a set of segmented images. The fusion weights are dynamically adjusted according to the regional features to finally generate a fusion-enhanced image. The present invention realizes the effective integration of medical imaging information of different modalities through feature extraction, spatial registration and deep learning fusion of multimodal medical images, improves the reference value of medical images, and can provide a more comprehensive and accurate imaging basis for subsequent clinical diagnosis of doctors. BRIEF DESCRIPTION OF THE DRAWINGS

[0095] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of this application. The exemplary embodiments and descriptions of this application are intended to explain this application and do not constitute an improper limitation on this application. In the accompanying drawings:

[0096] Figure 1 Schematic diagram of a method flow in an embodiment of the present invention;

[0097] Figure 2 Schematic diagram of the system structure of an embodiment of the present invention. DETAILED DESCRIPTION

[0098] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0099] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0100] Example 1

[0101] like Figure 1 As shown, this embodiment provides a medical image enhancement method based on multimodal fusion, including:

[0102] Acquire medical imaging data, including CT images, MRI images, PET images, and ultrasound images;

[0103] Based on CT images, MRI images, and PET images, density feature vectors, relaxation time feature vectors, and metabolic feature vectors are extracted in sequence, and a dimensionality reduction algorithm is used to generate a modal feature vector set.

[0104] Based on the modal feature vector set, a clustering algorithm is used to generate a cross-modal feature vocabulary containing semantic labels; the semantic labels include high-density clusters, soft tissue clusters, and high-metabolism clusters;

[0105] Calculate the mutual information value of each semantic label in the cross-modal feature vocabulary to generate the inter-modal feature correlation matrix;

[0106] According to the inter-modality feature correlation matrix, a spatial transformation algorithm is used to map ultrasound images, CT images, and MRI images into a unified three-dimensional space to generate a unified dimensional image set.

[0107] Perform non-rigid spatial registration on a uniform-dimensional image set to generate a geometrically consistent multimodal image set;

[0108] Extract bone region features, soft tissue region features, and high metabolic region features from the multimodal image set to generate a core feature set;

[0109] A deep learning algorithm is used to train the core feature set to generate a segmentation image set including segmentation masks;

[0110] The fusion weights are dynamically adjusted according to the regional features of the segmented image set to generate a fused enhanced image.

[0111] Furthermore, the process of acquiring medical imaging data includes:

[0112] Extract raw data sets from CT images, MRI images, PET images, and ultrasound images, parse them in DICOM format, and obtain standardized image data;

[0113] For the standardized image data, image preprocessing technology is used. If the image noise is higher than the preset threshold, the Gaussian filtering algorithm is applied to obtain the denoised image data;

[0114] Extract features from denoised image data and use a convolutional neural network algorithm to extract features from the pixel distribution of CT images, MRI images, PET images, and ultrasound images to obtain an image feature set;

[0115] According to the image feature set, the principal component analysis algorithm is used to reduce the feature dimension. If the feature redundancy exceeds the preset threshold, the low variance features are eliminated to obtain the optimized feature set.

[0116] Based on the optimized feature set, a classification model is constructed. If the model input is a CT image or MRI image, it is classified according to the preset anatomical structure label to obtain the image classification result;

[0117] Extract abnormal areas from the image classification results. If the pixel intensity of the abnormal area exceeds the preset range, it is marked as a potential lesion to obtain lesion marking data;

[0118] According to the lesion marking data, the image information of CT images, MRI images, PET images and ultrasound images is integrated, and data fusion technology is used to obtain a comprehensive information data set.

[0119] Specifically, during the acquisition of medical imaging data, CT images were first downloaded in batches from the hospital's PACS system via the DICOM protocol. Each image had a resolution of 512×512 pixels and was compressed using the JPEG2000 algorithm to reduce storage space. Next, MRI images were preprocessed using a deep learning model, such as U-Net, to remove noise and enhance contrast. The model was trained using the Adam optimizer with a learning rate of 0.001 and 1000 iterations. For PET images, a threshold-based segmentation algorithm was used to mark regions with an SUV value greater than 2.5, and 3D reconstruction techniques were used to generate stereo images. For ultrasound images, speckle noise was removed using an adaptive filtering algorithm with a 5×5 filter window size, and the filter parameters were dynamically adjusted based on image characteristics. Finally, all processed image data were matched for feature points using feature extraction algorithms, such as SIFT or SURF, to ensure spatial alignment between images of different modalities. The feature point matching threshold was set to 0.8 to ensure matching accuracy. The entire data processing process was automated using scripts to ensure efficiency and consistency.

[0120] Furthermore, the process of extracting eigenvectors and generating a modal eigenvector set using a dimensionality reduction algorithm includes:

[0121] The CT images, MRI images, and PET images are subjected to secondary denoising and normalization processing using a preprocessing algorithm to obtain the first density image, the first relaxation image, and the first metabolic image respectively;

[0122] For the first density image, a feature extraction algorithm is used to calculate the pixel gray value distribution and texture features to generate a density feature vector;

[0123] For the first relaxation image, a feature extraction algorithm is used to calculate the T1 and T2 relaxation time distributions and generate a relaxation time feature vector;

[0124] For the first metabolic image, a feature extraction algorithm is used to calculate the standardized uptake value distribution to generate a metabolic feature vector;

[0125] If the dimension of the density feature vector is higher than the preset threshold, the principal component analysis algorithm is used to reduce the dimension of the density feature vector to obtain a second density feature vector;

[0126] If the dimension of the relaxation time feature vector is higher than a preset threshold, a principal component analysis algorithm is used to reduce the dimension of the relaxation time feature vector to obtain a second relaxation time feature vector;

[0127] If the dimension of the metabolic feature vector is higher than a preset threshold, the principal component analysis algorithm is used to reduce the dimension of the metabolic feature vector to obtain a second metabolic feature vector;

[0128] According to the second density eigenvector, the second relaxation time eigenvector and the second metabolic eigenvector, the eigenvectors of each modality are integrated by using the feature splicing method to generate a multimodal eigenvector set;

[0129] Extracting the weight distribution of each modal feature vector from the multimodal feature vector set. If the variance of the weight distribution is higher than a preset threshold, a normalization algorithm is used to normalize the multimodal feature vector set to obtain a normalized feature vector set.

[0130] For the standardized feature vector set, a clustering algorithm is used to group the feature vectors to generate multimodal feature vector grouping results;

[0131] According to the grouping results of multimodal feature vectors, the feature mapping method is used to project the grouping results into a low-dimensional space to obtain the final set of modal feature vectors.

[0132] Specifically, in the field of medical image data processing, feature extraction and integration of CT, MRI and PET images are of great significance.

[0133] For example, CT images are stored at a resolution of 512×512, reflecting tissue density; MRI images contain T1 and T2 weighted sequences, reflecting relaxation characteristics; and PET images record the distribution of radioactive tracers, reflecting metabolic activity.

[0134] For example, a hospital generates 1,000 CT images, 500 MRI images, and 200 PET images every day, and needs to ensure data integrity and format consistency.

[0135] In one possible implementation, preprocessing involves denoising and normalizing images from each modality. CT images can be filtered using non-local means to remove Gaussian noise while preserving edge details. MRI images can be transformed using wavelet transforms to remove artifacts and ensure clear T1 and T2 signals. PET images can be filtered using Gaussian filtering to smooth noise and maintain accurate SUV values. Normalization maps grayscale values ​​to a range of 0–255 for ease of subsequent analysis.

[0136] For example, after the grayscale value of a CT image is normalized, the value of the bone region is close to 200, and the value of the soft tissue region is close to 50.

[0137] Specifically, feature extraction generates a feature vector for each modality image. For CT images, grayscale histogram and GLCM texture features are calculated to generate a density feature vector containing information such as mean, variance, and contrast.

[0138] For example, the grayscale mean of a tumor region is 150, and the variance is 20. For MRI images, the T1 and T2 relaxation time distributions are extracted to form a relaxation time feature vector, such as a T1 mean of 800ms and a T2 mean of 100ms. For PET images, the mean and peak SUV values ​​are calculated to form a metabolic feature vector, such as a mean SUV of 3.0 and a peak SUV of 5.0.

[0139] Preferably, if the eigenvector dimension is too high, such as a density eigenvector with 100 dimensions, principal component analysis can be used to reduce the dimensionality, retaining 95% of the variance and generating a second density eigenvector of approximately 20 dimensions. Similarly, the relaxation time eigenvector can be reduced from 80 to 15 dimensions, and the metabolic eigenvector from 50 to 10 dimensions. This dimensionality reduction reduces computational complexity while preserving essential information.

[0140] For example, the density feature vector after dimensionality reduction can still reflect the high grayscale characteristics of the tumor area.

[0141] In one embodiment, feature stitching integrates the second density feature vector, the second relaxation time feature vector, and the second metabolic feature vector into a multimodal feature vector set.

[0142] For example, concatenation generates a 45-dimensional vector containing density, relaxation, and metabolic information. If the weight distribution has a high variance, such as a 60% weight for density features, a normalization algorithm can be used to equalize the weights of each modality to a mean of 1, generating a standardized set of feature vectors. This process ensures a balanced contribution from each modality.

[0143] It can be understood that clustering algorithms such as K-means group the standardized feature vector set to generate multimodal feature vector grouping results.

[0144] For example, the feature vectors are divided into 3 groups, and the grouping results are mapped to a 2D space through t-SNE to form the final feature vector set, which is convenient for visual analysis.

[0145] For example, tumor tissues are clustered in low-dimensional space, which is conducive to subsequent analysis.

[0146] For example, after feature mapping, the final feature vector set can be used for lesion classification, with the mapping results showing a concentrated distribution of density and metabolic features in the tumor region. This multimodal feature integration and grouping approach significantly improves the comprehensiveness and accuracy of feature representation.

[0147] Furthermore, the process of generating a cross-modal feature vocabulary containing semantic labels using a clustering algorithm based on the modal feature vector set includes:

[0148] The K-means clustering algorithm is used to cluster the modal feature vector set to determine the initial division of high-density clusters, soft tissue clusters, and high-metabolism clusters;

[0149] Based on the clustering results, a cross-modal feature vocabulary is generated, and semantic labels of high-density clusters, soft tissue clusters, and high-metabolism clusters are assigned to obtain a preliminary vocabulary.

[0150] If the intra-cluster variance in the preliminary vocabulary is greater than the preset threshold, the DBSCAN algorithm is used to perform secondary clustering on the outliers, update the cluster division, and obtain the optimized vocabulary;

[0151] Based on the optimized vocabulary, a semantic label set of cross-modal features is extracted to determine the accuracy of label assignment.

[0152] According to the label assignment results, the boundaries of each cluster in the feature vocabulary are adjusted to obtain the final cross-modal feature vocabulary;

[0153] According to the final feature vocabulary, the consistency between the feature vectors of each modality and the semantic labels is verified, and the cross-modal semantic mapping relationship is determined.

[0154] Specifically, feature vectors were first extracted from the multimodal data. For example, voxel features with a HU value range of -1000 to 3000 were extracted from CT images, metabolic features with a standardized uptake value (SUVmax) between 2.5 and 15 were extracted from PET images, and texture features were extracted from T1- and T2-weighted MRI images. After z-score normalization, these features were reduced to 50 dimensions using principal component analysis, retaining 95% of the variance. Clustering was then performed using a modified DBSCAN algorithm with a neighborhood radius (eps) of 0.5 and a minimum number of samples (min_samples) of 10. Clusters were formed using density reachability analysis. For each cluster, a silhouette coefficient greater than 0.6 was calculated as a valid cluster. Semantic annotation was then performed based on the statistical distribution of the features: clusters with an average HU value greater than 200 and an SUVmax greater than 5 were labeled as high-density clusters, those with HU values ​​between -50 and 100 and a texture entropy greater than 3.5 were labeled as soft tissue clusters, and those with an SUVmax greater than 8 and a metabolic volume greater than 5 cm³ were labeled as high-metabolism clusters. Finally, when constructing the cross-modal feature vocabulary, the term frequency-inverse document frequency method was used to calculate the weight of each semantic tag. For example, the TF-IDF value for high-density clusters was 1.2, and for soft tissue clusters it was 0.8. Inter-modal feature associations were established using a cosine similarity threshold greater than 0.85. To optimize the feature vocabulary, the expectation-maximization algorithm was used, iterating 20 times until the log-likelihood function changed by less than 0.01, ensuring that the alignment accuracy of each modality's features in the latent space reached above 90%.

[0155] Furthermore, the process of calculating the mutual information value of each semantic label in the cross-modal feature vocabulary and generating the inter-modal feature correlation matrix includes:

[0156] Obtain a set of feature vectors corresponding to each semantic tag from a cross-modal feature vocabulary, and determine the modal distribution of the feature vectors;

[0157] If the intra-cluster variance of the modal distribution is greater than the preset threshold, the principal component analysis algorithm is used to reduce the dimension of the feature vector to obtain a reduced-dimensional feature set;

[0158] Calculate the mutual information value between each semantic label based on the dimensionality reduction feature set;

[0159] If the mutual information value is lower than the preset threshold, the corresponding label pair is eliminated to obtain a set of highly correlated labels;

[0160] Based on the highly correlated tag set, the tag distribution characteristics are extracted to generate a preliminary correlation matrix;

[0161] If the symmetry deviation of the preliminary correlation matrix is ​​greater than a preset threshold, the matrix elements are adjusted using symmetry processing to obtain an optimized correlation matrix;

[0162] Obtain inter-modal feature association weights from the optimized correlation matrix and determine the modal association structure;

[0163] If the connectivity of the modal association structure is lower than a preset threshold, virtual association edges are added to optimize the structure to obtain an enhanced association structure;

[0164] For the enhanced correlation structure, the feature extraction contribution of each mode is calculated, the information interaction strength between the modes is determined, and the interaction strength distribution is obtained;

[0165] Through the interaction intensity distribution, the weight distribution in the correlation matrix is ​​adjusted and optimized to generate the final inter-modal feature correlation matrix.

[0166] Specifically, in the cross-modal feature vocabulary, the mutual information value of each semantic label is calculated to quantify the correlation of features between modalities. First, based on the annotated semantic labels such as high-density clusters, soft tissue clusters, and high-metabolism clusters, their occurrence frequencies in CT, PET, and MRI modalities are counted.

[0167] For example, the frequency of high-density clusters in the CT modality is 0.35, the frequency in the PET modality is 0.28, and the frequency of soft tissue clusters in the MRI modality is 0.42. The mutual information value of each semantic label is calculated through the joint probability distribution, using the formula MI(X,Y)=∑P(x,y)log(P(x,y) / (P(x)P(y))), where P(x,y) is the joint probability of the semantic label in the two modalities, and P(x) and P(y) are its marginal probabilities in a single modality.

[0168] For example, the mutual information value of high-density clusters in CT and PET modalities is 0.12, and the mutual information value of soft tissue clusters in CT and MRI modalities is 0.09. Next, an inter-modality feature correlation matrix is ​​constructed with a dimension of 3×3, corresponding to the CT, PET, and MRI modalities. Each element in the matrix represents the mutual information value of the semantic label between the modalities. For example, the correlation between CT and PET modalities is 0.12, the correlation between CT and MRI modalities is 0.09, and the correlation between PET and MRI modalities is 0.15. To further optimize the correlation matrix, a spectral clustering algorithm is used to decompose the matrix, setting the number of clusters to 2. The eigenvectors of the Laplacian matrix are used to analyze potential inter-modality associations. Ultimately, the generated feature correlation matrix can be used to guide the fusion and analysis of multimodal data, providing a basis for subsequent cross-modal feature matching.

[0169] Furthermore, the process of generating a uniform dimensional image set includes:

[0170] Based on the inter-modality feature correlation matrix, the shared feature vectors of ultrasound images, CT images, and MRI images are extracted by matrix decomposition method to obtain the feature extraction results.

[0171] Based on the feature extraction results, a preset spatial transformation algorithm is used to calibrate the coordinate systems of ultrasound images, CT images, and MRI images to generate a preliminary aligned three-dimensional image set.

[0172] Determine the preliminarily aligned 3D image set. If there are local mismatch areas, adjust the spatial deviation between images using an iterative closest point algorithm to obtain a spatially aligned image set.

[0173] Extract the boundary features of each modality image from the spatially aligned image set, generate a boundary description vector of unified dimension through the feature fusion algorithm, and determine the fusion feature set;

[0174] Based on the fusion feature set, the stereoscopic microscopy algorithm is used to perform dimension normalization on the 3D image set to generate a uniform dimension image set.

[0175] For a unified dimensional image set, the consistency between modalities within the image set is detected using a stereo depth analysis algorithm to determine the consistency verification results;

[0176] If the consistency verification result is lower than the preset threshold, the fusion feature set is optimized through the feature weighting adjustment algorithm to regenerate a unified dimensional image set.

[0177] Specifically, first, the inter-modality feature correlation matrix is ​​constructed by calculating the correlation coefficient between the eigenvectors of ultrasound images, CT images, and MRI images. For example, the Pearson correlation coefficient is used to calculate the value of each element in the matrix. Assume that the correlation coefficient between ultrasound images and CT images is 0.85, the correlation coefficient between ultrasound images and MRI images is 0.78, and the correlation coefficient between CT images and MRI images is 0.92. Next, a spatial transformation algorithm is used to map the images of different modalities to a unified three-dimensional space. Specifically, an affine transformation matrix is ​​used to transform the image coordinates. For example, the pixel coordinates (x, y, z) of the ultrasound image are mapped to the coordinates (x', y', z') in the unified space using the affine transformation matrix T1, where T1 is a 3×4 matrix whose element values ​​are determined according to the resolution and spatial position of the image, such as:

[0178] T1=[1.2,0,0,10;0,1.2,0,15;0,0,1.2,20]

[0179] Similarly, CT and MRI images are mapped using affine transformation matrices T2 and T3, respectively. The values ​​of T2 and T3 are adjusted based on the spatial characteristics of their respective modalities. During the mapping process, the images are resampled using a bilinear interpolation algorithm to ensure consistent resolution within the unified space. For example, the resolution of an ultrasound image is adjusted from 0.5 mm × 0.5 mm × 1.0 mm to 0.3 mm × 0.3 mm × 0.3 mm. Finally, the mapped images are fused to generate a unified dimensional image set. For example, pixel values ​​from ultrasound, CT, and MRI images are fused using a weighted average method with weights of 0.4, 0.3, and 0.3, respectively, to ensure that the fused image set retains the characteristic information of each modality.

[0180] Furthermore, the process of performing non-rigid spatial registration on the uniform-dimensional image set to generate a geometrically consistent multimodal image set includes:

[0181] Based on the unified dimension image set, a preprocessing method is used to unify the image dimensions to obtain a first image set;

[0182] If the resolution or size of the first image set is inconsistent, an interpolation algorithm is used to adjust the first image set to determine the second image set;

[0183] For the second image set, non-rigid spatial registration is performed using the B-spline registration algorithm to obtain registration parameters;

[0184] Applying a geometric transformation to the second image set using the registration parameters to generate a third image set;

[0185] If the geometric consistency of the third image set does not reach a preset threshold, the registration parameters are iteratively optimized to obtain a fourth image set;

[0186] Based on the fourth image set, the multimodal data is integrated using image fusion technology to generate a fifth image set;

[0187] The spatial consistency of the fifth image set is evaluated, and finally a geometrically consistent multimodal image set is obtained.

[0188] Specifically, during the non-rigid spatial registration process, a B-spline-based deformable model is first used to initially register the uniform-dimensional image set. The node spacing of the deformable mesh is set to 8 pixels to ensure flexibility in local deformation. The spatial transformation parameters between the images are calculated by optimizing the objective function and incorporating mutual information as a similarity metric. The initial number of iterations is 50 with a step size of 0.1. Subsequently, a multi-resolution strategy is used to gradually optimize the registration results from low to high resolution. The number of low-resolution layers is 3 and the number of high-resolution layers is 1, with 30 and 20 iterations per layer, respectively. A regularization term is introduced during the registration process to control the smoothness of the deformation. The regularization coefficient is set to 0.01 to avoid distortion caused by excessive deformation. After registration, the registration performance is evaluated by calculating the registration error and similarity metrics (such as the Dice coefficient) of the overlapping regions. The Dice coefficient must be above 0.85 to ensure geometric consistency. Finally, the registered multimodal image sets are fused using a weighted averaging method. The weights are dynamically adjusted according to the signal-to-noise ratio of each modality image. The weight of the modality with a high signal-to-noise ratio is set to 0.6, and the weight of the modality with a low signal-to-noise ratio is set to 0.4 to generate a high-quality geometrically consistent multimodal image set.

[0189] Furthermore, the process of generating the core feature set includes:

[0190] Using a multimodal image segmentation algorithm, the boundaries of the bone region, soft tissue region, and high metabolic region are obtained from the multimodal image set to obtain the regional segmentation results;

[0191] A convolutional neural network is used to extract features from the segmented bone area, soft tissue area, and high metabolic area to obtain the initial feature set of each area.

[0192] If the dimension of the initial feature set is higher than the preset threshold, the dimension is reduced by principal component analysis to obtain a compressed feature set;

[0193] Based on the compressed feature set, the features of the bone area, soft tissue area and high metabolic area are fused to generate a preliminary feature set;

[0194] If the feature correlation in the preliminary feature set is lower than the preset threshold, redundant features are eliminated through the feature selection algorithm to obtain the optimized feature set;

[0195] Through data integration method, the optimized feature set is standardized to obtain the core feature set;

[0196] For the core feature set, a clustering algorithm is used to group the features to obtain the classified feature set.

[0197] Specifically, when extracting bone region features from a multimodal image set, this embodiment uses a deep learning-based U-Net network for bone segmentation. By inputting a CT image, the pre-trained U-Net model is used to accurately segment the bone region, with a segmentation accuracy of over 95%. After segmentation, morphological features of the bone region, such as bone density and bone volume fraction, are extracted. Bone density can be calculated using Hounsfield units (HU). The HU value of normal bones ranges from 200 to 1000. The bone density feature is obtained by calculating the average HU value of pixels in the region.

[0198] To extract soft tissue region features, this embodiment uses the ResNet50 model based on a convolutional neural network (CNN) to perform soft tissue segmentation on MRI images. After segmentation, the texture features of the soft tissue are extracted, such as contrast, energy, and entropy in the gray-level co-occurrence matrix (GLCM). The contrast value typically ranges from 0 to 100, the energy value ranges from 0 to 1, and the entropy value ranges from 0 to 10. By calculating these feature values, the texture characteristics of the soft tissue can be described.

[0199] High metabolic region features can be extracted from PET images. Using a threshold-based segmentation method, regions with SUV values ​​greater than 2.5 are defined as high metabolic regions. Features such as the maximum SUV, mean SUV, and metabolic volume (MTV) of these regions are then extracted. The maximum SUV typically ranges from 5 to 20, the mean SUV ranges from 2 to 10, and the MTV ranges from 1 to 100 cubic centimeters. These features can reflect the metabolic activity of the tumor. Finally, the extracted bone region features, soft tissue region features, and high metabolic region features are fused to generate a core feature set. Feature fusion can use principal component analysis (PCA) to reduce the high-dimensional features to 10 to 20 principal components, retaining more than 90% of the information to form the final core feature set.

[0200] Furthermore, the process of generating the fused enhanced image includes:

[0201] Obtain segmented regions according to the segmented image set, determine region boundaries using a preset segmentation algorithm, and obtain a segmented region set;

[0202] For the segmented region set, extract the feature vector of each region and determine the regional features through feature extraction algorithm;

[0203] According to regional characteristics, a dynamic weight allocation method is used to determine the fusion weight of each region and obtain a weight set;

[0204] If the weight set meets the preset threshold, the segmented region set is fused and the initial fused image is generated through the image fusion algorithm;

[0205] Obtain pixel distribution characteristics from the initial fused image, use pixel analysis method to determine enhancement requirements, and obtain enhancement parameters;

[0206] According to the enhancement parameters, the initial fused image is enhanced and an enhanced fused image is generated by an image enhancement algorithm;

[0207] The final feature vector is extracted from the enhanced fusion image, and it is judged whether it meets the preset fusion quality standard to obtain the final enhanced fusion image.

[0208] Exemplarily, when acquiring segmented regions from an image set, a preset segmentation algorithm such as a threshold-based segmentation method is used.

[0209] Specifically, by analyzing the grayscale histogram of the image, a dynamic threshold is set to distinguish the foreground and background areas.

[0210] For example, in medical imaging scenarios, for CT images, a threshold range of 150 to 255 can be set based on the grayscale difference between bone and soft tissue to generate an initial set of segmented regions. This method is simple and efficient, quickly determining region boundaries and providing a foundation for subsequent feature extraction.

[0211] In a possible implementation, feature vectors are extracted for the segmented region set, and a texture-based feature extraction algorithm, such as a local binary pattern, may be used.

[0212] Preferably, for each segmented area, the grayscale comparison results of its 8 neighboring pixels are calculated to generate a 256-dimensional feature vector.

[0213] For example, in the same medical image scene, the texture features of the bone region may show high contrast, while the soft tissue region is relatively smooth. This feature vector can effectively represent the regional characteristics and provide a basis for subsequent weight allocation.

[0214] Specifically, the dynamic weight assignment method adopts a weighting strategy based on feature similarity.

[0215] In one embodiment, the Euclidean distance between each region feature vector and a preset template is compared to calculate a similarity score, and a weight is assigned accordingly.

[0216] For example, a bone region may be assigned a weight of 0.7 due to its significant features, while a soft tissue region may be assigned a weight of 0.3. This set of weights reflects the importance of the region and ensures that key regions are more prominent during fusion.

[0217] It should be noted that if the weight set meets the preset threshold, such as the sum is greater than 0.8, the fusion process is performed. The weighted average fusion algorithm is used to superimpose the pixel values ​​of each region according to the weight to generate the initial fused image.

[0218] For example, high grayscale values ​​in the bone region are more prominent in the fused image due to their higher weight. This method can preserve key information and improve the overall consistency of the image.

[0219] For example, pixel distribution characteristics are obtained through histogram analysis.

[0220] In one embodiment, the grayscale distribution of the initial fused image is statistically analyzed to determine whether there are too dark or too bright areas. If the grayscale is detected to be concentrated in the range of 0 to 50, it indicates that contrast enhancement is required, and enhancement parameters such as a stretch factor of 1.5 are generated.

[0221] In one possible implementation, the image enhancement algorithm may use adaptive histogram equalization to perform local contrast adjustment on the initial fused image according to the enhancement parameters.

[0222] For example, for areas with uneven grayscale distribution, the grayscale range is expanded to 0 to 200 after enhancement. This method can improve the visibility of image details and facilitate subsequent feature extraction.

[0223] It can be understood that the extraction of the final feature vector combines global and local features.

[0224] For example, a comprehensive feature vector is generated by calculating the color histogram and edge density of the enhanced fused image.

[0225] In one embodiment, if the edge density is greater than 0.6 and the color distribution is uniform, the fusion quality standard is determined to be met. This method ensures that the final image meets the requirements in terms of details and overall effect.

[0226] Preferably, the final enhanced fused image can be used for subsequent medical diagnosis support.

[0227] For example, enhanced bone regions have clearer boundaries, making it easier for doctors to identify potential lesions. This process, from segmentation to enhancement, forms a complete chain, with each step supporting each other to ensure high-quality and practical output images.

[0228] Example 2

[0229] like Figure 2 As shown, based on the same inventive concept, this embodiment also provides a medical image enhancement system based on multimodal fusion, including:

[0230] A medical imaging data acquisition module is used to acquire medical imaging data, including CT images, MRI images, PET images, and ultrasound images;

[0231] The feature vector extraction module is used to extract density feature vectors, relaxation time feature vectors, and metabolic feature vectors based on CT images, MRI images, and PET images, and generate a modal feature vector set using a dimensionality reduction algorithm;

[0232] A cross-modal feature vocabulary generation module is used to generate a cross-modal feature vocabulary containing semantic labels based on the modal feature vector set using a clustering algorithm; wherein the semantic labels include high-density clusters, soft tissue clusters, and high-metabolism clusters;

[0233] The inter-modal feature correlation matrix generation module is used to calculate the mutual information value of each semantic label in the cross-modal feature vocabulary and generate the inter-modal feature correlation matrix;

[0234] A unified dimensional image set generation module is used to map ultrasound images, CT images, and MRI images into a unified three-dimensional space using a spatial transformation algorithm based on the inter-modality feature correlation matrix to generate a unified dimensional image set;

[0235] A multimodal image set generation module is used to perform non-rigid spatial registration on a uniform-dimensional image set to generate a geometrically consistent multimodal image set;

[0236] A core feature set extraction module is used to extract bone region features, soft tissue region features, and high metabolic region features from a multimodal image set to generate a core feature set;

[0237] A segmentation image set generation module is used to train the core feature set using a deep learning algorithm to generate a segmentation image set containing a segmentation mask;

[0238] The fusion enhanced image generation module is used to dynamically adjust the fusion weight according to the regional features of the segmented image set to generate a fusion enhanced image.

[0239] The medical image enhancement system based on multimodal fusion provided in this embodiment has all the advantages of the medical image enhancement method based on multimodal fusion provided in the first embodiment.

[0240] Example 3

[0241] This embodiment further discloses a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described in the first embodiment.

[0242] Example 4

[0243] This embodiment further discloses a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in the first embodiment are implemented.

[0244] Example 5

[0245] This embodiment further discloses a computer program product, including a computer program, which implements the steps of the method described in the first embodiment when executed by a processor.

[0246] The above are merely preferred embodiments of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A medical image enhancement method based on multimodal fusion, characterized in that: include: Acquiring medical imaging data, wherein the medical imaging data includes CT images, MRI images, and PET images; Based on the CT image, MRI image, and PET image, density feature vectors, relaxation time feature vectors, and metabolic feature vectors are extracted in sequence, and a dimensionality reduction algorithm is used to generate a modal feature vector set; Based on the modal feature vector set, a clustering algorithm is used to generate a cross-modal feature vocabulary containing semantic labels; wherein the semantic labels include high-density clusters, soft tissue clusters, and high-metabolism clusters; Calculating the mutual information value of each semantic tag in the cross-modal feature vocabulary to generate an inter-modal feature correlation matrix; According to the inter-modality feature correlation matrix, a spatial transformation algorithm is used to map the CT image, MRI image, and PET image to a unified three-dimensional space to generate a unified dimensional image set; performing non-rigid spatial registration on the uniform-dimensional image set to generate a geometrically consistent multimodal image set; Extracting bone region features, soft tissue region features, and high metabolic region features from the multimodal image set to generate a core feature set; Using a deep learning algorithm to train the core feature set to generate a segmentation image set including a segmentation mask; The fusion weights are dynamically adjusted according to the regional features of the segmented image set to generate a fused enhanced image.

2. The method according to claim 1, characterized in that The process of acquiring medical imaging data includes: Extract original data sets from CT images, MRI images, and PET images, parse them in DICOM format, and obtain standardized image data; For the standardized image data, image preprocessing technology is used. If the image noise is higher than the preset threshold, the Gaussian filtering algorithm is applied to obtain the denoised image data; Extracting features from the denoised image data, using a convolutional neural network algorithm to extract features based on pixel distribution of CT images, MRI images, and PET images to obtain an image feature set; According to the image feature set, a principal component analysis algorithm is used to reduce the feature dimension. If the feature redundancy exceeds a preset threshold, low-variance features are eliminated to obtain an optimized feature set. A classification model is constructed for the optimized feature set. If the model input is a CT image or an MRI image, the image is classified according to the preset anatomical structure label to obtain an image classification result. Extracting an abnormal area from the image classification result, and marking the abnormal area as a potential lesion if the pixel intensity exceeds a preset range, thereby obtaining lesion marking data; According to the lesion marking data, the image information of the CT image, the MRI image, and the PET image is integrated, and a data fusion technology is used to obtain a comprehensive information data set.

3. The method according to claim 1, characterized in that The process of extracting eigenvectors and generating a set of modal eigenvectors using a dimensionality reduction algorithm includes: The CT images, MRI images, and PET images are subjected to secondary denoising and normalization processing using a preprocessing algorithm to obtain the first density image, the first relaxation image, and the first metabolic image respectively; For the first density image, using a feature extraction algorithm to calculate pixel grayscale value distribution and texture features to generate a density feature vector; For the first relaxation image, using a feature extraction algorithm to calculate T1 and T2 relaxation time distributions to generate a relaxation time feature vector; For the first metabolic image, a feature extraction algorithm is used to calculate a standardized uptake value distribution to generate a metabolic feature vector; If the dimension of the density feature vector is higher than a preset threshold, a principal component analysis algorithm is used to reduce the dimension of the density feature vector to obtain a second density feature vector; If the dimension of the relaxation time feature vector is higher than a preset threshold, a principal component analysis algorithm is used to reduce the dimension of the relaxation time feature vector to obtain a second relaxation time feature vector; If the dimension of the metabolic feature vector is higher than a preset threshold, a principal component analysis algorithm is used to reduce the dimension of the metabolic feature vector to obtain a second metabolic feature vector; According to the second density feature vector, the second relaxation time feature vector, and the second metabolic feature vector, the feature vectors of each modality are integrated using a feature splicing method to generate a multimodal feature vector set; Extracting a weight distribution of each modal feature vector from the multimodal feature vector set, and if the variance of the weight distribution is higher than a preset threshold, normalizing the multimodal feature vector set using a normalization algorithm to obtain a normalized feature vector set; For the standardized feature vector set, a clustering algorithm is used to group the feature vectors to generate multimodal feature vector grouping results; According to the multimodal feature vector grouping results, a feature mapping method is used to project the grouping results into a low-dimensional space to obtain a final set of modal feature vectors.

4. The method according to claim 1, wherein The process of generating a cross-modal feature vocabulary containing semantic labels using a clustering algorithm based on the modal feature vector set includes: The K-means clustering algorithm was used to cluster the modal feature vector set to determine the initial division of high-density clusters, soft tissue clusters, and high-metabolism clusters. Based on the clustering results, a cross-modal feature vocabulary is generated, and semantic labels of high-density clusters, soft tissue clusters, and high-metabolism clusters are assigned to obtain a preliminary vocabulary. If the intra-cluster variance in the preliminary vocabulary is greater than a preset threshold, the DBSCAN algorithm is used to perform secondary clustering on the outliers, update the cluster division, and obtain an optimized vocabulary; Extracting a set of semantic labels for cross-modal features based on the optimized vocabulary and determining the accuracy of label assignment; According to the label assignment results, the boundaries of each cluster in the feature vocabulary are adjusted to obtain the final cross-modal feature vocabulary; According to the final cross-modal feature vocabulary, the consistency between each modality feature vector and the semantic label is verified to determine the cross-modal semantic mapping relationship.

5. The method according to claim 1, characterized in that The process of calculating the mutual information value of each semantic tag in the cross-modal feature vocabulary to generate an inter-modal feature correlation matrix includes: Obtain a set of feature vectors corresponding to each semantic tag from a cross-modal feature vocabulary, and determine the modal distribution of the feature vectors; If the intra-cluster variance of the modal distribution is greater than the preset threshold, the principal component analysis algorithm is used to reduce the dimension of the feature vector to obtain a reduced-dimensional feature set; Calculating mutual information values ​​between semantic labels based on the dimension reduction feature set; If the mutual information value is lower than a preset threshold, the corresponding label pair is eliminated to obtain a highly correlated label set; Extracting label distribution features based on the highly correlated label set and generating a preliminary correlation matrix; If the symmetry deviation of the preliminary correlation matrix is ​​greater than a preset threshold, the matrix elements are adjusted by symmetry processing to obtain an optimized correlation matrix; Obtain inter-modal feature association weights from the optimized correlation matrix and determine the modal association structure; If the connectivity of the modal association structure is lower than a preset threshold, virtual association edges are added to optimize the structure to obtain an enhanced association structure; For the enhanced correlation structure, the feature extraction contribution of each modality is calculated, the information interaction strength between the modalities is determined, and the interaction strength distribution is obtained; The interaction intensity distribution is used to adjust and optimize the weight distribution in the correlation matrix to generate a final inter-modality feature correlation matrix.

6. The method according to claim 1, characterized in that The process of generating a uniform dimensional image set includes: Based on the inter-modality feature correlation matrix, shared feature vectors of the CT image, the MRI image, and the PET image are extracted by a matrix decomposition method to obtain a feature extraction result; Based on the feature extraction results, a preset spatial transformation algorithm is used to calibrate the coordinate systems of the CT images, MRI images, and PET images to generate a preliminarily aligned three-dimensional image set; Determining the preliminarily aligned three-dimensional image set, if there is a local mismatch area, adjusting the spatial deviation between the images by an iterative closest point algorithm to obtain a spatially aligned image set; Extract the boundary features of each modality image from the spatially aligned image set, generate a boundary description vector of unified dimension through the feature fusion algorithm, and determine the fusion feature set; According to the fusion feature set, a stereoscopic microscopy algorithm is used to perform dimensional normalization processing on the three-dimensional image set to generate a uniform dimensional image set; For a unified dimensional image set, the consistency between modalities within the image set is detected using a stereo depth analysis algorithm to determine the consistency verification results; If the consistency verification result is lower than the preset threshold, the fusion feature set is optimized through the feature weighting adjustment algorithm to regenerate a unified dimensional image set.

7. The method according to claim 1, characterized in that The process of performing non-rigid spatial registration on the uniform dimensional image set to generate a geometrically consistent multimodal image set includes: Based on the unified dimensional image set, a preprocessing method is used to unify the image dimensions to obtain a first image set; If the resolution or size of the first image set is inconsistent, adjusting it through an interpolation algorithm to determine a second image set; For the second image set, performing non-rigid spatial registration using a B-spline registration algorithm to obtain registration parameters; Applying a geometric transformation to the second image set using the registration parameters to generate a third image set; If the geometric consistency of the third image set does not reach a preset threshold, iteratively optimizing the registration parameters to obtain a fourth image set; integrating multimodal data using image fusion technology based on the fourth image set to generate a fifth image set; The spatial consistency of the fifth image set is evaluated to ultimately obtain a geometrically consistent multimodal image set.

8. The method according to claim 1, characterized in that The process of generating the core feature set includes: Using a multimodal image segmentation algorithm, the boundaries of the bone region, soft tissue region, and high metabolic region are obtained from the multimodal image set to obtain the regional segmentation results; A convolutional neural network is used to extract features from the segmented bone area, soft tissue area, and high metabolic area to obtain the initial feature set of each area. If the dimension of the initial feature set is higher than a preset threshold, the dimension is reduced by principal component analysis to obtain a compressed feature set; Based on the compressed feature set, the features of the bone area, soft tissue area and high metabolic area are fused to generate a preliminary feature set; If the feature correlation in the preliminary feature set is lower than a preset threshold, redundant features are eliminated through a feature selection algorithm to obtain an optimized feature set; By using a data integration method, the optimized feature set is standardized to obtain a core feature set; For the core feature set, a clustering algorithm is used to group the features to obtain the classified feature set.

9. The method according to claim 1, characterized in that The process of generating a fused enhanced image includes: Acquire segmented regions according to the segmented image set, determine region boundaries using a preset segmentation algorithm, and obtain a segmented region set; For the segmented region set, extracting a feature vector of each region and determining the region features using a feature extraction algorithm; According to the regional characteristics, a dynamic weight allocation method is adopted to determine the fusion weight of each region to obtain a weight set; If the weight set meets the preset threshold, the segmented region set is fused and an initial fused image is generated by an image fusion algorithm; Obtaining pixel distribution features from the initial fused image, determining enhancement requirements using a pixel analysis method, and obtaining enhancement parameters; performing enhancement processing on the initial fused image according to the enhancement parameters, and generating an enhanced fused image by an image enhancement algorithm; The final feature vector is extracted from the enhanced fusion image, and it is judged whether it meets the preset fusion quality standard to obtain the final enhanced fusion image.

10. A medical image enhancement system based on multimodal fusion, characterized in that: include: A medical image data acquisition module is used to acquire medical image data, wherein the medical image data includes CT images, MRI images, and PET images; A feature vector extraction module is used to extract density feature vectors, relaxation time feature vectors, and metabolic feature vectors based on the CT image, MRI image, and PET image, and generate a modal feature vector set using a dimensionality reduction algorithm; a cross-modal feature vocabulary generation module, configured to generate a cross-modal feature vocabulary containing semantic labels based on the modal feature vector set using a clustering algorithm; wherein the semantic labels include high-density clusters, soft tissue clusters, and high-metabolism clusters; an inter-modal feature correlation matrix generation module, configured to calculate the mutual information value of each semantic tag in the cross-modal feature vocabulary to generate an inter-modal feature correlation matrix; a unified dimensional image set generation module, configured to map the CT images, MRI images, and PET images to a unified three-dimensional space using a spatial transformation algorithm based on the inter-modality feature correlation matrix to generate a unified dimensional image set; a multimodal image set generation module, configured to perform non-rigid spatial registration on the uniform-dimensional image set to generate a geometrically consistent multimodal image set; A core feature set extraction module is used to extract bone region features, soft tissue region features and high metabolic region features from the multimodal image set to generate a core feature set; A segmentation image set generation module is used to train the core feature set using a deep learning algorithm to generate a segmentation image set including a segmentation mask; The fusion enhanced image generation module is used to dynamically adjust the fusion weight according to the regional features of the segmented image set to generate a fusion enhanced image.

Images