Multi-modal medical image auxiliary diagnosis method, diagnosis model and model training method

Abstract

The invention provides a multi-mode medical image auxiliary diagnosis method, a diagnosis model and a model training method. The method comprises the following steps: acquiring medical image data of at least two different imaging modalities; preprocessing the medical image data to generate standardized medical image data; performing multi-modal feature extraction and fusion on the standardized medical image data to generate fusion feature representation; wherein the multi-modal feature extraction and fusion comprises the steps of feature whitening and attention regularization; and based on the fused feature representation, generating an auxiliary diagnosis result through a classifier. According to the method, high-dimensional singular value interference can be effectively relieved or eliminated, multi-modal feature fusion robustness and diagnosis result reliability are ensured, and the urgent clinical demand for a high-robustness and high-credibility intelligent auxiliary diagnosis system is met.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, and in particular to a multi-modal medical image auxiliary diagnosis method, a diagnosis model and a model training method. BACKGROUND

[0002] Multi-modal medical image auxiliary diagnosis is one of the core development directions in the field of modern medicine, which integrates the information of multiple imaging modalities such as computed tomography (CT), magnetic resonance imaging (MRI) and positron emission tomography (PET), and provides a more comprehensive basis for clinical decision-making. Based on artificial intelligence, especially deep learning technology, has been widely used in this field. The existing technology usually uses convolutional neural network (CNN) to automatically extract features from each modality image, and uses attention mechanism and other strategies for feature fusion, in order to improve the automation level and accuracy of diagnosis.

[0003] However, the above-mentioned existing technology has potential technical defects in actual deployment and application: due to the distribution difference of multi-modal medical image features in high-dimensional space, the attention mechanism may be interfered by the "high-dimensional singular value" between the feature vectors when calculating the weight, resulting in unstable weight distribution of the fused features. Specifically, when the feature dimension is extremely high, the dot product operation between feature vectors will cause a few abnormal large or small singular values to be amplified too much, thereby destroying the stability of the weight distribution and making it abnormally sensitive to the small perturbations of the input features.

[0004] This technical defect will cause a series of adverse consequences in actual clinical deployment. First, it directly reduces the robustness and generalization ability of the diagnosis model, resulting in significant fluctuations in the diagnosis performance of the model when facing image data from different medical institutions and different scanning devices. Second, unstable weight distribution may cause the model to ignore the weak features of key lesions and instead pay excessive attention to certain non-significant image patterns, thereby increasing the risk of misdiagnosis and missed diagnosis. Finally, this problem will weaken the trust of clinicians in artificial intelligence auxiliary diagnosis systems, and will intensify the work burden due to the need for frequent manual review, hindering the effective promotion and application of the technology in real medical scenarios.

[0005] Therefore, it is necessary for us to provide a multi-modal medical image auxiliary diagnosis scheme that can overcome high-dimensional singular value interference and ensure the robustness of multi-modal feature fusion and the reliability of diagnosis results, in order to meet the urgent needs of clinicians for high-robustness and high-credibility intelligent auxiliary diagnosis systems. SUMMARY

[0006] In view of the deficiencies in the prior art, the present application provides a multi-modal medical image auxiliary diagnosis method, a diagnosis model and a model training method, which can effectively alleviate or eliminate high-dimensional feature singular value interference, and ensure the stability and reliability of the multi-modal feature fusion process and the final diagnosis result.

[0007] To achieve the above object, in a first aspect, the present application provides a multi-modal medical image auxiliary diagnosis method, comprising the following steps:

[0008] Obtaining medical image data of at least two different imaging modalities;

[0009] Preprocessing the medical image data to generate standardized medical image data;

[0010] Multi-modal feature extraction and fusion are performed on the standardized medical image data to generate a fusion feature representation; wherein the multi-modal feature extraction and fusion include feature whitening and attention regularization;

[0011] Based on the fusion feature representation, an auxiliary diagnosis result is generated by a classifier.

[0012] As a further solution, the medical image data is further configured as three-dimensional volume data, and data enhancement processing is performed based on the three-dimensional volume data, including:

[0013] Rotation enhancement processing: linear transformation of voxel point position by three-dimensional space coordinate transformation matrix to maintain local structure continuity and change overall direction distribution of the three-dimensional volume data;

[0014] Scaling enhancement processing: calculating new voxel intensity values by three-dimensional interpolation function, so that the three-dimensional volume data produces diverse changes in spatial resolution without destroying structural consistency;

[0015] Elastic deformation enhancement processing: non-linear deformation of the three-dimensional volume data by constructing a continuous displacement field to realize flexible simulation of local disturbance; wherein the displacement field is generated by a Gaussian smoothed random field;

[0016] After rotation enhancement processing, scaling enhancement processing and elastic deformation enhancement processing, the generated multi-version volume data is merged to form an enhanced medical image data set.

[0017] As a further solution, the medical image data is further configured as an original image tensor, and data standardization processing is performed based on the original image tensor, including:

[0018] Denoising processing: using non-local mean filtering or bilateral filtering to realize noise smoothing suppression by calculating pixel neighborhood similarity;

[0019] Normalization: Linear mapping of image pixel values based on the mean μ and standard deviation σ of the current image to unify pixel value distribution and adjust image spatial size to a fixed resolution;

[0020] Contrast Enhancement: Histogram equalization or adaptive histogram equalization is used to adjust the pixel distribution through cumulative distribution function transformation.

[0021] As a further solution, feature extraction is performed for each imaging modality through multiple convolutional neural network branches arranged in parallel, and each branch outputs a modality-specific feature map corresponding to the modality.

[0022] As a further solution, it also includes feature whitening processing: performing feature decomposition based on the covariance matrix of each modality feature map to construct a whitening matrix, and performing linear transformation on the corresponding modality-specific feature map using the whitening matrix.

[0023] As a further solution, it also includes attention regularization processing: mapping the whitened feature vector into a query vector, a key vector, and a value vector, and calculating an attention score by introducing a regularization term based on the weight distribution entropy, to generate an attention-weighted modality feature vector.

[0024] As a further solution, the attention-weighted modality feature vectors are fused through a concatenation operation to generate a unified fusion feature representation.

[0025] As a further solution, the fusion feature representation is used to generate an auxiliary diagnosis result through a classifier:

[0026] The fusion feature representation is input into a fully connected neural network containing at least one hidden layer;

[0027] The input feature is linearly mapped and nonlinearly transformed layer by layer through the hidden layer to realize hierarchical feature abstraction

[0028] The abstracted high-dimensional feature is classified and transformed through the output layer of the network, and the disease class probability distribution is generated through the Softmax function;

[0029] Based on the probability distribution, the disease class with the highest probability is determined as the final auxiliary diagnosis result.

[0030] In a second aspect, the present application also provides a multi-modal medical image auxiliary diagnosis model, which is an end-to-end deep learning architecture, and the model comprises:

[0031] A feature extraction module extracts modality-specific feature maps from standardized medical image data of different imaging modalities through parallel neural network branches, respectively.

[0032] a feature whitening module, configured to perform whitening processing on the modality-specific feature map to eliminate linear correlation between feature dimensions;

[0033] an attention fusion module, configured to introduce an attention mechanism to the whitened feature and add a regularization term to balance weight distribution, and fuse weighted modality-specific features to generate a fused feature representation;

[0034] a classification module, configured to perform classification diagnosis on the fused feature representation and output an auxiliary diagnosis result.

[0035] In a third aspect, the present application further provides a training method of a multi-modal medical image auxiliary diagnosis model, which is used to train the multi-modal medical image auxiliary diagnosis model as described in the above technical solution, and the training method comprises the following steps:

[0036] obtaining a multi-modal medical image training sample and a corresponding disease label thereof;

[0037] inputting the training sample into an initial model, wherein the initial model comprises a feature extraction module, a feature whitening module, an attention fusion module and a classification module;

[0038] performing forward propagation to obtain a disease prediction probability by sequentially passing through each module of the initial model;

[0039] calculating a loss between the disease prediction probability and the disease label;

[0040] jointly optimizing trainable parameters of all modules in the initial model in an end-to-end manner through gradient back propagation to minimize the loss; and repeating the above steps until the loss function converges, thereby obtaining a trained auxiliary diagnosis model.

[0041] Compared with the prior art, the present application has the following beneficial effects:

[0042] 1. For the problem of unstable attention weight distribution caused by high-dimensional singular value interference in the prior art, the present application introduces feature whitening processing before the attention mechanism, effectively eliminates the linear correlation between feature dimensions, and ensures that the feature distribution satisfies the stable state of the unit covariance matrix after whitening. From the source, the interference of abnormal singular values on attention calculation is inhibited, so that the multi-modal feature fusion process is no longer sensitive to the change of input data, and the stability of the system is significantly improved.

[0043] 2、The application introduces an entropy regularization term in traditional attention calculation, effectively constraining the distribution concentration degree of attention weight. This mechanism prompts the model to automatically learn a more balanced feature weight allocation strategy during training, ensuring the stability of feature weighting and avoiding excessive dependence on a few abnormal features, thereby significantly improving the reliability of diagnostic decision while maintaining the sensitivity of the model to key lesion features.

[0044] 3、The application forms a complete solution by constructing a complete end-to-end processing pipeline, organically combining data enhancement, standardization preprocessing, improved feature fusion and classification diagnosis. This scheme shows excellent generalization ability and robustness when facing medical image data of different sources and different devices, effectively overcoming the performance fluctuation problem caused by unstable weight allocation in the prior art, and providing a reliable auxiliary diagnosis tool for clinical practice.

[0045] 4、The application constructs a complete technical system from raw images to final diagnosis, and each module adopts an end-to-end joint training method to optimize all parameters through a cross-entropy loss function, significantly reducing the risk of misdiagnosis and missed diagnosis caused by unstable models. The stable feature fusion mechanism ensures the consistency of the diagnosis results, reducing the need for manual review, which not only relieves the workload of doctors, but also enhances the trust of clinical artificial intelligence auxiliary diagnosis system, and effectively promotes the landing application of intelligent diagnosis technology in real medical scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 is a flow chart of the prior art multi-modal medical image diagnosis technology;

[0047] Figure 2 is a flow chart of the multi-modal medical image auxiliary diagnosis method in one specific embodiment of the application;

[0048] Figure 3 is a flow chart of the multi-modal medical image data enhancement processing step in one specific embodiment of the application;

[0049] Figure 4 is a flow chart of the multi-modal feature whitening and attention regularization in one specific embodiment of the application;

[0050] Figure 5 is a flow chart of the training method of the medical image auxiliary diagnosis model in one specific embodiment of the application. DETAILED DESCRIPTION

[0051] To make the purpose, technical scheme and advantages of the embodiments of the application clearer, the technical scheme in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are part of the embodiments of the application, not all embodiments.

[0052] In recent years, deep learning technology has made significant progress in medical image assisted diagnosis, providing a new technical path for efficient and accurate lesion recognition and diagnosis analysis. In the prior art, as shown in Figure 1 , a typical implementation scheme usually includes the following steps: first, acquiring multi-modal original medical image data and preprocessing; then extracting feature maps of different modalities using convolutional neural networks; then introducing attention mechanism to weight and fuse multi-modal features; finally, outputting auxiliary diagnosis results based on the fused features through a classifier.

[0053] However, the inventors have found through in-depth research and analysis that the above method has obvious limitations when dealing with high-dimensional features: when the feature dimension is high, the dot product operation in the attention mechanism is easily disturbed by a few abnormal singular values in the feature vector, resulting in unstable weight distribution. Especially when the feature dimension exceeds 1000, the dot product operation will amplify the influence of abnormal singular values, making the attention weight too sensitive to small changes in input features, thereby affecting the stability of feature fusion.

[0054] Therefore, in order to effectively alleviate or eliminate high-dimensional singular value interference, ensure the stability and reliability of multi-modal feature fusion, and promote the further development of artificial intelligence assisted diagnosis technology, the embodiments of the present application provide a multi-modal medical image assisted diagnosis method, as shown in Figure 2 , the method includes the following steps:

[0055] S101, acquiring medical image data of at least two different imaging modalities;

[0056] S102, preprocessing the medical image data to generate standardized medical image data;

[0057] S103, multi-modal feature extraction and fusion of the standardized medical image data to generate a fused feature representation; wherein the multi-modal feature extraction and fusion includes feature whitening and attention regularization;

[0058] S104, generating an auxiliary diagnosis result based on the fused feature representation through a classifier.

[0059] In specific implementation, the medical image data acquired by the embodiments includes but is not limited to data of different imaging modalities such as computed tomography (CT), magnetic resonance imaging (MRI), and positron emission tomography (PET). The preprocessing includes denoising, image size standardization, contrast enhancement and other operations to eliminate device differences and improve image quality.

[0060] In the feature extraction and fusion stage, a deep learning-based artificial intelligence model is used to process the standardized medical image data. This process specifically includes feature whitening to eliminate linear correlations between feature dimensions, and attention regularization to balance the distribution of attention weights by introducing regularization constraints. This effectively solves the problem of singular value interference in high-dimensional feature spaces and improves the stability of feature fusion.

[0061] Finally, the obtained fused feature representation is input into a classifier to generate auxiliary diagnostic results. Optional classifiers include support vector machines (SVM) and fully connected neural networks, with outputs in the form of disease probability distributions and / or classification labels, providing quantitative reference for clinical diagnosis.

[0062] like Figure 3 As shown, as a further solution, the medical image data is further configured as three-dimensional volumetric data, and data augmentation processing is performed based on the three-dimensional volumetric data, including:

[0063] Rotation enhancement processing: The positions of voxel points are linearly transformed by a three-dimensional spatial coordinate transformation matrix to maintain the local structural continuity of the three-dimensional volume data and change the overall directional distribution;

[0064] Scaling enhancement processing: New voxel intensity values ​​are calculated using a three-dimensional interpolation function, so that the three-dimensional volume data produces diverse changes in spatial resolution without destroying structural consistency;

[0065] Elastic deformation enhancement processing: Nonlinear deformation of the three-dimensional volume data is performed by constructing a continuous displacement field to achieve flexible simulation of local disturbances; wherein, the displacement field is generated by Gaussian smoothing of a random field;

[0066] After rotation enhancement, scaling enhancement, and elastic deformation enhancement, the generated multi-version volume data are merged to form an enhanced medical image dataset.

[0067] In this embodiment, the data augmentation process based on the three-dimensional volume data specifically includes the following operations:

[0068] The three-dimensional volume data is represented as V∈ ; where H v W v D v These represent the height, width, and depth of the three-dimensional volume data, respectively; C v The number of channels corresponds to the signal channels of different imaging modalities; voxels, as the basic units of three-dimensional volume data, contain information on tissue density, metabolic activity, and structural features.

[0069] The data augmentation process includes the following steps:

[0070] Rotational enhancement processing: through the rotational transformation formula V rot =R θ (V) Perform spatial transformation on the volume data, where R θ The rotation operator is represented by θ = (θx, θy, θz), which is the rotation angle vector around the x, y, and z axes. This operation achieves a linear transformation of the voxel position through a three-dimensional coordinate transformation matrix. While maintaining the continuity of the local structure, it changes the overall orientation distribution, thereby eliminating data directionality deviation and enabling the identification of different acquisition postures.

[0071] Scaling enhancement processing: Adjusting the spatial proportion of the image through scale transformation, using the scaling transformation formula V. scale =S s1 (V) Adjust the space ratio, where S s1 The scaling operator is denoted by s1, which represents the scaling factor. When s1 > 1, a scaling operation is performed; when s1 < 1, a scaling operation is performed. The scaling process calculates new voxel intensity values ​​using a 3D interpolation function, allowing the data to exhibit diverse spatial resolution variations without compromising structural consistency. This enhances the ability to identify lesions of different sizes and tissue proportions. The scaled volume maintains the same input format as the original data. This ensures compatibility with the input structure of subsequent processing steps.

[0072] Elastic deformation enhancement treatment: using the deformation formula V deform =V+D, by constructing a continuous displacement field to nonlinearly deform the volumetric image, flexible simulation of local disturbances is achieved, where D∈ For a three-dimensional displacement field tensor, each voxel corresponds to a three-dimensional displacement vector (d x ,d y ,x z ), representing the offset in spatial coordinates; the displacement field is generated by a Gaussian smoothed random field to ensure the spatial continuity of deformation, thereby simulating the flexible changes of biological tissues and the deformation characteristics of organs, so as to learn the diversity and complexity of lesion morphology.

[0073] As an alternative embodiment of the present invention, the elastic deformation enhancement processing can be replaced by a random flip enhancement operation. Unlike displacement-field-based elastic deformation, random flip performs a mirror transformation of the volume data along the coordinate axes using a random flip operator, specifically including horizontal and vertical flip operations. By changing the spatial orientation of the image rather than local deformation, data diversity is increased, achieving robust learning of pose changes while maintaining the realism of the anatomical structure.

[0074] As another alternative embodiment of the present invention, Gaussian noise addition can also be used as an enhancement method. This is achieved by generating a low-intensity Gaussian noise tensor and superimposing it onto the original volume data. Unlike the structural changes caused by elastic deformation, Gaussian noise enhancement improves the model's resistance to noise interference by introducing random perturbations at the pixel level, thereby enhancing its diagnostic stability under low-quality image conditions.

[0075] The three data augmentation methods described above enhance the model's generalization ability from different perspectives: elastic deformation augmentation focuses on morphological changes by simulating tissue deformation; random flipping augmentation enhances orientation invariance through spatial transformation; and Gaussian noise augmentation improves noise robustness by introducing random perturbations. Depending on the needs of practical applications, these augmentation techniques can be used individually or in combination to provide diverse training samples for the multimodal medical image-assisted diagnostic framework.

[0076] After completing the above augmentation operations, the generated multi-version volumetric data are merged to form an augmented medical image dataset, providing a high-quality input foundation for subsequent preprocessing and feature extraction. This data augmentation scheme significantly improves the model's adaptability to acquisition pose, lesion size, and tissue deformation by systematically introducing geometric and deformation diversity, thereby enhancing the robustness and generalization performance of the entire diagnostic framework.

[0077] As a further solution, the medical image data is further configured into raw image tensors, and data normalization processing is performed based on the raw image tensors, including:

[0078] Noise reduction: Non-local mean filtering or bilateral filtering is used to smooth and suppress noise by calculating the similarity of pixel neighborhoods;

[0079] Normalization: Based on the pixel mean μ and standard deviation σ of the current image, the image pixel values ​​are linearly mapped to unify the pixel value distribution and adjust the image spatial size to a fixed resolution.

[0080] Contrast enhancement processing: Histogram equalization or adaptive histogram equalization is used to adjust the pixel distribution through cumulative distribution function transformation.

[0081] As a further optimization of the present invention, the medical image data is configured as a raw image tensor, and data standardization processing is performed based on this tensor. The raw image tensor is represented as I. raw ∈ Where H0, W0, and D0 represent the height, width, and number of channels of the original image, respectively, corresponding to the number of modalities of the input image.

[0082] The data standardization process includes the following three sub-steps:

[0083] Denoising: A nonlocal mean filtering method is used to smoothly suppress noise by calculating neighborhood similarity within the pixel space. The mathematical expression of the denoising process is as follows:

[0084]

[0085] In the formula, I deneoised I represents the denoised image tensor. raw Ω represents the original image tensor, where p and q represent pixel positions in the image; Ω represents the neighborhood set centered at pixel p.

[0086] w(p,q) represents the similarity weight between pixels: Where h represents the filtering parameter, used to control the weight decay rate, |I raw (p)-I raw (q) represents the intensity difference between pixels, Z(p) is the normalization factor, and it is guaranteed that the sum of all weights is 1.

[0087] This processing improves image clarity by smoothing noise while preserving tissue edges and detailed textures through content-based weighting.

[0088] As an alternative embodiment of the present invention, the denoising process can also employ a bilateral filtering method. Unlike nonlocal mean filtering, which constructs a large-scale neighborhood set, bilateral filtering calculates pixel weights by combining spatial distance and gray-level similarity criteria.

[0089] Nonlocal mean filtering achieves excellent denoising results through a large-scale similarity search, while bilateral filtering achieves efficient edge preservation through spatial-grayscale dual constraints. Depending on the image quality requirements and computational resources required in the practical application, a suitable denoising method can be flexibly selected.

[0090] Normalization: This process adjusts the numerical distribution of image pixels to a standard range using a linear mapping. The specific process is as follows:

[0091]

[0092] In the formula, I deneoised I represents the denoised image tensor. normalized σ represents the normalized image tensor, μ represents the pixel mean of the current image, and σ represents the standard deviation. This standardization process ensures that the statistical distribution of brightness and contrast is consistent among different image modalities, thereby reducing the deviation caused by imaging conditions of different devices.

[0093] After normalization, the image spatial size is uniformly adjusted to a fixed resolution H*W, where H and W are preset standard input sizes. The size adjustment adopts the bilinear interpolation method, which calculates the intensity of the target pixel by weighting the gray values ​​of the four surrounding pixels in the two-dimensional pixel plane, thereby maintaining structural continuity and edge smoothness, and keeping the multimodal data consistent in spatial resolution.

[0094] Contrast enhancement processing: A histogram equalization method is used, which adjusts the pixel distribution through a cumulative distribution function transformation to make the grayscale value distribution of the output image more uniform. The mapping function is defined as:

[0095]

[0096] In the formula, I standardized Represents standardized medical imaging data, I normalized denoted as the normalized image tensor, and T represents the cumulative distribution function transformation based on the normalized histogram. Its function is to increase the proportion of high-frequency grayscale range and significantly improve the local contrast of tissue boundaries and lesion areas.

[0097] This processing step ensures that input images of different modalities achieve a unified standard in terms of brightness dynamic range and texture detail representation, ultimately resulting in standardized medical image data of uniform specifications. standardized This provides high-quality, low-noise, and scale-consistent input data for subsequent multimodal feature extraction and fusion.

[0098] As an alternative embodiment of the present invention, the contrast enhancement processing can also employ the Adaptive Histogram Equalization (CLAHE) method. Unlike global histogram equalization, CLAHE divides the image into multiple continuous non-overlapping block regions, performs histogram equalization operations independently on each region, and limits the intra-block contrast gain, thus replacing the global histogram mapping operation.

[0099] The two contrast enhancement methods have different applicable scenarios: histogram equalization is computationally efficient and suitable for global contrast enhancement; adaptive histogram equalization (CLAHE) can better enhance the contrast of fine structures through local processing, making it particularly suitable for displaying local lesion features, while effectively avoiding the problem of over-enhancing the overall image. The appropriate contrast enhancement method can be flexibly selected based on the image quality requirements and computational resource considerations in the specific application.

[0100] As a further solution, multimodal feature extraction and fusion are performed on the standardized medical image data to generate a fused feature representation, wherein the multimodal feature extraction and fusion includes feature whitening processing and attention regularization processing.

[0101] In the specific implementation process, standardized medical image data from the preprocessing stage is received as input, and this input data is represented as a multidimensional tensor I∈ Where H and W represent the image height and width, respectively, and C represents the number of channels, corresponding to the image channel dimension under different imaging modalities.

[0102] For each imaging modality, multiple independent and parallel convolutional neural network branches are used to extract features from the input tensor. Each branch captures local spatial structure information through convolutional layers and extracts hierarchical features through pooling layers, outputting a modality-specific feature map F. m ∈ Where m represents the modality index, and D m This represents the feature dimension of the corresponding modality. This feature extraction structure, through parameter sharing and convolution kernel sliding operations, obtains spatial texture and edge features that match the characteristics of the imaging modality, providing high-dimensional input for subsequent fusion processing.

[0103] like Figure 4 As shown, Figure 4 This is a schematic diagram of the multimodal feature whitening and attention regularization process in a specific implementation example of this application.

[0104] As a further optimization of the present invention, feature whitening processing is also included: feature decomposition is performed based on the covariance matrix of each modality feature map to construct a whitening matrix, and the whitening matrix is ​​used to perform linear transformation on the corresponding modality-specific feature map to ensure that the covariance matrix of the output feature is close to the identity matrix, and to ensure that the variance of each dimension is consistent, thereby improving the stability and numerical controllability of the model in high-dimensional space.

[0105] In a specific embodiment, for each modality-specific feature map F m The purpose of feature whitening is to eliminate the linear correlation between feature dimensions, reduce the interference caused by the differences in singular values ​​in high-dimensional features, and obtain statistically independent input features.

[0106] ZCA whitening:

[0107] Whitening matrix W m ∈ The whitening calculation is based on the ZCA (Zero-phase Component Analysis Whitening) principle, using the formula:

[0108]

[0109] In the formula, E represents the eigenvector matrix of the characteristic covariance matrix. This represents the corresponding eigenvalue diagonal matrix. It represents the reciprocal of the square root of each element of the eigenvalue, and T represents the transpose of the matrix.

[0110] Input mode-specific feature map F m ∈ D m This represents the feature dimension of the modality.

[0111] First, calculate the covariance matrix of the features along the sample dimension, expressed as:

[0112]

[0113] In the formula, Let F represent the covariance matrix, N represent the number of samples for this modality feature, and F represent the variance matrix. m,i Let μ represent the eigenvector of the i-th sample in the m-th modality, T represent the transpose of the matrix, and μ represent the eigenvector of the i-th sample in the m-th modality. m This represents the feature mean vector, whose elements μ m,j The mean of the j-th dimension of the corresponding feature is used to calculate the linear correlation structure between features.

[0114] Then, eigenvalue decomposition was performed on the covariance matrix to obtain:

[0115]

[0116] In the formula, Let E represent the covariance matrix. m Let e ​​be the eigenvector matrix and its column vectors. m,k This represents the principal direction corresponding to the k-th eigenvalue. It is an eigenvalue diagonal matrix, reflecting the distribution intensity of the eigenvalue variance in different directions.

[0117] Based on this decomposition result, a whitening matrix W is constructed. m :

[0118]

[0119] In the formula, The square root of the inverse of the eigenvalue matrix is ​​given by the following diagonal elements: This operation normalizes the variance of each principal component in the feature space so that its standard deviation is one.

[0120] The whitening matrix is ​​applied to the input features to obtain the whitened output:

[0121]

[0122] In the formula, W represents the feature vector after whitening. m Let F be the whitening matrix.m For a mode-specific feature map, its covariance matrix satisfy:

[0123]

[0124] In the formula, I∈ It is an identity matrix that represents complete decorrelation between features.

[0125] PCA whitening:

[0126] As an alternative embodiment of the present invention, feature whitening can also employ the PCA whitening method. Unlike ZCA whitening, which retains all feature dimensions, PCA whitening constructs a whitening matrix by selecting the first k principal components (k ≤ original feature dimensions) after eigenvalue decomposition of the feature covariance matrix, thereby achieving dimensionality reduction while eliminating feature correlation.

[0127] In this embodiment, feature whitening eliminates redundant coupling relationships and singular value biases of each modality feature in high-dimensional space through statistical decorrelation and scale normalization, making the feature distribution of different imaging modalities tend to be consistent. This whitening process reduces the statistical bias during the fusion of different modalities by eliminating the linear dependence and scale differences within modal features, making the feature representation in the fusion space more stable and separable.

[0128] As a further optimization of the present invention, attention regularization processing is also included: the whitened feature vector is mapped to a query vector, a key vector and a value vector, and an attention score is calculated by introducing a regularization term based on the weight distribution entropy value to generate an attention-weighted modal feature vector.

[0129] In the fusion network, the whitening output from the above steps is passed as input to the attention regularization module. Through a dynamic weight allocation mechanism, high diagnostic value feature channels are strengthened, forming a fusion feature representation with consistent structure and balanced distribution. .

[0130] In a specific embodiment, an attention mechanism module is introduced into the whitened feature vector to achieve intramodal feature weight allocation. The attention mechanism uses a learnable query vector Q. m Key vector K m Sum vector V m Mapping implements weight calculation:

[0131] Q m =W q F white,m K m =W k F white,m V m =W v F white,m

[0132] In the formula, F white,m Q represents the whitened feature vector. m K represents the matching requirement for the current feature. m Semantic description of candidate features, V m Used for weighted aggregation of information;

[0133] W q W k W v ∈ These are trainable projection matrices for the query, key, and value, respectively, responsible for whitening the feature vectors. Project onto a low-dimensional attention subspace; For matching the dimensions of the weight matrix, d k D represents the dimension of the attention computation space. m D represents the feature dimension of the corresponding modality. k This represents the dimension of the attention computation space, that is, the dimension of the attention subspace to which the whitening feature vector is projected.

[0134] The attention score matrix is ​​calculated using the dot product operation:

[0135]

[0136] In the formula, This is a scaling factor used to suppress gradient instability caused by excessively large dot product results in high-dimensional space.

[0137] The weight distribution is then obtained by normalization using the Softmax function:

[0138]

[0139] In the formula, a m ∈R N Let a be the attention weight vector, and its element a m,i This represents the importance weight of the i-th position of the input feature in the global feature aggregation.

[0140] To avoid excessive focus on a few regions, an entropy regularization term is set:

[0141]

[0142] In the formula, H(a) m The entropy value represents the uncertainty measure of the weight distribution; a larger entropy value indicates a more uniform weight distribution. N represents the number of feature elements. m,iThis represents the importance weight of the i-th position of the input feature in the global feature aggregation. The entropy regularization term suppresses the instability of the attention distribution by adding a smoothing constraint, ensuring a more balanced distribution of weights across feature dimensions and preventing excessive concentration of local feature weights.

[0143] Subsequently, an entropy regularization term is introduced into the attention score calculation:

[0144]

[0145] In the formula, Score is the attention output matrix, representing the feature response distribution after regularization and weighting; the query vector Q... m The key vector K represents the matching requirement for the current feature. m The semantic description representing the candidate features, the value vector V m Used for information weighted aggregation, λ represents the regularization coefficient, used to control the intensity of the entropy penalty for attention distribution.

[0146] By correcting the attention score through entropy regularization, the balance of feature weighting is enhanced, resulting in the final corrected formula:

[0147]

[0148] In the formula, λ is the regularization parameter, which controls the influence of the entropy term in the attention optimization process.

[0149] This mechanism enables a mapping from a high-dimensional feature whitening space to a smooth attention space, avoiding the polarization effect of abnormal channel weights and ensuring that each modality has a similar response scale during the fusion stage.

[0150] As an alternative embodiment of the present invention, attention regularization can also employ the L2 regularization method. Unlike entropy regularization, which calculates the information entropy of the weight distribution, L2 regularization constrains the concentration of the weight distribution by calculating the L2 norm of the attention weight vector and applying a penalty term, without needing to calculate the entropy value of the weight distribution.

[0151] Attention regularization is used to map from a high-dimensional feature whitening space to a smooth attention space, avoiding the polarization effect of abnormal channel weights. Both regularization methods ensure that each modality has a similar response scale during the fusion stage. The final output is a weighted feature representation. As one of the fusion inputs, it together with the weighted features of other modalities to form a balance-sensitive multimodal fusion representation. This indicates that information balance and structural sensitivity are considered at the global level.

[0152] As a further optimization of the present invention, the attention-weighted feature vectors of each modality are fused through a concatenation operation to generate a unified fused feature representation F.fused :

[0153]

[0154] In the formula, ,and Concat(*) represents the weighted feature vectors of each modality. The features are concatenated to generate a unified, fused feature representation, F weighted,M Let M represent the weighted feature vectors of each modality, and M represent the number of modalities involved in the fusion. The concatenated feature vectors retain complementary information of different imaging modalities in the multimodal space. Combined with the processing results of the aforementioned whitening and attention mechanisms, a fusion feature representation with both discriminative power and stability is obtained.

[0155] As a further optimization of the present invention, the auxiliary diagnostic results are generated by a classifier based on the fused feature representation:

[0156] The fused feature representation is input into a fully connected neural network containing at least one hidden layer;

[0157] The hidden layer performs layer-by-layer linear mapping and non-linear transformation on the input features to achieve hierarchical feature abstraction.

[0158] The network's output layer performs classification transformations on the abstracted high-dimensional features, and the Softmax function generates the disease category probability distribution.

[0159] Based on the probability distribution, the disease category with the highest probability is determined as the final auxiliary diagnostic result.

[0160] In this embodiment, the fused feature representation As input, F fused It also represents a high-dimensional vector formed by concatenating multimodal features, containing structural and functional information from different imaging modalities such as computed tomography (CT), magnetic resonance imaging (MRI), and positron emission tomography (PET). A pre-trained fully connected neural network is used as the classifier, and feature space mapping and discriminative feature extraction are achieved through multi-layer nonlinear transformations. The network structure specifically includes:

[0161] The input layer passes the fused feature vector to the first hidden layer for linear mapping and non-linear activation:

[0162] h1=ReLU(W h1 F fused +b h1 )

[0163] In the formula, W h1 ∈ The weight matrix is ​​input to the hidden layer. Let D be the bias vector. h1 The number of neurons in the first hidden layer is represented by ; the ReLU activation function is defined as ReLU(x)=max(0,x), which introduces nonlinear features and prevents gradient vanishing, enabling the model to have nonlinear discrimination ability in the mapping of complex medical image features.

[0164] After processing by the first hidden layer, the output vector h1 enters the subsequent multi-layer hidden network for deep feature transformation. The output of the l-th hidden layer is:

[0165] hl=ReLU(W hl hl-1+b hl )

[0166] In the formula, W hl ∈ This is the weight matrix between the l-th layer and the previous layer. For the corresponding bias term, D hl This represents the number of neurons in the l-th layer, with layer indices l=2,3,...,L, where L is the total number of hidden layers.

[0167] This structure enhances the semantic separability of fused features through layer-by-layer feature abstraction and nonlinear mapping, thereby forming a more discriminative feature representation in a high-dimensional diagnostic space.

[0168] The network's output layer performs a classification transformation on the high-dimensional features after multiple nonlinear mappings, and generates a disease category probability distribution through the Softmax function.

[0169] P=softmax(W o h L +b o )

[0170] in, Let b be the output layer weight matrix. o ∈R K D is the output layer bias vector. hL Let K be the dimension of the last hidden layer, K be the number of disease categories, and h be the number of disease categories. L W is the hidden feature vector of the last layer. o and b o These are the output layer weight matrix and bias vector, respectively; the output vector P[P1,P2,...,P] K The value represents the predicted probability that the input image belongs to each disease category, with the category with the highest probability corresponding to the final auxiliary diagnostic result.

[0171] The Softmax function is defined as follows:

[0172]

[0173] The linear output is converted into a probability distribution, so that each disease category corresponds to a probability value between 0 and 1, and the sum of all probabilities is 1.

[0174] This classification process achieves accurate mapping from multimodal fusion features to disease identification through multi-layer feature abstraction and probability normalization. It can effectively distinguish different types of medical imaging patterns such as tumors, vascular lesions and neurodegenerative diseases, providing reliable intelligent auxiliary judgment for clinical diagnosis.

[0175] Through the above technical solutions, the present invention constructs a complete processing flow from multimodal feature fusion to disease classification and diagnosis, ensuring that the auxiliary diagnostic system maintains high accuracy while having good clinical applicability.

[0176] According to another embodiment of the present invention, a multimodal medical image-assisted diagnostic model is provided, the model being an end-to-end deep learning architecture, the model comprising:

[0177] The feature extraction module extracts modality-specific feature maps from standardized medical image data of different imaging modalities through parallel neural network branches.

[0178] The feature whitening module performs whitening processing on the modality-specific feature map to eliminate the linear correlation between feature dimensions;

[0179] The attention fusion module introduces an attention mechanism and adds a regularization term to the whitened features to balance the weight distribution, and fuses the weighted features of each modality to generate a fused feature representation.

[0180] The classification module performs classification and diagnosis on the fused feature representation and outputs auxiliary diagnostic results.

[0181] Feature extraction module: Input is standardized medical image data Where H is the image height, W is the image width, and C is the number of channels.

[0182] The module employs a convolutional neural network branch structure, with each branch specializing in processing a specific imaging modality. Each branch contains convolutional layers, batch normalization layers, and ReLU activation functions. The convolution operation formula is as follows:

[0183]

[0184] In the formula, This represents the output feature map of the l-th layer. and These represent the kernel weights and bias parameters, respectively. * indicates the convolution operation. BN is the batch normalization operation, used to stabilize the gradient distribution and improve the training convergence speed. ReLU is the rectified linear unit activation function.

[0185] In this feature extraction process, each modality branch performs feature extraction independently, outputting a modality-specific feature map F. m .

[0186] Feature whitening module: This module receives the modality-specific feature map F output by the feature extraction module. m ZCA whitening transformation is performed to eliminate statistical correlation between modes.

[0187] Calculate the covariance matrix:

[0188]

[0189] In the formula, Let N be the modal feature mean vector, and N be the number of samples. Then, eigenvalue decomposition is performed on the covariance matrix to obtain... ,in The eigenvector matrix, It is a diagonal matrix of eigenvalues.

[0190] Constructing the whitening matrix: Used to generate whitening feature vectors W m Let F be the whitening matrix. m For modality-specific feature maps, whitening ensures that the covariance of the output features is an identity matrix, thereby eliminating the interference of singular values ​​in high-dimensional features and enhancing feature stability.

[0191] Attention fusion module: Weighted aggregation of whitening features from each modality is performed, and a multi-head attention structure is introduced to achieve multi-layer semantic alignment, with each head independently calculating attention weights.

[0192] A multi-head attention structure is adopted. For the h-th attention head, the query vector is first generated. Key vector Sum value vector :

[0193]

[0194] in For attention projection matrix, For the projection dimension.

[0195] Attention score matrix The normalized weights are obtained through the Softmax function. To balance the weight distribution, an entropy regularization term is introduced. The final regularized attention output is .

[0196] The outputs of all attention heads are concatenated and linearly mapped to form a fused feature representation F.fused :

[0197]

[0198] Where H represents the number of attention heads. For the fusion mapping matrix, F weighted,m Let represent the weighted feature vectors of each modality, and m represent the number of modalities involved in the fusion.

[0199] Classification module: Represents the fused features F fused As input, classification and diagnosis are performed through a fully connected neural network.

[0200] The network contains multiple hidden layers, and the output of the l-th hidden layer is:

[0201] hl=ReLU(W hl hl-1+bhl)

[0202] In the formula, Whl∈ This is the weight matrix between the l-th layer and the previous layer. For the corresponding bias term, D hl This represents the number of neurons in the l-th layer, with layer indices l=2,3,...,L, where L is the total number of hidden layers.

[0203] Finally, the disease probability distribution is generated using the Softmax function, and the output layer calculation formula is as follows:

[0204]

[0205] In the formula, To output the weight matrix, Let K be the output layer bias vector, and K be the number of disease categories. W is the hidden feature vector of the last layer. o and b o These are the output layer weight matrix and bias vector, respectively.

[0206] The diagnostic model in this embodiment uses an end-to-end deep learning architecture to efficiently fuse and assist in the diagnosis of multimodal medical images. Through the collaborative work of four core modules—feature extraction, feature whitening, attention fusion, and classification—it achieves accurate medical image-assisted diagnosis.

[0207] like Figure 5 As shown, according to another embodiment of the present invention, a training method for a multimodal medical image-assisted diagnostic model is provided for training the multimodal medical image-assisted diagnostic model as described in the above technical solution. The training method includes the following steps:

[0208] Obtain multimodal medical image training samples and their corresponding disease labels;

[0209] The training samples are input into an initial model, which includes a feature extraction module, a feature whitening module, an attention fusion module, and a classification module.

[0210] Perform forward propagation, sequentially passing through each module of the initial model to obtain the disease prediction probability;

[0211] Calculate the loss between the predicted disease probability and the disease label;

[0212] In an end-to-end manner, the trainable parameters of all modules in the initial model are jointly optimized through gradient backpropagation to minimize the loss; the above steps are repeated until the loss function converges to obtain the trained auxiliary diagnostic model.

[0213] The training method proposed in this embodiment employs an end-to-end joint optimization strategy. The training objective is to optimize model parameters using gradient descent. During training, the optimal parameter configurations for feature extraction, feature whitening, attention fusion, and classification diagnosis are learned simultaneously, ultimately improving the prediction probability. With real labels Minimize the differences.

[0214] Training data preparation: Obtain multimodal medical image training samples and their corresponding disease labels, and input the training samples into the initial model. The multimodal medical image training samples include a medical image dataset obtained through enhancement processing and standardization preprocessing. The labels are represented as one-hot vectors. Where K is the number of disease categories, and each vector element Indicates whether the i-th sample belongs to the k-th disease category.

[0215] Forward propagation process: Standardized medical images will then be... Modality-specific feature maps are generated sequentially through the feature extraction module. Then, the feature whitening module generates decorrelated whitening feature vectors. Then, the attention fusion module calculates the weighted fusion features. Finally, the classification module outputs the predicted probability. During forward propagation, the convolution kernel weights Bias vector Whitening matrix and attention weight matrix All of them participate in feature mapping and information aggregation.

[0216] Loss function calculation: The cross-entropy loss function is used to measure the difference between the predicted probability and the true label.

[0217]

[0218] Where N represents the total number of training samples, Let be the probability that the model predicts the i-th sample as belonging to the k-th class.

[0219] Backpropagation and parameter optimization: based on the loss function The gradient of the model parameters is calculated using the following formula:

[0220] ,

[0221] in, This represents any trainable parameters, including convolution weights, whitening matrix, and attention projection matrix.

[0222] Parameter updates are performed using gradient descent or its variants (such as the Adam optimizer).

[0223]

[0224] in, The learning rate controls the step size for updating parameters.

[0225] The attention regularization parameter λ is adjusted during training through grid search or on the validation set to balance the concentration of weight distribution with the degree of information retention.

[0226] Determine if the loss function has converged. If it has not converged, repeat the forward propagation, loss calculation, backpropagation, and parameter update steps until the loss function converges, and finally obtain the trained auxiliary diagnostic model.

[0227] Based on the model obtained using the above training method, multimodal feature extraction and fusion are performed, and finally, auxiliary diagnostic results are generated through a classifier. This model can stably generate fused features and provide reliable disease probability distributions under multimodal input conditions, significantly improving diagnostic accuracy and cross-modal robustness. The entire training process ensures the model's stability in high-dimensional feature space and enhances its generalization ability to unseen samples by jointly optimizing the parameters of feature extraction, whitening, attention fusion, and classification modules, providing a highly efficient and reliable medical image-assisted diagnostic solution for clinical use.

[0228] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A multi-modal medical image aided diagnosis method, characterized in that, The method comprises the following steps: Obtaining medical image data of at least two different imaging modalities; Preprocessing the medical image data to generate standardized medical image data; Multi-modal feature extraction and fusion are performed on the standardized medical image data to generate a fused feature representation; wherein the multi-modal feature extraction and fusion includes feature whitening and attention regularization; Based on the fused feature representation, an auxiliary diagnosis result is generated by a classifier.

2. The multi-modal medical image aided diagnosis method of claim 1, wherein, The medical image data is further configured as three-dimensional volume data, and data enhancement processing is performed based on the three-dimensional volume data, including: Rotation enhancement processing: linear transformation of voxel point positions through a three-dimensional spatial coordinate transformation matrix to maintain the local structure continuity of the three-dimensional volume data and change the overall direction distribution; Scaling enhancement processing: new voxel intensity values are calculated through a three-dimensional interpolation function, so that the three-dimensional volume data produces diverse changes in spatial resolution without destroying the structural consistency; Elastic deformation enhancement processing: non-linear deformation of the three-dimensional volume data is performed by constructing a continuous displacement field to realize flexible simulation of local disturbance; wherein the displacement field is generated by a Gaussian smoothed random field; After rotation enhancement processing, scaling enhancement processing and elastic deformation enhancement processing, the generated multi-version volume data is merged to form an enhanced medical image data set.

3. The multi-modal medical image aided diagnosis method of claim 1, wherein, The medical image data is further configured as an original image tensor, and data standardization processing is performed based on the original image tensor, including: Denoising processing: non-local mean filtering or bilateral filtering is used to realize noise smoothing and suppression by calculating the similarity of pixel neighborhoods; Normalization processing: linear mapping of image pixel values based on the mean and standard deviation of the current image to unify the pixel value distribution and adjust the image space size to a fixed resolution; Contrast enhancement processing: histogram equalization or adaptive histogram equalization is used to adjust the pixel distribution through cumulative distribution function transformation.

4. The multi-modal medical image aided diagnosis method of claim 1, wherein, Each imaging modality is feature-extracted by a plurality of convolutional neural network branches arranged in parallel, and each branch outputs a modality-specific feature map corresponding to the modality.

5. The multi-modal medical image aided diagnosis method of claim 4, wherein, It also includes feature whitening processing: feature decomposition is performed based on the covariance matrix of each modality feature map to construct a whitening matrix, and the corresponding modality-specific feature map is linearly transformed using the whitening matrix.

6. The multi-modal medical image aided diagnosis method of claim 5, wherein, It also includes attention regularization processing: the whitened feature vector is mapped to a query vector, a key vector and a value vector, and an attention score is calculated by introducing a regularization term based on the weight distribution entropy value, generating an attention-weighted modality feature vector.

7. The multi-modal medical image aided diagnosis method of claim 6, wherein, The attention-weighted modality feature vectors are fused by concatenation operation to generate a unified fused feature representation.

8. The multi-modal medical image aided diagnosis method of claim 1, wherein, The fused feature representation is input into a fully connected neural network comprising at least one hidden layer; The input features are linearly mapped and nonlinearly transformed layer by layer through the hidden layer to realize hierarchical feature abstraction; ​ The high-dimensional abstracted features are classified and transformed by an output layer of a network, and a disease category probability distribution is generated by a Softmax function; Based on the probability distribution, a disease category with the highest probability is determined as a final auxiliary diagnosis result.

9. A multi-modal medical image assisted diagnosis model, characterized in that, The model is an end-to-end deep learning architecture, and the model comprises: a feature extraction module that extracts modality-specific feature maps from standardized medical image data of different imaging modalities through parallel neural network branches; a feature whitening module that whitens the modality-specific feature maps to eliminate linear correlation between feature dimensions; an attention fusion module that introduces an attention mechanism to the whitened features, adds a regularization term to balance weight distribution, and fuses weighted modality-specific features to generate a fused feature representation; a classification module that classifies the fused feature representation to output an auxiliary diagnosis result. 10.A method for training a multi-modal medical image aided diagnosis model, characterized in that, The training method for training the multi-modal medical image auxiliary diagnosis model of claim 9 comprises the following steps: obtaining multi-modal medical image training samples and their corresponding disease labels; inputting the training samples into an initial model, wherein the initial model comprises a feature extraction module, a feature whitening module, an attention fusion module, and a classification module; performing forward propagation to obtain disease prediction probabilities through the modules of the initial model in sequence; calculating the loss between the disease prediction probabilities and the disease labels; and in an end-to-end manner, jointly optimizing the trainable parameters of all modules in the initial model through gradient backpropagation to minimize the loss; repeating the above steps until the loss function converges to obtain a trained auxiliary diagnosis model.

Images