A deep learning-based multi-modal medical image intelligent analysis method

Abstract

The application relates to the technical field of medical image analysis, and discloses a kind of multi-modal medical image intelligent analysis method based on deep learning, the application adopts full deep learning end-to-end architecture: the grey scale normalization, spatial resampling and noise suppression preprocessing are carried out to the collected multi-modal original medical image, and each modality standardized medical image is obtained;Multi-modal image non-rigid spatial alignment is realized through a deep learning registration network, and multi-modal alignment images are generated;Each modality multi-scale deep feature is extracted using a multi-branch deep feature extraction network, and adaptive weighted fusion and multi-scale splicing are completed through a cross-modal attention fusion network to generate a multi-modal fusion feature map;Finally, the lesion position information, size parameter and pathological grading result are synchronously output through a lesion analysis deep learning network;The application solves the problems of insufficient multi-modal medical image analysis registration accuracy, low feature fusion adaptability and disjointedness of each link in the prior art.

Description

Technical Field

[0001] This invention relates to the field of medical image analysis technology, and in particular to a multimodal intelligent medical image analysis method based on deep learning. Background Technology

[0002] Multimodal medical image intelligent analysis is a core research direction in the field of medical image-assisted diagnosis, and it has significant application value in clinical scenarios such as tumor screening, pathological grading, and efficacy evaluation. Different imaging modalities possess differentiated imaging mechanisms and information advantages. For example, computed tomography (CT) can clearly present the anatomical structure of bones and high-density tissues, magnetic resonance imaging (MRI) can accurately distinguish the gray-scale differences and boundary morphology of soft tissues, and functional imaging can reflect the metabolic activity and functional state of tissues. The complementary information of multimodal images can provide a more comprehensive basis for clinical disease diagnosis. With the popularization of the concept of precision medicine and the rapid growth of medical image data, the traditional diagnostic mode relying on manual image interpretation has limitations such as low efficiency, high subjectivity, and insufficient consistency of diagnostic results. Therefore, the development of high-precision, automated multimodal medical image intelligent analysis methods has significant clinical significance and application value.

[0003] Existing multimodal medical image analysis technologies have mainly gone through two development stages: traditional image processing and deep learning-assisted analysis. Early analysis schemes often employed a processing framework combining traditional rigid registration algorithms with manually designed feature operators. Image spatial alignment was achieved through affine transformations, followed by the extraction of manually designed features such as gray-level co-occurrence matrices and histograms of oriented gradients for lesion identification. These schemes relied on manually set rules and features, limiting their ability to represent complex lesions and low-contrast lesions. Furthermore, registration accuracy was easily affected by individual anatomical differences and image acquisition parameter deviations, and their generalization ability and robustness failed to meet clinical application requirements. In recent years, deep learning technology has been widely applied in the field of medical imaging. Some schemes use convolutional neural networks for lesion detection and segmentation, but most still employ a step-by-step processing architecture, with registration, feature extraction, and lesion analysis steps operating independently. Moreover, multimodal feature fusion often uses simple channel stitching, failing to fully exploit the complementary correlation information between different modalities, leaving significant room for improvement in overall analysis accuracy.

[0004] Chinese invention patent application CN112561893A discloses a multimodal medical image lesion segmentation method. This method first uses a rigid registration algorithm to achieve spatial alignment of multimodal images, and then extracts fusion features through a single-branch convolutional neural network to complete lesion segmentation. The rigid registration method used in this scheme can only achieve global spatial transformation and cannot adapt to the non-rigid deformation of human soft tissue, easily leading to spatial offset of lesion regions in different modalities, thus affecting subsequent segmentation accuracy. Simultaneously, this scheme uses direct channel stitching to achieve multimodal feature fusion, which cannot adaptively adjust the weights of each modality based on the feature effectiveness of the lesion region. Redundant features can interfere with lesion recognition, resulting in low feature fusion efficiency. Chinese invention patent application CN113222796A discloses a deep learning-based medical image diagnosis method. This scheme trains a registration network and a diagnosis network separately, completing the image alignment and lesion diagnosis steps in stages. In this scheme, the registration network and the diagnostic network are independent and optimized separately, which cannot achieve collaborative optimization at the feature level. The error generated in the registration process will be directly transmitted to the diagnostic process and have a cumulative effect. Moreover, this scheme can only output a single diagnostic classification result and cannot provide quantitative information such as lesion location and size at the same time, making it difficult to support the multi-dimensional diagnostic needs of clinical practice.

[0005] In summary, existing multimodal medical image analysis technologies generally suffer from several technical limitations: insufficient spatial registration accuracy, making it difficult to adapt to the non-rigid deformation of soft tissues, thus becoming a fundamental bottleneck restricting analytical accuracy; simple multimodal feature fusion methods, lacking adaptive feature selection and weighting mechanisms, resulting in low utilization of modal complementary information; fragmented processing stages, failing to achieve end-to-end joint optimization, and prone to error accumulation effects; and single-dimensional output results, making it difficult to simultaneously meet the comprehensive clinical needs for lesion localization, quantitative measurement, and pathological grading. These shortcomings make it difficult for existing technologies to simultaneously achieve analytical accuracy and processing efficiency, failing to fully realize the clinical value of multimodal medical images. Therefore, there is an urgent need to propose a novel intelligent multimodal medical image analysis scheme to address the numerous problems existing in current technologies. Summary of the Invention

[0006] The present invention aims to provide a deep learning-based intelligent analysis method for multimodal medical images to solve the problems of insufficient registration accuracy, low adaptability of feature fusion, and inability to jointly optimize the fragmented process of multimodal medical image analysis in the prior art.

[0007] To achieve the above objectives, the present invention provides the following method:

[0008] This invention provides a deep learning-based intelligent analysis method for multimodal medical images:

[0009] S1: Acquire multimodal raw medical images of the target area of ​​the subject, and perform grayscale normalization, spatial resampling and noise suppression processing on the multimodal raw medical images to obtain standardized medical images of each modality;

[0010] S2: Input the standardized medical images of each modality into a deep learning registration network, select one modality from each standardized medical image as a reference modality, and the remaining modalities as floating modalities. Using the standardized medical image corresponding to the reference modality as a reference image, output the spatial alignment images corresponding to all floating modalities, and combine the reference image with all spatial alignment images into a multimodal alignment image.

[0011] S3: Input the multimodal aligned image into a multi-branch deep feature extraction network. Through the feature extraction branches that correspond one-to-one with each modality, perform multi-scale deep feature extraction on the aligned image of the corresponding modality to obtain the multi-scale deep feature map corresponding to each modality.

[0012] S4: Input the multi-scale depth feature maps corresponding to each modality into the cross-modal attention fusion network, perform adaptive weighted fusion of the depth feature maps of different modalities at the same scale to obtain the single-scale fusion feature maps corresponding to each scale, and perform multi-scale dimension unification and channel splicing on all single-scale fusion feature maps to generate multi-modal fusion feature maps.

[0013] S5: Input the multimodal fusion feature map into the lesion analysis deep learning network, and output the location information, size parameters and pathological grading results of the target lesion through the feature mapping and multi-task computation of the lesion analysis deep learning network.

[0014] Furthermore, the grayscale normalization process in S1 specifically employs a minimum-maximum linear normalization method to linearly map the voxel grayscale values ​​of the original multimodal medical images to a numerical range of 0 to 1; the spatial resampling process specifically resamples all modal medical images to a uniform voxel spacing and a uniform image size; the noise suppression process specifically employs a three-dimensional Gaussian filter to smooth and denoise each modal image, with the filter kernel size set to 3×3×3.

[0015] Furthermore, the deep learning registration network in S2 adopts a three-dimensional U-shaped convolutional neural network structure that includes an encoding path, a decoding path, and a skip connection structure. The input of the deep learning registration network is the stitching feature obtained by stitching the reference image and a single floating image in the channel dimension, and the output is the three-dimensional deformation field of the corresponding floating image. Based on the three-dimensional deformation field, the spatial offset transformation of all pixel coordinates of the corresponding floating image is performed, and the transformed voxel gray values ​​are sampled using trilinear interpolation to generate a spatially aligned image that is completely aligned with the spatial dimension of the reference image.

[0016] Furthermore, the number of branches in the multi-branch deep feature extraction network in S3 is equal to the number of modalities. Each feature extraction branch contains four layers of feature extraction units. Each layer of feature extraction units contains two convolutional layers with a kernel size of 3×3×3, a stride of 1, and padding of 1, and a max pooling layer with a kernel size of 2×2×2 and a stride of 2. Each feature extraction branch outputs four depth feature maps with different resolutions, forming a multi-scale depth feature map of the corresponding modality.

[0017] Furthermore, the adaptive weighted fusion calculation formula for depth feature maps of different modalities at the same scale in S4 is as follows:

[0018] ;

[0019] in, For the first Single-scale fused feature map corresponding to the scale. For the first The mode in the th ... Feature weight map at different scales For the first The mode in the th ... Depth feature map at this scale Perform element-wise summation on the weighted feature map for all modes.

[0020] Furthermore, in S4, the feature weight maps of each modality at each scale are generated by the cross-modal attention fusion network through a joint calculation of channel attention and spatial attention mechanisms. The channel attention mechanism performs global average pooling and global max pooling on the input single-modal single-scale depth feature map, and inputs the two pooling results into a shared fully connected layer. The shared fully connected layer includes a dimensionality reduction layer and a dimensionality increase layer, with a dimensionality reduction ratio set to 16, and outputs a channel saliency weight vector consistent with the dimension of the feature channel number. The spatial attention mechanism performs average pooling and max pooling on the input single-modal single-scale depth feature map along the channel dimension, and concatenates the two pooling results along the channel dimension before inputting them into a convolutional layer with a kernel size of 7×7×7, a stride of 1, and padding of 3, and outputs a spatial saliency weight map consistent with the spatial dimension of the feature map. The channel saliency weight vector and the spatial saliency weight map are dimension-matched and multiplied element-wise to obtain a feature weight map consistent with the dimension of the corresponding modal depth feature map.

[0021] Furthermore, the specific method for performing multi-scale dimension unification and channel stitching on all single-scale fusion feature maps in S4 is as follows: all single-scale fusion feature maps are uniformly adjusted to the size of the highest resolution single-scale fusion feature map, wherein single-scale fusion feature maps with resolutions lower than the highest resolution are enlarged to the target size through trilinear interpolation upsampling operation, and then all fusion feature maps after size unification are stitched along the channel dimension to generate a multimodal fusion feature map.

[0022] Furthermore, the lesion analysis deep learning network in S5 sequentially includes a convolutional mapping layer, a global average pooling layer, a fully connected layer, and a multi-task output layer. The convolutional mapping layer is used to perform deep feature mapping on the multimodal fusion feature map. The global average pooling layer is used to perform global compression of the channel dimension of the mapped features. The fully connected layer is used for nonlinear transformation of the features. The multi-task output layer outputs lesion location information, size parameters, and pathological grading results, respectively. The lesion location information is the center coordinates of the lesion's three-dimensional bounding box, the size parameters are the length, width, and height dimensions of the lesion's three-dimensional bounding box, and the pathological grading result is the pathological grade classification probability vector corresponding to the lesion.

[0023] Furthermore, the method also includes a joint training step for deep learning networks, specifically: constructing a multimodal medical image training dataset, wherein a single sample in the training dataset is a paired multimodal original medical image of the target site of the same subject, and all samples are accompanied by annotation information on lesion location, size and pathological grade; performing grayscale normalization, spatial resampling and noise suppression processing on all training samples in the same manner as in step S1; using an adaptive moment estimation optimizer to jointly optimize and train all parameters of the deep learning registration network, the multi-branch deep feature extraction network, the cross-modal attention fusion network and the lesion analysis deep learning network in an end-to-end backpropagation manner, wherein the initial learning rate of the adaptive moment estimation optimizer is set to 0.001 and the weight decay coefficient is set to 0.0001, and one training cycle is to traverse the complete training dataset once, until the difference between the maximum and minimum values ​​of the network loss function on the validation set within ten consecutive training cycles is less than 0.001.

[0024] Furthermore, the formula for calculating the network loss function is as follows:

[0025] ;

[0027] in, This is the total loss value. The registration accuracy loss value of the deep learning registration network is composed of a weighted sum of image similarity loss and deformation field smoothing loss, with a weight of 1 for image similarity loss and a weight of 0.01 for deformation field smoothing loss. The image similarity loss uses mean square error loss to calculate the difference in voxel gray values ​​between the reference image and the deformed floating image. The deformation field smoothing loss uses first derivative L2 smoothing loss to calculate the sum of squared spatial gradients of the three-dimensional deformation field. For the regression task of lesion location information and size parameters, the smoothed L1 loss value is used; The cross-entropy loss value is used for pathological grading and classification tasks. The balance coefficient for the regression loss is set to 1; This is the balance coefficient for the classification loss, with a value of 0.5.

[0028] The beneficial effects of this invention are reflected in the following aspects: This invention constructs an end-to-end multimodal medical image intelligent analysis architecture based on deep learning, which has significant technical advantages over existing technologies in many aspects. This scheme employs a three-dimensional U-shaped deep learning registration network containing encoding paths, decoding paths, and skip connection structures to achieve unsupervised, non-rigid registration of multimodal images. It can adaptively adapt to differences in soft tissue deformation in the human body, significantly improving spatial alignment accuracy compared to traditional rigid registration, providing a precise spatial foundation for subsequent analysis, and avoiding registration errors interfering with analysis results. This scheme uses a multi-branch, multi-scale deep feature extraction network corresponding one-to-one with each modality, adapting to the differences in feature distribution across different modalities. It can simultaneously cover fine-grained texture and global anatomical structure information, fully mining the unique information of each modality. Combined with a channel and spatial dual-dimensional cross-modal attention fusion mechanism, it can adaptively adjust modal weights according to the semantic saliency of features, strengthening effective lesion features and suppressing redundant noise. Compared to simple channel stitching, it significantly improves the efficiency of multimodal complementary information utilization and feature discriminability. This solution employs an end-to-end joint training mechanism, integrating registration, feature extraction, attention fusion, and lesion analysis networks into a unified optimization framework. Through backpropagation, all network parameters are adjusted synchronously, achieving collaborative optimization across modules. This effectively avoids the error accumulation problem inherent in step-by-step architectures, thus improving overall lesion analysis accuracy. The solution utilizes a multi-task output architecture, simultaneously outputting lesion 3D location, size parameters, and pathological grading results. It provides a one-stop solution with multi-dimensional quantitative information required for clinical diagnosis, eliminating the need for multiple analysis tasks performed step-by-step, improving processing efficiency while providing comprehensive support for clinical diagnosis. Furthermore, the overall architecture of the solution exhibits good generalization and scalability, adaptable to various imaging modalities and lesion analysis scenarios at different locations, demonstrating high clinical application value. Attached Figure Description

[0029] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn to scale.

[0030] Figure 1 This is a schematic diagram of a deep learning-based multimodal medical image intelligent analysis method provided in an embodiment of the present invention. Detailed Implementation

[0031] To enable those skilled in the art to better understand the present invention, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, product, or end that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or ends.

[0033] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0034] Existing multimodal medical image analysis technologies have mainly gone through two development stages: traditional image processing and deep learning-assisted analysis. Early analysis schemes often employed a processing framework combining traditional rigid registration algorithms with manually designed feature operators. Image spatial alignment was achieved through affine transformations, followed by the extraction of manually designed features such as gray-level co-occurrence matrices and histograms of oriented gradients for lesion identification. These schemes relied on manually set rules and features, limiting their ability to represent complex lesions and low-contrast lesions. Furthermore, registration accuracy was easily affected by individual anatomical differences and image acquisition parameter deviations, and their generalization ability and robustness failed to meet clinical application requirements. In recent years, deep learning technology has been widely applied in the field of medical imaging. Some schemes use convolutional neural networks for lesion detection and segmentation, but most still employ a step-by-step processing architecture, with registration, feature extraction, and lesion analysis steps operating independently. Moreover, multimodal feature fusion often uses simple channel stitching, failing to fully exploit the complementary correlation information between different modalities, leaving significant room for improvement in overall analysis accuracy.

[0035] Chinese invention patent application CN112561893A discloses a multimodal medical image lesion segmentation method. This method first uses a rigid registration algorithm to achieve spatial alignment of multimodal images, and then extracts fusion features through a single-branch convolutional neural network to complete lesion segmentation. The rigid registration method used in this scheme can only achieve global spatial transformation and cannot adapt to the non-rigid deformation of human soft tissue, easily leading to spatial offset of lesion regions in different modalities, thus affecting subsequent segmentation accuracy. Simultaneously, this scheme uses direct channel stitching to achieve multimodal feature fusion, which cannot adaptively adjust the weights of each modality based on the feature effectiveness of the lesion region. Redundant features can interfere with lesion recognition, resulting in low feature fusion efficiency. Chinese invention patent application CN113222796A discloses a deep learning-based medical image diagnosis method. This scheme trains a registration network and a diagnosis network separately, completing the image alignment and lesion diagnosis steps in stages. In this scheme, the registration network and the diagnostic network are independent and optimized separately, which cannot achieve collaborative optimization at the feature level. The error generated in the registration process will be directly transmitted to the diagnostic process and have a cumulative effect. Moreover, this scheme can only output a single diagnostic classification result and cannot provide quantitative information such as lesion location and size at the same time, making it difficult to support the multi-dimensional diagnostic needs of clinical practice.

[0036] In summary, existing multimodal medical image analysis technologies generally suffer from several technical limitations: insufficient spatial registration accuracy, making it difficult to adapt to the non-rigid deformation of soft tissues, thus becoming a fundamental bottleneck restricting analytical accuracy; simple multimodal feature fusion methods, lacking adaptive feature selection and weighting mechanisms, resulting in low utilization of modal complementary information; fragmented processing steps, failing to achieve end-to-end joint optimization, and prone to error accumulation effects; and single-dimensional output results, making it difficult to simultaneously meet the comprehensive clinical needs for lesion localization, quantitative measurement, and pathological grading. These shortcomings make it difficult for existing technologies to simultaneously achieve analytical accuracy and processing efficiency, failing to fully realize the clinical value of multimodal medical images. Therefore, there is an urgent need to propose a novel intelligent multimodal medical image analysis scheme to address the numerous problems existing in current technologies.

[0037] The present invention aims to provide a deep learning-based intelligent analysis method for multimodal medical images to solve the problems of insufficient registration accuracy, low adaptability of feature fusion, and inability to jointly optimize the fragmented process of multimodal medical image analysis in the prior art.

[0038] like Figure 1 As shown in the figure, a specific embodiment of the present invention provides a multimodal medical image intelligent analysis method based on deep learning, the specific steps of which are as follows:

[0039] S1: Acquire multimodal raw medical images of the target area of ​​the subject, and perform grayscale normalization, spatial resampling and noise suppression processing on the multimodal raw medical images to obtain standardized medical images of each modality.

[0040] In this embodiment of the invention, the multimodal raw medical images specifically include three modalities: computed tomography (CT) images, magnetic resonance imaging (MRI) T1-weighted images, and MRI T2-weighted images. All images are in three-dimensional volumetric data format. Gray-level normalization processing adopts a minimum-maximum linear normalization method, and the calculation formula is as follows:

[0041] ;

[0043] in, The normalized voxel gray values ​​are... This represents the grayscale value of the original voxel. This represents the minimum grayscale value of the corresponding modal image. To obtain the maximum grayscale value of the corresponding modal image, all voxel grayscale values ​​are linearly mapped to [value]. The numerical range is defined. Spatial resampling processing resamples all modal images to a uniform 1mm×1mm×1mm voxel spacing and a uniform 128×128×64 image size to eliminate spatial dimensional differences between different modal images. Noise suppression processing uses a three-dimensional Gaussian filter to smooth and denoise each modal image. The filter kernel size is set to 3×3×3, and the standard deviation of the Gaussian function is set to 1 to remove random noise and artifact interference generated during image acquisition.

[0044] S2: Input the standardized medical images of each modality into the deep learning registration network, select one modality from each standardized medical image as the reference modality, and the remaining modalities as floating modalities. Using the standardized medical image corresponding to the reference modality as the reference image, output the spatially aligned images corresponding to all floating modalities, and combine the reference image with all spatially aligned images into a multimodal aligned image.

[0045] In this embodiment of the invention, a standardized medical image corresponding to a T1-weighted image from magnetic resonance imaging is selected as the reference image, and the other two modal images are sequentially input into a deep learning registration network as floating images. The deep learning registration network adopts a three-dimensional U-shaped convolutional neural network structure containing an encoding path, a decoding path, and skip connections. The encoding path contains four downsampling levels, each of which sequentially sets two convolutional layers with a kernel size of 3×3×3, a stride of 1, and padding of 1, and a max pooling layer with a kernel size of 2×2×2 and a stride of 2. The decoding path contains four upsampling levels, each of which sequentially sets a trilinear interpolation upsampling layer and two convolutional layers with a kernel size of 3×3×3, a stride of 1, and padding of 1. Skip connections are set between corresponding levels of the encoding and decoding paths to transmit fine-grained spatial features. The deep learning registration network takes as input a two-channel stitched feature obtained by concatenating a reference image and a single floating image along the channel dimension. The output is a three-dimensional deformation field of the corresponding floating image, with its spatial dimension perfectly aligned to the floating image. Each voxel location stores the x, y, and z-axis offsets of its corresponding coordinates. Based on the three-dimensional deformation field, a spatial offset transformation is performed on all voxel coordinates of the corresponding floating image. The transformed voxel grayscale values ​​are then sampled using trilinear interpolation, generating a spatially aligned image that is perfectly aligned with the reference image's spatial dimension. Finally, the reference image and the two spatially aligned images are combined along the channel dimension to form a three-channel multimodal aligned image.

[0046] S3: Input the multimodal aligned image into the multi-branch deep feature extraction network. Through the feature extraction branches that correspond one-to-one with each modality, multi-scale deep feature extraction is performed on the aligned image of the corresponding modality to obtain the multi-scale deep feature map corresponding to each modality.

[0047] In this embodiment of the invention, the number of branches in the multi-branch deep feature extraction network is equal to the number of modalities. Three independent feature extraction branches are set up, each corresponding to an aligned image of one of the three modalities. The network structure of each branch is consistent, and parameters are not shared, to adapt to the image feature distribution of different modalities. Each feature extraction branch contains four levels of feature extraction units. Each level of feature extraction unit sequentially includes two convolutional layers with a kernel size of 3×3×3, a stride of 1, and padding of 1, and a max-pooling layer with a kernel size of 2×2×2 and a stride of 2. The feature maps output from the four levels correspond to 1x, 1 / 2x, 1 / 4x, and 1 / 8x the resolution of the original input image, with sizes of 128×128×64, 64×64×32, 32×32×16, and 16×16×8, respectively, corresponding to 16, 32, 64, and 128 channels. Each feature extraction branch outputs four depth feature maps at different resolutions, forming a multi-scale depth feature map for the corresponding modality. The feature maps at different levels correspond to image features with different receptive fields, which can cover multi-dimensional feature information from fine-grained texture details to global anatomical structures.

[0048] S4: Input the multi-scale depth feature maps corresponding to each modality into the cross-modal attention fusion network, perform adaptive weighted fusion of depth feature maps of different modalities at the same scale to obtain single-scale fusion feature maps corresponding to each scale, and perform multi-scale dimension unification and channel concatenation on all single-scale fusion feature maps to generate multi-modal fusion feature maps.

[0049] In this embodiment of the invention, the cross-modal attention fusion network sets up four independent attention calculation units for feature maps at four scales, with each unit corresponding to the same-scale depth feature maps of the three modalities. The feature weight maps for each modality at each scale are generated by the cross-modal attention fusion network through a joint calculation of channel attention and spatial attention mechanisms. The channel attention mechanism performs global average pooling and global max pooling on the input single-modal single-scale depth feature map, obtaining two feature vectors with a dimension of channel number × 1 × 1 × 1. The two pooling results are input into a shared fully connected layer, which includes one dimensionality reduction layer and one dimensionality increase layer, with a dimensionality reduction ratio set to 16. The output is a channel saliency weight vector consistent with the feature channel number dimension. The spatial attention mechanism performs average pooling and max pooling on the input single-modal single-scale depth feature map along the channel dimension, obtaining two spatial feature maps with one channel. The two pooling results are concatenated along the channel dimension and input into a convolutional layer with a kernel size of 7 × 7 × 7, a stride of 1, and padding of 3, outputting a spatial saliency weight map consistent with the spatial dimension of the feature map. After dimensionality matching between the channel saliency weight vector and the spatial saliency weight map, they are multiplied element-wise to obtain a feature weight map with the same dimension as the corresponding modality depth feature map. The adaptive weighted fusion calculation formula for depth feature maps of different modalities at the same scale is as follows:

[0050] ;

[0051] in, For the first Single-scale fused feature map corresponding to the scale. For the first The mode in the th ... Feature weight map at different scales For the first The mode in the th ... Depth feature map at this scale To perform element-wise summation on the weighted feature maps of all modalities, the specific method for unifying the multi-scale dimensions and stitching the channels of all single-scale fusion feature maps is as follows: all single-scale fusion feature maps are uniformly adjusted to the size of the highest resolution single-scale fusion feature map. Single-scale fusion feature maps with resolutions lower than the highest resolution are enlarged to the target size through trilinear interpolation upsampling. Then, all the fusion feature maps with unified dimensions are stitched together along the channel dimension to generate a multi-modal fusion feature map. In this embodiment, the total number of channels after stitching the fusion feature maps of the four scales is 240, and the spatial size is 128×128×64.

[0052] S5: Input the multimodal fusion feature map into the lesion analysis deep learning network. Through feature mapping and multi-task computation of the lesion analysis deep learning network, output the location information, size parameters and pathological grading results of the target lesion.

[0053] In this embodiment of the invention, the deep learning network for lesion analysis sequentially comprises a convolutional mapping layer, a global average pooling layer, a fully connected layer, and a multi-task output layer. The convolutional mapping layer contains two convolutional layers with a kernel size of 3×3×3, a stride of 1, and padding of 1, used for deep feature mapping of the multimodal fusion feature map, further integrating the semantic information of the multi-scale multimodal fusion features. The global average pooling layer is used to globally compress the channel dimension of the mapped features, compressing the three-dimensional feature map of each channel into a single value, and outputting a one-dimensional feature vector with a dimension of channel number × 1. The fully connected layer contains two fully connected units, each followed by a ReLU activation function to achieve non-linear transformation of the features and improve the network's feature fitting ability. The multi-task output layer is divided into three parallel output branches, which output lesion location information, size parameters, and pathological grading results, respectively. The lesion location information is the center coordinates of the lesion's 3D bounding box, output in the form of three-axis coordinate values; the size parameters are the length, width, and height dimensions of the lesion's 3D bounding box, output in the form of length values; the pathological grading result is the pathological grade classification probability vector corresponding to the lesion, with the vector dimension consistent with the number of pathological grading categories, and the value of each dimension corresponding to the predicted probability of the corresponding grade.

[0054] S6: Construct a multimodal medical image training dataset, and use an end-to-end backpropagation method to jointly optimize and train all deep learning networks to obtain a trained analysis model.

[0055] In this embodiment of the invention, a single sample in the training dataset is a paired multimodal original medical image of the target area of ​​the same subject. All samples are accompanied by annotation information on lesion location, size, and pathological grade. Gray-level normalization, spatial resampling, and noise suppression processing are performed on all training samples in the same manner as in step S1. The training dataset is divided into training, validation, and test sets in an 8:1:1 ratio. An adaptive moment estimation optimizer is used to jointly optimize and train all parameters of the deep learning registration network, multi-branch deep feature extraction network, cross-modal attention fusion network, and lesion analysis deep learning network using end-to-end backpropagation. The initial learning rate of the adaptive moment estimation optimizer is set to 0.001, the weight decay coefficient is set to 0.0001, and the training batch size is set to 4. One training cycle is one complete traversal of the training dataset. During training, the total loss value is calculated on the validation set after each training cycle until the difference between the maximum and minimum values ​​of the network loss function on the validation set within ten consecutive training cycles is less than 0.001. At this point, the network is considered to have converged, and training is terminated. The formula for calculating the network loss function is:

[0056] ;

[0058] in, This is the total loss value. The registration accuracy loss value for the deep learning registration network is composed of a weighted sum of image similarity loss and deformation field smoothing loss. The weight of image similarity loss is 1, and the weight of deformation field smoothing loss is 0.01. Image similarity loss adopts mean square error loss, which calculates the difference in voxel gray values ​​between the reference image and the deformed floating image. Deformation field smoothing loss adopts first derivative L2 smoothing loss, which calculates the sum of squared spatial gradients of the three-dimensional deformation field and constrains the spatial smoothness of the deformation field to avoid unreasonable local deformation. For the regression task of lesion location information and size parameters, the smoothed L1 loss value is used; The cross-entropy loss value is used for pathological grading and classification tasks. The balance coefficient for the regression loss is set to 1; This is the balance coefficient for the classification loss, with a value of 0.5.

[0059] The beneficial effects of this invention are reflected in the following aspects: This invention constructs an end-to-end multimodal medical image intelligent analysis architecture based on deep learning, which has significant technical advantages over existing technologies in many aspects. This scheme employs a three-dimensional U-shaped deep learning registration network containing encoding paths, decoding paths, and skip connection structures to achieve unsupervised, non-rigid registration of multimodal images. It can adaptively adapt to differences in soft tissue deformation in the human body, significantly improving spatial alignment accuracy compared to traditional rigid registration, providing a precise spatial foundation for subsequent analysis, and avoiding registration errors interfering with analysis results. This scheme uses a multi-branch, multi-scale deep feature extraction network corresponding one-to-one with each modality, adapting to the differences in feature distribution across different modalities. It can simultaneously cover fine-grained texture and global anatomical structure information, fully mining the unique information of each modality. Combined with a channel and spatial dual-dimensional cross-modal attention fusion mechanism, it can adaptively adjust modal weights according to the semantic saliency of features, strengthening effective lesion features and suppressing redundant noise. Compared to simple channel stitching, it significantly improves the efficiency of multimodal complementary information utilization and feature discriminability. This solution employs an end-to-end joint training mechanism, integrating registration, feature extraction, attention fusion, and lesion analysis networks into a unified optimization framework. Through backpropagation, all network parameters are adjusted synchronously, achieving collaborative optimization across modules. This effectively avoids the error accumulation problem inherent in step-by-step architectures, thus improving overall lesion analysis accuracy. The solution utilizes a multi-task output architecture, simultaneously outputting lesion 3D location, size parameters, and pathological grading results. It provides a one-stop solution with multi-dimensional quantitative information required for clinical diagnosis, eliminating the need for multiple analysis tasks performed step-by-step, improving processing efficiency while providing comprehensive support for clinical diagnosis. Furthermore, the overall architecture of the solution exhibits good generalization and scalability, adaptable to various imaging modalities and lesion analysis scenarios at different locations, demonstrating high clinical application value.

[0060] The above descriptions are merely embodiments of the present invention. Commonly known technical solutions or characteristics are not described in detail here. It should be noted that those skilled in the art can make various modifications and improvements without departing from the scope of the present invention, and these should also be considered within the protection scope of the present invention. These modifications and improvements will not affect the effectiveness of the present invention or the practicality of the patent. The scope of protection claimed in this application should be determined by the content of its claims, and the specific embodiments described in the specification can be used to interpret the content of the claims.

Claims

1. A multimodal medical image intelligent analysis method based on deep learning, characterized in that, The method includes: S1: Acquire multimodal raw medical images of the target area of ​​the subject, and perform grayscale normalization, spatial resampling and noise suppression processing on the multimodal raw medical images to obtain standardized medical images of each modality; S2: Input the standardized medical images of each modality into a deep learning registration network, select one modality from each standardized medical image as a reference modality, and the remaining modalities as floating modalities. Using the standardized medical image corresponding to the reference modality as a reference image, output the spatial alignment images corresponding to all floating modalities, and combine the reference image with all spatial alignment images into a multimodal alignment image. S3: Input the multimodal aligned image into a multi-branch deep feature extraction network. Through the feature extraction branches that correspond one-to-one with each modality, perform multi-scale deep feature extraction on the aligned image of the corresponding modality to obtain the multi-scale deep feature map corresponding to each modality. S4: Input the multi-scale depth feature maps corresponding to each modality into the cross-modal attention fusion network, perform adaptive weighted fusion of the depth feature maps of different modalities at the same scale to obtain the single-scale fusion feature maps corresponding to each scale, and perform multi-scale dimension unification and channel splicing on all single-scale fusion feature maps to generate multi-modal fusion feature maps. S5: Input the multimodal fusion feature map into the lesion analysis deep learning network, and output the location information, size parameters and pathological grading results of the target lesion through the feature mapping and multi-task computation of the lesion analysis deep learning network.

2. The method for intelligent analysis of multimodal medical images based on deep learning according to claim 1, characterized in that: The gray-level normalization process in S1 specifically employs a minimum-maximum linear normalization method to linearly map the voxel gray values ​​of the original multimodal medical images to a numerical range of 0 to 1; the spatial resampling process specifically resamples all modal medical images to a uniform voxel spacing and a uniform image size; the noise suppression process specifically employs a three-dimensional Gaussian filter to smooth and denoise each modal image, with the filter kernel size set to 3×3×3.

3. The method for intelligent analysis of multimodal medical images based on deep learning according to claim 1, characterized in that: The deep learning registration network in S2 adopts a three-dimensional U-shaped convolutional neural network structure that includes an encoding path, a decoding path, and a skip connection structure. The input of the deep learning registration network is the stitching feature obtained by stitching the reference image and a single floating image in the channel dimension, and the output is the three-dimensional deformation field of the corresponding floating image. Based on the three-dimensional deformation field, the spatial offset transformation of all pixel coordinates of the corresponding floating image is performed, and the transformed voxel gray values ​​are sampled using trilinear interpolation to generate a spatially aligned image that is completely aligned with the spatial dimension of the reference image.

4. The method for intelligent analysis of multimodal medical images based on deep learning according to claim 1, characterized in that: The number of branches in the multi-branch deep feature extraction network in S3 is equal to the number of modalities. Each feature extraction branch contains four levels of feature extraction units. Each level of feature extraction unit contains two convolutional layers with a kernel size of 3×3×3, a stride of 1, and a padding of 1, and a max pooling layer with a kernel size of 2×2×2 and a stride of 2. Each feature extraction branch outputs four depth feature maps at different resolutions, forming a multi-scale depth feature map for the corresponding modality.

5. The method for intelligent analysis of multimodal medical images based on deep learning according to claim 1, characterized in that: The adaptive weighted fusion calculation formula for depth feature maps of different modalities at the same scale in S4 is as follows: ； in, For the first Single-scale fused feature map corresponding to the scale. For the first The mode in the th ... Feature weight map at different scales For the first The mode in the th ... Depth feature map at this scale Perform element-wise summation on the weighted feature map for all modes.

6. The method for intelligent analysis of multimodal medical images based on deep learning according to claim 1, characterized in that: In S4, the feature weight maps of each modality at each scale are generated by the cross-modal attention fusion network through a joint calculation of channel attention mechanism and spatial attention mechanism. The channel attention mechanism performs global average pooling and global max pooling on the input single-modal single-scale depth feature map, respectively, and inputs the two pooling results into a shared fully connected layer. The shared fully connected layer includes a dimensionality reduction layer and a dimensionality increase layer. The dimensionality reduction ratio is set to 16, and the output is a channel saliency weight vector consistent with the dimension of the feature channel number. The spatial attention mechanism performs average pooling and max pooling on the input single-modality single-scale deep feature map along the channel dimension. The two pooling results are concatenated along the channel dimension and then input into a convolutional layer with a kernel size of 7×7×7, a stride of 1, and padding of 3. The output is a spatial saliency weight map with the same spatial dimension as the feature map. After matching the dimensions of the channel saliency weight vector and the spatial saliency weight map, they are multiplied element by element to obtain a feature weight map with the same dimensions as the corresponding modality depth feature map.

7. The method for intelligent analysis of multimodal medical images based on deep learning according to claim 1, characterized in that: The specific method for unifying and stitching channels of all single-scale fusion feature maps in S4 is as follows: all single-scale fusion feature maps are uniformly adjusted to the size of the highest resolution single-scale fusion feature map. Among them, single-scale fusion feature maps with resolutions lower than the highest resolution are enlarged to the target size through trilinear interpolation upsampling operation. Then, all fusion feature maps after size unification are stitched along the channel dimension to generate a multimodal fusion feature map.

8. The method for intelligent analysis of multimodal medical images based on deep learning according to claim 1, characterized in that: The lesion analysis deep learning network in S5 sequentially includes a convolutional mapping layer, a global average pooling layer, a fully connected layer, and a multi-task output layer. The convolutional mapping layer is used to perform deep feature mapping on the multimodal fusion feature map. The global average pooling layer is used to perform global compression of the channel dimension of the mapped features. The fully connected layer is used for nonlinear transformation of the features. The multi-task output layer outputs lesion location information, size parameters, and pathological grading results, respectively. The lesion location information is the center coordinates of the lesion's three-dimensional bounding box, the size parameters are the length, width, and height dimensions of the lesion's three-dimensional bounding box, and the pathological grading result is the pathological grade classification probability vector corresponding to the lesion.

9. The method for intelligent analysis of multimodal medical images based on deep learning according to claim 1, characterized in that: The method also includes a joint training step for deep learning networks, specifically: constructing a multimodal medical image training dataset, wherein a single sample in the training dataset is a paired multimodal original medical image of the target site of the same subject, and all samples are accompanied by annotation information on lesion location, size and pathological grade, and performing grayscale normalization, spatial resampling and noise suppression processing on all training samples in the same manner as in step S1; An adaptive moment estimation optimizer is used to jointly optimize and train all parameters of the deep learning registration network, the multi-branch deep feature extraction network, the cross-modal attention fusion network, and the lesion analysis deep learning network using an end-to-end backpropagation method. The initial learning rate of the adaptive moment estimation optimizer is set to 0.001, and the weight decay coefficient is set to 0.0001. One training cycle is to traverse the complete training dataset once, until the difference between the maximum and minimum values ​​of the network loss function on the validation set is less than 0.001 within ten consecutive training cycles.

10. The method for intelligent analysis of multimodal medical images based on deep learning according to claim 9, characterized in that: The formula for calculating the network loss function is as follows: ； in, This is the total loss value. The registration accuracy loss value of the deep learning registration network is composed of a weighted sum of image similarity loss and deformation field smoothing loss, with the image similarity loss having a weight of 1 and the deformation field smoothing loss having a weight of 0.

01. The image similarity loss uses mean square error loss to calculate the difference in voxel gray values ​​between the reference image and the deformed floating image; the deformation field smoothing loss uses first derivative L2 smoothing loss to calculate the sum of squared spatial gradients of the three-dimensional deformation field. For the regression task of lesion location information and size parameters, the smoothed L1 loss value is used; The cross-entropy loss value is used for pathological grading and classification tasks. The balance coefficient for the regression loss is set to 1; This is the balance coefficient for the classification loss, with a value of 0.5.

Images