Intelligent lesion detection method based on multi-modal medical image fusion

Abstract

This invention relates to the field of medical image detection technology, specifically to an intelligent lesion detection method based on multimodal medical image fusion. The method includes: acquiring multimodal medical image data of a target anatomical region; generating standardized image data blocks through cross-modal spatial registration and grayscale normalization preprocessing; performing multi-channel feature fusion enhancement on the image data, integrating texture, morphological, and functional metabolic features; performing three-dimensional semantic segmentation using a deep learning detection network to extract suspected lesion regions and obtain initial detection results; calculating multi-dimensional quantitative descriptors for each suspected lesion in the original image; and using a lesion classification and discrimination system to determine benignity / malignancy and pathological type. This method fully leverages the effective information from multimodal images, enriches the criteria for lesion discrimination, improves the precision of lesion identification, and accurately outputs results related to lesion localization, malignancy probability scoring, and pathological type.

Description

Technical Field

[0001] This invention relates to the field of medical image detection technology, and in particular to an intelligent lesion detection method based on multimodal medical image fusion. Background Technology

[0002] In clinical lesion screening and diagnosis scenarios, preliminary lesion identification and assessment are generally based on medical images. Conventional intelligent lesion detection methods often rely on single-type medical images for computational analysis. Traditional cross-modal image processing only performs basic spatial alignment and simple grayscale unification operations, and the feature mining process only extracts and calculates parameters for a single fixed type of image, failing to achieve collaborative utilization of the unique information of various modalities. Traditional deep learning detection architectures rely solely on fusing shallow image features to complete lesion segmentation and region selection, resulting in a relatively simple feature fusion mode that cannot take into account the diverse image attribute information presented during human tissue imaging.

[0003] A singular feature extraction and fusion approach can lead to the loss of differentiated lesion association information contained in multimodal images, resulting in region segmentation bias during 3D image semantic segmentation and inaccurate screening of suspected lesions. Relying solely on network-extracted fusion features to directly determine lesion attributes, detached from the original multimodal image data foundation, results in a limited dimension of lesion region information representation, making it difficult to finely distinguish pathological differences, blurring the boundaries between benign and malignant lesions, insufficient identification of pathological types, and a relatively simplistic lesion detection output. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies by proposing an intelligent lesion detection method based on multimodal medical image fusion.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: an intelligent lesion detection method based on multimodal medical image fusion, comprising:

[0006] The multimodal raw medical image datasets of the target anatomical region are acquired, merged, and preprocessed with cross-modal spatial registration and grayscale normalization to generate multimodal image data blocks with unified spatial coordinates and grayscale standards.

[0007] Based on the multimodal image data block, multi-channel feature fusion and enhancement calculations are performed to generate a fused feature image data block. The fused feature image data block includes texture features, morphological features, and functional metabolic features extracted and fused from different modal images.

[0008] The fused feature image data block is input into a preset deep learning detection network, which performs three-dimensional semantic segmentation and candidate region extraction on the fused feature image data block to generate an initial detection result containing several suspected lesion regions.

[0009] For each suspected lesion region in the initial detection results, calculate the multi-dimensional quantitative descriptor of the corresponding image of the suspected lesion region in the multimodal raw medical image data set;

[0010] The multi-dimensional quantitative descriptor is input into a pre-trained lesion classification and identification device, which then performs benign or malignant discrimination and pathological type identification on the suspected lesion area, generating a final identification result for the lesion area. The final identification result includes lesion location information, malignancy probability score, and suggested pathological type.

[0011] As a further aspect of the present invention, the acquisition of multimodal raw medical image datasets of the target anatomical region is combined and subjected to cross-modal spatial registration and grayscale normalization preprocessing operations to generate multimodal image data blocks with unified spatial coordinates and grayscale standards, including:

[0012] The multimodal raw medical image data set includes at least computed tomography (CT) image data, magnetic resonance imaging (MRI) image data, and positron emission tomography (PET) image data.

[0013] Select an image of one modality from the multimodal raw medical image dataset as a reference image for spatial registration;

[0014] Spatial feature points of key anatomical structures in the reference image are extracted to form a reference feature point set;

[0015] For each image to be registered in the multimodal raw medical image dataset, excluding the reference image, the following steps are performed:

[0016] Extract spatial feature points from the image to be registered that correspond to the key anatomical structure to form a set of feature points to be registered.

[0017] Based on the reference feature point set and the feature point set to be registered, spatial transformation parameters are calculated, including rotation matrix, translation vector and scaling factor;

[0018] The spatial transformation parameters are used to perform spatial transformation interpolation calculations on the image to be registered, so that the image to be registered is spatially aligned with the reference image, thereby generating a spatially aligned image.

[0019] For all spatially aligned images, including reference images and spatially aligned images, voxel-level grayscale normalization is performed to map grayscale values ​​from different imaging devices and different imaging protocols to a unified standardized grayscale range, generating multimodal image data blocks.

[0020] As a further aspect of the present invention, based on the multimodal image data block, multi-channel feature fusion and enhancement calculations are performed to generate a fused feature image data block, including:

[0021] For each modality of the multimodal image data block, a set of predefined feature extraction filters are used to perform convolution operations to extract the corresponding modality-specific feature map.

[0022] The modality-specific feature maps from different modalities are spliced ​​together according to the channel dimension based on the corresponding voxel positions to generate a multi-channel spliced ​​feature map.

[0023] Channel attention weighting calculation is performed on the multi-channel stitched feature map, and the channel attention weighting calculation dynamically allocates fusion weights according to the importance of each channel feature map to the lesion detection task;

[0024] The weighted multi-channel stitched feature map is input into the feature enhancement network. The feature enhancement network enhances the edge, texture and density difference features related to the lesion through asymmetric convolution and residual connection operations, while suppressing irrelevant tissue background features.

[0025] The feature map processed by the feature enhancement network is output as a fused feature image data block.

[0026] As a further aspect of the present invention, the fused feature image data block is input into a preset deep learning detection network, and the deep learning detection network performs three-dimensional semantic segmentation and candidate region extraction on the fused feature image data block to generate an initial detection result containing several suspected lesion regions, including:

[0027] The deep learning detection network adopts an encoder-decoder structure;

[0028] The encoder performs multi-level, downsampling feature extraction on the input fused feature image data block to generate deep semantic feature maps at different scales.

[0029] The decoder progressively upsamples the deepest semantic feature map and fuses feature maps from the corresponding level of the encoder during the upsampling process to restore the size of the feature map and supplement its details.

[0030] At the output of the decoder, each voxel is classified and predicted to generate a three-dimensional probability map, which represents the probability that each voxel belongs to the lesion region.

[0031] An adaptive threshold segmentation algorithm is applied to the three-dimensional probability map to extract connected regions with probability values ​​higher than a preset threshold, and each connected region is marked as a suspected lesion region.

[0032] The three-dimensional bounding box coordinates and voxel set of each suspected lesion region are calculated to form the initial detection results.

[0033] As a further aspect of the present invention, calculating a multi-dimensional quantitative descriptor of the suspected lesion region corresponding to the image in the multimodal raw medical image data set includes:

[0034] Based on the three-dimensional bounding box coordinates of the suspected lesion area, corresponding image blocks are extracted from the computed tomography (CT) image data, magnetic resonance imaging (MRI) image data, and positron emission tomography (PET) image data, respectively.

[0035] For computed tomography image data blocks, calculate the gray-level histogram statistics, gray-level co-occurrence matrix texture features, and shape irregularity of the voxels within them;

[0036] For magnetic resonance imaging image data blocks, based on images of different sequences, the signal intensity distribution of T1-weighted images, the signal intensity distribution of T2-weighted images, and the apparent diffusion coefficient distribution characteristics of diffusion-weighted images are calculated.

[0037] For positron emission tomography (PET) image data blocks, calculate the maximum, average, peak, and lesion metabolic volume of the standard uptake values;

[0038] All calculated feature values, including gray-level histogram statistics, gray-level co-occurrence matrix texture features, shape irregularity, T1 signal intensity distribution, T2 signal intensity distribution, apparent diffusion coefficient distribution features, maximum, average, peak values ​​of standard uptake values, and lesion metabolic volume, are concatenated in a predetermined order into a high-dimensional feature vector, which serves as the multi-dimensional quantization descriptor.

[0039] As a further aspect of the present invention, the multi-dimensional quantitative descriptor is input into a pre-trained lesion classification and identification device, which then performs benign / malignant differentiation and pathological type identification on the suspected lesion region, generating a final identification result for the lesion region, including:

[0040] The pre-trained lesion classification and identification device contains multiple fully connected layers and a classification output layer;

[0041] The high-dimensional feature vector is input into the lesion classification and discriminator, and after nonlinear transformation and feature dimensionality reduction in the fully connected layer, a deep abstract feature representation of the lesion is obtained.

[0042] Based on the deep abstract feature representation, the classification output layer of the lesion classification discriminator calculates two outputs in parallel: the first output is the benign or malignant discrimination probability, which is the probability value of the suspected lesion area being judged as a malignant tumor;

[0043] The second output is a pathological type probability distribution vector, which contains the probability that the suspected lesion area belongs to each predefined pathological type;

[0044] When the probability of benign or malignant differentiation exceeds the preset malignant determination threshold, the pathological type with the highest probability is selected from the pathological type probability distribution vector as the suggested pathological type.

[0045] The three-dimensional bounding box coordinates, benign / malignant discrimination probability, and suggested pathological type of the suspected lesion area are integrated to generate the final identification result of the lesion area.

[0046] As a further aspect of the present invention, the method further includes:

[0047] Based on the lesion location information and the multimodal image data block, a three-dimensional visualization model of the lesion is constructed. This three-dimensional visualization model fuses and presents the features of the lesion region in different modal images, specifically including:

[0048] Based on the lesion location information in the final identification result, the spatial range of the lesion region in the multimodal image data block is determined;

[0049] Within the aforementioned spatial range, volume rendering technology is used on computed tomography image data to display the density structure of lesions and their relationship with surrounding anatomy;

[0050] Under the same spatial coordinate system, the magnetic resonance imaging data is rendered using surface rendering technology to highlight the soft tissue boundaries and internal features of the lesions;

[0051] The metabolic information of the lesion area in the positron emission tomography (PET) image data is fused and displayed on the anatomical background composed of computed tomography (CT) images and magnetic resonance imaging (MRI) images in a pseudo-color transparent overlay.

[0052] The interactive interface allows users to adjust the display transparency, color mapping, and cutting planes of different modal images, and observe the 3D visualization model of the lesion from any angle.

[0053] As a further aspect of the present invention, spatial transformation interpolation calculation is performed on the image to be registered using the spatial transformation parameters to spatially align the image to be registered with the reference image, generating a spatially aligned image, specifically including:

[0054] Obtain the calculated spatial transformation parameters, which include rotation matrix, translation vector and scaling factor;

[0055] For each voxel in the image to be registered, the theoretical coordinates of the voxel in the target space coordinate system are calculated based on its original three-dimensional spatial coordinates using the rotation matrix, translation vector and scaling factor.

[0056] Based on the theoretical coordinates, determine the positions of adjacent voxels in the three-dimensional voxel grid of the reference image;

[0057] A trilinear interpolation algorithm is used to calculate the interpolated gray value at the theoretical coordinates based on the gray values ​​of adjacent voxel positions at the theoretical coordinates.

[0058] The calculated interpolated gray values ​​are assigned to the new positions of the corresponding voxels in the image to be registered in the target spatial coordinate system. All voxels in the image to be registered are traversed to complete the resampling of gray values ​​and spatial position mapping.

[0059] The image after spatial transformation interpolation calculation is evaluated for quality, the accuracy of spatial alignment is checked, and a spatially aligned image with the same spatial resolution, spatial range and consistent spatial coordinates as the reference image is generated.

[0060] As a further aspect of the present invention, channel attention weighting calculation is performed on the multi-channel stitched feature map. The channel attention weighting calculation dynamically allocates fusion weights based on the importance of each channel feature map to the lesion detection task, specifically including:

[0061] Input the multi-channel stitched feature map and obtain its channel dimensions;

[0062] Global average pooling is performed on the multi-channel stitched feature map in the spatial dimension to compress the feature map of each channel into a scalar and generate a channel descriptor vector.

[0063] The channel descriptor vector is input into a multilayer perceptron, which includes a dimensionality reduction layer and a dimensionality increase layer.

[0064] The correlation between the feature map of each channel and the lesion detection task is learned through the nonlinear transformation of the multilayer perceptron, and the initial weight coefficient of each channel is output.

[0065] The initial weight coefficients are input into the Sigmoid activation function, and the weight coefficients are normalized to between zero and one to obtain the attention weights for each channel.

[0066] The attention weights of each channel are multiplied one by one with the corresponding original channel feature maps to achieve weighted adjustment of each channel feature map;

[0067] The output is a feature map after channel attention weighting, in which channel features that contribute significantly to lesion detection are enhanced, while channel features that contribute less are suppressed.

[0068] As a further aspect of the present invention, the training method of the deep learning detection network includes:

[0069] Collect a training dataset containing multimodal medical images and corresponding expert-annotated lesion masks;

[0070] Perform cross-modal spatial registration and grayscale normalization preprocessing on the training dataset to generate multimodal image data blocks for training.

[0071] Perform multi-channel feature fusion and enhancement calculations on the multimodal image data blocks used for training to generate fused feature image data blocks used for training.

[0072] Using training fused feature image data blocks as input and expert-annotated lesion masks as supervision labels, the gradient descent algorithm is used to optimize the parameters of the deep learning detection network.

[0073] During training, a focusing loss function for difficult samples is introduced to improve the network's ability to segment lesions with blurred boundaries and small size.

[0074] The training process continues until the network's segmentation accuracy on the independent validation set reaches the preset standard, thus obtaining the preset deep learning detection network.

[0075] Compared with the prior art, the advantages and positive effects of the present invention are as follows:

[0076] Leveraging multi-channel feature fusion and enhanced computational models, this approach simultaneously extracts and integrates texture, morphological, and functional metabolic features from images across different modalities. The combination and complementarity of various heterogeneous image features improve the compositional structure of image features, addressing information gaps inherent in single-feature extraction and reducing effective information loss during cross-modal data processing. The integration of multi-dimensional image features reveals the comprehensive state of tissue physiology and structure, enhancing the information expression hierarchy of image data, strengthening the deep data correlation expression, and enabling image data to fully encompass tissue detail information from different imaging modalities, thereby improving the comprehensiveness and completeness of image feature mining.

[0077] For individual suspected lesion areas, multi-dimensional quantitative descriptors are calculated and statistically analyzed based on multimodal raw image data, and lesion attributes are distinguished using an independently configured lesion classification and discrimination device. This changes the processing method that relies solely on single features from fused images for judgment. Instead, it uses original, complete image data as the basis for quantitative analysis, refining the data representation granularity of local lesion areas and the differentiation levels of lesion attributes. Multi-angle data parameters participate in the lesion judgment process, refining the differentiation boundaries of different lesion states, enriching the reference dimensions for lesion identification, expanding the output information content of lesion detection, refining the differentiation accuracy of different pathological states, and improving the overall resolution level of lesion identification. Attached Figure Description

[0078] Figure 1 This is a flowchart of the intelligent lesion detection method based on multimodal medical image fusion described in this invention;

[0079] Figure 2 A flowchart for multi-channel feature fusion and enhancement calculation;

[0080] Figure 3 This is a flowchart of 3D semantic segmentation and candidate region extraction using a deep learning detection network. Detailed Implementation

[0081] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0082] In the description of this invention, it should be understood that the terms "length," "width," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientation or positional relationships, are based on the orientation or positional relationships shown in the accompanying drawings and are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, in the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0083] See Figure 1This paper describes a specific implementation method for intelligent lesion detection based on multimodal medical image fusion. It combines multimodal medical image data and achieves accurate detection, quantitative analysis, and differential diagnosis of lesions through a series of automated processing and intelligent analysis steps. The following details each step and preferred implementation method: First, the original multimodal medical image datasets of the target anatomical region are acquired and merged. Cross-modal spatial registration and grayscale normalization preprocessing operations are then performed to generate a multimodal image data block with unified spatial coordinates and grayscale standards. This step ensures that images from different imaging devices are comparable in space and grayscale standards. Based on this multimodal image data block, multi-channel feature fusion and enhancement calculations are performed to generate a fused feature image data block. This fused feature image data block contains texture features, morphological features, and functional metabolic features extracted and fused from images of different modalities. The fused feature image data block is then input into a pre-set deep learning detection network. This network performs 3D semantic segmentation and candidate region extraction on the fused feature image data block, generating an initial detection result containing several suspected lesion regions. For each suspected lesion region in the initial detection results, a multi-dimensional quantitative descriptor for its corresponding image in the multimodal raw medical image dataset is calculated. This multi-dimensional quantitative descriptor is then input into a pre-trained lesion classification and discriminator, which performs benign / malignant differentiation and pathological type identification on the suspected lesion regions, generating a final identification result for the lesion region. This final identification result includes lesion location information, malignancy probability score, and suggested pathological type.

[0084] In one embodiment of the present invention, a multimodal raw medical image dataset of the target anatomical region is acquired, merged, and preprocessed with cross-modal spatial registration and grayscale normalization to generate a multimodal image data block with unified spatial coordinates and grayscale standards. The specific implementation includes the following steps: The multimodal raw medical image dataset includes at least computed tomography (CT) image data, magnetic resonance imaging (MRI) image data, and positron emission tomography (PET) image data. One modality of image is selected from the multimodal raw medical image dataset as a reference image for spatial registration. Spatial feature points of key anatomical structures in the reference image are extracted to form a reference feature point set. For each image to be registered in the multimodal raw medical image dataset other than the reference image, the following steps are performed: Spatial feature points corresponding to key anatomical structures in the image to be registered are extracted to form a registration feature point set; Based on the reference feature point set and the registration feature point set, spatial transformation parameters are calculated, including rotation matrix, translation vector, and scaling factor.

[0085] Spatial transformation interpolation is performed on the image to be registered using spatial transformation parameters to align the image to be registered with the reference image in space, generating a spatially aligned image. Specifically, this involves: obtaining the calculated spatial transformation parameters, which include rotation matrices, translation vectors, and scaling factors; for each voxel in the image to be registered, calculating its theoretical coordinates in the target spatial coordinate system based on its original 3D spatial coordinates using the rotation matrix, translation vector, and scaling factor; determining the positions of adjacent voxels in the 3D voxel grid of the reference image based on the theoretical coordinates; using a trilinear interpolation algorithm, calculating the interpolated grayscale value at the theoretical coordinates based on the grayscale values ​​of adjacent voxel positions; assigning the calculated interpolated grayscale value to the corresponding voxel in the image to be registered in the new position in the target spatial coordinate system, traversing all voxels in the image to be registered to complete the resampling of grayscale values ​​and spatial position mapping; and performing a quality assessment on the image after spatial transformation interpolation to check the accuracy of spatial alignment, generating a spatially aligned image with the same spatial resolution, spatial range, and consistent spatial coordinates as the reference image. For all spatially aligned images, including reference images and spatially aligned images, voxel-level grayscale normalization is performed to map grayscale values ​​from different imaging devices and different imaging protocols to a unified standardized grayscale range, generating multimodal image data blocks.

[0086] In practice, the liver region is used as the target anatomical region. The acquired multimodal raw medical image data set includes computed tomography (CT) images, magnetic resonance imaging (MRI) T1-weighted images, MRI T2-weighted images, and fluorodeoxyglucose positron emission tomography (PET) images. CT images are selected from this set as reference images for spatial registration. Spatial feature points of key anatomical structures such as the liver contour, portal vein bifurcation, and main hepatic vein junctions are extracted from the CT images to form a reference feature point set. For MRI T1-weighted images, which are the images to be registered, spatial feature points corresponding to the aforementioned key anatomical structures are extracted to form a feature point set to be registered. Based on the reference feature point set and the feature point set to be registered, spatial transformation parameters are calculated using an iterative nearest-point algorithm. These spatial transformation parameters include rotation matrix, translation vector, and scaling factor.

[0087] The spatial transformation parameters, including the calculated rotation matrix, translation vector, and scaling factor, are obtained. For each voxel in the T1-weighted MRI image, the theoretical coordinates of the voxel in the target spatial coordinate system are calculated based on its original 3D spatial coordinates using the rotation matrix, translation vector, and scaling factor. Based on the theoretical coordinates, the positions of eight adjacent voxels in the 3D voxel grid of the computed tomography (CT) image are determined. A trilinear interpolation algorithm is used to calculate the interpolated grayscale value at the theoretical coordinates based on the grayscale values ​​of the eight adjacent voxel positions. The calculated interpolated grayscale value is assigned to the new position of the corresponding voxel in the T1-weighted MRI image in the target spatial coordinate system. This process is repeated for all voxels in the T1-weighted MRI image, completing grayscale resampling and spatial position mapping. The image after spatial transformation interpolation is evaluated for quality. The accuracy of spatial alignment is checked by calculating the normalized mutual information index, generating a spatially aligned image with the same spatial resolution, spatial range, and consistent spatial coordinates as the CT image.

[0088] In some embodiments, for MRI T2-weighted image data and fluorodeoxyglucose positron emission tomography (PET) image data, the same processing flow as for MRI T1-weighted image data is adopted, namely, sequentially extracting the feature point set to be registered, calculating spatial transformation parameters, performing spatial transformation interpolation calculation, and quality assessment to generate their respective spatially aligned images. Optionally, the quality assessment can be performed by combining manual visual inspection with automatic quantitative index calculation. For all spatially aligned images, including CT images used as reference images, MRI T1-weighted spatially aligned images, MRI T2-weighted spatially aligned images, and fluorodeoxyglucose PET spatially aligned images, voxel-level grayscale standardization processing is performed. This standardization process maps grayscale values ​​with different physical meanings under different imaging devices and protocols to a unified standardized grayscale range, specifically using the following formula:

[0089]

[0090] in: Represents the grayscale value of the original image. This represents the average grayscale value obtained statistically from the background tissue outside the target anatomical region. This represents the corresponding standard deviation of gray levels. The standardized grayscale values ​​are used to generate multimodal image data blocks through the above mapping.

[0091] In one embodiment of the present invention, multi-channel feature fusion and enhancement calculations are performed based on multimodal image data blocks to generate fused feature image data blocks. The specific implementation includes the following process. (See also...) Figure 2For each modality of the multimodal image data block, convolution operations are performed using a set of predefined feature extraction filters to extract the corresponding modality-specific feature map. The modality-specific feature maps from different modalities are then stitched together according to the channel dimension at the corresponding voxel positions to generate a multi-channel stitched feature map. Channel attention weighting is performed on the multi-channel stitched feature map. This calculation dynamically assigns fusion weights based on the importance of each channel feature map to the lesion detection task. Specifically, the process includes: inputting the multi-channel stitched feature map and obtaining its channel dimensions; performing global average pooling on the multi-channel stitched feature map in the spatial dimension to compress each channel's feature map into a scalar, generating a channel descriptor vector; inputting the channel descriptor vector into a multilayer perceptron (MLP), which includes a dimensionality reduction layer and an dimensionality increase layer; learning the correlation between each channel feature map and the lesion detection task through the LPP's nonlinear transformation, outputting the initial weight coefficients for each channel; inputting the initial weight coefficients into a sigmoid activation function to normalize the weight coefficients to between zero and one, obtaining the attention weight for each channel; multiplying the obtained attention weight for each channel with the corresponding original channel feature map channel by channel to achieve weighted adjustment of each channel feature map; and outputting the feature map after channel attention weighting, where channel features that contribute significantly to lesion detection are enhanced, while channel features that contribute less are suppressed. The weighted multi-channel stitched feature map is input into the feature enhancement network. The feature enhancement network uses asymmetric convolution and residual connection operations to enhance edge, texture, and density difference features related to lesions, while suppressing irrelevant tissue background features. The output feature map processed by the feature enhancement network is used as a fused feature image data block.

[0092] In practice, for computed tomography (CT) images, magnetic resonance imaging (MRI) T1-weighted images, MRI T2-weighted images, and positron emission tomography (PET) images in the multimodal image data block, convolution operations are performed using a set of predefined feature extraction filters to extract corresponding modality-specific feature maps. Specifically, the feature extraction filter set for CT images focuses on extracting edge features of high-density structures such as bone and calcification; the filter set for MRI images focuses on extracting soft tissue contrast and edema signal features; and the filter set for PET images focuses on extracting texture and gradient features of high-metabolic regions. The modality-specific feature maps from different modalities are then stitched together along the channel dimension according to their corresponding voxel positions to generate a multi-channel stitched feature map. The number of channels in the multi-channel stitched feature map is equal to the sum of the number of channels in each modality feature map.

[0093] Channel attention weighting is performed on the multi-channel stitched feature map. This calculation dynamically assigns fusion weights based on the importance of each channel feature map to the lesion detection task. Specifically, this involves: inputting the multi-channel stitched feature map, obtaining its channel dimensions, performing global average pooling on the feature map in its spatial dimension to compress each channel's feature map into a scalar, and generating a channel descriptor vector. This channel descriptor vector is then input into a multilayer perceptron (MLP), which includes a dimensionality reduction layer and an dimensionality increase layer. Through the LPP's nonlinear transformation, the correlation between each channel feature map and the lesion detection task is learned, and the initial weight coefficients for each channel are output. Finally, these initial weight coefficients are input into a sigmoid activation function, normalizing them to between zero and one to obtain the attention weight for each channel. The calculation process for the attention weights can be described as follows:

[0094]

[0095] in: This represents the channel descriptor scalar for the c-th channel. This represents the weight matrix of the dimensionality reduction layer. Represents a non-linear activation function. This represents the weight matrix of the dimension-upgrading layer. This represents the Sigmoid activation function. This represents the calculated attention weight for the c-th channel. The attention weight for each channel is multiplied channel-by-channel with the corresponding original channel feature map to achieve weighted adjustment of each channel feature map. The output is the feature map after channel attention weighting, where channel features that contribute significantly to lesion detection are enhanced, while those that contribute less are suppressed.

[0096] In some embodiments, the weighted multi-channel stitched feature map is input into a feature enhancement network. The feature enhancement network enhances edge, texture, and density difference features related to lesions through asymmetric convolution and residual connection operations, while suppressing irrelevant tissue background features. Optionally, the asymmetric convolution operation can use combinations of 1x3x3, 3x1x3, and 3x3x1 convolution kernels to efficiently extract anisotropic features in three-dimensional space. The residual connection operation adds and fuses the feature maps from the shallow and deep layers of the feature enhancement network to retain more detailed information. The output feature map processed by the feature enhancement network serves as a fused feature image data block. It can be understood that the fused feature image data block maintains the same spatial dimension as the input multimodal image data block, but it includes selected and enhanced cross-modal fusion features in the channel dimension.

[0097] In one embodiment of the present invention, a fused feature image data block is input into a preset deep learning detection network. The deep learning detection network performs three-dimensional semantic segmentation and candidate region extraction on the fused feature image data block, generating an initial detection result containing several suspected lesion regions. The specific implementation includes the following process. (See reference...) Figure 3 The deep learning detection network employs an encoder-decoder structure. The encoder performs multi-level, downsampling feature extraction on the input fused feature image data blocks, generating deep semantic feature maps at different scales. The decoder progressively upsamples the deepest semantic feature map, fusing feature maps from corresponding levels of the encoder during the upsampling process to restore the size and supplement details of the feature maps. At the decoder's output, each voxel is classified and predicted, generating a 3D probability map. This 3D probability map represents the probability that each voxel belongs to a lesion region. An adaptive threshold segmentation algorithm is applied to the 3D probability map to extract connected regions with probability values ​​higher than a preset threshold, and each connected region is marked as a suspected lesion region.

[0098] For each suspected lesion region, the 3D bounding box coordinates and voxel set are calculated to form the initial detection result. The training method of the deep learning detection network includes: collecting a training dataset containing multimodal medical images and corresponding expert-annotated lesion masks; performing cross-modal spatial registration and grayscale normalization preprocessing operations on the training dataset to generate multimodal image data blocks for training; performing multi-channel feature fusion and enhancement calculations on the multimodal image data blocks for training to generate fused feature image data blocks for training; using the fused feature image data blocks for training as input and the expert-annotated lesion masks as supervision labels, the gradient descent algorithm is used to optimize the parameters of the deep learning detection network; during the training process, a focusing loss function for difficult samples is introduced to improve the network's ability to segment lesions with blurred boundaries and small sizes; the training process continues until the network's segmentation accuracy on the independent validation set reaches the preset standard, thus obtaining the preset deep learning detection network.

[0099] In practice, the fused feature image data block is input into a pre-defined deep learning detection network, which employs an encoder-decoder structure. The encoder performs multi-level, downsampling feature extraction on the input fused feature image data block, generating deep semantic feature maps at different scales. The encoder consists of multiple convolutional downsampling modules connected in series. Each convolutional downsampling module performs two convolution operations followed by a max pooling operation with a stride of two. The decoder progressively upsamples the deepest semantic feature map, fusing feature maps from the corresponding levels of the encoder during the upsampling process to restore the size and supplement details of the feature map. After each upsampling, the decoder concatenates the feature map at the corresponding scale with the encoder's feature map along the channel dimension. At the decoder's output, each voxel is classified and predicted. A three-dimensional probability map is generated through a 1x1x1 convolutional layer and a sigmoid activation function. The three-dimensional probability map represents the probability that each voxel belongs to a lesion region.

[0100] An adaptive threshold segmentation algorithm is applied to the 3D probabilistic map to extract connected regions with probability values ​​higher than a preset threshold. Each connected region is marked as a suspected lesion region. The adaptive threshold segmentation algorithm employs a connected component labeling method based on 3D region growing. The 3D bounding box coordinates and voxel set are calculated for each suspected lesion region to form the initial detection result. The training method for the deep learning detection network includes: collecting a training dataset containing multimodal medical images and corresponding expert-annotated lesion masks; performing cross-modal spatial registration and grayscale normalization preprocessing operations on the training dataset to generate multimodal image data blocks for training; performing multi-channel feature fusion and enhancement calculations on the multimodal image data blocks for training to generate fused feature image data blocks for training; using the fused feature image data blocks for training as input and expert-annotated lesion masks as supervision labels, the gradient descent algorithm is used to optimize the parameters of the deep learning detection network, with the Adam algorithm used as the optimizer.

[0101] During training, a focusing loss function is introduced for difficult samples to improve the network's ability to segment lesions with blurred boundaries and small sizes. This can be understood as the focusing loss function... The calculation method is as follows:

[0102]

[0103] in: This represents the model's predicted probability for the correct category. It is a weighting factor used for class balancing. This is the focusing parameter, used to adjust the degree of focus on difficult samples. The training process continues until the network's segmentation accuracy on the independent validation set reaches a preset standard, resulting in the preset deep learning detection network. The segmentation accuracy on the independent validation set can be evaluated using metrics such as the Dessian similarity coefficient and the Jaccard coefficient. Table 1 shows one metric for evaluating the training process of the deep learning detection network, recording the changes in the network's segmentation accuracy metric on the validation set after multiple training epochs.

[0104] Table 1: Segmentation accuracy of deep learning detection networks on the validation set varies with training cycles.

[0105] Training Epoch Dissemination similarity coefficient (DSC) Jaccard coefficient 10 0.732 0.632 20 0.815 0.723 30 0.863 0.795 40 0.881 0.826 50 0.889 0.833

[0106] In some embodiments, the specific parameter settings for the gradient descent algorithm may include an initial learning rate of 0.001, with a cosine annealing strategy used to dynamically adjust the learning rate. Optionally, the training process may employ an early stopping strategy, stopping training when the segmentation accuracy of the validation set no longer improves within a specified number of consecutive periods. The batch size of the fused feature image data blocks used for training is set to four. It can be understood that the network training process is iterative optimization, with the aim of continuously approximating the expert-annotated lesion mask with the predicted 3D probability map.

[0107] In one embodiment of the present invention, the multi-dimensional quantitative descriptor of the suspected lesion region in the multimodal raw medical image data set is calculated, and its specific implementation includes the following process: Based on the three-dimensional bounding box coordinates of the suspected lesion region, corresponding image blocks are extracted from the computed tomography (CT) image data, magnetic resonance imaging (MRI) image data, and positron emission tomography (PET) image data, respectively.

[0108] For computed tomography (CT) image data blocks, the gray-level histogram statistics, gray-level co-occurrence matrix texture features, and shape irregularity of their internal voxels are calculated. For magnetic resonance imaging (MRI) image data blocks, based on different image sequences, the signal intensity distribution of T1-weighted images, T2-weighted images, and apparent diffusion coefficient distribution features of diffusion-weighted images are calculated. For positron emission tomography (PET) image data blocks, the maximum, average, and peak values ​​of the standard uptake and the metabolic volume of lesions are calculated. All calculated feature values, including gray-level histogram statistics, gray-level co-occurrence matrix texture features, shape irregularity, T1 signal intensity distribution, T2 signal intensity distribution, apparent diffusion coefficient distribution features, maximum, average, and peak values ​​of the standard uptake, and metabolic volume of lesions, are concatenated in a predetermined order into a high-dimensional feature vector, which serves as a multi-dimensional quantization descriptor. The multi-dimensional quantitative descriptor is input into a pre-trained lesion classification and identification device, which then performs benign or malignant differentiation and pathological type identification on suspected lesion areas, generating the final identification result of the lesion area. The specific implementation includes the following process.

[0109] The pre-trained lesion classification discriminator comprises multiple fully connected layers and a classification output layer. High-dimensional feature vectors are input into the lesion classification discriminator, undergoing nonlinear transformation and feature dimensionality reduction in the fully connected layers to obtain a deep abstract feature representation of the lesion. Based on this deep abstract feature representation, the classification output layer of the lesion classification discriminator computes two outputs in parallel: the first output is the benign / malignant discrimination probability, which is the probability value of a suspected lesion area being classified as a malignant tumor; the second output is a pathological type probability distribution vector, which contains the probability that the suspected lesion area belongs to each predefined pathological type. When the benign / malignant discrimination probability exceeds a preset malignancy threshold, the pathological type with the highest probability is selected from the pathological type probability distribution vector as the suggested pathological type. The three-dimensional bounding box coordinates of the suspected lesion area, the benign / malignant discrimination probability, and the suggested pathological type are integrated to generate the final identification result of the lesion area.

[0110] In practice, for a suspected lesion area obtained from the initial detection results, corresponding image blocks are extracted from computed tomography (CT) images, magnetic resonance imaging (MRI) T1-weighted images, MRI T2-weighted images, and positron emission tomography (PET) images based on the three-dimensional bounding box coordinates of the suspected lesion area. For each CT image block, the gray-level histogram statistics, gray-level co-occurrence matrix (GLCM) texture features, and shape irregularity of its internal voxels are calculated. The gray-level histogram statistics include mean, standard deviation, skewness, and kurtosis; the GLCM texture features include contrast, correlation, energy, and homogeneity; and the shape irregularity is obtained by calculating the ratio of the lesion surface area to the surface area of ​​a sphere of the same volume. For a T1-weighted image data block of magnetic resonance imaging, calculate the signal intensity distribution characteristics of the T1-weighted image, including the mean and variance of the signal intensity. For a T2-weighted image data block of magnetic resonance imaging, calculate the signal intensity distribution characteristics of the T2-weighted image. For a diffusion-weighted image data block, calculate the apparent diffusion coefficient distribution characteristics, including the mean, minimum, and histogram entropy of the apparent diffusion coefficient values.

[0111] For positron emission tomography (PET) image data blocks, the maximum, average, peak, and lesion metabolic volumes of the standard uptake values ​​are calculated. The peak standard uptake value is defined as the average standard uptake value within a spherical region of one cubic centimeter centered on the point with the highest standard uptake value within the lesion. The lesion metabolic volume is obtained by thresholding the lesion region using an appropriate standard uptake value threshold and then calculating the total voxel volume. All calculated feature values, including gray-level histogram statistics, gray-level co-occurrence matrix texture features, and shape irregularity of the computed tomography images; the T1-weighted and T2-weighted signal intensity distributions and apparent diffusion coefficient distributions of the magnetic resonance imaging (MRI) images; and the maximum, average, peak, and lesion metabolic volumes of the PET images, are concatenated in a predetermined order into a high-dimensional feature vector, serving as a multi-dimensional quantization descriptor. Table 2 shows some feature values ​​from the multi-dimensional quantization descriptor calculated for a suspected lesion region.

[0112] Table 2: Multidimensional Quantitative Descriptor Feature Value Table

[0113] Feature categories Specific feature name Eigenvalues CT grayscale statistics Mean (HU) 45.2 CT texture (GLCM) Contrast 12.7 CT shape Irregularity 1.85 MRIT1 signal Mean (au) 620.3 MRI ADC distribution Minimum value (mm² / s) 0.00078 PET metabolism SUV peak 9.4 PET volume MTV(cm³) 15.6

[0114] The high-dimensional feature vector is input into a pre-trained lesion classification discriminator. This discriminator contains multiple fully connected layers and a classification output layer. The fully connected layers use the ReLU activation function and include dropout operations to prevent overfitting. The high-dimensional feature vector is input into the lesion classification discriminator and undergoes nonlinear transformation and feature dimensionality reduction in the fully connected layers to obtain a deep abstract feature representation of the lesion. Based on this deep abstract feature representation, the classification output layer of the lesion classification discriminator calculates two outputs in parallel. The first output is the benign / malignant discrimination probability, which is the probability value of a suspected lesion area being classified as a malignant tumor. The second output is a pathological type probability distribution vector, which contains the probability that the suspected lesion area belongs to each predefined pathological type, including adenocarcinoma, squamous cell carcinoma, and small cell carcinoma. When the benign / malignant discrimination probability exceeds a preset malignancy threshold, the pathological type with the highest probability is selected from the pathological type probability distribution vector as the suggested pathological type. The calculation of the classification output layer can be understood as follows:

[0115]

[0116] in, This represents the deep abstract feature representation vector input to the classification output layer. This represents the weight matrix of the classification output layer. This represents the bias vector. This represents the Softmax activation function. This represents the output probability distribution vector. The three-dimensional bounding box coordinates of the suspected lesion region, the probability of benign or malignant differentiation, and the suggested pathological type are integrated to generate the final identification result of the lesion region.

[0117] In some embodiments, the probability distribution vectors for benign / malignant classification and pathological type can be obtained by parallel computation of two independent fully connected branches. Optionally, the loss function used during training of the pre-trained lesion classification discriminator can be a weighted sum of binary cross-entropy loss and multi-class cross-entropy loss. It is understood that the malignancy threshold can be determined before model deployment using the receiver operating characteristic curve of the validation set.

[0118] In one embodiment of the present invention, the method further includes constructing a three-dimensional visualization model of the lesion based on lesion localization information and multimodal image data blocks. The three-dimensional visualization model of the lesion fuses and presents the features of the lesion region in different modal images, specifically including the following processes: Based on the lesion localization information in the final identification result, the spatial range of the lesion region in the multimodal image data blocks is determined. Within this spatial range, volume rendering technology is used on the computed tomography (CT) image data to display the density structure of the lesion and its surrounding anatomical relationships. Under the same spatial coordinate system, surface rendering technology is used on the magnetic resonance imaging (MRI) image data to highlight the soft tissue boundaries and internal features of the lesion. The metabolic information of the lesion region in the positron emission tomography (PET) image data is fused and displayed on the anatomical background composed of the CT and MRI images using a pseudo-color transparent overlay. Through an interactive interface, users can adjust the display transparency, color mapping, and cutting planes of different modal images to observe the three-dimensional visualization model of the lesion from any angle.

[0119] In practice, based on the lesion location information in the final identification results of the lesion area, the spatial range of the lesion area within the multimodal image data block is determined. The lesion location information includes the coordinates of the lesion's three-dimensional bounding box. Within the determined spatial range, volume rendering technology is applied to the computed tomography (CT) image data to display the density structure of the lesion and its surrounding anatomical relationships. Volume rendering technology maps the grayscale values ​​of the CT images to color and transparency using a specified transfer function. Under the same spatial coordinate system, surface rendering technology is applied to the T1-weighted and T2-weighted magnetic resonance imaging (MRI) image data to highlight the soft tissue boundaries and internal features of the lesion. Surface rendering technology reconstructs the three-dimensional surface model of the lesion through three-dimensional edge detection and isosurface extraction algorithms.

[0120] Metabolic information of lesion regions from positron emission tomography (PET) images is fused and displayed on an anatomical background composed of computed tomography (CT) and magnetic resonance imaging (MRI) images using a pseudo-color transparent overlay. This pseudo-color mapping maps the standard uptake values ​​as a gradient from blue to red. An interactive interface allows users to adjust the display transparency, color mapping, and cutting planes of different image modalities, observing the 3D visualization model of the lesion from any angle. The interactive interface provides translation, rotation, and zoom functions. In some embodiments, the fusion of volume rendering and surface rendering involves the calculation of transparency mixing across multiple layers, with the final color of each pixel... The following formula can be used for synthesis:

[0121]

[0122] in: Indicates the total number of overlaid image layers. Indicates the first Color vectors of each layer Indicates the first The opacity value of each layer at the current pixel. This represents the cumulative product of the opacities of all preceding layers. In reverse order, they are the computed tomography volume rendering layer, the magnetic resonance imaging (MRI) plane rendering layer, and the positron emission tomography (PET) pseudo-color overlay layer. Optionally, the color mapping scheme and transparency curve can be customized by the user through an interactive interface. In some embodiments, the interactive interface may also provide 3D annotation and measurement tools, allowing users to measure length, angle, and volume on the 3D visualization model of the lesion. It is understood that the cutting plane function allows users to define planes in any direction, thereby dissecting the 3D model to observe its internal structure.

[0123] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments that can be applied to other fields. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A lesion intelligent detection method based on multi-modal medical image fusion, characterized in that, The method includes: The multimodal raw medical image datasets of the target anatomical region are acquired, merged, and preprocessed with cross-modal spatial registration and grayscale normalization to generate multimodal image data blocks with unified spatial coordinates and grayscale standards. Based on the multimodal image data block, multi-channel feature fusion and enhancement calculations are performed to generate a fused feature image data block. The fused feature image data block includes texture features, morphological features, and functional metabolic features extracted and fused from different modal images. The fused feature image data block is input into a preset deep learning detection network, which performs three-dimensional semantic segmentation and candidate region extraction on the fused feature image data block to generate an initial detection result containing several suspected lesion regions. For each suspected lesion region in the initial detection results, calculate the multi-dimensional quantitative descriptor of the corresponding image of the suspected lesion region in the multimodal raw medical image data set; The multi-dimensional quantitative descriptor is input into a pre-trained lesion classification and identification device, which then performs benign or malignant discrimination and pathological type identification on the suspected lesion area, generating a final identification result for the lesion area. The final identification result includes lesion location information, malignancy probability score, and suggested pathological type. 2.The intelligent lesion detection method based on multi-modal medical image fusion according to claim 1, characterized in that, The multimodal raw medical image datasets of the target anatomical region are merged and preprocessed with cross-modal spatial registration and grayscale normalization to generate multimodal image data blocks with unified spatial coordinates and grayscale standards, including: The multimodal raw medical image data set includes at least computed tomography (CT) image data, magnetic resonance imaging (MRI) image data, and positron emission tomography (PET) image data. Select an image of one modality from the multimodal raw medical image dataset as a reference image for spatial registration; Spatial feature points of key anatomical structures in the reference image are extracted to form a reference feature point set; For each image to be registered in the multimodal raw medical image dataset, excluding the reference image, the following steps are performed: Extract spatial feature points from the image to be registered that correspond to the key anatomical structure to form a set of feature points to be registered. Based on the reference feature point set and the feature point set to be registered, spatial transformation parameters are calculated, including rotation matrix, translation vector and scaling factor; The spatial transformation parameters are used to perform spatial transformation interpolation calculations on the image to be registered, so that the image to be registered is spatially aligned with the reference image, thereby generating a spatially aligned image. For all spatially aligned images, including reference images and spatially aligned images, voxel-level grayscale normalization is performed to map grayscale values ​​from different imaging devices and different imaging protocols to a unified standardized grayscale range, generating multimodal image data blocks. 3.The intelligent lesion detection method based on multi-modal medical image fusion according to claim 2, characterized in that, Based on the multimodal image data block, multi-channel feature fusion and enhancement calculations are performed to generate a fused feature image data block, including: For each modality of the multimodal image data block, a set of predefined feature extraction filters are used to perform convolution operations to extract the corresponding modality-specific feature map. The modality-specific feature maps from different modalities are spliced ​​together according to the channel dimension based on the corresponding voxel positions to generate a multi-channel spliced ​​feature map. Channel attention weighting calculation is performed on the multi-channel stitched feature map, and the channel attention weighting calculation dynamically allocates fusion weights according to the importance of each channel feature map to the lesion detection task; The weighted multi-channel stitched feature map is input into the feature enhancement network. The feature enhancement network enhances the edge, texture and density difference features related to the lesion through asymmetric convolution and residual connection operations, while suppressing irrelevant tissue background features. The feature map processed by the feature enhancement network is output as a fused feature image data block. 4.The method of claim 3, wherein, The fused feature image data block is input into a preset deep learning detection network, which performs 3D semantic segmentation and candidate region extraction on the fused feature image data block to generate an initial detection result containing several suspected lesion regions, including: The deep learning detection network adopts an encoder-decoder structure; The encoder performs multi-level, downsampling feature extraction on the input fused feature image data block to generate deep semantic feature maps at different scales. The decoder progressively upsamples the deepest semantic feature map and fuses feature maps from the corresponding level of the encoder during the upsampling process to restore the size of the feature map and supplement its details. At the output of the decoder, each voxel is classified and predicted to generate a three-dimensional probability map, which represents the probability that each voxel belongs to the lesion region. An adaptive threshold segmentation algorithm is applied to the three-dimensional probability map to extract connected regions with probability values ​​higher than a preset threshold, and each connected region is marked as a suspected lesion region. The three-dimensional bounding box coordinates and voxel set of each suspected lesion region are calculated to form the initial detection results.

5. The intelligent lesion detection method based on multi-modal medical image fusion according to claim 4, characterized in that, Calculate the multi-dimensional quantitative descriptor of the suspected lesion region in the multimodal raw medical image dataset, including: Based on the three-dimensional bounding box coordinates of the suspected lesion area, corresponding image blocks are extracted from the computed tomography (CT) image data, magnetic resonance imaging (MRI) image data, and positron emission tomography (PET) image data, respectively. For computed tomography image data blocks, calculate the gray-level histogram statistics, gray-level co-occurrence matrix texture features, and shape irregularity of the voxels within them; For magnetic resonance imaging image data blocks, based on images of different sequences, the signal intensity distribution of T1-weighted images, the signal intensity distribution of T2-weighted images, and the apparent diffusion coefficient distribution characteristics of diffusion-weighted images are calculated. For positron emission tomography (PET) image data blocks, calculate the maximum, average, peak, and lesion metabolic volume of the standard uptake values; All calculated feature values, including gray-level histogram statistics, gray-level co-occurrence matrix texture features, shape irregularity, T1 signal intensity distribution, T2 signal intensity distribution, apparent diffusion coefficient distribution features, maximum, average, peak values ​​of standard uptake values, and lesion metabolic volume, are concatenated in a predetermined order into a high-dimensional feature vector, which serves as the multi-dimensional quantization descriptor. 6.The intelligent lesion detection method based on multi-modal medical image fusion according to claim 5, characterized in that, The multi-dimensional quantitative descriptor is input into a pre-trained lesion classification and identification device, which then performs benign / malignant and pathological type differentiation on the suspected lesion areas, generating the final identification result of the lesion areas, including: The pre-trained lesion classification and identification device contains multiple fully connected layers and a classification output layer; The high-dimensional feature vector is input into the lesion classification and discriminator, and after nonlinear transformation and feature dimensionality reduction in the fully connected layer, a deep abstract feature representation of the lesion is obtained. Based on the deep abstract feature representation, the classification output layer of the lesion classification discriminator calculates two outputs in parallel: the first output is the benign or malignant discrimination probability, which is the probability value of the suspected lesion area being judged as a malignant tumor; The second output is a pathological type probability distribution vector, which contains the probability that the suspected lesion area belongs to each predefined pathological type; When the probability of benign or malignant differentiation exceeds the preset malignant determination threshold, the pathological type with the highest probability is selected from the pathological type probability distribution vector as the suggested pathological type. The three-dimensional bounding box coordinates, benign / malignant discrimination probability, and suggested pathological type of the suspected lesion area are integrated to generate the final identification result of the lesion area.

7. The intelligent lesion detection method based on multimodal medical image fusion according to claim 1, characterized in that, The method further includes: Based on the lesion location information and the multimodal image data block, a three-dimensional visualization model of the lesion is constructed. This three-dimensional visualization model fuses and presents the features of the lesion region in different modal images, specifically including: Based on the lesion location information in the final identification result, the spatial range of the lesion region in the multimodal image data block is determined; Within the aforementioned spatial range, volume rendering technology is used on computed tomography image data to display the density structure of lesions and their relationship with surrounding anatomy; Under the same spatial coordinate system, the magnetic resonance imaging data is rendered using surface rendering technology to highlight the soft tissue boundaries and internal features of the lesions; The metabolic information of the lesion area in the positron emission tomography (PET) image data is fused and displayed on the anatomical background composed of computed tomography (CT) images and magnetic resonance imaging (MRI) images in a pseudo-color transparent overlay. The interactive interface allows users to adjust the display transparency, color mapping, and cutting planes of different modal images, and observe the 3D visualization model of the lesion from any angle.

8. The intelligent lesion detection method based on multimodal medical image fusion according to claim 2, characterized in that, The spatial transformation parameters are used to perform spatial transformation interpolation calculations on the image to be registered, so that the image to be registered is spatially aligned with the reference image, generating a spatially aligned image, specifically including: Obtain the calculated spatial transformation parameters, which include rotation matrix, translation vector and scaling factor; For each voxel in the image to be registered, the theoretical coordinates of the voxel in the target space coordinate system are calculated based on its original three-dimensional spatial coordinates using the rotation matrix, translation vector and scaling factor. Based on the theoretical coordinates, determine the positions of adjacent voxels in the three-dimensional voxel grid of the reference image; A trilinear interpolation algorithm is used to calculate the interpolated gray value at the theoretical coordinates based on the gray values ​​of adjacent voxel positions at the theoretical coordinates. The calculated interpolated gray values ​​are assigned to the new positions of the corresponding voxels in the image to be registered in the target spatial coordinate system. All voxels in the image to be registered are traversed to complete the resampling of gray values ​​and spatial position mapping. The image after spatial transformation interpolation calculation is evaluated for quality, the accuracy of spatial alignment is checked, and a spatially aligned image with the same spatial resolution, spatial range and consistent spatial coordinates as the reference image is generated.

9. The intelligent lesion detection method based on multimodal medical image fusion according to claim 3, characterized in that, Channel attention weighting is performed on the multi-channel stitched feature maps. This channel attention weighting calculation dynamically assigns fusion weights based on the importance of each channel feature map to the lesion detection task, specifically including: Input the multi-channel stitched feature map and obtain its channel dimensions; Global average pooling is performed on the multi-channel stitched feature map in the spatial dimension to compress the feature map of each channel into a scalar and generate a channel descriptor vector. The channel descriptor vector is input into a multilayer perceptron, which includes a dimensionality reduction layer and a dimensionality increase layer. The correlation between the feature map of each channel and the lesion detection task is learned through the nonlinear transformation of the multilayer perceptron, and the initial weight coefficient of each channel is output. The initial weight coefficients are input into the Sigmoid activation function, and the weight coefficients are normalized to between zero and one to obtain the attention weights for each channel. The attention weights of each channel are multiplied one by one with the corresponding original channel feature maps to achieve weighted adjustment of each channel feature map; The output is a feature map after channel attention weighting, in which channel features that contribute significantly to lesion detection are enhanced, while channel features that contribute less are suppressed.

10. The intelligent lesion detection method based on multimodal medical image fusion according to claim 4, characterized in that, The training method for the deep learning detection network includes: Collect a training dataset containing multimodal medical images and corresponding expert-annotated lesion masks; Perform cross-modal spatial registration and grayscale normalization preprocessing on the training dataset to generate multimodal image data blocks for training. Perform multi-channel feature fusion and enhancement calculations on the multimodal image data blocks used for training to generate fused feature image data blocks used for training. Using training fused feature image data blocks as input and expert-annotated lesion masks as supervision labels, the gradient descent algorithm is used to optimize the parameters of the deep learning detection network. During training, a focusing loss function for difficult samples is introduced to improve the network's ability to segment lesions with blurred boundaries and small size. The training process continues until the network's segmentation accuracy on the independent validation set reaches the preset standard, thus obtaining the preset deep learning detection network.

Images