Intelligent detection method and device for fusing medical image learning image

Abstract

The invention discloses an intelligent detection method and device for fusing a medical image learning image, and relates to the technical field of medical image processing. The method comprises the following steps: acquiring and preprocessing a bimodal medical image, and extracting a feature map through multi-scale decomposition; constructing a cross-modal correlation model, and setting a modal attention mechanism (embedding anatomical structure prior guidance feature complementation) and a morphological attention mechanism (setting lesion morphological constraint weight); the method comprises the following steps: collecting multiple types of image samples, pairing according to a focus form and an imaging mode to construct a bimodal joint data set, and correlating and labeling to generate a training data set with modal attributes; after a multi-stage iteration training model, inputting the preprocessed image to carry out feature fusion so as to obtain a fused image; and generating a lesion probability graph according to the fused image, positioning a lesion area through multi-threshold segmentation, and outputting a detection result. The system comprises a data acquisition module, a preprocessing module and the like. The method improves the accuracy and reliability of medical image detection, and is suitable for clinical multi-modal image analysis.

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, and in particular to an intelligent detection method and device that integrates medical image learning images. Background Technology

[0002] In the field of medical diagnostics, medical imaging is a key means for the early detection and accurate diagnosis of diseases. With the rapid development of medical imaging technology, various imaging modalities such as computed tomography (CT) and magnetic resonance imaging (MRI) are widely used in clinical practice. However, single-modality imaging often has information limitations. For example, CT clearly displays bones and calcifications but lacks contrast in soft tissues, while MRI excels at presenting soft tissue details but is not sensitive to calcified structures. Furthermore, different modalities of imaging face many challenges in terms of spatial alignment, noise interference, and feature complementarity.

[0003] Traditional medical imaging detection methods primarily rely on single-modal image analysis, making it difficult to fully utilize the complementary information from multimodal images, thus limiting the accuracy and reliability of lesion detection. For example, in early tumor screening, a single CT image may miss small lesions due to low soft tissue contrast, while MRI, although capable of detecting lesions, lacks clear anatomical landmarks. Furthermore, current technologies have limited effectiveness in noise suppression and contrast enhancement during image preprocessing, easily leading to information distortion during subsequent feature extraction. In feature fusion, the lack of effective utilization of prior information on different modal anatomical structures and lesion morphological features makes it difficult to construct accurate cross-modal correlation models. Regarding dataset construction, the lack of systematic management of multi-dimensional attributes (such as imaging equipment, lesion stage, acquisition parameters, etc.) results in low sample matching efficiency and significant modal differences during model training, affecting the improvement of detection performance.

[0004] Meanwhile, with the deepening application of artificial intelligence technology in the field of medical imaging, detection methods based on deep learning have gradually become a research hotspot. However, existing models generally suffer from problems such as insufficient fusion of multimodal data, simplistic attention mechanism design, and insufficient adaptability to complex lesion morphology and anatomical structures. For example, traditional attention mechanisms only focus on feature differences between modalities and do not fully integrate prior information about anatomical structures to guide feature complementarity, resulting in a lack of anatomical semantic consistency in the fused feature map. When locating lesion regions, single threshold segmentation methods struggle to balance density differences and morphological integrity of different lesions, easily leading to over- or under-segmentation and affecting the accuracy of detection results.

[0005] Furthermore, clinical practice places higher demands on the timeliness and interpretability of image detection. Existing systems often require complex manual intervention for image registration and feature calibration when processing multimodal images, which is time-consuming and dependent on the operator's experience. At the same time, the model output lacks an intuitive interpretation of the distribution of anatomical structures and the morphological characteristics of lesions, making it difficult to meet clinicians' needs for traceability and reliability of detection results. Summary of the Invention

[0006] To address the technical problem that existing technologies lack intuitive interpretations of anatomical structure distribution and lesion morphological features in their model outputs, thus failing to meet clinicians' needs for traceability and reliability of test results, this invention provides an intelligent detection method and device that integrates medical image learning. The technical solution is as follows:

[0007] On the one hand, an intelligent detection method integrating medical image learning is provided. This method is implemented by an intelligent detection device integrating medical image learning, and includes:

[0008] S1: Acquire medical image data of two different modalities and preprocess them separately to obtain denoised dual-modal images;

[0009] S2, perform multi-scale decomposition on the preprocessed dual-modal image and extract multi-resolution feature maps;

[0010] S3. Construct a cross-modal association model based on dynamic feature fusion, and set up modal attention mechanism and morphological attention mechanism in the model; wherein, the modal attention mechanism embeds prior information of anatomical structure of different modalities to guide feature complementarity; the morphological attention mechanism sets morphological constraint weights of lesion area.

[0011] S4: Collect image samples with different lesion types and imaging parameters, perform modal pairing of the samples according to lesion morphology and imaging mode, and construct a dual-modal joint dataset;

[0012] S5, associates and labels the bimodal joint dataset with image acquisition parameters to generate a training dataset with modal attributes;

[0013] S6. The cross-modal association model is trained in multiple stages using a training dataset with modal attributes. The preprocessed bimodal images are input into the trained model for feature fusion, and the enhanced fused images are output.

[0014] S7 generates a lesion probability map based on the anatomical structure distribution of the fused image, locates the lesion area through the multi-threshold segmentation module, and outputs the detection results.

[0015] Optionally, in step S1, two different modalities of medical image data are acquired and preprocessed separately, including:

[0016] S11: Acquire medical image data in two different modalities; use an anisotropic diffusion algorithm to suppress noise in the original image; use an adaptive histogram equalization algorithm to enhance the contrast of the denoised image to obtain the enhanced image;

[0017] S12 uses a pre-trained ResUNet segmentation model to segment organ regions in enhanced images. Morphological closing operations are used to divide the images into suspected lesion regions and normal tissue regions, generating anatomical structure mask maps.

[0018] S13, on the anatomical structure mask map, the watershed algorithm is used to divide the sub-regions, and the pixels with continuously changing gray levels are aggregated into homogeneous sub-regions by gradient thresholding to extract the topological structure data of the lesion edge.

[0019] S14, On the lesion topology data, the spatial correspondence of multimodal images is reconstructed using the mutual information registration algorithm, generating bimodal registration parameters that include anatomical landmarks and lesion locations;

[0020] S15 aligns the dual-modal registration parameters with the image coordinate system through spatial transformation, and generates a cross-modal anatomical structure mapping matrix by combining the three-dimensional reconstruction model.

[0021] Optionally, in step S3, a cross-modal association model based on dynamic feature fusion is constructed, including:

[0022] S31 uses the VGG network as the backbone network of the cross-modal association model. The backbone network contains convolutional blocks and pooling layers to extract multi-level features.

[0023] S32 introduces a modal attention mechanism after the convolutional block of the backbone network to construct a modal attention layer. This layer generates structure-guided attention weights by fusing feature maps and structural masks of different modalities.

[0024] S33, in the modal attention layer, the Canny operator is used to extract anatomical edge features of different modalities, and the edge feature map is spatially superimposed with the feature map output by the backbone network to obtain a feature map that enhances anatomical perception.

[0025] S34, In the modal attention layer, the region growing algorithm is used to locate the potential lesion region, and the lesion region proposal box is spatially registered with the anatomical perception feature map to generate a composite feature map that integrates anatomical structure and lesion distribution.

[0026] S35, a morphological attention layer is set after the pooling layer of the backbone network. The morphological attention layer calculates the morphological significance weight by fusing lesion morphological features and density distribution features.

[0027] S36, in the morphological attention layer, the gray-level gradient statistical algorithm is used to extract the edge sharpness and density uniformity features of the lesion area, and the gradient statistical features are fused with the feature map output by the backbone network across layers.

[0028] S37, in the morphological attention layer, morphological erosion operation is used to eliminate the interference of artifact regions and generate a heat map of the core lesion region. The heat map and the morphological feature map are then spliced ​​together to obtain a feature map that integrates the prior information of the lesion.

[0029] Optionally, in step S34, a composite feature map of the fused anatomical structure and lesion distribution is generated, including:

[0030] A sliding window scanning algorithm is used to traverse the input image at multiple scales and calculate the probability distribution of anatomical landmarks within each window.

[0031] Morphological opening operations are performed on windows with probability distributions higher than a threshold to filter out candidate lesion regions;

[0032] Map the coordinates of the candidate region to the feature map space, and extract the composite feature map of the corresponding region.

[0033] The extracted feature map is scaled to a preset size using nearest neighbor interpolation, and the correlation weights between the anatomy and the lesion are calculated by inputting the modal attention layer.

[0034] Optionally, in step S36, the gradient statistical features are fused with the feature map output by the backbone network across layers, including:

[0035] The three parameters extracted by the gray-scale gradient statistical algorithm—edge intensity, gradient direction, and curvature—are concatenated into a morphological description vector.

[0036] The morphological description vector is expanded to the same channel dimension as the backbone network feature map through convolutional layers;

[0037] The expanded morphological feature vectors are weighted and fused with the backbone network feature map according to the spatial dimension, and the weights are dynamically adjusted by the morphological saliency score.

[0038] Optionally, in step S37, generating a heat map of the core area of ​​the lesion includes:

[0039] Threshold segmentation is performed on the input image to extract candidate lesion regions;

[0040] A connected component labeling algorithm is used to merge broken regions and remove isolated noise points to form a complete lesion region.

[0041] Calculate the maximum density projection from each pixel to the lesion core to generate a density attenuation heatmap;

[0042] Multiply the heatmap with the feature map of the morphological attention layer channel by channel to enhance the response intensity of the core area of ​​the lesion.

[0043] Optionally, in step S4, a dual-modal joint dataset is constructed, including:

[0044] S41 collects medical image data from different imaging devices, including plain CT scans, enhanced CT scans, and T1 / T2 weighted MRI.

[0045] S42 collects imaging data of different lesion stages, including early lesions, progressive lesions, and late lesions;

[0046] S43, paired datasets combined by imaging device and lesion stage, with each combination category containing at least ten samples from different anatomical locations;

[0047] S44: Extract the acquisition parameters and lesion coordinates of each image, and generate three-dimensional coordinate data of the lesion center point through spatial transformation;

[0048] S45, image data is associated with imaging parameters and lesion stage by triplets to form data units with multi-dimensional attribute labels;

[0049] S46 constructs a tree-like index structure based on multi-dimensional attribute tags to enable rapid sample retrieval by device type and lesion stage.

[0050] Optionally, in step S46, a tree-like index structure is constructed based on multi-dimensional attribute labels, including:

[0051] The data units are input into the improved DBSCAN clustering algorithm, and the imaging parameter similarity threshold and spatial distribution radius are set.

[0052] For samples that meet the threshold conditions, a primary cluster is generated according to the imaging device, and a secondary sub-cluster is performed within each primary cluster according to the lesion stage;

[0053] Each sub-cluster is assigned a unique identifier, which includes the device type code and the lesion stage code;

[0054] It uses a B+ tree structure to store the mapping relationship between identifier codes and data unit addresses, and supports multi-condition combined retrieval.

[0055] Optionally, in step S6, the cross-modal association model is trained in multiple stages using a training dataset with modal attributes, including:

[0056] S61, extract the imaging device and lesion stage labels of the input image, and retrieve matching historical training samples through B+ tree index;

[0057] S62, the feature distribution of historical samples is aligned with the new samples for domain adaptation, and an adversarial generative algorithm is used to eliminate modal differences;

[0058] S63, calculate the Euclidean distance between the feature spaces of the new and old samples, and dynamically adjust the iterative learning rate. The larger the distance, the higher the learning rate.

[0059] In step S7, the lesion area is located and the detection results are output, including:

[0060] The confidence level is determined based on the probability map of the lesion, and a multimodal review mechanism is triggered when the probability is lower than the threshold.

[0061] In the multi-threshold segmentation module, three criteria are defined: density threshold, morphological integrity, and anatomical conformity.

[0062] The lesion segmentation results are generated by merging regions to take into account both sensitivity and specificity.

[0063] The segmentation results are converted into 3D labeled images, and the consistency between the segmentation boundaries and the anatomical structures is verified in real time.

[0064] On the other hand, an intelligent detection device integrating medical image learning images is provided. This device is applied to the intelligent detection method integrating medical image learning images, and the device includes:

[0065] The data acquisition and preprocessing module is used to acquire medical image data of two different modalities and preprocess them separately to obtain denoised dual-modal images.

[0066] The feature extraction module is used to perform multi-scale decomposition on the preprocessed dual-modal image and extract multi-resolution feature maps.

[0067] The model building module is used to build a cross-modal association model based on dynamic feature fusion. Modal attention mechanism and morphological attention mechanism are set in the model. The modal attention mechanism embeds prior information of anatomical structure of different modalities to guide feature complementarity, and the morphological attention mechanism sets morphological constraint weights of lesion areas.

[0068] The dataset construction module is used to collect image samples of different lesion types and imaging parameters. Based on the lesion morphology and imaging mode, the samples are modally paired to construct a dual-modal joint dataset. The dual-modal joint dataset is then associated and labeled with the image acquisition parameters to generate a training dataset with modal attributes.

[0069] The model training and fusion module is used to perform multi-stage iterative training of the cross-modal association model using a training dataset with modal attributes. It inputs the preprocessed bimodal images into the trained model for feature fusion and outputs the enhanced fused image.

[0070] The lesion detection module is used to generate a lesion probability map based on the anatomical structure distribution of the fused image, locate the lesion area through the multi-threshold segmentation module, and output the detection results.

[0071] On the other hand, an intelligent detection device integrating medical image learning images is provided. The intelligent detection device integrating medical image learning images includes: a processor; a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, any one of the methods described above for intelligent detection integrating medical image learning images is implemented.

[0072] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored therein, the at least one instruction being loaded and executed by a processor to implement any of the above-described intelligent detection methods for integrating medical image learning images.

[0073] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following:

[0074] In the image preprocessing stage, the combination of anisotropic diffusion algorithm and adaptive histogram equalization algorithm effectively suppressed noise in the original images and enhanced contrast, providing high-quality image data for subsequent feature extraction. The pre-trained ResUNet segmentation model, combined with morphological closing operations, achieved accurate segmentation of organ regions and preliminary localization of suspected lesion regions. The generated anatomical structure mask map provided important anatomical references for cross-modal registration and feature fusion. The application of the watershed algorithm further extracted the topological structure data of lesion edges. Combined with mutual information registration algorithm and 3D reconstruction model, it ensured accurate alignment of the bimodal images in the spatial coordinate system, laying the foundation for cross-modal feature fusion.

[0075] In constructing cross-modal association models, the VGG-based backbone network design effectively extracts multi-level features. The introduction of a modal attention mechanism is innovative; by fusing feature maps, structural masks, and anatomical edge features from different modalities, structure-guided attention weights are generated. This not only achieves complementarity of prior anatomical information between different modalities but also accurately locates potential lesion regions through region growing and sliding window scanning algorithms. The generated composite feature map fully integrates anatomical structure and lesion distribution information, significantly improving the model's ability to perceive anatomical semantics. The morphological attention mechanism, by extracting morphological features such as edge sharpness and density uniformity, combined with morphological erosion operations, effectively suppresses artifact interference. The generated lesion core region heatmap enhances the model's response intensity to the core lesion region, enabling the model to more accurately capture the morphological features of the lesion.

[0076] During dataset construction, image data from multiple devices and lesion stages were collected and triplet-based association annotations were performed to construct a bimodal joint dataset containing multi-dimensional attribute labels. Based on an improved DBSCAN clustering algorithm and a B+ tree-structured indexing mechanism, efficient sample retrieval and domain-adaptive alignment were achieved. Combined with adversarial generative algorithms and a dynamically adjusted learning rate strategy, modal differences were effectively eliminated, improving the model's generalization ability to different imaging conditions and lesion stages. A multi-stage iterative training strategy further optimized the model's feature learning process, ensuring that the model could fully utilize the feature distribution of historical and newly added samples, dynamically adjusting the learning strategy to improve training efficiency and detection accuracy.

[0077] In the lesion detection phase, a lesion probability map generated from fused images, combined with a multi-threshold segmentation module, achieves precise localization and segmentation of lesion areas through multiple criteria, including density threshold, morphological integrity, and anatomical conformity. The generation of 3D annotation maps and the consistency verification of segmentation boundaries with anatomical structures not only improve the spatial accuracy of the detection results but also provide clinicians with more intuitive and traceable diagnostic evidence. The multimodal verification mechanism triggers further analysis when the detection confidence level is insufficient, effectively reducing the risk of missed diagnoses and misdiagnoses and improving the reliability of the detection results.

[0078] Furthermore, the system architecture of this invention, through modular design, achieves fully automated processing from data acquisition, preprocessing, feature extraction, model training to lesion detection, reducing manual intervention and improving detection efficiency. Simultaneously, by introducing prior information on anatomical structures and lesion morphology constraints, the interpretability of the model is enhanced, enabling detection results to be better integrated with clinical anatomical knowledge and meet actual diagnostic needs. Attached Figure Description

[0079] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0080] Figure 1 This is a flowchart of an intelligent detection method that integrates medical image learning images provided in an embodiment of the present invention;

[0081] Figure 2 This is a schematic diagram illustrating the working principle of medical image data preprocessing provided in this embodiment of the invention;

[0082] Figure 3 This is a flowchart of the construction of a cross-modal association model based on dynamic feature fusion provided in an embodiment of the present invention;

[0083] Figure 4 This is a flowchart illustrating the generation of a composite feature map that integrates anatomical structures and lesion distribution, as provided in an embodiment of the present invention.

[0084] Figure 5 This is a flowchart of the construction of a dual-modal joint dataset provided in an embodiment of the present invention;

[0085] Figure 6 This is a block diagram of an intelligent detection device that integrates medical image learning images according to an embodiment of the present invention;

[0086] Figure 7 This is a schematic diagram of the structure of an intelligent detection device that integrates medical image learning images according to an embodiment of the present invention. Detailed Implementation

[0087] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0088] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0089] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.

[0090] In this embodiment of the invention, sometimes a subscript such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0091] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0092] This invention provides an intelligent detection method that integrates medical image learning images. This method can be implemented by an intelligent detection device that integrates medical image learning images, which can be a terminal or a server. Figure 1 The flowchart shown is for an intelligent detection method that integrates medical images and learns from them. The processing flow of this method may include the following steps:

[0093] S1: Acquire medical image data of two different modalities and preprocess them separately to obtain denoised dual-modal images;

[0094] In one feasible implementation, step S1 involves acquiring two different modalities of medical image data and preprocessing them separately, including:

[0095] S11: Acquire medical image data in two different modalities; use an anisotropic diffusion algorithm to suppress noise in the original image; use an adaptive histogram equalization algorithm to enhance the contrast of the denoised image to obtain the enhanced image;

[0096] S12 uses a pre-trained ResUNet segmentation model to segment organ regions in enhanced images. Morphological closing operations are used to divide the images into suspected lesion regions and normal tissue regions, generating anatomical structure mask maps.

[0097] S13, on the anatomical structure mask map, the watershed algorithm is used to divide the sub-regions, and the pixels with continuously changing gray levels are aggregated into homogeneous sub-regions by gradient thresholding to extract the topological structure data of the lesion edge.

[0098] S14, On the lesion topology data, the spatial correspondence of multimodal images is reconstructed using the mutual information registration algorithm, generating bimodal registration parameters that include anatomical landmarks and lesion locations;

[0099] S15 aligns the dual-modal registration parameters with the image coordinate system through spatial transformation, and generates a cross-modal anatomical structure mapping matrix by combining the three-dimensional reconstruction model.

[0100] S2 performs multi-scale decomposition on the preprocessed dual-modal image and extracts multi-resolution feature maps.

[0101] In one feasible implementation, Figure 2 This diagram illustrates the working principle of medical image data preprocessing. First, two different modalities of medical image data are acquired, such as CT and MRI images. For each modality, preprocessing is performed sequentially: an anisotropic diffusion algorithm is used to suppress noise in the original image, and then an adaptive histogram equalization algorithm is used to enhance the contrast of the denoised image, resulting in an enhanced image.

[0102] Multi-scale decomposition of the preprocessed dual-modal image is performed, for example, through wavelet transform or Gaussian pyramid decomposition, to obtain feature maps at different resolutions in order to capture structural information at different scales in the image.

[0103] A cross-modal association model based on dynamic feature fusion is constructed, in which modal attention and morphological attention mechanisms are set. The modal attention mechanism embeds prior information of anatomical structures from different modalities to guide the complementarity of features from different modalities; the morphological attention mechanism sets morphological constraint weights for lesion regions to highlight the morphological features of lesions.

[0104] Image samples of different lesion types (e.g., tumors, inflammation) and imaging parameters (e.g., CT slice thickness, MRI sequence type) are collected. Modal pairing is performed based on lesion morphology (e.g., nodules, masses) and imaging mode (e.g., plain scan, enhanced scan) to form a dual-modal joint dataset. The dataset is then associated and labeled with image acquisition parameters (e.g., device model, scan time) to generate a training dataset with modal attributes (e.g., modality type, parameter values).

[0105] A multi-stage iterative training method was used to train the cross-modal association model using a training dataset with modal attributes. After training, preprocessed bimodal images were input into the model, and the enhanced fused images were output through the model's internal feature fusion mechanism to highlight the differences between lesions and normal tissues.

[0106] Based on the anatomical structure distribution of the fused images, a lesion probability map is generated. By comprehensively considering factors such as density and morphology through a multi-threshold segmentation module, the lesion area is located and the detection results, such as the location, size, and type of the lesion, are output.

[0107] In one feasible implementation, based on the above overall scheme, the preprocessing of medical image data in step S1 specifically includes the following operations:

[0108] An anisotropic diffusion algorithm is used to suppress noise in the original image. This algorithm effectively removes Gaussian noise or salt-and-pepper noise by preserving edge information during image smoothing, avoiding the edge blurring problem caused by traditional filtering algorithms. After denoising, an adaptive histogram equalization algorithm is used to enhance the image contrast. This algorithm divides the image into multiple sub-regions and performs histogram equalization on each sub-region separately, thereby enhancing local contrast while avoiding global overexposure or underexposure, resulting in an enhanced image with clearer details.

[0109] A pre-trained ResUNet segmentation model was used to segment organ regions in enhanced images. The ResUNet model combines the residual connections of ResNet with the encoding / decoding structure of UNet, effectively extracting semantic information from images and accurately segmenting organ regions such as the liver and lungs. After segmentation, morphological closing operations (i.e., dilation followed by erosion) were applied to process the segmentation results, dividing the image into suspected lesion regions and normal tissue regions, generating an anatomical structure mask map that clearly identifies areas where lesions may exist. Then, the watershed algorithm was used to sub-regionally divide the image on the anatomical structure mask map. The watershed algorithm treats the image as a terrain surface, with pixel gray values ​​corresponding to altitude. By simulating the process of water spreading from low-lying areas, pixels with continuously changing gray values ​​are aggregated into homogeneous sub-regions, thereby extracting topological data of lesion edges, such as edge continuity and tortuosity.

[0110] Based on lesion topology data, a mutual information registration algorithm is used to reconstruct the spatial correspondence of multimodal images. This algorithm maximizes the mutual information between two modalities to determine optimal spatial transformation parameters, generating bimodal registration parameters that include anatomical landmarks (such as bone endpoints and organ contour key points) and lesion locations, ensuring spatial alignment of different modalities. Finally, the bimodal registration parameters are spatially transformed and aligned with the image coordinate system, for example, through affine or nonlinear transformations, to unify the registration parameters with the physical coordinate system of the images. Combined with a 3D reconstruction model, a cross-modal anatomical structure mapping matrix is ​​generated. This matrix records the correspondence of anatomical structures in different modalities, providing a spatial reference for subsequent feature fusion.

[0111] In this embodiment, the preprocessing process uses bimodal data of lung CT and chest MRI images as an example. First, anisotropic diffusion denoising is performed on the raw CT images. This algorithm iteratively calculates the diffusion coefficient of each pixel to smooth noisy areas (such as salt-and-pepper noise in lung texture) while preserving gradient information of important structures such as lung nodule edges. For example, when processing lung CT images, the algorithm automatically identifies noise points and edge points for potential vascular cross-sectional noise in mediastinal window images, only smoothing the noise points to avoid blurring vascular edges. After denoising, an adaptive histogram equalization algorithm is used to enhance contrast, expanding the dynamic range of grayscale in the lung parenchyma, making low-density lesions such as ground-glass nodules easier to identify in the images. For instance, the lung field area originally concentrated in the 100-200 grayscale value range is expanded to the 50-300 range, improving the display effect of subtle density differences.

[0112] Next, a pre-trained ResUNet model is used to segment the lung region. Taking a CT image of a patient with a nodule in the right upper lobe of the lung as an example, after inputting the enhanced CT image, the model extracts the contour features of the lung parenchyma through multi-layer convolution of the encoding path. The decoding path gradually restores the spatial resolution and fuses with the encoding path features, finally outputting a binary mask image of the lung region. Morphological closing operations are performed on this mask image, using 3×3 rectangular structuring elements to first dilate and then erode, filling the tiny gaps in the lung fissure region, eliminating the interference of subpleural small blood vessel cross-sections, and clearly dividing the image into the suspected lesion area in the right upper lobe containing the nodule and the remaining normal lung tissue area.

[0113] Based on the anatomical structure mask image, a watershed algorithm is used for subregion division. Taking a ground-glass nodule in the right upper lobe of the lung as an example, the gray values ​​of the mask image are regarded as topographical height, the high gray-level gradient region at the edge of the nodule is regarded as the "watershed," and the lung tissue inside and around the nodule is regarded as different "basins." The algorithm starts from the lowest point (the gray-level minimum point) of each basin and spreads outward, aggregating pixels with continuously changing gray values ​​into homogeneous subregions. For example, the lung segment where the nodule is located is divided into a separate subregion, while retaining the jagged topological structure data of the nodule edge. This data can reflect malignant features such as lobulation of the nodule.

[0114] The mutual information registration algorithm is used to align CT and MRI modalities. Taking the chest CT and T1-weighted MRI images of the same patient as an example, anatomical landmarks such as hilar lymph nodes are first marked in the CT image, and landmarks at the same anatomical location are marked in the MRI image. By calculating the mutual information of the two modal images under different spatial transformations, the optimal translation, rotation, and scaling parameters are determined so that the error between the coordinates (x1, y1, z1) of the lung nodule center in CT and the corresponding coordinates (x2, y2, z2) in MRI is less than 1 mm. The dual-modal registration parameters containing these landmarks and nodule locations are then generated.

[0115] Finally, spatial transformation alignment and mapping matrix generation are performed. The registration parameters of CT and MRI are transformed with their respective DICOM coordinate systems, for example, by using an affine transformation matrix to convert the millimeter coordinate system of the CT image to the coordinate space of the MRI. Combined with the lung surface model generated by the 3D reconstruction algorithm, a cross-modal anatomical structure mapping matrix is ​​established. This matrix records the corresponding region index of each lung segment in CT in MRI, such as the voxel coordinate range of the corresponding region of the anterior segment of the right upper lobe in CT in MRI, providing accurate spatial correspondence for subsequent cross-modal feature fusion. Through the above steps, the preprocessed dual-modal images not only have reduced noise and enhanced contrast, but also achieve accurate alignment of anatomical structures, laying the foundation for subsequent multi-scale feature extraction and cross-modal correlation model analysis.

[0116] S3. Construct a cross-modal association model based on dynamic feature fusion, and set up modal attention mechanism and morphological attention mechanism in the model; wherein, the modal attention mechanism embeds prior information of anatomical structure of different modalities to guide feature complementarity; the morphological attention mechanism sets morphological constraint weights of lesion area.

[0117] In one feasible implementation, step S3 involves constructing a cross-modal association model based on dynamic feature fusion, including:

[0118] S31 uses the VGG network as the backbone network of the cross-modal association model. The backbone network contains convolutional blocks and pooling layers to extract multi-level features.

[0119] S32 introduces a modal attention mechanism after the convolutional block of the backbone network to construct a modal attention layer. This layer generates structure-guided attention weights by fusing feature maps and structural masks of different modalities.

[0120] S33, in the modal attention layer, the Canny operator is used to extract anatomical edge features of different modalities, and the edge feature map is spatially superimposed with the feature map output by the backbone network to obtain a feature map that enhances anatomical perception.

[0121] S34, In the modal attention layer, the region growing algorithm is used to locate the potential lesion region, and the lesion region proposal box is spatially registered with the anatomical perception feature map to generate a composite feature map that integrates anatomical structure and lesion distribution.

[0122] S35, a morphological attention layer is set after the pooling layer of the backbone network. The morphological attention layer calculates the morphological significance weight by fusing lesion morphological features and density distribution features.

[0123] S36, in the morphological attention layer, the gray-level gradient statistical algorithm is used to extract the edge sharpness and density uniformity features of the lesion area, and the gradient statistical features are fused with the feature map output by the backbone network across layers.

[0124] S37, in the morphological attention layer, morphological erosion operation is used to eliminate the interference of artifact regions and generate a heat map of the core lesion region. The heat map and the morphological feature map are then spliced ​​together to obtain a feature map that integrates the prior information of the lesion.

[0125] In one feasible implementation, step S34, generating a composite feature map integrating anatomical structures and lesion distribution, includes:

[0126] A sliding window scanning algorithm is used to traverse the input image at multiple scales and calculate the probability distribution of anatomical landmarks within each window.

[0127] Morphological opening operations are performed on windows with probability distributions higher than a threshold to filter out candidate lesion regions;

[0128] Map the coordinates of the candidate region to the feature map space, and extract the composite feature map of the corresponding region.

[0129] The extracted feature map is scaled to a preset size using nearest neighbor interpolation, and the correlation weights between the anatomy and the lesion are calculated by inputting the modal attention layer.

[0130] In one feasible implementation, step S36 involves cross-layer fusion of gradient statistical features and feature maps output by the backbone network, including:

[0131] The three parameters extracted by the gray-scale gradient statistical algorithm—edge intensity, gradient direction, and curvature—are concatenated into a morphological description vector.

[0132] The morphological description vector is expanded to the same channel dimension as the backbone network feature map through convolutional layers;

[0133] The expanded morphological feature vectors are weighted and fused with the backbone network feature map according to the spatial dimension, and the weights are dynamically adjusted by the morphological saliency score.

[0134] In one feasible implementation, step S37, generating a heat map of the core region of the lesion, includes:

[0135] Threshold segmentation is performed on the input image to extract candidate lesion regions;

[0136] A connected component labeling algorithm is used to merge broken regions and remove isolated noise points to form a complete lesion region.

[0137] Calculate the maximum density projection from each pixel to the lesion core to generate a density attenuation heatmap;

[0138] Multiply the heatmap with the feature map of the morphological attention layer channel by channel to enhance the response intensity of the core area of ​​the lesion.

[0139] In one feasible implementation, Figure 3 The flowchart illustrates the construction of a cross-modal association model based on dynamic feature fusion. The specific process for constructing the cross-modal association model based on dynamic feature fusion in step S3 is as follows:

[0140] First, the VGG network is used as the backbone network for the cross-modal association model. The VGG network contains multiple convolutional blocks and pooling layers. Each convolutional block consists of multiple stacked 3×3 convolutional layers, extracting multi-level features from the image through layer-by-layer convolution, from low-level edge and texture features to high-level semantic features. The pooling layers are used to reduce the spatial dimension of the feature maps, expand the receptive field, and improve the robustness of the model.

[0141] A modal attention mechanism is introduced after the convolutional blocks of the backbone network to construct a modal attention layer. This layer calculates structure-guided attention weights by fusing feature maps and structural masks from different modalities. Specifically, the Canny operator is used to extract anatomical edge features from different modalities. The Canny operator accurately detects edge pixels in the image through steps such as Gaussian filtering, gradient calculation, non-maximum suppression, and hysteresis thresholding, generating edge feature maps. The edge feature maps are then spatially superimposed with the feature maps output by the backbone network, i.e., feature values ​​at the same spatial location are added together to obtain a feature map that enhances anatomical perception, making the model more focused on the boundaries and contours of anatomical structures.

[0142] In the modal attention layer, a region growing algorithm is used to locate potential lesion regions. Starting from a seed point (such as a pre-defined initial lesion point segmented by a threshold), the algorithm gradually merges adjacent pixels based on pixel grayscale similarity and spatial continuity criteria to form candidate lesion regions. The lesion region proposal box (i.e., the bounding rectangle of the candidate region) is spatially registered with the anatomical perception feature map to ensure that the proposal box and the anatomical structure in the feature map are consistent, generating a composite feature map that fuses the anatomical structure and lesion distribution. This feature map simultaneously contains information about the anatomical background and potential lesions.

[0143] A morphological attention layer is set after the pooling layer of the backbone network. This layer calculates morphological saliency weights by fusing lesion morphological features (such as shape and edge smoothness) and density distribution features (such as gray-level uniformity) to highlight areas that conform to typical lesion morphology. Specifically, the edge sharpness (such as the average gradient magnitude) and density uniformity features (such as the variance of gray values) of the lesion region are extracted through a gray-level gradient statistical algorithm. These gradient statistical features are then fused across layers with the feature map output by the backbone network to enhance the model's sensitivity to lesion morphology.

[0144] Finally, in the morphological attention layer, morphological erosion is used to eliminate interference from artifact regions. The erosion operation removes edge pixels that do not match the structuring element by sliding the structuring element, thereby removing small noise regions or non-lesion protrusions and generating a heatmap of the lesion core region. The heatmap and the morphological feature map are then concatenated channel by channel to obtain a feature map that incorporates prior lesion information, providing cleaner lesion features for subsequent feature fusion.

[0145] The specific operation is illustrated using a dual-modal image analysis of liver CT and MRI. First, a VGG16 network is used as the backbone network, containing 13 convolutional layers and 3 pooling layers. Taking a liver CT arterial phase image as an example, with an input size of 512×512 pixels, the first convolutional block (two 3×3 convolutional layers, stride 1) extracts the edge features of the hepatic arteries, outputting a 64-channel feature map. This is then compressed to 256×256 pixels by a max pooling layer (pooling kernel 2×2, stride 2) to capture the coarse-grained structure of intrahepatic vessels. For contrast-enhanced T1-weighted MRI images at the same level, the backbone network processes the images to generate corresponding high-level semantic feature maps, such as the contrast features between the liver parenchyma and the tumor.

[0146] A modal attention mechanism is introduced after the convolutional block. Taking a hemangioma in the left lobe of the liver as an example, the Canny operator is used to extract the edge of the liver capsule from CT images and the tumor edge from MRI images, generating edge feature maps. The liver capsule edge feature map from CT is superimposed with the liver parenchyma feature map output by the backbone network to highlight the anatomical structure of the liver boundary; at the same time, the tumor edge feature map from MRI is superimposed with the corresponding feature map to enhance the perception of the tumor contour. For example, in the CT feature map, the gradient value of the liver capsule edge is enhanced by 30%, so that the model prioritizes the anatomical boundary of the liver during feature fusion, avoiding misclassification of extrahepatic tissue as lesions.

[0147] When using the region growing algorithm to locate potential lesion regions, taking a low-density nodule in the left lobe of the liver in CT images as an example, a seed point (gray value 20 HU lower than the liver parenchyma) is manually set at the center of the nodule, with a growth threshold of ±10 HU and a neighborhood range of 4 connected components. Starting from the seed point, the algorithm merges pixels with gray values ​​within the range of [seed gray value - 10, seed gray value + 10] in adjacent pixels, gradually forming candidate regions. After removing vascular cross-sectional noise within the region through morphological opening operations (structuring element is a 5×5 disk), a candidate region proposal box containing hemangioma is obtained. Its coordinates are mapped to the feature map space, and a composite feature map (64×64 pixels) of the corresponding region is extracted. This feature map simultaneously contains the low-density features of the nodule in the CT image and the anatomical structural features of the liver capsule.

[0148] A morphological attention layer is set after the pooling layer. Taking hepatocellular carcinoma in the right lobe of the liver in T2-weighted MRI images as an example, edge sharpness and density uniformity features are extracted using a gray-level gradient statistical algorithm. The mean gradient magnitude of the tumor edge pixels is calculated to be 15 (reflecting edge sharpness), and the gray-level variance is 80 (reflecting density uniformity). These two parameters are concatenated with the curvature parameter (calculated as 0.02 using Gaussian curvature) to form a morphological description vector. After expanding the vector to 128 channels, the same as the backbone network feature map, through a 1×1 convolutional layer, it is weighted and fused with the feature map output from the pooling layer. For example, in the tumor edge region, the morphological saliency score is high (0.8), and the weight of the morphological feature vector in this region is increased to 60%, enhancing the response of malignant morphological features such as spiculation at the tumor edge.

[0149] When using morphological erosion to eliminate artifacts, taking the hepatic hilar lymph nodes in CT enhanced venous phase images as an example, the initially extracted candidate lymph node regions are eroded (structural element is a 3×3 square, iterated twice) to remove blurring artifacts caused by partial volume effects at the edges, generating a heatmap of the lymph node core region. After concatenating the heatmap with the morphological feature map channel, a feature map fusing lymph node density and morphological priors is obtained. The pixel values ​​in the core region are increased by 40% compared to before erosion, while those in non-core regions are reduced by 20%, allowing the model to focus more on the solid portion of the lymph node and reducing interference from artifacts such as vascular cross-sections. Through these steps, the cross-modal association model can effectively fuse anatomical priors and lesion morphological features, improving the feature representation ability of lesions in bimodal images. Figure 4 A flowchart for generating a composite feature map that integrates anatomical structures and lesion distribution.

[0150] In one feasible implementation, the specific steps for generating the composite feature map integrating anatomical structures and lesion distribution in step S34 are as follows: A sliding window scanning algorithm is used to traverse the input image at multiple scales. The sliding window moves row by row and column by column on the image at different sizes (e.g., small windows capture details, large windows capture the overall structure), and the probability distribution of anatomical landmarks (such as those extracted in Example 1) within each window is calculated. The probability distribution is obtained by statistically analyzing the ratio of the number of anatomical landmarks within the window to the total area of ​​the window, reflecting the likelihood that the window region belongs to a key location of the anatomical structure.

[0151] Windowing with a probability distribution higher than a preset threshold is processed using morphological opening. This morphological opening operation, which involves erosion followed by dilation, removes isolated noise points at the window edges, smooths the window boundaries, and filters out more reliable lesion candidate regions. These candidate regions contain anatomical landmarks and exhibit high structural relevance, suggesting the potential presence of lesions related to anatomical structures.

[0152] The coordinates of the selected candidate regions are mapped to the feature map space. Due to the size difference between the input image and the feature map (e.g., the feature map size is reduced after pooling in the backbone network), a coordinate scaling formula is used to convert the candidate region coordinates in the image space to coordinates in the feature map space, ensuring accurate positional correspondence. Based on the mapped coordinates, a composite feature map of the corresponding region is extracted from the feature map. This feature map contains the anatomical structural features and potential lesion features of the candidate region.

[0153] Finally, the extracted feature map is scaled to a preset size (such as the standard size of the model input) using nearest neighbor interpolation and then input into the modal attention layer. In the modal attention layer, the association weights between anatomical structures and lesions are calculated based on the distribution of anatomical landmarks and the location of lesion candidate regions. This dynamically adjusts the importance of different regions during feature fusion, enhancing the model's ability to capture lesion features guided by anatomical structures.

[0154] The specific operation is illustrated using a dual-modal image analysis of brain CT and MRI. Assuming the input is a patient's plain CT scan image (showing a suspected lacunar infarction in the right basal ganglia) and a T2-weighted MRI image (showing a high-signal lesion in the corresponding area), a sliding window scan is first performed on the CT image. The window sizes are set to 10×10 pixels (capturing small blood vessel details), 20×20 pixels (capturing brain sulci and gyri), and 50×50 pixels (capturing the overall morphology of the brain lobes). Taking the 20×20 pixel window as an example, the scan moves line by line starting from the upper left corner of the image, with a step size of 5 pixels. The probability distribution of anatomical landmarks (such as the apex of the anterior horn of the lateral ventricle and the origin of the middle cerebral artery) within each window is calculated. For example, in a window in the right basal ganglia region, if the coordinates of the apex of the anterior horn of the lateral ventricle are detected within 3 pixels of the window center, the probability of this anatomical landmark being an anatomical landmark is calculated to be 0.8 (threshold set to 0.5).

[0155] For windows with a probability higher than 0.5, morphological opening operations were performed using 5×5 rhomboid structural elements. Taking a window with a probability of 0.8 as an example, the opening operation first eroded and removed isolated pixels within 3 pixels of the window edge (such as noise points in the cerebrospinal fluid space), then expanded to restore the window size, filtering out candidate lesion regions containing anatomical landmarks of the basal ganglia. This region is roughly circular, with a diameter of about 8 mm, corresponding to low-density lesions in CT images. The image coordinates (x=200-220, y=180-200, z=30) of this candidate region were mapped to the feature map space output by the backbone network (the feature map size is 1 / 4 of the input image, i.e., 128×128 pixels). The feature map coordinates (x=50-55, y=45-50, z=30) were obtained through the coordinate scaling formula (feature map coordinates = image coordinates / 4). The composite feature map of this region (size 15×15 pixels) was extracted, which includes the gray-white matter contrast features of the basal ganglia in CT images and the edema features with high T2 signal in MRI images.

[0156] The extracted feature map is scaled to a preset size (64×64 pixels) using nearest neighbor interpolation and input into the modal attention layer. In the modal attention layer, if the spatial distance between the anatomical landmark (anterior horn of the lateral ventricle) and the candidate lesion region is calculated to be 2 mm (less than the preset threshold of 5 mm), it is considered an anatomically relevant region, and an anatomically guided attention weight matrix is ​​generated. For example, at the corresponding position in the feature map, the weight of anatomical structural features (such as the low-density CT features of the lateral ventricle edge) is increased to 0.7, and the weight of lesion distribution features (such as high signal on MRI) is 0.3. Weighted fusion enhances the response of lesion features adjacent to anatomical structures. After this processing, when analyzing this region, the model prioritizes lesion features related to the anatomical location of the anterior horn of the lateral ventricle, suppressing artifact interference from areas far from anatomical structures, thereby more accurately identifying the boundary between lacunar infarct lesions and surrounding normal tissue. The entire process, through multi-scale window traversal, anatomical landmark probability analysis, and spatial registration, achieves dynamic fusion of anatomical structure and lesion distribution characteristics, thereby improving the cross-modal association model's sensitivity to lesion detection in the anatomical context.

[0157] In one feasible implementation, the specific process of cross-layer fusion of gradient statistical features and backbone network feature maps in step S36 is as follows:

[0158] Three parameters—edge intensity, gradient direction, and curvature—are extracted using a grayscale gradient statistical algorithm. Edge intensity is obtained by calculating the gradient magnitude of the pixel neighborhood, reflecting the prominence of the edge; gradient direction is the direction angle of the gradient vector, describing the direction of the edge; and curvature is calculated using the second derivative, reflecting the degree of edge bending. These three parameters are concatenated into a morphological description vector, which characterizes the morphological features of the lesion region from multiple dimensions.

[0159] The morphological description vector is processed by convolutional layers to expand its channel dimension to the same number of channels as the backbone network feature map. For example, if the backbone network feature map has 64 channels and the morphological description vector initially has 3 channels, it is expanded to 64 channels through a 1×1 convolutional layer to ensure that the two are consistent in channel dimension, which facilitates subsequent fusion operations.

[0160] The expanded morphological feature vectors are weighted and fused with the backbone network feature map according to the spatial dimension. Specifically, at each spatial location (e.g., each pixel in the feature map), the fusion weights are dynamically adjusted based on the morphological saliency score (calculated from the morphological description vector, reflecting the probability that the location belongs to a lesion). The higher the morphological saliency score, the greater the weight of the morphological feature vector at that location, thereby highlighting areas that conform to the lesion morphology in the fused feature map, suppressing interference from normal tissue, and improving the model's sensitivity and specificity to lesion morphological features.

[0161] The specific operation is carried out using a dual-modal analysis of mammography and ultrasound images as an example. Assuming the analysis object is a suspicious nodule in the upper outer quadrant of the left breast, the mammography image shows the nodule as high-density with blurred edges, while the ultrasound image shows unclear nodule boundaries and uneven internal echoes. First, the edge intensity, gradient direction, and curvature parameters of the nodule in the mammography image are extracted using a gray-scale gradient statistical algorithm: the edge intensity is taken as the average value of the pixel gradient amplitude at the nodule edge, which is 25 (reflecting the degree of edge blurring); the gradient direction is distributed in the range of 120°-150° (indicating irregular edge direction); and the curvature is calculated as 0.03 by fitting the nodule edge curve (reflecting the degree of edge concavity / convexity). These three parameters are concatenated into a morphological description vector [25, 135°, 0.03], which characterizes the irregular features of the nodule edge.

[0162] Since the backbone network (such as VGG16) has 128 channels in its feature map at this level, a 1×1 convolutional layer is used to expand the morphological description vector through these channels. The weight matrix of the convolutional layer is 128×3, and the bias vector is 128-dimensional. A linear transformation expands the 3-dimensional vector into a 128-dimensional morphological feature vector. For example, the edge intensity parameter 25 in the vector, after convolution, is mapped to edge-related feature values ​​(such as 0.6-0.8) in the first 40 channels of the new vector; the gradient direction parameter 135° is mapped to direction-sensitive feature values ​​(such as 0.7-0.9) in the middle 40 channels; and the curvature parameter 0.03 is mapped to curvature-related feature values ​​(such as 0.5-0.7) in the last 48 channels, forming a morphological feature representation with the same dimensions as the backbone network's feature map.

[0163] The expanded morphological feature vector is then fused with the feature map output by the backbone network using spatial weighting. Taking the feature map of the corresponding nodule region in the ultrasound image as an example, the feature vector at a certain spatial location (i,j) in the backbone network feature map is [0.4,0.6,0.5,...] (128 dimensions). The morphological saliency score of the morphological feature vector at this location is calculated to be 0.7 using the Softmax function (a higher score indicates a stronger correlation with malignant morphological features). During fusion, the feature vector at this location is updated to: backbone feature vector × (1-0.7) + morphological feature vector × 0.7, i.e., [0.4×0.3+0.6×0.7,0.6×0.3+0.7×0.7,...], which enhances the feature response of regions with blurred edges, chaotic orientations, and high curvature. In subsequent feature propagation, the fused feature map will highlight areas conforming to malignant morphological criteria, such as spiculation in mammography and angular features in ultrasound, while suppressing the smooth edge features of benign lesions such as fibroadenomas, thereby improving the model's ability to specifically identify breast cancer lesions. The entire fusion process, through dynamic weight adjustment, achieves an organic combination of gradient statistical features and deep semantic features, enhancing the model's ability to perceive and utilize lesion morphological details.

[0164] In one feasible implementation, for generating the heatmap of the core lesion region in step S37, the input image is first subjected to threshold segmentation. Based on the image's grayscale distribution, an appropriate threshold (such as the threshold automatically calculated by the Otsu algorithm) is selected to divide the image into foreground (candidate lesion region) and background (normal tissue), extracting preliminary candidate lesion regions. These candidate regions may contain multiple discontinuous sub-regions or isolated noise points. A connected component labeling algorithm is then used to process the threshold-segmented candidate regions. The algorithm iterates through each pixel, marking adjacent pixels with the same grayscale value as the same connected component, merging broken sub-regions, and eliminating isolated noise points with excessively small areas based on the area, perimeter, and other characteristics of the connected components, forming a complete lesion region and ensuring the accuracy of the heatmap.

[0165] Calculate the maximum density projection from each pixel to the lesion core. The lesion core is defined as the geometric center or the point of highest density in the lesion region. The maximum density projection is generated by calculating the maximum gray value along the path from the pixel to the core in a certain direction, producing a density decay heatmap. The higher the pixel value in the heatmap, the closer the location is to the lesion core, the greater the density, and the higher the probability that it belongs to a real lesion.

[0166] The generated heatmap is multiplied channel-by-channel with the feature map of the morphological attention layer. At each channel of the feature map, the corresponding pixel value is multiplied by the pixel value of the heatmap, thereby enhancing the response intensity of the core lesion region in the feature map, suppressing interference from edge artifacts and non-core regions, allowing the model to focus more on the core part of the lesion, and improving the accuracy and reliability of subsequent lesion detection.

[0167] S4: Collect image samples with different lesion types and imaging parameters, perform modal pairing of the samples according to lesion morphology and imaging mode, and construct a dual-modal joint dataset;

[0168] In one feasible implementation, Figure 5 A flowchart for constructing a bimodal joint dataset; in step S4, the bimodal joint dataset is constructed, including:

[0169] S41 collects medical image data from different imaging devices, including plain CT scans, enhanced CT scans, and T1 / T2 weighted MRI.

[0170] S42 collects imaging data of different lesion stages, including early lesions, progressive lesions, and late lesions;

[0171] S43, paired datasets combined by imaging device and lesion stage, with each combination category containing at least ten samples from different anatomical locations;

[0172] S44: Extract the acquisition parameters and lesion coordinates of each image, and generate three-dimensional coordinate data of the lesion center point through spatial transformation;

[0173] S45, image data is associated with imaging parameters and lesion stage by triplets to form data units with multi-dimensional attribute labels;

[0174] S46 constructs a tree-like index structure based on multi-dimensional attribute tags to enable rapid sample retrieval by device type and lesion stage.

[0175] In one feasible implementation, step S46, constructing a tree-like index structure based on multi-dimensional attribute labels, includes:

[0176] The data units are input into the improved DBSCAN clustering algorithm, and the imaging parameter similarity threshold and spatial distribution radius are set.

[0177] For samples that meet the threshold conditions, a primary cluster is generated according to the imaging device, and a secondary sub-cluster is performed within each primary cluster according to the lesion stage;

[0178] Each sub-cluster is assigned a unique identifier, which includes the device type code and the lesion stage code;

[0179] It uses a B+ tree structure to store the mapping relationship between identifier codes and data unit addresses, and supports multi-condition combined retrieval.

[0180] In one feasible implementation, when constructing the dual-modal joint dataset in step S4, medical image data from different imaging devices are collected, including plain CT scans, enhanced CT scans, and T1 / T2 weighted MRI, covering multiple imaging modes; image data from different lesion stages (early, advanced, and late stages) are also collected to ensure that the dataset contains information on the entire lifecycle of lesion development. Data sets are paired according to imaging device and lesion stage, with each combination category containing at least ten samples from different anatomical locations to ensure data diversity. Acquisition parameters (such as tube voltage and slice spacing) and lesion coordinates are extracted from each image. Three-dimensional coordinate data of the lesion center point is generated through spatial transformation. Image data is associated with imaging parameters and lesion stage using triplet annotations to form data units with multi-dimensional attribute labels. A tree-like index structure is constructed based on these multi-dimensional attribute labels. The data units are clustered using an improved DBSCAN clustering algorithm, generating first-level clusters according to imaging device, and second-level sub-clusters within each cluster according to lesion stage. A B+ tree structure is used to store the mapping relationship between identifier codes and data unit addresses, enabling fast retrieval.

[0181] S5, associates and labels the bimodal joint dataset with image acquisition parameters to generate a training dataset with modal attributes;

[0182] S6. The cross-modal association model is trained in multiple stages using a training dataset with modal attributes. The preprocessed bimodal images are input into the trained model for feature fusion, and the enhanced fused images are output.

[0183] In one feasible implementation, step S6 involves performing multi-stage iterative training of the cross-modal association model using a training dataset with modal attributes, including:

[0184] S61, extract the imaging device and lesion stage labels of the input image, and retrieve matching historical training samples through B+ tree index;

[0185] S62, the feature distribution of historical samples is aligned with the new samples for domain adaptation, and an adversarial generative algorithm is used to eliminate modal differences;

[0186] S63 calculates the Euclidean distance between the feature spaces of the new and old samples and dynamically adjusts the iterative learning rate; the larger the distance, the higher the learning rate.

[0187] S7 generates a lesion probability map based on the anatomical structure distribution of the fused image, locates the lesion area through the multi-threshold segmentation module, and outputs the detection results.

[0188] In one feasible implementation, step S7, locating the lesion area and outputting the detection result, includes:

[0189] The confidence level is determined based on the probability map of the lesion, and a multimodal review mechanism is triggered when the probability is lower than the threshold.

[0190] In the multi-threshold segmentation module, three criteria are defined: density threshold, morphological integrity, and anatomical conformity.

[0191] The lesion segmentation results are generated by merging regions to take into account both sensitivity and specificity.

[0192] The segmentation results are converted into 3D labeled images, and the consistency between the segmentation boundaries and the anatomical structures is verified in real time.

[0193] In one feasible implementation, during the model training phase (step S6), the imaging device and lesion stage labels of the input image are extracted. Matching historical training samples are retrieved using a B+ tree index. The historical samples are then aligned with the new samples through domain adaptation. An adversarial generative algorithm is used to eliminate modal differences. The Euclidean distance between the feature spaces of the old and new samples is calculated, and the iterative learning rate is dynamically adjusted. During the lesion detection phase (step S7), confidence levels are determined based on the lesion probability map. A multimodal verification mechanism is triggered for low-probability regions. A multi-threshold segmentation module integrates density thresholds, morphological integrity, and anatomical conformity criteria to generate lesion segmentation results, which are then converted into 3D labeled maps. The consistency between the segmentation boundaries and the anatomical structures is verified in real time.

[0194] The specific operation will be carried out using the detection of pulmonary nodules in lung CT images as an example. Assume the input is a patient's enhanced chest CT image, and the target lesion is a solid nodule in the right middle lobe of the lung (approximately 15mm in diameter, with visible lobulation at the margin). First, threshold segmentation is performed on the CT image, and the global threshold is calculated using the Otsu algorithm. The algorithm determines the optimal segmentation threshold by maximizing the inter-class variance between the foreground and background, using the following formula:

[0195]

[0196] in, and These are the probability densities of background and foreground pixels, respectively. and These are the average grayscale values ​​of the background and foreground pixels, respectively. The threshold is calculated. Regions with gray values ​​higher than T are classified as candidate nodule regions, and regions containing the main body of the nodule and part of the blood vessel cross-section are initially extracted.

[0197] A connected component labeling algorithm is used to process candidate regions. Each pixel in the image is traversed, and adjacent pixels with gray values ​​all higher than [a certain value] are selected. Pixels are labeled as belonging to the same connected region. Connected regions with an area less than 30 pixels² (such as blood vessel cross-sections) are considered noise points and removed; the largest connected region is retained as the complete lesion area, which contains the solid part of the nodule and a small amount of surrounding inflammatory exudate. Through this step, the originally scattered nodule pixels are merged into a single connected region, eliminating the breakage phenomenon caused by partial volume effect.

[0198] Calculate the maximum density projection from each pixel to the lesion core. Define the lesion core as the geometric center of the connected component. For each pixel Calculate the maximum density value from the core along the coordinate axis. The formula is:

[0199]

[0200] In this context, the CT value reflects tissue density, and the maximum value along the path represents the maximum density feature between that pixel and the core. For example, if a pixel at the edge of a nodule travels along the x-axis towards the core, passing through a blood vessel section (CT value 150 HU) and the nodule parenchyma (CT value 80 HU), then... The core pixel of the nodule .according to Generate a density decay heatmap; the higher the pixel value (the brighter the color), the closer it is to the core or the higher the density.

[0201] Finally, the heatmap is multiplied channel-by-channel with the feature map of the morphological attention layer. Taking a certain channel of the feature map as an example, this channel is sensitive to high-density structures. The feature value of the nodule core region in this channel is 0.9, while the feature value of the peripheral vascular cross-section region is 0.5. In the heatmap, the pixel value of the core region is 1.0, and that of the vascular cross-section region is 0.3. After multiplication, the feature value of the core region remains at 0.9, while that of the vascular cross-section region decreases to 0.15, effectively suppressing artifact interference. Through the above operations, the feature response of the lesion core region is significantly enhanced, and the interference of peripheral non-nodular structures is weakened. In subsequent weighted processing, the model can more accurately focus on the solid part of the nodule, improving the ability to distinguish between benign and malignant nodules.

[0202] Figure 6 This is a block diagram of an intelligent detection device that integrates medical image learning images according to an embodiment of the present invention. This device is used in an intelligent detection method that integrates medical image learning images. (Refer to...) Figure 6 The device includes a data acquisition and preprocessing module 310, a feature extraction module 320, a model building module 330, a dataset building module 340, a model training and fusion module 350, and a lesion detection module 360. Among them:

[0203] The data acquisition and preprocessing module 310 is used to acquire two different modalities of medical image data and preprocess them separately to obtain denoised dual-modal images.

[0204] The feature extraction module 320 is used to perform multi-scale decomposition on the preprocessed dual-modal image and extract multi-resolution feature maps.

[0205] The model building module 330 is used to build a cross-modal association model based on dynamic feature fusion. Modal attention mechanism and morphological attention mechanism are set in the model. The modal attention mechanism embeds prior information of anatomical structure of different modalities to guide feature complementarity, and the morphological attention mechanism sets morphological constraint weights of lesion areas.

[0206] The dataset construction module 340 is used to collect image samples of different lesion types and imaging parameters. Based on the lesion morphology and imaging mode, the samples are modally paired to construct a dual-modal joint dataset. The dual-modal joint dataset is then associated and labeled with the image acquisition parameters to generate a training dataset with modal attributes.

[0207] The model training and fusion module 350 is used to perform multi-stage iterative training of the cross-modal association model using a training dataset with modal attributes. It inputs the preprocessed bimodal images into the trained model for feature fusion and outputs the enhanced fused image.

[0208] The lesion detection module 360 ​​is used to generate a lesion probability map based on the anatomical structure distribution of the fused image, locate the lesion area through the multi-threshold segmentation module, and output the detection results.

[0209] Figure 7 This is a schematic diagram of the structure of an intelligent detection device that integrates medical image learning images according to an embodiment of the present invention, as shown below. Figure 7 As shown, the intelligent detection device that integrates medical image learning can include the above-mentioned Figure 6 The intelligent detection device shown is a medical image learning image fusion device. Optionally, the intelligent detection device fusion medical image learning image fusion device Figure 410 may include a first processor 2001.

[0210] Optionally, the intelligent detection device Figure 410, which integrates medical image learning images, may also include a memory 2002 and a transceiver 2003.

[0211] The first processor 2001, memory 2002, and transceiver 2003 can be connected via a communication bus.

[0212] The following is combined Figure 7 The components of the intelligent detection device (Figure 410) that integrates medical imaging and image learning are described in detail below:

[0213] The first processor 2001 is the control center of the intelligent detection device Figure 410 that integrates medical image learning. It can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0214] Optionally, the first processor 2001 can perform various functions of the intelligent detection device Figure 410 that integrates medical image learning images by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0215] In a specific implementation, as one example, the first processor 2001 may include one or more CPUs, for example... Figure 7 CPU0 and CPU1 are shown in the diagram.

[0216] In a specific implementation, as one example, the intelligent detection device Figure 410, which integrates medical image learning images, may also include multiple processors, for example... Figure 7 The first processor 2001 and the second processor 2004 are shown in the diagram. Each of these processors can be a single-core processor or a multi-core processor. Here, a processor can refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0217] The memory 2002 is used to store the software program that executes the present invention, and is controlled by the first processor 2001 to execute it. The specific implementation method can be referred to the above method embodiment, and will not be repeated here.

[0218] Optionally, the memory 2002 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 2002 may be integrated with the first processor 2001 or exist independently, and may be connected via the interface circuit of the intelligent detection device for learning images from medical images in Figure 410 (Figure 410). Figure 7 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0219] The transceiver 2003 is used to communicate with network devices or with terminal devices.

[0220] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0221] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0222] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0223] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An intelligent detection method integrating medical image learning images, characterized in that, The method includes: S1: Acquire medical image data of two different modalities and preprocess them separately to obtain denoised dual-modal images; S2, perform multi-scale decomposition on the preprocessed dual-modal image and extract multi-resolution feature maps; S3. Construct a cross-modal association model based on dynamic feature fusion, and set up modal attention mechanism and morphological attention mechanism in the model; wherein, the modal attention mechanism embeds prior information of anatomical structure of different modalities to guide feature complementarity; the morphological attention mechanism sets morphological constraint weights of lesion area. S4: Collect image samples with different lesion types and imaging parameters, perform modal pairing of the samples according to lesion morphology and imaging mode, and construct a dual-modal joint dataset; S5, associates and labels the bimodal joint dataset with image acquisition parameters to generate a training dataset with modal attributes; S6. The cross-modal association model is trained in multiple stages using a training dataset with modal attributes. The preprocessed bimodal images are input into the trained model for feature fusion, and the enhanced fused images are output. S7 generates a lesion probability map based on the anatomical structure distribution of the fused image, locates the lesion area through the multi-threshold segmentation module, and outputs the detection results.

2. The intelligent detection method for integrating medical image learning according to claim 1, characterized in that, In step S1, two different modalities of medical image data are acquired and preprocessed separately, including: S11: Acquire medical image data in two different modalities; use an anisotropic diffusion algorithm to suppress noise in the original image; use an adaptive histogram equalization algorithm to enhance the contrast of the denoised image to obtain the enhanced image; S12 uses a pre-trained ResUNet segmentation model to segment organ regions in enhanced images. Morphological closing operations are used to divide the images into suspected lesion regions and normal tissue regions, generating anatomical structure mask maps. S13, on the anatomical structure mask map, the watershed algorithm is used to divide the sub-regions, and the pixels with continuously changing gray levels are aggregated into homogeneous sub-regions by gradient thresholding to extract the topological structure data of the lesion edge. S14, On the lesion topology data, the spatial correspondence of multimodal images is reconstructed using the mutual information registration algorithm, generating bimodal registration parameters that include anatomical landmarks and lesion locations; S15 aligns the dual-modal registration parameters with the image coordinate system through spatial transformation, and generates a cross-modal anatomical structure mapping matrix by combining the three-dimensional reconstruction model.

3. The intelligent detection method for integrating medical image learning according to claim 1, characterized in that, In step S3, a cross-modal association model based on dynamic feature fusion is constructed, including: S31 uses the VGG network as the backbone network of the cross-modal association model. The backbone network contains convolutional blocks and pooling layers to extract multi-level features. S32 introduces a modal attention mechanism after the convolutional block of the backbone network to construct a modal attention layer. This layer generates structure-guided attention weights by fusing feature maps and structural masks of different modalities. S33, in the modal attention layer, the Canny operator is used to extract anatomical edge features of different modalities, and the edge feature map is spatially superimposed with the feature map output by the backbone network to obtain a feature map that enhances anatomical perception. S34, In the modal attention layer, the region growing algorithm is used to locate the potential lesion region, and the lesion region proposal box is spatially registered with the anatomical perception feature map to generate a composite feature map that integrates anatomical structure and lesion distribution. S35, a morphological attention layer is set after the pooling layer of the backbone network. The morphological attention layer calculates the morphological significance weight by fusing lesion morphological features and density distribution features. S36, in the morphological attention layer, the gray-level gradient statistical algorithm is used to extract the edge sharpness and density uniformity features of the lesion area, and the gradient statistical features are fused with the feature map output by the backbone network across layers. S37, in the morphological attention layer, morphological erosion operation is used to eliminate the interference of artifact regions and generate a heat map of the core lesion region. The heat map and the morphological feature map are then spliced ​​together to obtain a feature map that integrates the prior information of the lesion.

4. The intelligent detection method for integrating medical image learning according to claim 3, characterized in that, In step S34, a composite feature map of the fused anatomical structure and lesion distribution is generated, including: A sliding window scanning algorithm is used to traverse the input image at multiple scales and calculate the probability distribution of anatomical landmarks within each window. Morphological opening operations are performed on windows with probability distributions higher than a threshold to filter out candidate lesion regions; Map the coordinates of the candidate region to the feature map space, and extract the composite feature map of the corresponding region. The extracted feature map is scaled to a preset size using nearest neighbor interpolation, and the correlation weights between the anatomy and the lesion are calculated by inputting the modal attention layer.

5. The intelligent detection method for integrating medical image learning according to claim 3, characterized in that, In step S36, the gradient statistical features are fused with the feature map output by the backbone network across layers, including: The three parameters extracted by the gray-scale gradient statistical algorithm—edge intensity, gradient direction, and curvature—are concatenated into a morphological description vector. The morphological description vector is expanded to the same channel dimension as the backbone network feature map through convolutional layers; The expanded morphological feature vectors are weighted and fused with the backbone network feature map according to the spatial dimension, and the weights are dynamically adjusted by the morphological saliency score.

6. The intelligent detection method for integrating medical image learning according to claim 3, characterized in that, In step S37, a heat map of the core area of ​​the lesion is generated, including: Threshold segmentation is performed on the input image to extract candidate lesion regions; A connected component labeling algorithm is used to merge broken regions and remove isolated noise points to form a complete lesion region. Calculate the maximum density projection from each pixel to the lesion core to generate a density attenuation heatmap; Multiply the heatmap with the feature map of the morphological attention layer channel by channel to enhance the response intensity of the core area of ​​the lesion.

7. The intelligent detection method for fused medical image learning images according to any one of claims 2 to 6, characterized in that, In step S4, a bimodal joint dataset is constructed, including: S41 collects medical image data from different imaging devices, including plain CT scans, enhanced CT scans, and T1 / T2 weighted MRI. S42 collects imaging data of different lesion stages, including early lesions, progressive lesions, and late lesions; S43, paired datasets combined by imaging device and lesion stage, with each combination category containing at least ten samples from different anatomical locations; S44: Extract the acquisition parameters and lesion coordinates of each image, and generate three-dimensional coordinate data of the lesion center point through spatial transformation; S45, image data is associated with imaging parameters and lesion stage by triplets to form data units with multi-dimensional attribute labels; S46 constructs a tree-like index structure based on multi-dimensional attribute tags to enable rapid sample retrieval by device type and lesion stage.

8. The intelligent detection method for integrating medical image learning according to claim 7, characterized in that, In step S46, a tree-like index structure is constructed based on multi-dimensional attribute labels, including: The data units are input into the improved DBSCAN clustering algorithm, and the imaging parameter similarity threshold and spatial distribution radius are set. For samples that meet the threshold conditions, a primary cluster is generated according to the imaging device, and a secondary sub-cluster is performed within each primary cluster according to the lesion stage; Each sub-cluster is assigned a unique identifier, which includes the device type code and the lesion stage code; It uses a B+ tree structure to store the mapping relationship between identifier codes and data unit addresses, and supports multi-condition combined retrieval.

9. The intelligent detection method for integrating medical image learning according to claim 8, characterized in that, In step S6, the cross-modal association model is trained in multiple stages using a training dataset with modal attributes, including: S61, extract the imaging device and lesion stage labels of the input image, and retrieve matching historical training samples through B+ tree index; S62, the feature distribution of historical samples is aligned with the new samples for domain adaptation, and an adversarial generative algorithm is used to eliminate modal differences; S63, calculate the Euclidean distance between the feature spaces of the new and old samples, and dynamically adjust the iterative learning rate. The larger the distance, the higher the learning rate. In step S7, the lesion area is located and the detection results are output, including: The confidence level is determined based on the probability map of the lesion, and a multimodal review mechanism is triggered when the probability is lower than the threshold. In the multi-threshold segmentation module, three criteria are defined: density threshold, morphological integrity, and anatomical conformity. The lesion segmentation results are generated by merging regions to take into account both sensitivity and specificity. The segmentation results are converted into 3D labeled images, and the consistency between the segmentation boundaries and the anatomical structures is verified in real time.

10. An intelligent detection device for integrating medical image learning images, wherein the intelligent detection device for integrating medical image learning images is used to implement the intelligent detection method for integrating medical image learning images as described in any one of claims 1-9, characterized in that, The device includes: The data acquisition and preprocessing module is used to acquire medical image data of two different modalities and preprocess them separately to obtain denoised dual-modal images. The feature extraction module is used to perform multi-scale decomposition on the preprocessed dual-modal image and extract multi-resolution feature maps. The model building module is used to build a cross-modal association model based on dynamic feature fusion. Modal attention mechanism and morphological attention mechanism are set in the model. The modal attention mechanism embeds prior information of anatomical structure of different modalities to guide feature complementarity, and the morphological attention mechanism sets morphological constraint weights of lesion areas. The dataset construction module is used to collect image samples of different lesion types and imaging parameters. Based on the lesion morphology and imaging mode, the samples are modally paired to construct a dual-modal joint dataset. The dual-modal joint dataset is then associated and labeled with the image acquisition parameters to generate a training dataset with modal attributes. The model training and fusion module is used to perform multi-stage iterative training of the cross-modal association model using a training dataset with modal attributes. It inputs the preprocessed bimodal images into the trained model for feature fusion and outputs the enhanced fused image. The lesion detection module is used to generate a lesion probability map based on the anatomical structure distribution of the fused image, locate the lesion area through the multi-threshold segmentation module, and output the detection results.

Images