A multi-source data driven hierarchical dynamic fusion lesion recognition system

Abstract

The present application relates to the technical field of lesion recognition, and in particular to a hierarchical dynamic fusion lesion recognition system based on multi-source data driving. The system comprises a CT feature extraction module, an MRI feature extraction module, a PET feature extraction module, an ultrasound feature extraction module, an image semantic feature extraction module, a multi-task learning module, a first fusion module, a second fusion module and a lesion recognition module. The present application extracts CT spatial structure, MRI microscopic texture, PET metabolic activity heterogeneity and ultrasound dynamic motion features, and integrates clinical pathological prior knowledge. By using hierarchical attention, graph convolution and meta-learning strategies, the present application solves the defects of poor feature extraction specificity, lack of scene adaptation and weak small sample generalization ability of existing methods, and realizes fine fusion of multi-modal features and scene adaptive matching.

Description

Technical Field

[0001] This invention relates to the field of lesion identification technology, specifically to a hierarchical dynamic fusion lesion identification system driven by multi-source data. Background Technology

[0002] Medical imaging technology is a core tool for clinical lesion identification. Currently, commonly used clinical medical imaging modalities include computed tomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET), and ultrasound. Each modality has unique diagnostic value: CT excels at presenting the spatial structure and edge morphology of lesions; MRI can clearly display the microscopic texture and soft tissue contrast of lesions; PET can reflect the metabolic heterogeneity of lesions; and ultrasound can capture the dynamic motion characteristics of lesions. However, single-modality imaging has significant limitations. For example, CT has difficulty distinguishing the microscopic differences between benign and malignant lesions; MRI has low sensitivity to calcified lesions; PET is easily affected by scanning noise; and ultrasound has speckle noise and artifacts. Relying solely on a single modality of imaging can easily lead to missed or misdiagnosed lesions, failing to meet the needs of accurate clinical diagnosis.

[0003] Existing multimodal medical image fusion methods often employ simple stitching and fixed-weight fusion, which suffer from the following core defects: First, they fail to incorporate prior clinical pathological knowledge, relying solely on the image data itself, resulting in a lack of pathological relevance in the fused features and difficulty in matching the pathological logic of clinical diagnosis. Second, feature extraction from each modality lacks specificity and fails to adequately adapt to the characteristics of different modalities (such as the structural characteristics of CT, the texture characteristics of MRI, the metabolic characteristics of PET, and the dynamic characteristics of ultrasound), easily leading to feature redundancy or loss of core features. Third, the fusion strategies lack scene adaptability, failing to consider the feature differences of lesions of different sizes and pathological types, resulting in missed detection of microstructural features in small lesions and incomplete extraction of overall features from large lesions. Fourth, lesion annotation data in clinical practice is often small-sample, and existing methods have weak generalization ability in small-sample, multi-scenario scenarios, making it difficult to adapt to the actual needs of clinical applications. Fifth, during the multimodal feature fusion process, the complementarity of features from each modality is not fully explored, leading to problems such as feature dilution and ineffective fusion, affecting the accuracy of lesion identification.

[0004] Therefore, developing a lesion identification system that can incorporate prior pathological knowledge, accurately extract core features of each modality, possess scene adaptability, be suitable for small sample scenarios, and fully leverage the complementary advantages of multimodal features has become a key technical problem that urgently needs to be solved in the field of medical image processing. Summary of the Invention

[0005] To address the shortcomings of existing methods and the needs of practical applications, this invention provides a hierarchical dynamic fusion lesion identification system driven by multi-source data, comprising the following modules: The system comprises the following modules: a CT feature extraction module for extracting CT structural semantic feature maps using an adaptive spatial pyramid; an MRI feature extraction module for extracting MRI multi-sequence texture features using an attention-enhanced frequency-space dual-stream network; a PET feature extraction module for extracting PET metabolic heterogeneity map features using a graph convolutional network; an ultrasound feature extraction module for extracting ultrasound dynamic motion features using a time-aware recurrent network; an image semantic feature extraction module for extracting pathology-guided image semantic features using a medical knowledge graph generative adversarial network; a multi-task learning module for extracting clinically-guided comprehensive image features using multi-task learning; a first fusion module for fusing CT structural semantic feature maps, MRI multi-sequence texture features, and clinically-guided comprehensive image features using a hierarchical attention-based temporal graph convolutional network to obtain hierarchical attention fusion features; a second fusion module for obtaining metabolic dynamic complementary features by combining PET metabolic heterogeneity map features and ultrasound dynamic motion features using a gated cross-attention graph matching algorithm; and a lesion identification module for identifying lesions using meta-learning hierarchical dynamic fusion, utilizing pathology-guided image semantic features, hierarchical attention fusion features, and metabolic dynamic complementary features.

[0006] Optionally, the step of extracting the semantic feature map of CT structure through adaptive spatial pyramid includes the following steps: The pyramid hierarchy and receptive field are dynamically adjusted according to the size of the lesion candidate bounding box to construct an adaptive spatial pyramid; multi-scale structural features are extracted using the adaptive spatial pyramid; based on the multi-scale structural features, a CT structural semantic feature map is generated through cross-level feature residual fusion and semantic enhancement.

[0007] Optionally, the extraction of MRI multi-sequence texture features via an attention-enhanced frequency-space dual-stream network includes the following steps: Frequency domain branch texture features are extracted using two-dimensional fast Fourier transform; spatial domain branch texture features are extracted using a lightweight convolutional network; and sequence-level attention enhancement weighting is achieved using channel attention. The attention-weighted frequency domain texture features and spatial domain texture features are then element-wise summed and fused to generate MRI multi-sequence texture features.

[0008] Optionally, the step of extracting PET metabolic heterogeneity profile features through a graph convolutional network includes the following steps: Based on metabolic activity similarity and spatial adjacency constraints, a metabolic heterogeneity-oriented graph structure is constructed; node metabolic variation coefficients are introduced as attention weights, and a heterogeneity-enhancing graph convolutional layer is designed; based on the graph structure, global heterogeneity features are extracted using the graph convolutional layer to generate PET metabolic heterogeneity map features.

[0009] Optionally, the extraction of ultrasonic dynamic motion features through a time-aware recurrent network includes the following steps: Time-aware lesion keyframes are screened using motion feature values; based on the spatial features of the keyframes, a time-attention-enhanced LSTM model is used to obtain ultrasound dynamic motion features.

[0010] Optionally, the step of extracting pathology-guided image semantic features through a generative adversarial network based on a medical knowledge graph includes the following steps: The lesion subgraphs of the medical knowledge graph are filtered to obtain knowledge embedding vectors; a generator for pathological constraints and a discriminator for pathological perception are constructed; and the knowledge embedding vectors, the generator, and the discriminator are combined to extract pathological-guided image semantic features.

[0011] Optionally, the extraction of clinically guided image synthesis features through multi-task learning includes the following steps: Auxiliary tasks are selected based on the relevance of clinical tasks; a shared-branch multi-task feature extraction network is constructed, and the loss function is optimized according to the imbalance of medical data annotation and the difference in task difficulty; through multi-task joint training with dynamic gradient weighting, comprehensive image features for clinical guidance are obtained.

[0012] Optionally, the step of fusing CT structural semantic feature maps, MRI multi-sequence texture features, and clinically guided image synthesis features through a hierarchical attention temporal graph convolutional network to obtain hierarchical attention fusion features includes the following steps: A hierarchical heterogeneous graph is constructed to realize the structured association of features; an attention weight calculation mechanism is used to obtain the comprehensive attention weight of each node; multidimensional features are obtained through a temporal graph convolutional network, and the hierarchical attention fusion features are obtained by combining the comprehensive attention weight and the multidimensional features.

[0013] Optionally, the graph matching algorithm using gated cross-attention, combined with PET metabolic heterogeneity map features and ultrasound dynamic motion features, to obtain complementary metabolic dynamic features includes the following steps: The graph structure of PET metabolic heterogeneity map features and ultrasound dynamic motion features is reconstructed; bidirectional attention weights are calculated based on gated cross attention, and complementary constraints are introduced to realize topological matching of nodes in the two graphs; the features of the matched node pairs are aggregated, and global complementary fusion features are extracted to generate metabolic dynamic complementary features.

[0014] Optionally, the hierarchical dynamic fusion through meta-learning, utilizing pathology-guided image semantic features, hierarchical attention fusion features, and metabolic dynamic complementary features to complete lesion identification, includes the following steps: Based on functional priority and representation depth, hierarchical modeling of the semantic features of pathology-guided images, hierarchical attention fusion features, and metabolic dynamic complementary features is performed. A meta-learning task set and scene-adaptive hierarchical dynamic weights are constructed and divided. After training through model-independent meta-learning, hierarchical dynamic fusion and lesion identification inference are completed.

[0015] This invention first preprocesses images of each modality to eliminate noise and parameter differences, then extracts core features of each modality (CT spatial structure, MRI microtexture, PET metabolic activity heterogeneity, and ultrasound dynamic motion features). It then incorporates prior clinical and pathological knowledge and employs strategies such as hierarchical attention, graph convolution, and meta-learning to address the shortcomings of existing methods, such as poor feature extraction targeting, lack of scene adaptation, and weak generalization ability with small samples. This achieves refined fusion of multimodal features and scene-adaptive matching. It effectively compensates for the limitations of single-modal imaging, improves the accuracy and comprehensiveness of lesion identification, and is adaptable to multiple organs such as the lungs and liver, as well as various lesion types. It can be widely applied to clinical auxiliary diagnosis, lesion screening, and pathological classification, solving the problems of missed and misdiagnosed lesions in clinical practice. It adapts to the actual clinical needs of small samples and multiple scenarios, providing reliable technical support for precise clinical diagnosis and facilitating the deep integration of medical image processing and clinical diagnosis. Attached Figure Description

[0016] Figure 1 This is a framework diagram of a hierarchical dynamic fusion lesion identification system based on multi-source data driven by an embodiment of the present invention. Detailed Implementation

[0017] Specific embodiments of the present invention will now be described in detail. It should be noted that the embodiments described herein are for illustrative purposes only and are not intended to limit the invention. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the invention. However, it will be apparent to those skilled in the art that these specific details are not necessary to practice the invention. In other instances, well-known circuits, software, or methods have not been specifically described to avoid obscuring the invention.

[0018] Throughout this specification, references to an embodiment, example, or illustration mean that a particular feature, structure, or characteristic described in connection with that embodiment or example is included in at least one embodiment of the invention. Therefore, phrases appearing in various places throughout the specification, such as "in one embodiment," "in an embodiment," "an example," or "an illustration," do not necessarily refer to the same embodiment or example. Furthermore, specific features, structures, or characteristics can be combined in any suitable combination and / or sub-combination in one or more embodiments or examples. Moreover, those skilled in the art will understand that the illustrations provided herein are for illustrative purposes and are not necessarily drawn to scale.

[0019] Please see Figure 1 The present invention provides a hierarchical dynamic fusion lesion identification system based on multi-source data driving, comprising: CT feature extraction module 1, MRI feature extraction module 2, PET feature extraction module 3, ultrasound feature extraction module 4, image semantic feature extraction module 5, multi-task learning module 6, first fusion module 7, second fusion module 8, and lesion identification module 9.

[0020] The CT feature extraction module 1 is used to extract CT structural semantic feature maps through an adaptive spatial pyramid.

[0021] To address the heterogeneity of lesion size in CT images (e.g., small nodules, large masses) and the fact that spatial structure is the core feature of lesions (e.g., lobulation, spiculation, edge morphology), this method dynamically adjusts the pyramid hierarchy and receptive field by changing the candidate bounding box size. It also optimizes the convolution extraction method by considering the sparsity of CT grayscale features, ultimately obtaining a structural semantic feature map that retains the original spatial dimension, avoiding the loss of spatial location information in traditional flattening extraction. The extraction of CT structural semantic feature maps using an adaptive spatial pyramid includes the following steps: S11. Dynamically adjust the pyramid hierarchy and receptive field based on the size of the lesion candidate frame to construct an adaptive spatial pyramid.

[0022] Based on the actual needs of clinical lesion diagnosis, adaptive window width and window level calibration is implemented according to the organ type to which the lesion belongs: for lung lesions, a lung window (window width 1000-1500HU, window level -600--400HU) is used to focus on the gray-scale differences of small nodules in the lung parenchyma; for liver lesions, a liver window (window width 100-150HU, window level 30-50HU) is used to highlight the density distinction between the liver parenchyma and the lesion; during the calibration process, the window width and window level thresholds are dynamically adjusted based on the mean and variance of gray-scale values ​​in the organ region to ensure that the gray-scale details in the lesion region are clearly distinguishable.

[0023] Meanwhile, bilinear interpolation was used to interpolate the CT tomographic sequences to supplement the spatial gap information between the layers and avoid the breakage of inter-layer features. The z-score normalization method was used to normalize the grayscale of the interpolated sequences, mapping all pixel grayscale values ​​to the [-1,1] interval to eliminate the grayscale shift differences caused by different CT equipment and different scanning parameters.

[0024] Furthermore, the Otsu adaptive threshold segmentation algorithm was used to determine the grayscale thresholds of the lesions and the background, and the suspected lesion areas were initially segmented. Then, the grayscale peak points after segmentation were used as seed points, and the segmentation results were further optimized by the region growing method (the growth threshold was set to ±5 HU of the seed point grayscale value). Finally, candidate lesion boxes were selected, and the three-dimensional spatial coordinates (x, y, z) and length, width, and height pixel dimensions of the candidate boxes were accurately marked, providing the core input basis for the subsequent pyramid hierarchy construction.

[0025] Based on this, a dynamic hierarchical allocation mechanism is constructed according to the actual size of the lesion candidate frame: First, the equivalent diameter of the lesion candidate frame is calculated, and the lesions are divided into three categories according to the equivalent diameter: small lesions (equivalent diameter < 5 mm, such as micronodules), medium lesions (5 mm ≤ equivalent diameter ≤ 20 mm), and large lesions (equivalent diameter > 20 mm, such as liver masses); for small lesions, two fine-grained layers are added to the traditional 4 layers to form a 6-layer pyramid structure, among which the newly added 1×1 and 2×2 receptive field layers are specifically designed to capture pixel-level microstructures (such as nodules). Features include spiculation and microcalcification. For medium-sized lesions, the traditional four-layer structure is retained, and microstructure, local edges, regional structures, and overall morphological features are captured through 1×1, 2×2, 4×4, and 8×8 receptive fields respectively. For large lesions, three coarse-grained layers (4×4, 8×8, and 16×16 receptive fields) are retained to focus on capturing macroscopic features such as the overall structure and lobulated morphology of the lesion. For normal tissue areas outside the candidate box, only one basic layer (8×8 receptive field) is retained, and only global grayscale features are extracted, significantly reducing unnecessary computation. The entire layer allocation process responds in real time to differences in lesion size, ensuring that the core structural features of lesions of different sizes can be accurately captured.

[0026] S12. Extract multi-scale structural features using the adaptive spatial pyramid.

[0027] To address the characteristics of CT images, such as sparse grayscale texture and indistinct grayscale gradients at lesion edges, each pyramid layer employs depthwise separable convolution instead of traditional standard convolution to achieve efficient and targeted feature extraction: the traditional 3×3 standard convolution is split into 3×3 depthwise convolution and 1×1 pointwise convolution. The depthwise convolution performs convolution operations on each input feature channel separately, specifically capturing local features such as lesion edges and contours within a single channel; the 1×1 pointwise convolution is responsible for fusing the feature channels output by the depthwise convolution, integrating the structural information of different channels, while adjusting the number of feature channels to a reasonable range (e.g., reducing from 64 dimensions to 32 dimensions) to effectively reduce computational complexity.

[0028] To further enhance the feature response of lesion edges and contours, a Sobel edge enhancement operator is added after depthwise separable convolution. The operator weights are adjusted based on the gray-level gradient characteristics of CT, focusing on enhancing the gray-level difference between lesion edges and normal tissues and suppressing background noise interference. Each pyramid layer outputs a structural feature map that perfectly matches the spatial resolution of the original CT image (e.g., if the input CT tomography is 512×512 pixels, the output feature map is still 512×512 pixels). The number of feature map channels is dynamically adjusted according to different layers (32 channels for fine-grained layers and 64 channels for coarse-grained layers), which not only preserves complete spatial location information but also achieves hierarchical extraction of multi-scale structural features, laying the foundation for subsequent cross-layer fusion.

[0029] S13. Based on the multi-scale structural features, generate a CT structural semantic feature map through cross-level feature residual fusion and semantic enhancement.

[0030] Efficient fusion of multi-scale features is achieved through cross-level residual connections: edge and microstructure features extracted from fine-grained levels (1×1, 2×2 receptive fields) are directly passed to coarse-grained levels (4×4, 8×8 receptive fields) through residual edges. A 1×1 convolutional layer is added to the residual edge to adjust the number of feature channels, ensuring channel matching between fine-grained and coarse-grained features and avoiding feature loss due to channel incompatibility. The coarse-grained level performs element-wise addition and fusion of its own extracted overall structural features with the fine-grained features passed from the residual edge, so that accurate edge and microstructure information is incorporated into the coarse-grained features, effectively solving the problem of information loss in the multi-scale feature extraction process.

[0031] After fusion, a 1×1 convolutional layer is used to reduce the channel dimensionality of the high-dimensional fused features, reducing the 64-dimensional feature channels after multi-scale fusion to 32-dimensional. At the same time, the importance weight of each feature channel is calculated through a channel attention mechanism. Channels with high correlation to lesion structure (such as edge feature channels) are given high weights (0.7-0.9), while channels related to background noise are given low weights (0.1-0.3). This strengthens the core semantic information of lesion structure, suppresses interference from invalid background features, and makes the fused features more targeted.

[0032] The multi-scale features after cross-level fusion and semantic enhancement are high-dimensional feature vectors. These high-dimensional features are mapped back to the spatial dimension of the original CT image to achieve a precise correspondence between the structural semantic features and the spatial location of the original CT image. A transposed convolutional network is used to construct the feature mapping module. The transposed convolutional kernel is 3×3 in size and the stride is set to 1. Zero padding is used to avoid the checkerboard effect in the feature mapping process, ensuring that the mapped feature map is completely consistent with the spatial size of the original CT image (e.g., if the original CT sequence is 512×512×N pixels, where N is the number of tomographic layers, the generated feature map is also 512×512×N pixels).

[0033] During the mapping process, based on the spatial location of the lesion candidate box, the feature response value of the lesion region is enhanced (multiplied by 1.2 weight), and the feature response value of the normal tissue region is suppressed (multiplied by 0.8 weight), finally generating a CT structural semantic feature map. Each pixel in the feature map corresponds to a unique spatial location in the original CT image, and the pixel value is the structural semantic feature response value at that location (the value range is [0,1]). The higher the pixel value, the higher the probability that the location is a lesion structure. This accurately integrates multi-scale lesion structural features and spatial location information, providing a core spatial structural foundation for subsequent multimodal feature fusion.

[0034] The MRI feature extraction module 2 is used to extract MRI multi-sequence texture features through an attention-enhanced frequency-space dual-stream network.

[0035] The core lesion features of MRI are texture features (such as signal uniformity and gray-level variation patterns). Clinically, multiple sequence scans such as T1 / T2 / FLAIR / enhanced T1 are commonly used, resulting in information redundancy between sequences, artifact noise in some sequences, and texture features exhibiting both frequency distribution (periodic gray-level variation) and spatial distribution (spatial continuity of texture). Based on a frequency-spatial dual-stream network, this method incorporates sequence-level attention enhancement to filter redundant sequence noise, optimizes spatial domain extraction for MRI texture ambiguity, and optimizes frequency component selection for frequency domain noise, ultimately achieving complementary extraction of multi-sequence texture features. The extraction of MRI multi-sequence texture features using an attention-enhanced frequency-spatial dual-stream network includes the following steps: S21. Extract frequency domain branch texture features using two-dimensional fast Fourier transform.

[0036] During multi-sequence MRI scanning, slice shift and organ motion artifacts are easily caused by factors such as patient respiration, slight limb movement, and differences in scanning parameters. Furthermore, the grayscale distribution of different sequences (T1 / T2 / FLAIR / enhanced T1) varies significantly. Traditional rigid registration can only achieve simple translational and rotational alignment, failing to accommodate registration deviations caused by organ deformation. Overall normalization, on the other hand, masks the differences in grayscale characteristics between different sequences. Therefore, to address slice shift and organ motion artifacts in MRI scanning, a B-spline elastic registration algorithm is employed to achieve pixel-level spatial alignment between multiple sequences. First, the enhanced T1 sequence was selected as the reference sequence (this sequence has the highest contrast between lesions and normal tissues and the clearest structure), and other sequences (T1, T2, FLAIR) were used as floating sequences. Through a Gaussian pyramid multi-scale optimization strategy, registration was iteratively performed from low resolution to high resolution. During the registration process, mutual information was used as a similarity metric to accurately calculate the pixel displacement between the floating sequence and the reference sequence. Combined with B-spline basis functions, flexible alignment of organ deformation areas was achieved to ensure accurate pixel matching at the same lesion location in multiple sequences, and the registration error was controlled within 1 pixel.

[0037] To avoid interference from differences in grayscale distribution between different sequences, the traditional overall normalization method is abandoned, and grayscale normalization is performed separately for each sequence: an adaptive min-max normalization algorithm is adopted to automatically calculate the maximum and minimum grayscale values ​​of each sequence based on its grayscale distribution characteristics, mapping all pixel grayscale values ​​to the [0,1] interval, while preserving the grayscale texture features of each sequence (such as the high signal of the T2 sequence and the edema signal of the FLAIR sequence), thus preserving the original sequence difference information for subsequent texture feature extraction, and ensuring that the texture features of different sequences can be accurately distinguished and extracted.

[0038] Furthermore, the essence of MRI texture features is the spatial distribution pattern of gray values. This pattern manifests as different frequency components in the frequency domain. Among them, the texture features of medical lesions (such as signal uniformity and gray value change periodicity) are mainly concentrated in the low and medium frequency regions, while the high frequency regions are mostly scanning noise, artifacts and irrelevant details. If the full frequency components are retained, the texture features will be interfered with by noise, reducing the specificity of feature extraction.

[0039] Based on this, a two-dimensional fast Fourier transform (2D-FFT) was performed on each registered MRI sequence to transform the sequence from the spatial domain to the frequency domain, obtaining a frequency domain spectrogram; based on the distribution pattern of clinical MRI texture features, a screening threshold for mid- and low-frequency components was set: The components within 0-0.5 times the cutoff frequency from the center of the spectrum in the frequency domain are preserved, while high-frequency components above 0.5 times the cutoff frequency are filtered out. This threshold was determined through statistical analysis of a large number of clinical MRI samples. It can effectively filter scanning noise and artifacts while completely preserving the core texture frequency characteristics of the lesion.

[0040] To reduce the dimensionality of frequency domain features and decrease subsequent computation, 1×1 convolution is used to perform channel dimensionality reduction on the selected mid-to-low frequency features: the high-dimensional channels (e.g., 128 dimensions) of mid-to-low frequency features are compressed to 32 dimensions. During the convolution process, the feature channels related to the periodicity of gray-level changes are retained. Finally, the frequency domain texture features of each sequence are extracted. These features can accurately reflect the periodicity of gray-level changes in lesion texture (e.g., the non-uniform signal of tumor lesions is manifested as the irregular distribution of mid-to-low frequency components in the frequency domain), laying the foundation for subsequent complementary fusion with spatial domain texture features.

[0041] S22. Extract spatial domain branch texture features based on a lightweight convolutional network.

[0042] MRI textures are characterized by strong ambiguity and unclear boundaries. Traditional convolutional networks have limited receptive fields and are difficult to capture the continuous spatial texture features of lesions. Simply increasing the size of the convolutional kernel will lead to a decrease in the accuracy of feature extraction and will easily introduce irrelevant background features.

[0043] Based on this, a lightweight convolutional network (using a 3-layer convolutional structure) was constructed. To address the blurred texture characteristic of MRI, dilated convolutions were added to the convolutional layers to expand the receptive field, capturing continuous spatial texture features of a larger lesion area without increasing the kernel size. The dilation rate of the dilated convolutions was dynamically adjusted according to the MRI sequence type. For the T1 and Enhanced T1 sequences with rich texture details, a dilated convolution with a dilation rate of 1 is used to capture fine textures. For the T2 and FLAIR sequences with a wide texture range, a dilated convolution with a dilation rate of 2 is used to capture large-scale continuous textures, ensuring that the spatial texture features of different sequences can be accurately extracted.

[0044] To avoid feature sparsity caused by dilated convolution expanding the receptive field, batch normalization (BN) is added after each dilated convolutional layer to standardize the features output by the convolution, mapping the feature values ​​to a reasonable range, reducing the risk of gradient vanishing, and strengthening the response values ​​of lesion texture features while suppressing the interference of background noise. The last layer of the lightweight convolutional network outputs spatial domain texture features, which can accurately reflect the spatial continuity and distribution pattern of lesion texture (such as diffuse texture in edema areas and local clustered texture of tumor nodules), complementing the frequency domain texture features.

[0045] S23. Utilize channel attention to achieve sequence-level attention enhancement weighting, and perform element-wise addition and fusion of the attention-weighted frequency domain texture features and spatial domain texture features to generate MRI multi-sequence texture features.

[0046] MRI multi-sequence processing suffers from information redundancy, with different sequences contributing significantly to lesion texture features (e.g., enhanced T1 sequences contribute highly to tumor lesion texture recognition, while some T2 sequences containing artifacts contribute little). Considering these differences in contribution, a channel attention module (an improved version of SE-Net) is constructed to achieve sequence-level attention enhancement and weighting, emphasizing the features of effective sequences and suppressing interference from redundant and noisy sequences. The specific implementation process is as follows: First, the feature response values ​​of each sequence in the lesion region are extracted. The lesion region of each sequence is located by the registered lesion candidate box. The mean feature response value of all pixels in the region is calculated and used as the lesion response index of the sequence. Based on this index, the attention weight of each sequence is calculated by a fully connected layer and a sigmoid activation function. The weight value ranges from [0,1]. The higher the lesion response value, the greater the attention weight.

[0047] For noisy sequences with low lesion response (such as T2 sequences containing motion artifacts), their attention weight is reduced to 0.1-0.3 to reduce their interference with the overall features; for effective sequences with high lesion response (such as enhanced T1 sequences), their attention weight is increased to 0.7-0.9 to enhance their core texture features; finally, the frequency domain and spatial domain texture features of each sequence are multiplied by their corresponding attention weights to complete the weighted optimization of the features of each sequence extracted by dual streams, ensuring that the features fused in the subsequent process are more targeted.

[0048] Frequency domain texture features focus on reflecting the periodicity of grayscale changes in lesion texture, while spatial domain texture features focus on reflecting the spatial continuity of lesion texture. The two are complementary. Attention-weighted frequency domain texture features and spatial domain texture features are fused element-wise. During the fusion process, the complementary enhancement of the two features is achieved. The periodic information of the frequency domain features can make up for the problem that the spatial domain features are not good at capturing the grayscale change pattern, and the continuous information of the spatial domain features can make up for the problem that the frequency domain features are not good at capturing the spatial distribution. Element-wise fusion can ensure the deep fusion of the two features and avoid feature redundancy.

[0049] After fusion, the fused features are processed by channel fusion through 1×1 convolution, compressing the fused feature channels of multiple sequences (e.g., 32 dimensions for each sequence, 128 dimensions for 4 sequences) to 64 dimensions. At the same time, redundant information between sequences is filtered out, and the core complementary information of the texture features of each sequence is retained. This ensures that the fused features contain both the frequency distribution pattern of lesion texture and the spatial distribution features, providing high-quality input for the final generation of subsequent texture features.

[0050] The fused features are high-dimensional feature maps (e.g., the input MRI sequence is 256×256 pixels, and the fused feature map is 256×256×64 pixels), which cannot be directly used for subsequent multimodal feature fusion. Furthermore, it is necessary to retain the contribution information of different sequences to the texture features so that subsequent fusion steps can be weighted accordingly. Based on this, global average pooling is performed on the fused feature map: the pixel mean of each channel of the feature map is calculated, and the 256×256×64 high-dimensional feature map is converted into a 64-dimensional one-dimensional feature vector. This vector can accurately summarize the core texture features of each sequence after fusion.

[0051] Meanwhile, to preserve the feature identifiers of the sequence dimensions, during the global average pooling process, the pooling value for the fusion features of each MRI sequence is calculated separately. The pooling results of each sequence are then sorted according to sequence type (T1, T2, FLAIR, enhanced T1), and a sequence identifier label (implemented through one-hot encoding, occupying 4 feature bits) is added to the one-dimensional feature vector, ultimately generating a 72-dimensional MRI multi-sequence texture feature vector. This feature vector contains complementary information of the texture features of each sequence, and the sequence identifier label allows subsequent fusion steps to accurately identify the contribution of different MRI sequences to the texture features, providing crucial support for the refined fusion of subsequent multimodal features.

[0052] The PET feature extraction module 3 is used to extract PET metabolic heterogeneity map features through a graph convolutional network.

[0053] The core lesion feature of PET images is the heterogeneity of metabolic activity (such as the metabolic heterogeneity of tumors and the metabolic differences between benign and malignant lesions). Considering the topological correlation and global heterogeneity of metabolic activity, PET metabolic images are transformed into a graph structure. An improved Graph Convolutional Network (GCN) is used to construct the map using metabolic activity similarity, and a feature enhancement aggregation operation based on metabolic heterogeneity is introduced to achieve an improvement from pixel-level metabolic features to atlas-level heterogeneity features. The extraction of PET metabolic heterogeneity atlas features through a graph convolutional network includes the following steps: S31. Construct a graph structure guided by metabolic heterogeneity based on metabolic activity similarity and spatial adjacency constraints.

[0054] The standardized uptake value (SUV) of PET images directly reflects the metabolic activity of lesions. However, the raw SUV values ​​are subject to individual differences (such as being affected by patient weight, injection dosage, and scan time) and noise interference, making them unsuitable for direct feature extraction. Therefore, the SUV values ​​are first standardized using the z-score normalization method, calculated as follows: ,in Original SUV value, This is the average for all-camera SUVs. The standard deviation is used to normalize the SUV value to the [-1,1] interval, eliminating individual differences and the influence of dimensions, and ensuring that PET data from different patients and different scanning batches are comparable.

[0055] Subsequently, a lesion metabolic threshold is set. This threshold is not a fixed value, but is adaptively adjusted according to clinical diagnostic criteria and the type of organ in which the lesion is located. For example, for lung lesions, the SUV is set to ≥2.5 based on commonly used clinical standards; for liver lesions, the SUV is set to ≥2.0; and for breast lesions, the SUV is set to ≥1.8. Through statistical verification of clinical samples, low-metabolic background tissue pixels (such as normal organ tissue, blood vessels, and adipose tissue) with SUV values ​​lower than the threshold are filtered out, and only candidate metabolic pixels (i.e., pixels with SUV values ​​higher than the threshold and suspected lesion areas) are retained.

[0056] During the screening process, connected component analysis is introduced to remove isolated noise pixels (connected components with less than 3 pixels are considered noise), further reducing the interference of background noise on subsequent feature extraction. This ensures that the selected candidate pixels are all effective pixels related to lesion metabolism, providing a high-quality node foundation for subsequent graph structure construction.

[0057] The core of graph structure construction is to enable graph nodes and edges to accurately represent the metabolic heterogeneity of PET. A dual reconstruction graph strategy of metabolic activity similarity + spatial adjacency constraint is adopted. The selected candidate metabolic pixels are used as graph nodes. The feature vector of each node consists of two parts to ensure that it can fully reflect the metabolic characteristics of the node: one is the normalized SUV value of the pixel itself (reflecting the absolute metabolic activity at that position), and the other is the mean SUV value of all candidate metabolic pixels in the 3×3 neighborhood of the pixel (reflecting the local metabolic level at that position. The neighborhood range has been verified by clinical samples. The 3×3 neighborhood can most accurately capture local metabolic fluctuations). The two are concatenated with a 1:1 weight to form the node feature vector, and the dimension is adapted to the input of the subsequent GCN layer.

[0058] The calculation of edge weights is based on metabolic similarity. The specific process is as follows: First, calculate the Euclidean distance between the feature vectors of any two candidate pixel nodes (to measure the metabolic difference between the two nodes; the smaller the Euclidean distance, the smaller the metabolic difference). Then, normalize the Euclidean distance (mapped to the [0,1] interval). Take the reciprocal of the normalized Euclidean distance as the initial edge weight. That is, the smaller the metabolic difference, the higher the edge weight, ensuring that nodes with similar metabolic characteristics can form a close relationship.

[0059] Meanwhile, to avoid invalid connections formed by distant, unrelated metabolic nodes (such as nodes in different lesion regions or isolated nodes at the boundary between lesions and normal tissue), spatial adjacency constraints are introduced: Euclidean distance is used to calculate the spatial distance between nodes (in pixels), and only connections between nodes with a spatial distance of <3 pixels are retained. For nodes with a spatial distance of ≥3 pixels, the edge weight is directly set to 0 (considered as unconnected). Compared with the traditional method of building edges based on spatial location, this mapping method can more accurately capture the core associations of metabolic heterogeneity, making the graph structure more consistent with the distribution patterns of PET metabolic features, thus laying the foundation for subsequent heterogeneity feature extraction.

[0060] S32. Introduce the node metabolic variation coefficient as the attention weight to design a graph convolutional layer with enhanced heterogeneity.

[0061] Assigning the same weight to the features of all neighboring nodes makes it impossible to distinguish the contribution of different nodes to metabolic heterogeneity, which can easily lead to the dilution of metabolic heterogeneity features. Based on this, the node metabolic variation coefficient is introduced as an attention weight to achieve targeted aggregation that enhances heterogeneity.

[0062] The core of calculating the metabolic variation coefficient of a node is to reflect the degree of metabolic fluctuation in that node and its neighborhood, satisfying the following: ,in The standard deviation of the SUV values ​​of all candidate metabolic pixels within the 3×3 neighborhood of this node. The coefficient of variation (CV) represents the mean SUV value of the corresponding neighborhood. The larger the CV, the more drastic the metabolic fluctuations in the region where the node is located, indicating that it is the core region of metabolic heterogeneity of the lesion (such as the junction between the high metabolic area of ​​the tumor core and the surrounding low metabolic area). The smaller the CV, the more uniform the metabolism in the region, which is mostly a uniform metabolic region of normal tissue or lesion.

[0063] The calculated coefficient of variation is normalized (mapped to the [0,1] interval) and used as the attention weight of the node, which is then incorporated into the node aggregation formula of GCN: Aggregated node feature = its own node feature × 0.4 + Σ(neighboring node feature × neighboring node attention weight × edge weight) × 0.6. Nodes with large metabolic fluctuations and high coefficient of variation (attention weight ≥ 0.7) are aggregated in a focused manner to amplify their metabolic heterogeneity features; nodes with uniform metabolism and low coefficient of variation (attention weight ≤ 0.3) have their aggregation weight reduced to suppress their interference with the overall heterogeneity features, thereby achieving targeted enhancement of metabolic heterogeneity features.

[0064] S33. Based on the graph structure, global heterogeneity features are extracted using the graph convolutional layer to generate PET metabolic heterogeneity map features.

[0065] A single graph convolutional layer (GCN layer) can only capture metabolic heterogeneity correlations within a local area and cannot achieve feature aggregation from local metabolic fluctuations to global metabolic distribution. Therefore, multiple GCN layers need to be stacked to construct a hierarchical aggregation architecture from local to global. In this embodiment, considering the complexity of PET metabolic features and the need to avoid overfitting, and after validation with a large number of clinical samples, it was determined that 2-3 GCN layers should be stacked (2 layers are suitable for lesions with weak metabolic heterogeneity, and 3 layers are suitable for tumor lesions with strong metabolic heterogeneity). The core parameters of each GCN layer are tailored to the PET feature extraction requirements: the convolution kernel size is set to 3×3, and the number of channels is adjusted according to the pattern of input channel → input channel × 2 → input channel, ensuring the depth of feature extraction while avoiding dimensional redundancy.

[0066] To address the feature smoothing problem that easily occurs when stacking multiple GCNs (i.e., after multi-layer aggregation, metabolic heterogeneity features are averaged, making it impossible to distinguish between the core and surrounding regions), residual connections (shortcut paths) are added after each GCN layer. The specific design of the residual connections is as follows: the input features of each GCN layer are element-wise summed with the output features of that GCN layer. Before summing, the number of channels in the input features is adjusted using a 1×1 convolution to ensure consistency with the number of channels in the output features, avoiding channel incompatibility. The core function of residual connections is to preserve the original metabolic node features, allowing the features after each aggregation layer to incorporate metabolic association information from the neighborhood without losing their own metabolic characteristics. This effectively alleviates the feature smoothing problem and ensures that after 2-3 layers of aggregation, a complete extraction of everything from local metabolic fluctuations (such as metabolic differences in a single pixel) to global metabolic heterogeneity (such as the metabolic distribution of the entire lesion, and the metabolic gradient between the core and infiltrative regions) can be achieved, providing comprehensive global metabolic association information for subsequent atlas feature generation.

[0067] The output of multi-layer GCN aggregation is a graph-structured node feature matrix (with dimensions N×C, where N is the number of candidate metabolic pixel nodes and C is the number of feature channels). This matrix is ​​still a scattered node-level feature and cannot be directly used for subsequent multimodal feature fusion. It needs to be transformed into a one-dimensional graph-level feature vector with a unified dimension through pooling operations to achieve the improvement from node-level features to graph-level features.

[0068] In line with the core requirement of PET metabolic heterogeneity, global max pooling is selected as the pooling method (compared to global average pooling, global max pooling can more accurately capture the core peak features of metabolic heterogeneity, such as the features of the high metabolic core area of ​​lesions). The specific operation process is as follows: global max pooling is performed on the node feature matrix output by the last layer GCN according to the feature channel dimension. That is, for each feature channel, the maximum value of all nodes in that channel is selected as the pooling result of that channel. Finally, the N×C node feature matrix is ​​transformed into a 1×C one-dimensional feature vector.

[0069] This atlas-level feature vector fully preserves the three core pieces of information about PET metabolism: first, global topological association (achieved through multi-layer GCN aggregation, reflecting the metabolic node association patterns throughout the lesion region); second, local heterogeneity (achieved through attention weight aggregation, preserving the core features of areas with dramatic metabolic fluctuations); and third, core metabolic region features (achieved through global max pooling, highlighting the features of the high-metabolic core region of the lesion). Simultaneously, the generated feature vector undergoes min-max normalization (mapping to the [0,1] interval) to eliminate numerical differences and ensure the stability and comparability of the feature vector.

[0070] The ultrasonic feature extraction module 4 is used to extract ultrasonic dynamic motion features through a time-aware cyclic network.

[0071] Ultrasound is a dynamic video sequence image. The core feature of lesions is their dynamic motion characteristics (such as lesion displacement under organ movement, rhythm of elastic movement, and motion differences with surrounding tissues). Traditional recurrent neural networks (LSTM / RNN) use isochronous full-frame extraction, which suffers from many redundant frames and dilution of key motion information. To address this, redundant frames are removed through keyframe filtering, and temporal attention weights are added to the gating units to focus on capturing peak frames and core temporal features of lesion motion, thus adapting to the domain characteristics of ultrasound dynamics. The extraction of ultrasound dynamic motion features using a time-aware recurrent network includes the following steps: S41. Filter time-aware lesion keyframes through motion feature values.

[0072] First, ultrasound images naturally contain speckle noise (caused by the scattering characteristics of sound waves). Fixed-window median filtering is prone to incomplete noise removal or over-filtering that blurs lesion edges. Therefore, an adaptive median filtering method is adopted, and the specific implementation logic is as follows: The size of the filtering window is dynamically adjusted based on the pixel grayscale differences within the ultrasound frame (a 3×3 window is used in areas with uniform grayscale, and a 5×5 window is automatically switched in areas with abrupt grayscale changes (such as the edge of the lesion). By iteratively calculating the median of pixels within the window, noise pixels are removed while grayscale gradient information of the lesion edge is preserved, ensuring a balance between noise removal and edge preservation.

[0073] To address the inter-frame artifacts caused by slight camera movement and operator hand tremors during ultrasound examinations, registration markers are selected based on organ anatomical features (such as blood vessel orientation and organ capsule contours). A rigid registration algorithm (using mutual information as a similarity metric, setting the number of iterations to 50, and controlling the registration error within 1 pixel) is employed to achieve precise alignment of frames in a dynamic sequence, eliminating the interference of displacement artifacts on subsequent motion feature extraction.

[0074] Considering the high frame rate (usually ≥15fps) and long sequence length (≥100 frames per segment) of ultrasound dynamic sequences, direct processing would generate a large amount of unnecessary computation. Therefore, the registered sequences are sampled at equal time intervals, with the sampling interval adaptively adjusted according to the total sequence length (3 frames for long sequences and 2 frames for short sequences). This compresses the original long sequences into short sequences that are 20%-30% of their original length. While preserving the complete lesion motion cycle, this significantly reduces the computational load for subsequent keyframe screening and feature aggregation, thus meeting the needs of real-time clinical processing.

[0075] The core of lesion keyframe screening is to accurately capture the core information of lesion movement and avoid redundant frame interference. Therefore, the motion feature value of the lesion region in each frame is calculated in multiple dimensions. The specific calculation method is as follows: the displacement of each pixel in the lesion region is calculated by optical flow method (Lucas-Kanade algorithm), and the average value of all pixel displacements is taken as the pixel displacement index; the gray level change rate is calculated by the average gray level difference between the lesion regions in two adjacent frames to reflect the gray level fluctuation caused by lesion movement; the motion difference between the lesion region and the surrounding normal tissue is calculated by the gray level variance ratio between the lesion region and the surrounding normal tissue to highlight the difference between the movement of the lesion and the normal tissue.

[0076] Based on the clinical organ movement patterns, differentiated movement thresholds were set: the movement threshold for cardiac ultrasound (intense lesion movement) was set to 0.8 (after feature value normalization), the threshold for abdominal ultrasound (slow lesion movement) was set to 0.4, and the threshold for breast ultrasound was set to 0.5. Keyframes with movement feature values ​​higher than the corresponding thresholds were selected, and core frames such as peak frames of lesion contraction / relaxation, extreme frames of elastic movement, and frames of maximum lesion displacement were retained.

[0077] Meanwhile, to avoid missing or over-screening keyframes, inter-frame correlation verification is added (the correlation coefficient of adjacent keyframes ≥0.7 is considered valid). In the end, 20%-30% of the keyframes in the original sequence are retained. This not only removes redundant frames with smooth motion (such as lesion still frames or slightly jittery frames), but also ensures that the core information of the complete motion cycle of the lesion is not lost, providing high-quality input for subsequent temporal feature aggregation.

[0078] S42. Based on the spatial features of the keyframes, obtain the ultrasonic dynamic motion features using a time-attention-enhanced LSTM model.

[0079] Spatial features of keyframes are the foundation for temporal motion feature aggregation. It is necessary to accurately capture the morphology, location, and spatial relationship with surrounding tissues of lesions, while also considering computational efficiency. A lightweight convolutional network is used to extract single-frame spatial features. This network consists of two convolutional layers, one pooling layer, and one fully connected layer. Specific parameters are designed to fit the characteristics of ultrasound images: the first convolutional layer uses a 3×3 kernel, 16 channels, and a stride of 1, with zero padding to ensure no reduction in feature map size, focusing on capturing low-level spatial features such as lesion edges and contours; the second convolutional layer uses a 3×3 kernel, 32 channels, and a stride of 1. The length is 1, which further extracts mid-level spatial features such as gray-scale distribution and local texture of lesions; the pooling layer adopts 2×2 max pooling, which retains the core information of the features while compressing the feature dimension; the fully connected layer maps the pooled features into a 64-dimensional feature vector. The final output spatial features not only include the shape contour of the lesion and pixel gray-scale distribution, but also cover the spatial adjacency relationship between the lesion and surrounding tissues (such as blood vessels and capsules), ensuring that the spatial information of each key frame is completely extracted, providing solid spatial feature support for subsequent LSTM temporal aggregation, and avoiding misjudgment of motion features caused by lack of spatial correlation of temporal features.

[0080] Furthermore, the gating unit of the LSTM is improved by incorporating temporal attention weights to achieve differentiated feature aggregation. The specific improvement logic is as follows: the motion feature values ​​of keyframes are normalized (mapped to the [0,1] interval) and used as temporal attention weights, which are then incorporated into the calculation of the input gate, forget gate, and output gate of the LSTM. In the input gate, the higher the motion feature value of the key frame, the greater the input weight of its spatial features, ensuring that the spatial features of the core key frame are given priority input; in the forget gate, the features of ordinary key frames with low motion feature values ​​are quickly forgotten, reducing interference from redundant information; in the output gate, the temporal correlation information of the core key frames with high motion feature values ​​is retained, strengthening the peak features and temporal rhythm of lesion motion.

[0081] The 64-dimensional spatial features of each keyframe are input into the improved LSTM in chronological order (frame order of the ultrasound dynamic sequence). The LSTM hidden layer dimension is set to 64, and the number of iterations is set to 100. Through differential processing of gating units, the temporal correlation features of lesion motion are captured, including changes in motion rhythm, temporal fluctuations in displacement, and temporal patterns of motion differences with surrounding tissues. At the same time, the original spatial features are preserved through residual connections to avoid the loss of spatial information during temporal aggregation, thus achieving deep fusion of spatial and temporal features.

[0082] The hidden layer features of the last layer of the LSTM contain complete temporal correlation information of lesion motion, but the dimensionality is high (64-dimensional hidden layer features, corresponding to multiple key frames) and there are some redundant features. It is necessary to generate a standardized one-dimensional temporal feature vector through global pooling and post-processing. Specifically, the hidden layer features of the last layer of the LSTM are subjected to hybrid pooling processing of global average pooling and global max pooling. Global average pooling calculates the mean of the hidden layer features to capture the global temporal rhythm of lesion motion (such as motion period and average displacement). Global max pooling extracts the maximum value of the hidden layer features to capture the local peak features of lesion motion (such as maximum displacement and most significant motion difference). The two pooling results are summed element by element with a 1:1 weight to generate a preliminary 64-dimensional temporal feature vector.

[0083] To further enhance the specificity of the features, the initial feature vectors were post-processed: z-score normalization was used to map the feature values ​​to the [-1,1] interval to eliminate numerical difference interference; the importance weight of each feature channel was calculated through a lightweight channel attention module, and high weights (0.7-0.9) were assigned to core feature channels such as motion rhythm and displacement changes, while low weights (0.1-0.3) were assigned to noise channels. Finally, a 64-dimensional ultrasound dynamic motion feature vector was generated, which fully preserved the motion rhythm, displacement changes, motion differences with surrounding tissues, and motion cycle characteristics of the lesion.

[0084] The image semantic feature extraction module 5 is used to extract pathology-guided image semantic features through a medical knowledge graph generative adversarial network.

[0085] Generative Adversarial Networks (GANs) for extracting medical image features are purely data-driven, lacking pathological prior constraints and prone to generating invalid features that do not conform to clinical pathological diagnoses. This paper addresses this by deeply integrating a medical knowledge graph (containing pathology-image association priors) with GANs, improving the generator and discriminator through pathological constraints. This guides the feature extraction process with pathological knowledge, achieving a dual feature extraction model of data-driven and knowledge-driven approaches, ensuring that the extracted image semantic features align with clinical pathological patterns. The extraction of pathology-guided image semantic features using a medical knowledge graph-based GAN includes the following steps: S51. Filter the lesion subgraphs of the medical knowledge graph to obtain the knowledge embedding vector.

[0086] First, from the global medical knowledge graph constructed in clinical practice (a complete set of triplets covering multiple organs and multiple pathological types), a dual strategy of triplet association screening and irrelevant knowledge pruning is strictly adopted based on the organ type of the lesion to be identified (such as lung, liver, breast, etc.) to screen out the knowledge subgraph related to the lesion, thereby minimizing the coding complexity while ensuring the preservation of core pathological-imaging association knowledge.

[0087] The specific screening logic aligns with clinical diagnostic relevance: prioritizing the retention of three core triplets: organ type-lesion type, lesion type-pathological features, and pathological features-imaging representation (e.g., retaining triplets such as lung-lung-squamous carcinoma, squamous carcinoma-lobulated-CT lobulated nodules for lung lesions). By traversing the adjacency matrix of the global knowledge graph, triplets unrelated to the target organ or lesion are pruned (e.g., directly pruning cardiovascular-related knowledge such as heart-coronary heart disease-ECG abnormalities for lung lesions, and pruning breast-related knowledge such as breast-fibroadenoma-MRI low signal for liver lesions).

[0088] After screening, the knowledge subgraph is deduplicated and denoised to remove duplicate triples and abnormal triples with incorrect annotations (such as triples where pathological features contradict imaging representations). Finally, a lesion knowledge subgraph with a concise structure and close connections is obtained, with the number of nodes controlled between 50 and 80 (balancing coding efficiency and knowledge integrity).

[0089] Furthermore, the TransE knowledge embedding algorithm is used to transform the triple knowledge in the subgraph into a low-dimensional dense vector to achieve feature encoding, with specific implementation details adapted to the characteristics of medical knowledge: The embedding dimension is set to 72-dimensional, the marginal parameter of the TransE algorithm is set to 1.0, the learning rate is 0.001, and the number of training iterations is 1000. By minimizing the distance loss of the triple (subject-relation-object), the subject, relation, and object are all mapped to a 72-dimensional vector space, ensuring that the encoded vector can accurately retain the correlation features of pathological type-pathological features-image representation (such as the accurate encoding of the correlation weight between the lobulated pathological features of squamous cell carcinoma and the lobulated image representation of CT in the vector).

[0090] After encoding, the validity of the generated knowledge embedding vectors is verified by calculating the correlation similarity between vectors (e.g., the cosine similarity between the vectors of squamous cell carcinoma and CT lobulated nodules). If the similarity is ≥0.7, it indicates that the correlation features are completely preserved and the vectors are considered valid embedding vectors. If the similarity is <0.7, the process returns to the subgraph filtering step, the filtering rules are re-optimized and re-encoded, and finally a 72-dimensional knowledge embedding vector is generated, providing high-quality pathological prior support for the dual input of the subsequent generator.

[0091] S52. Construct a generator for pathological constraints and a discriminator for pathological perception.

[0092] The generator uses the basic features of CT / MRI / PET / ultrasound single modality (all standardized to 64-dimensional feature vectors, which are then stitched together to form 256-dimensional vectors) and the aforementioned 72-dimensional knowledge embedding vector as dual inputs to achieve dual-driven processing of image data and pathological knowledge, ensuring that the generated features conform to pathological patterns.

[0093] The generator's main body adopts a lightweight convolutional block structure, adapting to the extraction requirements of medical multimodal features. The specific structural design is as follows: It contains three consecutive convolutional blocks, each consisting of a 3×3 convolution layer, batch normalization (BN, momentum 0.9, epsilon=1e-5), and a ReLU activation function. The convolutional stride is set to 1, and zero padding ensures that the feature map size is not reduced. The first convolutional block adjusts the dual input (256-dimensional image features + 72-dimensional knowledge embedding vector, concatenated to 328 dimensions) to 128 channels through a 1×1 convolution, realizing the fusion and dimensional calibration of the input features. The second convolutional block compresses the 128-dimensional features to 64 dimensions, focusing on extracting the correlation information between image features and pathological knowledge. The third convolutional block outputs 72-dimensional semantic features of pathological guidance candidate images, consistent with the dimension of the knowledge embedding vector, facilitating the subsequent calculation of pathological constraint loss.

[0094] To ensure that the generated candidate features are consistent with the pathology-image association knowledge, a pathology feature constraint loss is specifically added during the generation process. The design is tailored to clinical pathology: based on the knowledge embedding vector, a standard distribution of pathology features is defined (using a Gaussian distribution, with the mean being the mean of the knowledge embedding vector and the variance set to 0.01). The pathology feature constraint loss uses the mean squared error loss (MSE). This loss is jointly optimized with the generator's generation loss, forcing the candidate features generated by the generator to continuously approach the standard distribution of pathology features, avoiding the generation of invalid features that do not conform to clinical pathology (such as generating lung cancer adenocarcinoma features but not including the typical imaging characteristic of high signal on MRI T2).

[0095] Meanwhile, feature normalization processing (min-max normalization to the [0,1] interval) is added to the output of the generator to ensure that the candidate features are adapted to the input requirements of the subsequent discriminator, laying the foundation for subsequent adversarial training.

[0096] To address the limitation of GAN discriminators, which can only distinguish the authenticity of features but cannot take into account the rationality of pathology, the discriminator is reconstructed to achieve dual discrimination functions of authenticity discrimination and pathological matching discrimination, ensuring that the generated features not only fit the real multimodal image features but also conform to prior pathological knowledge.

[0097] The discriminator adopts a dual-branch parallel + fusion decision structure. The two branches share the underlying feature extraction network. The specific structure is as follows: The underlying shared network is a 2-layer 3×3 convolution + BN + LeakyReLU activation function with a convolution stride of 2. It compresses the input features (generated candidate features or real multimodal basic features, both of which are 72-dimensional) to 32-dimensional and extracts the core representation of the features.

[0098] Based on this, branch 1 (authenticity discrimination branch) outputs a 1-dimensional probability value (range [0,1]) through a fully connected layer. It uses cross-entropy loss to calculate the similarity between the generated feature and the real feature. If the probability value is ≥0.5, it is judged as a real feature, and if it is <0.5, it is judged as a false feature, thus realizing the authenticity discrimination function of traditional GAN.

[0099] Branch 2 (pathological matching and discrimination branch) maps 32-dimensional features to 72-dimensional features through a fully connected layer, calculates cosine similarity with the knowledge embedding vector, and determines whether the similarity is ≥0.6 and conforms to the pathological prior, and whether it is <0.6 and does not conform to the pathological prior.

[0100] Simultaneously, a pathological matching loss is introduced to quantify the matching degree. The pathological matching loss adopts the cross-entropy loss, and the calculation formula is as follows: The knowledge embedding vector labels are set according to the clinical pathology relevance (e.g., the knowledge embedding vector label related to the target lesion is 1, and the one that is not related is 0).

[0101] The fusion decision logic of the dual-discrimination function strictly conforms to clinical needs: if the generated feature only satisfies the authenticity discrimination (probability ≥ 0.5) but does not satisfy the pathological matching discrimination (similarity < 0.6), it is judged as a false feature, and the generator is guided to adjust the generation strategy through backpropagation of pathological matching loss; if the generated feature satisfies both discriminations (authenticity probability ≥ 0.5 and pathological similarity ≥ 0.6), it is judged as a true feature; if neither is satisfied, it is directly judged as a false feature.

[0102] In addition, gradient clipping (gradient norm threshold set to 0.5) is added to the output of the discriminator to avoid model instability caused by the discriminator training too fast, and to ensure that the two discriminator branches work together to form an effective adversarial balance with the generator.

[0103] S53. Combining the knowledge embedding vector, the generator, and the discriminator, extract the semantic features of the pathology-guided images.

[0104] Using multimodal basic features as real samples, the real samples are first preprocessed (to be consistent with the candidate features output by the generator, min-max normalized to the [0,1] interval, and the dimension adjusted to 72 dimensions), and then input together with the candidate features generated by the generator into the discriminator to carry out adversarial learning between the generator and the discriminator. The core is to jointly optimize the generation loss, discrimination loss, and pathological matching loss to achieve dual-constraint training driven by data and knowledge, ensuring training stability and feature effectiveness.

[0105] The specific training process and parameter design are as follows: ① Training initialization: The parameters of both the generator and the discriminator are initialized using the He normal distribution. The total number of training iterations is set to 5000, and the initial learning rate is 0.0001. The Adam optimizer (decay coefficient 0.9) is used to adaptively adjust the learning rate. The initial weight allocation of the three losses is in line with the training objective: generation loss weight 1.0, discriminant loss weight 1.0, and pathological matching loss weight 1.2, focusing on strengthening the role of pathological constraints. ② Iterative training process: In each iteration, the generator is trained first. Multimodal basic features and knowledge embedding vectors are input to generate candidate features. The generation loss (using cross-entropy loss to measure the difference between generated features and real features) and pathological feature constraint loss are calculated and summed to obtain the total generator loss. The generator parameters are updated through backpropagation. Then, the discriminator is trained. Real samples and candidate samples generated by the generator are input respectively. The discriminant loss (cross-entropy loss to measure the classification accuracy of the discriminator) and pathological matching loss are calculated and summed to obtain the total discriminant loss. The discriminator parameters are updated through backpropagation. The weights of the three losses are adjusted every 100 iterations. If the pathological matching loss is >0.2, its weight is increased to 1.5 until the pathological matching loss converges. ③ Training stability optimization: To avoid problems such as gradient explosion and mode collapse during adversarial training, a gradient pruning strategy (gradient norm threshold set to 1.0) is adopted to constrain the gradients of the generator and discriminator; a validation is performed every 500 iterations to calculate the feature similarity (cosine similarity between generated features and knowledge embedding vectors) and discrimination accuracy on the validation set; a convergence condition is set: when the generation loss on the validation set is <0.05, the discrimination loss is <0.03, and the pathological matching loss is <0.02, and the fluctuation of the three losses is less than 1% for three consecutive iterations, adversarial training is stopped. At this time, the generator can stably generate candidate features that conform to the pathological prior and fit the real image features, providing reliable support for the subsequent final feature generation.

[0106] After the GAN adversarial training is completed, the generator has the ability to generate dual features of image features and pathological knowledge. At this time, it is necessary to generate the final pathological-guided image semantic features based on new lesion samples to ensure that the features can integrate the core semantics of multimodal images and prior pathological knowledge.

[0107] The specific implementation process includes three stages: preprocessing, feature generation, and post-processing and validation, ensuring the effectiveness and clinical suitability of the features: ① New sample preprocessing: The single-modal basic features of new lesion samples (CT structural semantic features, MRI multi-sequence texture features, PET metabolic heterogeneity features, and ultrasound dynamic motion features) are preprocessed according to the above standards, standardized into 64-dimensional feature vectors, concatenated and adjusted to 256 dimensions, and concatenated with the knowledge embedding vector (re-selected and encoded according to the organ type of the new sample) to obtain 328-dimensional input features, which are completely consistent with the input format of the generator in the training stage, ensuring that the generator can stably output features. ② Feature generation: The preprocessed input features are input into the trained generator. The generator extracts features through internal convolutional blocks and pathological constraints, outputting 72-dimensional preliminary pathological-guided image semantic features. These features have initially integrated the core semantics of multimodal images (such as lobular structures in CT and texture details in MRI) and prior pathological knowledge (such as the correlation between pathological types and image representations). ③ Post-processing and validation: To further improve feature quality, the initially generated features are post-processed: z-score normalization is used to map feature values ​​to the [-1,1] interval to eliminate interference caused by numerical differences; the importance weight of each feature channel is calculated through the channel attention mechanism (SE-Net lightweight structure), and high weights (0.7-0.9) are given to channels with high pathological-image correlation (such as squamous cell carcinoma lobulation feature channel and adenocarcinoma T2 high signal channel), while low weights (0.1-0.3) are given to background noise channels to strengthen core semantic features.

[0108] After post-processing, a dual validity verification was implemented: first, the cosine similarity between the feature and the knowledge embedding vector was calculated, with a value ≥0.7 considered acceptable for pathological association; second, the average similarity between the feature and the basic features of each single modality was calculated, with a value ≥0.6 considered acceptable for image semantic preservation. Only when both verifications passed was the feature considered a valid pathological guidance image semantic feature. The final 72-dimensional pathological guidance image semantic feature not only fully preserves the core semantic information of multimodal images but also deeply integrates clinical pathological priors.

[0109] The multi-task learning module 6 is used to extract comprehensive image features for clinical guidance through multi-task learning.

[0110] Based on the clinical task correlation screening auxiliary task, the loss function and training strategy are optimized to address the imbalance in medical data annotation and the differences in task difficulty. Through joint training of the lesion identification main task and the clinical auxiliary task, the extracted features are integrated with key information of clinical diagnosis, realizing the integrated extraction of image features and clinical guidance. The extraction of comprehensive image features for clinical guidance through multi-task learning includes the following steps: S61. Screening auxiliary tasks based on clinical task relevance.

[0111] With lesion identification as the primary task (the core objective being to accurately determine the presence and location of lesions in images), we strictly adhere to the core logic of clinical diagnosis: first qualitative assessment, then grading, and finally subtyping. We select auxiliary tasks that are highly relevant to the primary task to ensure that they provide supplementary information and enhance feature representation for the primary task of lesion identification, rather than introducing irrelevant interference.

[0112] The specific screening logic aligns with clinical practice: In clinical diagnosis, after identifying lesions, doctors prioritize determining whether the lesion is benign or malignant, and then clarify the lesion stage (e.g., T1-T4 stage of tumor) and pathological type (e.g., squamous cell carcinoma and adenocarcinoma of lung cancer). These judgments all rely on the correlation between the imaging characteristics of the lesion and clinical indicators. Therefore, the binary classification of benign and malignant lesions, the multi-classification of lesion stages, and the multi-classification of pathological types are set as core auxiliary tasks. At the same time, irrelevant tasks that are not directly related to the characteristics of the lesion itself (e.g., patient age and gender classification) are strictly removed. This is because such information has no direct mapping relationship with the core features of the lesion, such as structure, texture, and metabolism. Including such information would introduce irrelevant noise and exacerbate the negative transfer between tasks (i.e., auxiliary tasks interfere with the feature learning of the main task).

[0113] To ensure that each task is optimized independently and without interference, each task corresponds to an independent network output head. The structure of the output head is adapted to the task requirements: the binary classification output head for benign and malignant diseases is 2-dimensional (corresponding to benign and malignant), and the multi-classification output head for lesion staging / pathological typing matches the specific number of categories (e.g., lung cancer staging output is 4-dimensional, and pathological typing output is 3-dimensional). Each output head is connected to an independent Softmax activation function to achieve task-specific probability output, laying the foundation for subsequent joint training.

[0114] S62. Construct a shared-branch multi-task feature extraction network and optimize the loss function based on the imbalance of medical data annotation and the difference in task difficulty.

[0115] Based on the complexity of medical multimodal features and the stability requirements of multi-task training, a dual-structure network with a shared backbone and task branches is constructed. This achieves feature sharing across multiple tasks, preserves the specificity of each task, and avoids the gradient vanishing problem of deep networks.

[0116] The shared backbone network design adopts a simplified version of the ResNet-18 residual convolutional structure, which contains 4 residual blocks. Each residual block consists of 2 layers of 3×3 convolutions, batch normalization (BN, momentum 0.9, epsilon=1e-5) and ReLU activation function. The residual connections directly pass low-level features through the shortcut path, which effectively alleviates the gradient vanishing problem in deep network training and ensures that multimodal features can be extracted at depth.

[0117] The structural semantic features of CT, multi-sequence texture features of MRI, metabolic heterogeneity features of PET, and dynamic motion features of ultrasound (all standardized to 64-dimensional feature vectors) are concatenated and input into a shared backbone network. Through layer-by-layer extraction of residual convolution, a 64-dimensional cross-task shared basic comprehensive feature is generated. This feature integrates the core information of four single modalities and is the basic feature support common to all tasks.

[0118] Furthermore, an independent task branch network is designed for each task (main task + 3 auxiliary tasks). The branch network structure is lightweight and specifically adapted to the task requirements—each branch consists of one 1×1 convolutional layer (adjusting the number of channels to 32 dimensions) and one fully connected layer. The 1×1 convolution is used to compress the number of channels and reduce redundancy, while the fully connected layer is used to map the shared basic comprehensive features to the output space of each task, realizing task-specific feature extraction. The parameters of each task branch are initialized and updated independently to ensure that the feature learning of different tasks does not interfere with each other. At the same time, the branch network and the shared backbone network are linked through gradient backpropagation, so that the shared features can be adapted to the learning needs of each task in a synchronous manner, achieving a synergistic effect of shared feature support and task feature complementarity.

[0119] To address the domain characteristics of unbalanced medical data annotation and significant differences in training difficulty across tasks, a composite loss function is constructed, which combines task-specific loss, sample weighting, and task weighting. This ensures the stability and effectiveness of joint training and prevents a particular task from dominating training or neglecting a small number of sample classes.

[0120] Task-specific loss functions refer to loss functions designed specifically for each task type, tailored to the evaluation needs of medical tasks: the main task (lesion identification) uses weighted cross-entropy loss, with weights set according to the ratio of lesions to background samples (1.5 for lesion samples and 0.5 for background samples), addressing the problem of scarce lesion samples and redundant background samples in clinical data; the benign / malignant binary classification task uses binary cross-entropy loss, incorporating LabelSmoothing (smoothing coefficient 0.1) to alleviate overfitting caused by sample annotation noise; the multi-class classification task of lesion staging and pathological typing uses multi-class cross-entropy loss, combined with a class balance factor, to adapt to the class distribution characteristics of multi-class tasks.

[0121] Sample-weighted loss refers to a loss method used to address imbalanced labeling in medical data (e.g., rare pathological subtypes accounting for less than 5% of the total sample). A separate sample-weighted loss is applied to each task, and the weights are calculated using the following formula: ,in, This represents the proportion of samples in this category within the total sample. Set to 2.0, this formula assigns high weights to classes with fewer samples and low weights to classes with more samples, ensuring that the model focuses on fewer and difficult samples during training, thereby improving the model's ability to identify rare lesions.

[0122] The task weighting coefficient refers to the initial task weighting coefficient added to each task to account for the differences in training difficulty of different tasks (such as the pathological classification task being more difficult than the benign and malignant classification due to its more subtle classification features). The initial values ​​are set based on the experience of clinical experts: the weight of the main task (lesion identification) is 0.5, the weight of the benign and malignant binary classification is 0.2, the weight of the lesion staging multi-class classification is 0.15, and the weight of the pathological classification multi-class classification is 0.15. Subsequently, the weighting coefficient is further optimized through a dynamic gradient weighting strategy to ensure that each task progresses in a coordinated manner without interfering with each other.

[0123] S63. Through dynamic gradient weighted multi-task joint training, clinically guided comprehensive image features are obtained.

[0124] By using a dynamic gradient weighting strategy, the loss weights are adaptively adjusted according to the real-time training difficulty of each task, which solves the problem of negative transfer between tasks (such as excessive training error in one task dominating gradient updates, leading to a decline in the training effect of other tasks), and ensures collaborative optimization of the shared backbone network and the branch networks of each task.

[0125] In the initial stage of training, the shared backbone network is pre-trained (1000 iterations, learning rate 0.001), and single-modal basic features are input to initially extract cross-modal shared features. After pre-training, the network is connected to each task branch network for multi-task joint training. In each iteration, the loss of the main task and each auxiliary task is calculated simultaneously, and the total loss is obtained by summing the dynamic weights. The parameters of the shared backbone and task branches are updated synchronously through backpropagation.

[0126] During each iteration, the gradient norm (using L2 norm) of the loss for each task is calculated in real time. A larger gradient norm indicates a higher training difficulty and worse current training performance, requiring a higher loss weight; conversely, a smaller gradient norm results in a lower weight. The weight update formula is as follows: (j is the task number), ensuring that the sum of the weights of all tasks is 1, achieving dynamic matching between training difficulty and weights. For example, if the gradient norm of the pathological classification task is twice that of the benign / malignant classification task, then the loss weight of the pathological classification task will be automatically adjusted to twice that of the benign / malignant classification task, ensuring that the model prioritizes optimizing tasks with higher training difficulty, while avoiding a single task dominating training.

[0127] To further improve training stability, dynamic gradient clipping (gradient norm threshold set to 1.0) is added to avoid gradient explosion; the Adam optimizer (initial learning rate 0.0001, decay coefficient 0.9) is used to adaptively adjust the learning rate; validation is performed every 500 iterations. If the accuracy of a certain task on the validation set does not improve for 3 consecutive times, the weight update of that task is paused, and other tasks are optimized in a focused manner until the validation accuracy of all tasks tends to stabilize.

[0128] After multi-task joint training, the core objective is to extract comprehensive features that combine multimodal imaging information and clinical guidance information, providing high-level clinical semantic support for hierarchical attention fusion. Therefore, the output of the last layer of the shared backbone network is selected as the final feature. The generated comprehensive clinical guidance image features are 64-dimensional one-dimensional feature vectors, which contain two core pieces of information: First, multimodal image fusion information, which integrates the spatial structure of CT, the texture details of MRI, the metabolic activity of PET, and the dynamic motion features of ultrasound to achieve complementarity of multimodal information; Second, clinical diagnostic guidance information, which, through multi-task joint training, encodes the correlation between benign / malignant, stage, and type and imaging features (such as the correlation weight between the edge features of malignant lesions, high metabolic features and benign / malignant labels, and the correspondence between tumor lobulation structure and stage), making the features relevant to clinical diagnosis.

[0129] The first fusion module 7 is used to obtain hierarchical attention fusion features by fusing CT structural semantic feature maps, MRI multi-sequence texture features and clinically guided image comprehensive features through a hierarchical attention temporal graph convolutional network.

[0130] A hierarchical attention-based temporal graph convolutional network is constructed, modeling three types of features as hierarchical heterogeneous graphs. Multi-dimensional attention weighting (modality / level / node) is designed, and temporal convolution captures cross-level feature interactions, achieving refined and hierarchical multi-feature fusion to highlight core lesion-related information. The process involves fusing CT structural semantic feature maps, MRI multi-sequence texture features, and clinically guided image synthesis features through the hierarchical attention-based temporal graph convolutional network to obtain hierarchical attention fusion features, including the following steps: S71. Construct a hierarchical heterogeneous graph to achieve structured association of features through the hierarchical heterogeneous graph.

[0131] Based on the core logic of clinical lesion diagnosis—microscopic texture observation → mesoscopic structural analysis → macroscopic clinical judgment—CT structural semantic feature maps, MRI multi-sequence texture features, and clinically guided comprehensive image features are precisely hierarchically modeled according to feature representation levels and clinical diagnostic value. This ensures that the hierarchical division aligns with clinical practice. Simultaneously, a hierarchical heterogeneous map is constructed to achieve structured association of features. The specific implementation is as follows: Hierarchical modeling: The bottom layer consists of MRI multi-sequence texture features (72-dimensional vectors), which are pixel / local texture-level features, focusing on reflecting the microscopic features of lesions (such as grayscale changes in MRI T1 / T2 sequences and edema texture details in FLAIR sequences), corresponding to the observation of lesion microscopic texture in clinical diagnosis; the middle layer consists of CT structural semantic feature maps (512×512×32 pixel dimensions), which are lesion spatial structure-level features, focusing on reflecting the mesoscopic features of lesions (such as lobulation, spiculation, edge morphology, and three-dimensional spatial distribution), corresponding to the analysis of lesion spatial morphology in clinical diagnosis; the top layer consists of clinically guided comprehensive image features (output from a shared backbone network, with dimensions adapted to the middle layer feature channels), which are global clinical semantic-level features, focusing on reflecting the macroscopic clinical features of lesions (such as benign / malignant correlation features, lesion staging / type-related features, lesion size / location, and other basic clinical information), corresponding to the comprehensive judgment based on clinical indicators in clinical diagnosis. The hierarchical division is not simply based on feature dimensions, but follows the clinical logic of micro supporting meso, meso relating to macro, and macro guiding judgment, ensuring the functional correlation of features at each level.

[0132] Hierarchical heterogeneous graph construction: Using feature units at each level as graph nodes, clearly defining node definitions and edge construction rules to ensure the graph structure aligns with feature characteristics and clinical relevance. First, there is the node definition. The nodes of the CT structural semantic feature map are each pixel point, and each node carries the spatial coordinates and structural semantic response value of the pixel. The nodes of the MRI multi-sequence texture feature map are each sequence unit (divided into 4 sequence units according to T1, T2, FLAIR, and enhanced T1), and each node carries the texture features and sequence identifier of the corresponding sequence. The nodes of the clinically guided image comprehensive feature map are clinical indicator units (divided into 5 core indicator units according to benign or malignant, stage, type, size, and location), and each node carries the feature value and importance weight of the corresponding clinical indicator.

[0133] Secondly, regarding edge construction, nodes within the same level are edged based on feature similarity. Cosine similarity is used for similarity calculation, with a value range of [0,1]. Edges with a similarity ≥ 0.6 are considered valid, and the weight is the similarity value, ensuring that nodes with similar features within the same level form close associations (such as adjacent lesion pixels in CT structures or similar sequence units in MRI). Nodes across levels are edged based on the spatial / clinical correlation of features. The correlation is weighted (0.1-0.9) through clinical priors. For example, the correlation between pixel nodes in the lesion region of a CT structure (middle layer) and high-response sequence units in MRI texture (bottom layer) is set to 0.8; the correlation between clinical benign / malignant indicator units (high layer) and CT lesion structure pixel nodes (middle layer) and MRI high-response sequence units (bottom layer) is set to 0.9; the correlation between clinical location indicator units (high layer) and CT spatial coordinate nodes (middle layer) is set to 0.85; and the correlation between normal tissue nodes and nodes at other levels is set to below 0.1 to avoid invalid associations.

[0134] The final constructed hierarchical heterogeneous graph contains three levels and the number of nodes is controlled between 1200 and 1500 (balancing computational efficiency and feature representation). It retains the independence of features at each level and enables cross-level and cross-modal feature interaction through edge association, laying the structural foundation for subsequent attention weighting and graph convolution aggregation.

[0135] S72. Calculate the overall attention weight for each node using the attention weight calculation mechanism.

[0136] A triple attention weighting mechanism is designed to achieve refined and targeted weighting of features. Each attention module is designed by combining clinical priors and feature response characteristics, as detailed below: Modal attention: The core is to distinguish the importance of features from three modalities: CT, MRI, and clinical, and to adapt to the modal dependence differences of different lesion types. First, the average response values ​​of the three modal features in the lesion region are extracted (the average response value of the lesion pixel node for CT modality, the average feature value of the high-response sequence unit for MRI modality, and the average weight value of the core clinical indicators for clinical modality). The average response value is input into a lightweight fully connected network (16 neurons), and the Sigmoid activation function outputs three normalized weight values ​​(within the range of [0,1], and the sum of the three is 1), corresponding to CT, MRI, and clinical modalities, respectively.

[0137] Based on clinical experience, weighting benchmarks are set. For example, in the identification of small nodules, the MRI texture modality is preferentially assigned a weight of 0.4-0.5, the CT structural modality 0.3-0.4, and the clinical modality 0.1-0.2; in the identification of large masses, the CT structural modality is preferentially assigned a weight of 0.4-0.5, the clinical modality 0.3-0.4, and the MRI texture modality 0.1-0.2. At the same time, modal consistency constraints are added. If the difference in lesion response values ​​between the two modalities is less than 0.2, the weights of the two modalities are appropriately increased to ensure modal complementarity.

[0138] Hierarchical attention: The core is to dynamically adjust the feature weights of the bottom, middle, and top layers according to the needs of lesion identification, aligning with the clinical micro-meso-macro diagnostic logic. First, a hierarchical weight adjustment function is constructed based on the equivalent diameter of the lesion candidate box and the preliminary pathological candidate type. For small nodules (equivalent diameter < 5 mm), the weights of bottom-layer MRI texture (weight 0.4-0.5) and middle-layer CT structure (0.3-0.4) are emphasized, while the weights of high-layer clinical features are set to 0.1-0.2 to highlight micro-texture and meso-texture. Diagnostic value of structure: For large lesions (equivalent diameter > 20 mm), high-level clinical features (0.4-0.5) and mid-level CT structures (0.3-0.4) are given priority, while the weight of low-level MRI texture is set to 0.1-0.2 to highlight macroscopic clinical information and overall structure; For medium-sized lesions, the weights of the three layers are evenly distributed (low-level 0.3-0.35, mid-level 0.35-0.4, high-level 0.25-0.3). After weight calculation, L2 normalization is performed to ensure reasonable weight distribution and avoid extremes.

[0139] Node attention: The core is to select effective nodes related to lesions, filter out normal tissue nodes and noise nodes, and improve feature purity. First, for each node in the hierarchical heterogeneous map, the similarity between its feature response value and the core nodes of the lesion (the gray-scale peak nodes of lesions in CT, the high-response sequence units in MRI, and the clinical benign and malignant indicator nodes) is calculated. The similarity is calculated using Euclidean distance normalization. A node selection threshold is set (similarity ≥ 0.5). Nodes that meet the threshold are assigned high weights (0.7-0.9), nodes below the threshold are assigned low weights (0.1-0.3), and nodes with similarity < 0.2 are directly assigned 0 weights (considered invalid nodes and not participating in subsequent aggregation). At the same time, spatial constraints are added, and the weights of adjacent lesion nodes in the CT structure are mutually reinforced (weight increased by 0.1) to avoid isolated lesion nodes being misclassified as noise, ensuring that node attention can accurately capture the features of the lesion-related area and filter out irrelevant background interference.

[0140] The triple attention weights are fused by element-wise multiplication to obtain the comprehensive attention weight for each node, achieving refined weighting of the three dimensions of modality, hierarchy, and node, and providing targeted guidance for subsequent feature aggregation.

[0141] S73. Obtain multidimensional features through a temporal graph convolutional network, and combine the comprehensive attention weights and the multidimensional features to obtain hierarchical attention fusion features.

[0142] Based on the structural characteristics of hierarchical heterogeneous graphs, a temporal graph convolutional network (TGCN) is used to achieve feature aggregation. This not only preserves the core features of each level but also captures the temporal sequence of feature interactions across levels. This aligns with the temporal logic in clinical diagnosis, which involves first observing microscopic textures, then analyzing mesoscopic structures, and finally combining them with clinical judgment. The specific implementation is as follows: Intra-layer graph convolutional aggregation: For each layer of the hierarchical heterogeneous graph, a lightweight graph convolutional layer (1-layer graph convolution to avoid feature smoothing caused by excessively deep networks) is constructed separately to achieve aggregation of node features within the layer. The core parameters of the graph convolution are designed to fit the feature characteristics of each layer: The graph convolution kernel size of the bottom-layer MRI sequence unit node is set to 3×3, and the edge weights adopt the feature similarity weights within the same layer. During the aggregation process, the texture difference features of each sequence are mainly preserved (such as high signal of T2 sequence and edema signal of FLAIR sequence). After aggregation, a 32-dimensional bottom-layer fusion feature is output; The graph convolution kernel size of the middle-layer CT pixel node is set to 5×5, and the edge weights combine spatial adjacency and feature similarity. An edge enhancement operator is added during the aggregation process to enhance structural features such as lesion edges and lobulation. After aggregation, a 64-dimensional middle-layer fusion feature is output; The graph convolution kernel size of the high-layer clinical indicator node is set to 1×1, and the edge weights adopt the importance weights of clinical indicators. During the aggregation process, the core clinical information such as benign or malignant and stage is mainly preserved. After aggregation, a 32-dimensional high-layer fusion feature is output. Batch normalization (BN) is added after each layer of graph convolution to reduce feature sparsity, suppress overfitting, and preserve the feature independence of each layer.

[0143] Cross-layer temporal convolutional interaction: To capture the temporal interaction of features from the bottom, middle, and top layers, a temporal convolutional layer (TCN) is introduced after aggregation within each layer. The features aggregated from the three layers are input in the temporal order of bottom layer → middle layer → top layer. The kernel size of the temporal convolution is set to 3, and the stride is set to 1. Zero padding is used to ensure consistency between input and output dimensions. The focus is on capturing three types of interaction temporal sequences: first, the support of bottom-layer MRI texture features for middle-layer CT structural features (e.g., texture uniformity corresponds to the benign or malignant tendency of the structure); second, the verification of middle-layer CT structural features for high-layer clinical features (e.g., lobulated structures correspond to tumor staging); and third, the selection of bottom-layer and middle-layer features by high-layer clinical features (e.g., benign or malignant indicators select corresponding texture and structural features).

[0144] During temporal convolution, temporal attention weights are added, assigning high weights (0.7-0.8) to key interaction sequences (such as the verification of structural features by clinical features) and low weights (0.2-0.3) to secondary interaction sequences, ensuring the relevance of interaction features.

[0145] Residual Connectivity and Feature Preservation: To avoid loss of original features during aggregation, a dual residual connectivity mechanism is introduced: First, intra-layer residual connectivity, where the original node features of each layer are directly passed to the output of the convolutional layer of that layer through residual edges, and summed element-wise with the aggregated features, preserving the detailed information of the original features (such as the sequence texture details of MRI and the pixel-level structural information of CT); Second, cross-layer residual connectivity, where the aggregated features of the bottom and middle layers are passed to the output of the temporal convolutional layer of the higher layers through residual edges, and summed with the cross-layer interactive features, ensuring that the core features of the bottom and middle layers are not diluted by the features of the higher layers. Simultaneously, a 1×1 convolutional layer is added to the residual edges to adjust the number of feature channels, ensuring that the feature channels of different layers match, avoiding feature loss due to channel incompatibility. The final output is a high-dimensional hybrid feature that integrates intra-layer aggregation and cross-layer interaction.

[0146] After cross-level aggregation, the high-dimensional hybrid features need further filtering of redundant information and strengthening of core features. Specifically, a dual fusion strategy of attention-weighted summation and channel fusion is adopted to achieve deep fusion of high-dimensional features. First, the three layers of features after intra-level aggregation and cross-level interaction are multiplied by the hierarchical attention weights to obtain the weighted features of each layer. Then, the weighted bottom, middle, and high-level features are summed element-wise to achieve deep feature complementarity. Feature normalization (z-score normalization) is added during the summation process to map the feature values ​​to the [-1,1] interval to avoid the numerical differences of features at different levels from interfering with the fusion effect. Finally, a 1×1 convolutional layer is used to perform channel fusion on the summed features, integrating redundant channels and retaining the core feature channels, initially compressing the 128-dimensional feature channels to 80 dimensions.

[0147] Further channel-based dimensionality reduction was performed on the 80-dimensional features using a lightweight channel filtering network (two fully connected layers, 64 neurons in the first layer and 32 neurons in the second layer). This network incorporated clinical priors to select core feature channels—prioritizing channels related to lesion structure, texture, and clinical diagnosis (such as CT edge feature channels, MRI texture uniformity channels, and clinical benign / malignant indices) while filtering channels related to background noise and irrelevant tissues. Channel attention calibration was incorporated during dimensionality reduction, assigning high weights (0.7-0.9) to core channels and low weights (0.1-0.3) to secondary channels to ensure that the representational power of the features was not lost after dimensionality reduction. Ultimately, the feature channels were compressed to 64 dimensions, yielding a preliminary hierarchical attention-fusion feature vector.

[0148] To ensure that the generated features meet clinical needs and are free from noise interference, the 64-dimensional feature vector is post-processed: first, min-max normalization is used to map the feature values ​​to the [0,1] interval; second, feature validity verification is added by calculating the similarity between the feature vector and CT structural features, MRI texture features, and clinical comprehensive features. If the similarity is ≥0.6 (indicating that the feature retains the core information of the previous steps), it is considered a valid feature; if the similarity is <0.6, the process returns to the hierarchical aggregation step to readjust the graph convolution parameters to ensure the consistency and validity of the features.

[0149] The final 64-dimensional hierarchical attention fusion feature vector fully preserves the spatial structural features of CT, the multi-sequence texture features of MRI, and the core information for clinical guidance. Furthermore, it has undergone triple attention-based weighting and multiple rounds of optimization, giving it extremely strong targeting and representation capabilities.

[0150] The second fusion module 8 is used to obtain complementary metabolic dynamics features by combining PET metabolic heterogeneity map features and ultrasound dynamic motion features through a gated cross-attention graph matching algorithm.

[0151] PET metabolic features (static, metabolic heterogeneity) and ultrasound dynamic features (dynamic, rhythmic motion) are highly complementary lesion features (e.g., high-metabolic tumor lesions are often accompanied by differences in motion compared to surrounding tissues). Based on a graph matching algorithm, by incorporating gated cross-attention, bidirectional information interaction and weighting of the two features are achieved. Complementarity-guided graph matching finds the associated matching nodes between the two features, allowing metabolic and motion features to mutually reinforce each other, achieving true complementary fusion. The graph matching algorithm using gated cross-attention, combined with PET metabolic heterogeneity map features and ultrasound dynamic motion features, obtains complementary metabolic dynamic features, including the following steps: S81, Reconstructing the graph structure of PET metabolic heterogeneity map features and ultrasound dynamic motion features.

[0152] Considering the differences in characterization between PET metabolic heterogeneity mapping features and ultrasound dynamic motion features, PET features emphasize static metabolic topological correlations, while ultrasound features emphasize dynamic motion temporal correlations. Therefore, it is necessary to first reconstruct both into adaptive, complementary, and matched feature sub-maps to ensure the accuracy of subsequent matching. The specific implementation is as follows: PET Subgraph Reconstruction: Based on the PET metabolic heterogeneity map features and clinical PET metabolic diagnostic criteria, metabolic heterogeneity feature units are divided into three types of nodes according to the SUV value threshold: high metabolic core area (SUV≥2.5, corresponding to the core tumor lesion), metabolic gradient area (1.5≤SUV<2.5, corresponding to the tumor infiltration area), and low metabolic edge area (SUV<1.5, corresponding to edema / normal tissue around the lesion). The node features are the mean SUV value and metabolic variation coefficient of the region. The edge construction continues the metabolic-oriented approach, using the metabolic similarity between nodes (the reciprocal of the normalized Euclidean distance) as the basic weight, while adding spatial adjacency constraints (only connecting nodes with a spatial distance of <3 pixels) to avoid invalid connections between distant and unrelated metabolic nodes. The edge weight is calculated by weighting metabolic similarity (70%) and spatial adjacency (30%) to ensure that the edge can reflect both metabolic and spatial associations, echoing the graph structure.

[0153] Ultrasound sub-image reconstruction: Based on the dynamic motion characteristics of ultrasound, dynamic motion feature units are divided into three types of nodes according to the threshold of motion feature values ​​(pixel displacement, grayscale change rate): motion peak region (motion feature value ≥ 0.8, corresponding to the motion region of the lesion contraction / dilution extreme frame), rhythm change region (0.4 ≤ motion feature value < 0.8, corresponding to the region of rhythmic change of lesion motion), and motion plateau region (motion feature value < 0.4, corresponding to the region of motion of normal tissue around the lesion). The node features are the motion feature value of the region and the inter-frame motion correlation. The construction of edges focuses on temporal correlation, using the inter-frame motion correlation between nodes as the weight, and only connecting the corresponding motion units in adjacent temporal frames to capture the temporal continuity of lesion motion, which is consistent with the temporal feature extraction logic.

[0154] Both subgraphs control the number of nodes to be between 50 and 100 (balancing computational efficiency and feature representation capability), and the graph structure dimension matches the original feature dimension (32 dimensions) to ensure the complete preservation of the topological association of PET metabolism and the temporal association of ultrasound motion, laying the structural foundation for subsequent dual-graph matching and complementary fusion.

[0155] S82. Based on gated cross-attention, calculate bidirectional attention weights and introduce complementary constraints to achieve topology matching of nodes in the two graphs.

[0156] Conventional cross-attention only enables unidirectional information exchange between nodes in two images and lacks a targeted noise filtering mechanism, easily introducing invalid information from nodes with low complementarity, thus failing to meet the clinical complementary needs of PET metabolic and ultrasound motion features. Therefore, this invention constructs a bidirectional cross-attention layer, combined with a gating unit, to achieve precise interaction of highly complementary information. The specific design is as follows: The core structure of bidirectional cross-attention is as follows: the node features of the PET sub-image and the ultrasound sub-image are used as inputs for bidirectional interaction. The PET sub-image node features (32-dimensional) serve as both query (Q) and key (K), while the ultrasound sub-image node features (32-dimensional) serve as value (V), thus completing the attention weighting of ultrasound features by PET. At the same time, the ultrasound sub-image node features serve as Q and K, and the PET sub-image node features serve as V, thus completing the attention weighting of PET features by ultrasound, forming a bidirectional interactive closed loop.

[0157] Attention weights are calculated using a scaled dot product attention method. Dividing by the square root of the feature dimension alleviates the gradient vanishing problem and ensures the stability of weight calculation. After Softmax normalization, the weight values ​​are mapped to the [0,1] interval, accurately quantifying the complementary correlation between nodes in the two graphs.

[0158] Complementary correlation point capture: Based on clinicopathological priors (such as high-metabolic tumor lesions usually accompanied by differences in motion with surrounding tissues, and benign nodules having smooth metabolism and motion rhythm consistent with surrounding tissues), the focus is on capturing the correlation of highly complementary nodes, such as nodes in the high-metabolic core area of ​​PET sub-images and nodes in the motion peak area of ​​ultrasound sub-images, nodes in the metabolic gradual change area of ​​PET sub-images and nodes in the rhythm change area of ​​ultrasound sub-images. The information interaction of such nodes is strengthened by attention weight (weight ≥ 0.7), while the interaction of low-complementary nodes (such as low-metabolic areas of PET and smooth motion areas of ultrasound) is weakened (weight ≤ 0.3).

[0159] Gating Unit Design: A sigmoid gating unit is added after the cross-attention layer. The input of the gating unit is the complementarity score of the nodes in both images. This score is obtained by calculating the cosine similarity between the features of PET and ultrasound nodes, and a complementarity threshold is set based on clinical priors (≥0.7 for high complementarity, <0.7 for low complementarity). When the complementarity score is ≥0.7, the output of the gating unit approaches 1, allowing the attention interaction information of that node to pass through; when the score is <0.7, the output of the gating unit approaches 0, blocking the flow of low-complementarity noise information, effectively filtering irrelevant noise, and ensuring that the information that passes through is all high-complementarity core information, further improving the targeting and accuracy of attention weighting.

[0160] Furthermore, based on the bidirectional attention weights calculated using gated cross-attention, direct node matching can easily lead to mismatches and invalid matches (such as matching high-metabolic nodes in PET with nodes exhibiting smooth motion in ultrasound), failing to fully leverage the complementary advantages of both. Therefore, this invention introduces a complementarity constraint and employs the Hungarian algorithm to achieve accurate topological matching of nodes in both graphs, as detailed below: Topology matching initialization: Based on the bidirectional attention weight matrix output by gated cross-attention, a matching cost matrix for nodes in both graphs is constructed. The elements of the cost matrix are 1 - attention weight, that is, the higher the attention weight, the lower the matching cost, ensuring that nodes with high complementarity are matched first. At the same time, topology constraints are added, requiring that the matched nodes have consistent topological positions in their respective subgraphs (e.g., nodes in the core area of ​​the PET subgraph only match nodes in the core area of ​​the ultrasound subgraph to avoid invalid cross-region matching). The constraint weight is set to 0.2 to balance complementarity and topology consistency.

[0161] Node matching execution: The Hungarian algorithm is used to optimize the matching cost matrix. This algorithm can find the globally optimal matching scheme in polynomial time, avoiding the complementarity loss caused by local optimal matching. During the matching process, a matching threshold is set (matching cost ≤ 0.3, corresponding attention weight ≥ 0.7). Only node pairs that meet the threshold requirements are retained as valid matching node pairs. For nodes that do not meet the threshold, matching is not performed temporarily. Their original features are supplemented through residual connections later to avoid the loss of effective features.

[0162] Complementarity Matching Loss Optimization: To further ensure high complementarity of matched nodes, a complementary matching loss function is introduced. This function is constructed based on the feature cosine similarity of the matched node pairs, satisfying: The matching loss is calculated as 1 - mean(cos(PET matching node features, ultrasound matching node features)), where mean refers to the average function and cos refers to the cosine similarity. This loss is minimized through backpropagation, forcing the feature similarity of matching node pairs to continuously increase, ensuring that all matched nodes are highly complementary. Simultaneously, an L1 regularization term is added to limit the fluctuation range of the matching weights and avoid overfitting; the regularization coefficient is set to 0.001. When the matching loss converges to below 0.3 and the proportion of effective matching node pairs is ≥80%, the matching iteration stops, completing the accurate matching of nodes in both graphs. This effectively avoids invalid matching and mismatch issues, laying the foundation for subsequent feature aggregation.

[0163] S83. Aggregate the features of the matching node pairs, extract global complementary fusion features, and then generate metabolic dynamic complementary features.

[0164] After accurate matching of nodes in both graphs, the features of the matched node pairs need to be aggregated to extract globally complementary fused features. An improved graph convolution is used for aggregation, combined with residual connections to achieve feature optimization. The specific logic is as follows: Matching feature preprocessing: The PET and ultrasound features of the effective matching node pairs are summed and fused element by element to obtain preliminary matching fusion features. Feature normalization (z-score normalization) is added during the fusion process to map the feature values ​​to the [-1,1] interval to avoid the numerical difference between the two features from interfering with the aggregation effect. For unmatched nodes, their original features (PET or ultrasound features) are retained separately and subsequently supplemented into the fusion features through residual connections to ensure that no effective features are lost.

[0165] Improved Graph Convolutional Aggregation: A lightweight graph convolutional layer (one layer to avoid feature smoothing caused by excessively deep networks) is constructed. The matched two graphs are used as input, and the node features of the graph convolution serve as the initial matching and fusion features. The edge weights are the attention weights from the gated cross-attention output. During graph convolutional aggregation, node complementarity weights (calculated based on complementarity scores) are introduced. High aggregation weights (0.8-0.9) are assigned to highly complementary node pairs (complementarity score ≥ 0.8), while low aggregation weights (0.5-0.7) are assigned to low complementary matching node pairs (0.7 ≤ complementarity score < 0.8), achieving targeted aggregation and highlighting the core information of highly complementary nodes. After aggregation, a global preliminary fusion feature is output, which integrates the complementary information of the matched node pairs and the topological / temporal associations between the two graphs.

[0166] Residual connection optimization: To avoid the loss of original features during the aggregation process, dual residual connections are introduced: First, local residual connections, which directly pass the preliminary matching fusion features to the graph convolution output through residual edges, and sum them element-wise with the aggregated features, preserving the original complementary information of the matching nodes; Second, global residual connections, which supplement the original features (PET or ultrasound) of unmatched nodes into the global preliminary fusion features through residual edges, ensuring that all effective features are utilized.

[0167] Meanwhile, batch normalization (BN) is added after the graph convolutional layer to reduce feature sparsity and suppress overfitting. The momentum of batch normalization is set to 0.9. Finally, 1×1 convolution is used to reduce the channel dimension of the aggregated features, compressing the number of feature channels to 32 dimensions, and obtaining the optimized global matching fusion features.

[0168] The optimized global matching fusion features are high-dimensional graph structure features (50-100 nodes, 32-dimensional feature channels). Global pooling and feature calibration are used to generate the final metabolic dynamic complementary features, as detailed below: Global Hybrid Pooling: A hybrid pooling method combining global average pooling and global max pooling is adopted to take into account both global features and local core features: global average pooling calculates the pixel mean of each feature channel to capture the global topological correlation of PET metabolism and the global temporal rhythm of ultrasound motion; global max pooling extracts the maximum value of each feature channel to capture the local core features of high metabolic core areas and motion peak areas; the results of the two pooling are summed element-wise with a 1:1 weight to obtain a preliminary 32-dimensional feature vector, which retains global complementary information while highlighting local core features.

[0169] Feature calibration optimization: To further enhance the specificity of complementary features, a feature calibration module is added. The importance weight of each channel in the 32-dimensional feature vector is calculated through the channel attention mechanism. The channel attention adopts the SE-Net lightweight structure (1 fully connected layer + Sigmoid activation). Channels with strong metabolic-motor complementarity (such as the channel associated with PET high metabolism and ultrasound motion peak) are given high weight (0.7-0.9), while channels with weak complementarity (such as background noise channels) are given low weight (0.1-0.3), thereby achieving feature calibration and improving feature representation capabilities.

[0170] Final Feature Generation: The calibrated 32-dimensional feature vector is normalized (min-max normalized to the [0,1] interval) to obtain the final metabolic dynamic complementary feature vector. This feature vector fully realizes the complementary enhancement of PET metabolic heterogeneity (static) and ultrasound dynamic motion characteristics (dynamic), preserving both the global topological correlation and local heterogeneity features of PET metabolism, and integrating the temporal rhythm and motion difference features of ultrasound dynamics, while filtering out irrelevant noise to ensure the relevance and effectiveness of the features.

[0171] The lesion identification module 9 is used to identify lesions by using hierarchical dynamic fusion of meta-learning and utilizing pathology-guided image semantic features, hierarchical attention fusion features, and metabolic dynamic complementary features.

[0172] In this embodiment, the hierarchical dynamic fusion through meta-learning, utilizing pathology-guided image semantic features, hierarchical attention fusion features, and metabolic dynamic complementary features to complete lesion identification, includes the following steps: S91. Based on functional priority and representation depth, hierarchical modeling of representation hierarchy is performed on pathology-guided image semantic features, hierarchical attention fusion features, and metabolic dynamic complementary features.

[0173] The pathology-guided imaging semantic features, hierarchical attention fusion features, and metabolic dynamic complementary features are divided into a clear three-layer structure according to their functional priority and representational depth in lesion identification. The positioning, functional boundaries, and clinical value of each layer are clearly defined to ensure that the hierarchical division aligns with the core logic of clinical diagnosis: pathology first, imaging as evidence, and supplementary methods as auxiliary. Core layer: Pathology-guided image semantic features. These features integrate the pathology-image association priors from the medical knowledge graph with multimodal image core semantics, guiding the pathological logic judgment for lesion identification and providing core pathological priors for lesion identification (such as the correspondence between pathological types and image representations). They are the key basis for distinguishing between benign and malignant lesions and clarifying lesion classification. The depth of their feature representation reaches the association level of clinical pathological diagnosis and can directly relate to the pathological essence of lesions.

[0174] The base layer features a layered attention fusion feature that has completed the layered and refined fusion of CT structural semantics, MRI multi-sequence texture, and clinical guidance information. It can provide spatial structure of lesions (such as lobulation and spiculation), microscopic texture (such as signal uniformity), and basic clinical information (such as lesion size and location), providing solid imaging support for the pathological logic judgment of the core layer. It fills the gap between the pathological prior and the actual imaging features and is the core foundation for lesion localization and preliminary feature screening.

[0175] Supplementary Layer: Metabolic dynamic complementary features. This feature achieves complementary enhancement of PET metabolic heterogeneity (static) and ultrasound dynamic motion features (dynamic). It can provide supplementary information such as the distribution of metabolic activity of lesions, motion rhythm, and motion differences with surrounding tissues. It supplements and enhances lesions with metabolic abnormalities and dynamic motion abnormalities (such as metabolically active tumors and abnormally moving benign nodules) that are not covered by the core layer and the basic layer, further improving the comprehensiveness of lesion identification and reducing the probability of missed diagnosis and misdiagnosis.

[0176] The hierarchical division of the three features is not arbitrary, but based on the clinical diagnostic process and the value of feature representation, forming a closed-loop logic of pathology-led, imaging-supported, and supplementary enhancement, ensuring that each feature performs its own function and works synergistically.

[0177] S92. Construct and divide the meta-learning task set and scene-adaptive hierarchical dynamic weights, and complete the hierarchical dynamic fusion and lesion identification reasoning after training through model-independent meta-learning.

[0178] Considering the significant scenario heterogeneity in lesion identification in clinical practice (lesions in different organs and with different pathological types have significantly different feature distributions and identification difficulties), and that clinical labeled data are mostly small samples (e.g., very few labeled samples for rare lesions), this invention constructs a meta-learning task set that fits real-world application scenarios based on actual clinical datasets. The specific implementation is as follows: First, we collected a dataset of lesion annotations covering multiple organs and pathological types (covering common organs such as the lung, liver, breast, and pancreas, and including more than 10 common and rare lesion types such as squamous cell carcinoma, adenocarcinoma, hepatocellular carcinoma, benign nodules, and inflammatory lesions). The dataset contains the above three features and corresponding clinical annotations (lesion presence, pathological type, benign or malignant, and organ type).

[0179] Secondly, based on the combination of organ type and pathological type, the overall dataset is divided into multiple independent meta-learning task sets. Each task set corresponds to a specific lesion recognition scenario (such as lung adenocarcinoma recognition task, benign liver nodule recognition task, and breast inflammatory lesion recognition task), ensuring that the scenario differences between different task sets are significant and simulating the recognition needs of different lesions in clinical practice.

[0180] Finally, each meta-learning task set is divided into a support set: query set ratio of 1:3 (to match the small sample scenario in clinical practice). The support set consists of small sample data (10-20 samples per task set, simulating the small sample annotation scenario of rare lesions and new organ lesions in clinical practice), used to quickly adjust model parameters and adapt the model to the specific scenario. The query set consists of validation samples (30-50 samples per task set), used to verify the effect of parameter adjustment and evaluate the model's recognition performance in the scenario.

[0181] Meanwhile, to avoid overfitting of the task sets, slight data augmentation (such as feature perturbation and slight rotation) is performed on the support set of each task set to ensure that the model still has good generalization ability in small sample scenarios. The task set division process strictly follows the distribution law of clinical data to ensure the representativeness and practicality of the task sets.

[0182] Furthermore, a scene-adaptive hierarchical dynamic weight module is added to the meta-learning backbone network (using a residual convolution + fully connected structure) to achieve real-time adaptive adjustment of the three-layer feature fusion weights under different scenarios. The specific design is as follows: First, the module's input is the scene features of the task set. The scene features consist of unique thermal encoding of organ type + pathology type. For 6 common organs, 6-dimensional unique thermal encoding is used; for 12 common pathology types, 12-dimensional unique thermal encoding is used. The two are concatenated to form an 18-dimensional scene feature vector, ensuring that the scene features can accurately represent the current lesion identification scene.

[0183] Secondly, the main body of the module adopts a lightweight fully connected network (2 hidden layers, with 64 neurons in the first hidden layer, 32 in the second hidden layer, and 3 in the output layer). The fully connected network performs feature mapping and weight calculation on scene features, and finally outputs 3 normalized weight values, which correspond to the fusion weights of the core layer, the basic layer, and the supplementary layer, respectively. The weight values ​​range from [0,1], and the sum of the three is 1, so as to avoid extreme weight distribution.

[0184] Furthermore, a weight constraint mechanism is added, which limits the range of weight fluctuations through L2 regularization. At the same time, a gating unit is introduced to appropriately reduce the weight adjustment range based on the confidence level of the scene characteristics (such as when the confidence level of rare lesion scenes is low), so as to ensure the rationality of weight allocation.

[0185] Finally, adaptive weight adjustment logic is set for different scenarios, and weight adjustment benchmarks are set based on clinical diagnostic experience (e.g., in the tumor lesion recognition scenario, the core layer pathological feature weight is preferentially allocated to 0.4-0.6, the basic layer to 0.2-0.3, and the supplementary layer to 0.1-0.2; in the benign nodule recognition scenario, the basic layer imaging feature weight is preferentially allocated to 0.4-0.5, the core layer to 0.2-0.3, and the supplementary layer to 0.1-0.2; in the metabolic abnormality lesion recognition scenario, the supplementary layer metabolic dynamic feature weight is preferentially allocated to 0.3-0.4, the core layer to 0.3-0.4, and the basic layer to 0.2-0.3), replacing the traditional fixed fusion weights, so that the weight allocation is more in line with the recognition needs of specific scenarios and achieves accurate matching between scenarios and weights.

[0186] Considering the need for lesion identification in small samples and multiple scenarios in clinical practice, an improved Model-Independent Meta-Learning (MAML) training strategy is adopted to optimize the problem of fixed learning rates in the inner and outer loops of traditional MAML, enabling the model to quickly adapt to different lesion identification scenarios. The specific training process is as follows: Global Meta-Training Phase: Using all constructed meta-learning task sets as training objects, a batch sampling strategy is adopted (sampling 4-6 different task sets each time). The support set and query set features of each task set are input into the meta-learning model. The recognition loss of each task set is calculated through forward propagation (using cross-entropy loss combined with sample weighting to solve the problem of imbalanced labeling in small samples). The general parameters of the model (including the basic parameters of the meta-learning backbone network and hierarchical dynamic weight module) are updated through backpropagation. The model learns general fusion rules and recognition patterns under different scenarios, enabling the model to have basic lesion recognition capabilities. The outer loop learning rate of global meta-training is set to 0.0001, and the number of training iterations is 5000. After each iteration, the performance of the model on the unsampled task set is verified to ensure the general generalization ability of the model.

[0187] Task-level rapid fine-tuning phase: For each independent meta-learning task set, rapid gradient descent fine-tuning is performed using the support set (small sample) of that task set. During the fine-tuning process, only the parameters of the hierarchical dynamic weight module are adjusted, without changing the general parameters of the meta-learning backbone network, ensuring fine-tuning efficiency while avoiding damage to the model's general recognition ability. The learning rate of the inner loop of the task-level fine-tuning is set to 0.001, and the number of fine-tuning iterations is 10-20 times (fitting the requirement of rapid adaptation with small samples). The goal of fine-tuning is to enable the hierarchical dynamic weight module to accurately output the optimal fusion weights in this scenario.

[0188] Training optimization and termination conditions: During training, dynamic gradient clipping (gradient norm threshold set to 1.0) is introduced to avoid gradient explosion; Dropout regularization is added (dropout probability set to 0.3) to reduce model overfitting; when the average recognition accuracy of the meta-test set (scene task set independent of the meta-training set) is stable above 95% and the recognition accuracy fluctuation of each task set is less than 2%, training is stopped, and the trained meta-learning model is obtained.

[0189] The core advantage of this training strategy is that by combining global general training with task-level small sample fine-tuning, the model can not only master the general recognition rules of different scenarios, but also quickly adapt to specific lesion recognition scenarios. This solves the problem of poor model generalization ability in small sample scenarios in clinical practice. Moreover, compared with traditional meta-learning, the training efficiency is improved by more than 30%, which is more in line with the actual clinical application needs.

[0190] After training, it can be directly used for the identification of new lesion samples, realizing scene-adaptive hierarchical dynamic fusion and inference. The specific process follows the clinical diagnostic process, ensuring that the inference results are accurate and usable. The steps are as follows: Scene feature extraction: For newly input lesion samples, the scene features are first extracted. If the sample contains clinical annotations (organ type, pathological candidate type), the annotation information is directly one-hot encoded to generate an 18-dimensional scene feature vector. If the sample does not have clear clinical annotations, the basic clinical information in the basic layer features (hierarchical attention fusion features) is combined with a lightweight classifier to adaptively identify the organ type and preliminary pathological candidate type of the sample and generate scene features, ensuring that unannotated samples can also achieve scene adaptation.

[0191] Adaptive weight calculation: Input scene features into the trained hierarchical dynamic weight module. The module quickly calculates the fusion weights of the core layer, basic layer and supplementary layer in the scene. The weight values ​​are automatically normalized to ensure that the sum of the three is 1.

[0192] Layered dynamic fusion: The three layers of features are weighted and fused according to the calculated fusion weights. The specific fusion formula is: Global fusion feature = Core layer feature × Core layer weight + Basic layer feature × Basic layer weight + Supplementary layer feature × Supplementary layer weight. Feature normalization is added during the fusion process to avoid the numerical differences of features in different dimensions from interfering with the fusion effect, and the final global fusion feature is generated. This feature integrates multi-dimensional information such as pathology, imaging, metabolism, and dynamics, and is in line with the recognition needs of the current scenario.

[0193] Lesion identification and result output: The global fusion features are input into a lightweight classifier (using a fully connected layer + Softmax activation function), and the lesion identification results are output, including the presence (present / absent), pathological type, and benign or malignant nature of the lesion. At the same time, the confidence level of each identification result is output (value range [0,1]). Results with a confidence level below 0.8 are marked as suspected, prompting clinicians to further verify. Finally, the identification results are post-processed and combined with clinical diagnostic thresholds (e.g., a confidence level of ≥0.8 for benign or malignant determination can confirm the diagnosis, 0.6-0.8 is suspected, and <0.6 is excluded) to filter false positive results, ensuring that the output results conform to the clinical diagnostic criteria and can directly provide a reference for clinical diagnosis.

[0194] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be covered within the scope of the present invention.

Claims

1. A hierarchical dynamic fusion lesion identification system based on multi-source data, characterized in that, Includes the following modules: The CT feature extraction module is used to extract semantic feature maps of CT structures through an adaptive spatial pyramid. An MRI feature extraction module is used to extract MRI multi-sequence texture features through an attention-enhanced frequency-space dual-stream network. The PET feature extraction module is used to extract PET metabolic heterogeneity profile features through graph convolutional networks. The ultrasound feature extraction module is used to extract dynamic motion features of ultrasound through a time-aware recurrent network. The image semantic feature extraction module is used to extract pathology-guided image semantic features through a generative adversarial network based on a medical knowledge graph. The multi-task learning module is used to extract comprehensive image features for clinical guidance through multi-task learning; The first fusion module is used to fuse CT structural semantic feature maps, MRI multi-sequence texture features, and clinically guided image comprehensive features through a hierarchical attention temporal graph convolutional network to obtain hierarchical attention fusion features; The second fusion module is used to obtain complementary metabolic dynamic features by combining PET metabolic heterogeneity map features and ultrasound dynamic motion features through a gated cross-attention graph matching algorithm. The lesion identification module is used to identify lesions by using hierarchical dynamic fusion through meta-learning, utilizing pathology-guided image semantic features, hierarchical attention fusion features, and metabolic dynamic complementary features.

2. The hierarchical dynamic fusion lesion identification system based on multi-source data as described in claim 1, characterized in that, The extraction of CT structural semantic feature maps through adaptive spatial pyramids includes the following steps: The pyramid hierarchy and receptive field are dynamically adjusted based on the size of the lesion candidate bounding box to construct an adaptive spatial pyramid. The adaptive spatial pyramid is used to extract multi-scale structural features; Based on the multi-scale structural features, a semantic feature map of CT structure is generated through cross-level feature residual fusion and semantic enhancement.

3. The hierarchical dynamic fusion lesion identification system based on multi-source data as described in claim 1, characterized in that, The extraction of MRI multi-sequence texture features using an attention-enhanced frequency-space dual-stream network includes the following steps: Frequency domain branch texture features are extracted using two-dimensional fast Fourier transform; Spatial domain branch texture features are extracted using a lightweight convolutional network; Channel attention is used to achieve sequence-level attention enhancement weighting. The frequency domain texture features and spatial domain texture features after attention weighting are summed and fused element-wise to generate MRI multi-sequence texture features.

4. The hierarchical dynamic fusion lesion identification system based on multi-source data driving according to claim 1, characterized in that, The extraction of PET metabolic heterogeneity profile features using graph convolutional networks includes the following steps: Based on metabolic activity similarity and spatial adjacency constraints, a graph structure oriented towards metabolic heterogeneity is constructed. By introducing the node metabolic variation coefficient as an attention weight, a graph convolutional layer with enhanced heterogeneity is designed. Based on the graph structure, global heterogeneity features are extracted using the graph convolutional layer to generate PET metabolic heterogeneity map features.

5. The hierarchical dynamic fusion lesion identification system based on multi-source data as described in claim 1, characterized in that, The extraction of ultrasonic dynamic motion features through a time-aware recurrent network includes the following steps: Filter time-aware lesion keyframes using motion feature values; Based on the spatial features of the keyframes, the ultrasonic dynamic motion features are obtained using a temporally attention-enhanced LSTM model.

6. The hierarchical dynamic fusion lesion identification system based on multi-source data driving according to claim 1, characterized in that, The extraction of pathology-guided image semantic features using a generative adversarial network based on a medical knowledge graph includes the following steps: Filter the lesion subgraphs of the medical knowledge graph to obtain knowledge embedding vectors; Construct a generator for pathological constraints and a discriminator for pathological perception; By combining the knowledge embedding vector, the generator, and the discriminator, pathology-guided image semantic features are extracted.

7. The hierarchical dynamic fusion lesion identification system based on multi-source data as described in claim 1, characterized in that, The extraction of clinically guided image integration features through multi-task learning includes the following steps: Screening auxiliary tasks based on their relevance to clinical tasks; A shared-branch multi-task feature extraction network was constructed, and the loss function was optimized based on the imbalance of medical data annotation and the difference in task difficulty. Clinically-guided imaging features are obtained through multi-task joint training with dynamic gradient weighting.

8. The hierarchical dynamic fusion lesion identification system based on multi-source data as described in claim 1, characterized in that, The method of obtaining hierarchical attention fusion features by fusing CT structural semantic feature maps, MRI multi-sequence texture features, and clinically guided image comprehensive features through a hierarchical attention temporal graph convolutional network includes the following steps: Construct a hierarchical heterogeneous graph, and realize the structured association of features through the hierarchical heterogeneous graph; The attention weight of each node is obtained by using the attention weight calculation mechanism; Multidimensional features are obtained through a temporal graph convolutional network, and hierarchical attention fusion features are obtained by combining the comprehensive attention weights and the multidimensional features.

9. The hierarchical dynamic fusion lesion identification system based on multi-source data as described in claim 1, characterized in that, The graph matching algorithm using gated cross-attention, combined with PET metabolic heterogeneity map features and ultrasound dynamic motion features, obtains complementary metabolic dynamic features, including the following steps: Reconstructing the graphical structure of PET metabolic heterogeneity profiles and ultrasound dynamic motion characteristics; Based on gated cross-attention calculation of bidirectional attention weights, complementary constraints are introduced to achieve topological matching of nodes in the two graphs; The features of matching node pairs are aggregated, and global complementary fusion features are extracted to generate metabolic dynamic complementary features.

10. The hierarchical dynamic fusion lesion identification system based on multi-source data as described in claim 1, characterized in that, The hierarchical dynamic fusion through meta-learning utilizes pathology-guided image semantic features, hierarchical attention fusion features, and metabolic dynamic complementary features to complete lesion identification, including the following steps: Based on functional priority and representation depth, hierarchical modeling of representation hierarchy is performed on pathology-guided image semantic features, hierarchical attention fusion features, and metabolic dynamic complementary features. We construct and partition a meta-learning task set and a scene-adaptive hierarchical dynamic weight, and complete the hierarchical dynamic fusion and lesion identification inference after training through model-independent meta-learning.

Images