Non-aligned multi-modal skin lesion classification system based on graph attention network

Abstract

The application discloses a non-aligned multi-modal skin lesion classification system based on a graph attention network, which comprises a lesion perception significant enhancement feature extraction module, a multi-modal heterogeneous graph topology construction module, a double-layer graph attention dynamic fusion module and a multi-task classification prediction module.The non-aligned multi-modal skin lesion classification system based on the graph attention network solves the technical defects of poor anti-interference ability of lesion feature extraction, poor modal non-aligned fusion effect and shallow utilization of patient metadata of the existing multi-modal skin lesion classification method, realizes adaptive fusion of multi-source information without explicit alignment, improves the comprehensive diagnosis accuracy of a seven-point checklist multi-subtask and skin lesion, and reduces the clinical missed diagnosis and misdiagnosis rate.

Description

Technical Field

[0001] This invention belongs to the fields of intelligent medical image diagnosis, multimodal deep learning, and graph neural network technology, and specifically relates to a non-aligned multimodal skin lesion classification system based on graph attention network. Background Technology

[0002] Skin cancer is one of the fastest-growing malignant tumors globally. While malignant melanoma accounts for a small percentage, it carries a high risk of death, making early and accurate classification crucial for improving patient prognosis. In clinical diagnosis, clinical images provide macroscopic information such as the overall outline of the lesion, color distribution, and surrounding skin background, while dermoscopic images magnify and reveal microscopic pathological features such as pigment networks, blue-white veils, and vascular structures. These two types of image information are complementary, but they naturally exhibit misalignment in imaging perspective, scale, and semantic granularity. Existing multimodal skin lesion classification technologies suffer from the following main limitations: 1. The feature extraction module lacks the ability to adapt to irregular lesion morphology: Existing studies generally use general convolutional networks such as ResNet to complete feature extraction. Standard convolution uses a fixed rectangular sampling grid, which cannot adapt to the irregular shape and blurred boundaries of skin lesions. At the same time, common artifacts such as hair, oil, and skin folds can seriously interfere with the feature extraction process. General networks have difficulty focusing on the effective area of ​​lesions, resulting in the loss of key diagnostic features.

[0003] 2. The multimodal fusion mechanism has obvious limitations: Clinical images and dermoscopy images exhibit dual misalignment characteristics: firstly, spatial misalignment, as the two types of images are captured from different perspectives and have different areas of interest, lacking pixel-level or region-level spatial correspondence; secondly, semantic misalignment, as dermoscopy focuses on subsurface microstructures through optical magnification, while clinical images present the macroscopic morphology of lesions, resulting in a scale difference in semantic granularity. Existing fusion methods often force alignment relationships through attention or adversarial learning, which easily introduces redundant matching and noise propagation under conditions of spatial geometric mismatch and semantic scale differences. Furthermore, patient metadata is usually only superficially stitched together before the classifier and does not deeply participate in the feature interaction process.

[0004] In summary, existing technologies are insufficient to meet the high-precision, low-miss-diagnosis-misdiagnosis requirements for skin lesion classification in real-world, non-aligned, multimodal clinical scenarios. There is an urgent need for an intelligent classification system that can enhance the visual representation of lesions, adapt to the non-aligned relationship between clinical images and dermoscopic images, and enable patient metadata to be deeply involved in structured fusion. Summary of the Invention

[0005] The main objective of this invention is to provide a non-aligned multimodal skin lesion classification system based on graph attention networks, addressing technical problems in existing multimodal skin lesion classification methods such as insufficient anti-interference capability of lesion feature extraction, poor fusion effect of non-aligned clinical images and dermoscopic images, and superficial utilization of patient metadata. This invention significantly enhances lesion perception and achieves adaptive fusion of multi-source information without explicit spatial or semantic alignment through heterogeneous graph attention fusion, improving the accuracy of seven-point checklist criterion classification and comprehensive diagnosis of skin lesions.

[0006] To achieve the above objectives, this invention provides a non-aligned multimodal skin lesion classification system based on graph attention networks, comprising a lesion perception significantly enhanced feature extraction module, a multimodal heterogeneous graph topology construction module, a two-layer graph attention dynamic fusion module, and a multi-task classification prediction module, wherein: The lesion perception significantly enhanced feature extraction module is configured with a dual-branch visual backbone network, which receives clinical images and dermoscopic images respectively. The LASE module is embedded after multiple convolution stages of the backbone network. The lesion perception channel is enhanced through lesion perception channel screening, deformable convolution offset sampling and spatial-channel significant gating to enhance the visual representation of lesions. The features of the two images are uniformly mapped into a 512-dimensional visual feature vector. The multimodal heterogeneous graph topology construction module is used to construct a heterogeneous graph containing multiple nodes, including A clinical image nodes, A dermoscopy image nodes, and B metadata nodes. The dual-layer graph attention dynamic fusion module uses a two-layer multi-head graph attention network to adaptively aggregate and fuse the initial features of multiple nodes, and outputs the fused 512-dimensional features of each node. The multi-task classification prediction module combines the fusion features of clinical image nodes and dermoscopy image nodes with the same semantic meaning, inputs them into a parallel independent two-layer fully connected classification head, and simultaneously outputs the classification results of the examination list sub-tasks and the comprehensive diagnostic classification results of skin lesions.

[0007] As a further preferred technical solution to the above technical solution, the internal processing flow of the LASE module specifically includes: The lesion perception channel screening unit performs channel compression and screening on the input features, reduces redundant channel responses, and retains key features related to lesion texture, pigment network, and vascular pattern. Deformable convolutional offset sampling units learn the spatial offset of convolutional sampling points and construct an adaptive sampling grid to match skin lesion areas with blurred boundaries and irregular shapes; The spatial-channel salient gating unit generates salient gating weights based on spatial location and channel response, and applies the salient gating weights to the original input features to enhance the response of the lesion area and suppress hair, oil, skin wrinkles and background noise.

[0008] As a further preferred technical solution to the above technical solution, for the multimodal heterogeneous graph topology construction module: Each image node uses the 512-dimensional visual feature vector output by the lesion perception significantly enhanced feature extraction module as its initial node feature; the clinical image nodes correspond one-to-one with the dermoscopy image nodes, respectively corresponding to the pathological feature nodes and the comprehensive diagnosis DIAG nodes; the metadata nodes are used to encode clinical metadata such as patient age, gender, and lesion anatomical location.

[0009] As a further preferred technical solution to the above technical solution, the multimodal heterogeneous graph topology construction module constructs three types of directed edges: intramodal criterion convergence edges, cross-modal supplementary edges, and metadata interaction edges. The construction rules for these three types of directed edges are as follows: Intramodal criterion convergence edge: The comprehensive diagnosis DIAG node of a single modality serves as the central node and establishes a bidirectional connection with the pathological feature nodes within this modality; Cross-modal supplementary edges: All nodes in the dermoscopy image establish unidirectional connections to corresponding nodes with the same semantics in the clinical image, passing on micro-texture detail information; Metadata interaction edge: Metadata nodes establish unidirectional connections to all image nodes, injecting epidemiological clinical prior knowledge into all visual feature nodes.

[0010] As a further preferred technical solution to the above technical solution, for the dual-layer graph attention dynamic fusion module: The first layer of GAT uses 8 attention heads, each of which outputs 8-dimensional features. The features from multiple heads are concatenated to obtain a 64-dimensional intermediate feature representation. The second layer of GAT uses a single attention head to map the 64-dimensional intermediate features back to global fusion features of the same dimension as the initial node features; Attention coefficients between nodes are calculated using learnable weights, and high fusion weights are assigned to strongly correlated lesion nodes to automatically filter out unaligned weakly correlated noise nodes; after multi-head aggregation, the ELU activation function is used to alleviate gradient vanishing.

[0011] As a further preferred technical solution to the above technical solution, the workflow of the multi-task classification and prediction module is as follows: By concatenating the fusion features of nodes of the same semantic category in clinical images and dermoscopy images, multiple sets of equal-length fusion feature vectors are generated. Eight independent two-layer fully connected networks are set up as classification heads. The first layer compresses the dimension of the concatenated fused features and uses ReLU activation. The second layer outputs the probability distribution of the corresponding classification task. The cross-entropy loss of all checklist sub-tasks and comprehensive diagnostic tasks is used as the total loss function, and all network module parameters are trained end-to-end in a unified manner.

[0012] As a further preferred technical solution to the above technical solution, the overall working process of the system is as follows: Step S1: Data preprocessing: The input clinical images and dermoscopy images are uniformly scaled and normalized, and the patient metadata is one-hot encoded and mapped to the dimension that matches the 512-dimensional visual feature vector; Step S2: Significantly enhanced lesion perception and feature extraction: The dual-branch visual backbone network, combined with the embedded module LASE, processes clinical images and dermoscopic images respectively, and outputs a 512-dimensional standardized visual feature vector after global average pooling or feature vectorization. Step S3: Heterogeneous graph representation construction: Initialize image nodes with standardized visual feature vectors and metadata nodes with metadata encoding vectors. Construct a dynamic graph topology according to three types of edges: intra-modal, cross-modal, and metadata. Step S4: Graph Attention Adaptive Fusion: A two-layer multi-head GAT network completes node message passing and feature aggregation, realizing the fusion of non-aligned multimodal information; Step S5: Multi-task classification output: Concatenate semantically bimodal node features, and output the seven pathological features of the checklist subtask and the comprehensive diagnosis results of skin lesions through the parallel classification head; Step S6: Model training and inference: Optimize network parameters based on multi-task joint loss. After training, input patient multimodal data to complete automated classification and diagnosis of skin lesions.

[0013] The beneficial effects of this invention are as follows: 1. The LASE lesion perception enhancement mechanism was applied to multimodal classification of skin lesions. By filtering lesion perception channels, deformable convolutional offset sampling, and spatial-channel significant gating, the representation of irregular lesion regions was adaptively enhanced and interference from artifacts such as hair and shine was suppressed. 2. Construct a 17-node heterogeneous graph containing independent metadata nodes. Model the relationship between local criteria, comprehensive diagnosis and clinical priors through intramodal criterion convergence edges, cross-modal supplementary edges and metadata interaction edges, solving the problem of shallow metadata splicing in traditional methods. The two-layer GAT can dynamically adjust the fusion weights of different modalities and different nodes without explicit spatial or semantic alignment. 3. High average accuracy on the 7PC multi-task public SPC dataset, significantly reducing missed diagnoses and misdiagnoses; 4. The dual-branch and heterogeneous graph architecture decouples feature extraction and multimodal fusion, adapts to real clinical images and dermoscopy images with natural misalignment, has strong robustness, and has clinical application value. Attached Figure Description

[0014] Figure 1This is a diagram of the overall system architecture of LASE-UGF of the present invention; Figure 2 This is a diagram of the ResNet-50 backbone network structure with embedded LASE modules according to the present invention. Figure 3a This is a schematic diagram of the multimodal heterogeneous graph structure of the present invention; Figure 3b This is a schematic diagram of the adjacency matrix of the present invention; Figure 4 This is an example diagram of the SPC dataset samples and label annotations of the present invention; Figure 5 This is a comparison of the ResNet-50 baseline and the lesion thermograms enhanced by the LASE module in this invention; Figure 6 This is a bar chart comparing the performance of multiple indicators in the ablation experiment of this invention. Detailed Implementation

[0015] The following description is intended to disclose the present invention and enable those skilled in the art to implement it. The preferred embodiments described below are merely examples, and other obvious variations will occur to those skilled in the art. The basic principles of the invention defined in the following description can be applied to other embodiments, modifications, improvements, equivalents, and other technical solutions that do not depart from the spirit and scope of the invention.

[0016] In the preferred embodiments of the present invention, those skilled in the art should note that the clinical images and dermoscopy images involved in the present invention can be considered as prior art.

[0017] Preferred embodiment.

[0018] The LASE-UGF (Un-aligned Graph Fusion) model proposed in this embodiment aims to address the shortcomings of existing methods in adaptive enhancement of lesion features and unaligned multimodal fusion. Figure 1 As shown, the model as a whole includes two collaborative modules: a lesion perception enhancement module and a non-aligned graph structured fusion module. The non-aligned graph structured fusion module includes a multimodal heterogeneous graph topology construction module and a two-layer graph attention dynamic fusion module.

[0019] First, considering the modal differences and common interferences between clinical images and dermoscopy images, separate ResNet-50 backbone networks were used for feature extraction, with LASE modules embedded in several key convolutional stages. The LASE modules enhance the response of lesion-related regions through channel filtering, offset sampling, and significant gating, suppressing background noise and non-diagnostic artifacts, thereby improving the discriminative power of single-modal visual representations.

[0020] Secondly, to overcome the limitations of existing fusion methods in terms of reliance on spatial alignment and insufficient utilization of metadata, the criterion-level semantic features of the two types of images and patient metadata are uniformly modeled as heterogeneous graph nodes. Based on this, a graph topology is constructed containing intra-modal criterion convergence edges, cross-modal supplementary edges, and metadata interaction edges, and adaptive message passing between nodes is achieved through two layers of GAT. This design eliminates the need for explicit feature alignment, allowing for dynamic adjustment of the contribution weights of visual modalities and metadata based on different samples and tasks.

[0021] Finally, the fused cross-modal node features are concatenated and input into a parallel classification head to simultaneously predict the attributes of the seven-point checklist and the final diagnostic category. This framework effectively decouples the feature extraction and fusion processes, enhancing the quality of single-modal representations while enabling flexible modeling of unaligned, multi-source heterogeneous information.

[0022] Specifically, this invention discloses a non-aligned multimodal skin lesion classification system based on graph attention networks, including a lesion perception significantly enhanced feature extraction module, a multimodal heterogeneous graph topology construction module, a two-layer graph attention dynamic fusion module, and a multi-task classification prediction module, wherein: The lesion perception significantly enhanced feature extraction module is the front-end processing module of LASE-UGF, responsible for extracting lesion-related representations from clinical images and dermoscopic images that can be used for subsequent image modeling. As shown in Figure 2, this stage uses ResNet-50 as the backbone network, and the LASE module is embedded after the four key convolutional stages from layer 1 to layer 4. The two visual modalities are encoded using independent backbone networks to preserve the macroscopic morphological information of clinical images and the microscopic structural information of dermoscopic images.

[0023] Skin lesion images exhibit significant domain specificity. Illumination conditions, skin color differences, and imaging equipment can introduce non-diagnostic channel variations; lesion boundaries often present blurred, irregular, and multi-scale morphologies; various visual artifacts can also produce local responses similar to the lesions. The LASE module sequentially reduces redundancy through channel filtering, adapts to irregular morphologies through offset sampling, and suppresses artifact interference through spatial-channel gating. Relying on the hierarchical structure of ResNet-50, the LASE module adjusts responses related to lesion edges and local textures at the superficial level, and enhances semantic features related to pigment distribution, vascular morphology, and comprehensive discrimination at the higher level.

[0024] Ultimately, the output features of both types of images are subjected to global average pooling to obtain a 512-dimensional vector, which is used as the initial representation for subsequent criterion nodes and diagnostic nodes.

[0025] The LASE module is a lesion perception saliency enhancement module for skin lesion images, designed to improve the model's ability to focus on diagnostically relevant lesion regions. This module optimizes visual feature representation in stages through three collaborative units: lesion perception channel selection, deformable convolutional offset sampling, and spatial-channel saliency gating. First, it selects key channels related to lesions and reduces redundant responses. Then, it dynamically adjusts the convolutional sampling position to adapt to irregular lesion morphologies. Finally, it enhances lesion regions and suppresses background artifacts through combined spatial and channel gating. The internal processing flow of the LASE module specifically includes: Step 1: Lesion Perception Channel Screening. Input features are compressed and screened to reduce redundant channel responses and retain key channel features related to lesion texture, pigment networks, and vascular patterns. Specifically, input features can be processed through depthwise separable convolution or equivalent lightweight channel transformations to decouple spatial texture extraction and cross-channel feature fusion, thereby improving the expression intensity of lesion-related channels while controlling computational overhead.

[0026] This process reduces redundant responses while preserving key semantic features such as pigments and blood vessels, providing more diagnostically relevant spatial feature inputs for subsequent deformable convolutional offset sampling. The processing procedure is shown in Equation (1): (1); in, As input features, For depthwise separable convolution weights, For instance normalization, GELU is the activation function. These are the compressed features.

[0027] Step 2: Deformable convolution offset sampling. Deformable convolution learns the spatial offset of each sampling point and constructs an adaptive sampling grid to overcome the limitations of the fixed regular grid of traditional convolution. For skin lesions with blurred boundaries, irregular shapes, or significant size differences, deformable convolution can dynamically adjust the sampling position according to local features, more completely capturing the edge and internal texture changes of the lesion. Its sampling process is shown in formula (2): (2); in, The kernel size is the convolution kernel size. For convolution kernel weights, These are spatial attention weights (used to suppress background region responses). To fix the reference sampling point, The learned offset is used to effectively capture lesion features outside the reference kernel range.

[0028] Step 3: Spatial-channel salient gating. The deformable convolution output features are normalized and activated to generate salient gating weights constrained by both spatial location and channel dimension. These salient gating weights are then multiplied point-by-point with the original input features of the module to enhance the response to the lesion region and suppress background noise. This process can reduce the impact of non-diagnostic artifacts such as hair, oiliness, and skin wrinkles on classification features, and finally output optimized lesion-related visual features. The relevant processes are shown in formulas (3) and (4): (3); (4); in, Used to unify feature scale for function, This is a point-by-point multiplication operation.

[0029] In this embodiment, the LASE module is embedded after layers 1 to 4 of ResNet-50, significantly enhancing lesion perception at each level: lower-level features emphasize the correlation between edges and textures in lesions, while higher-level features emphasize higher-order semantic responses such as abnormal pigment networks and disordered vascular patterns. This strategy improves feature discriminativeness while maintaining the stability of the basic backbone network structure, and is particularly suitable for skin lesion images with heavy background interference and irregular lesion morphology.

[0030] The multimodal heterogeneous graph topology construction module is used to overcome the strong assumptions of existing methods regarding multimodal feature space or semantic alignment, and to achieve deep collaboration between patient metadata and visual features. In this embodiment, patient metadata is explicitly modeled as independent graph nodes, and the seven-point checklist criteria and comprehensive diagnostic nodes are uniformly organized into a heterogeneous graph structure. Through GAT learning, dynamic associations between nodes are achieved, enabling adaptive fusion without explicit alignment.

[0031] The multimodal heterogeneous graph topology construction module aims to transform the multimodal information required for the diagnosis of skin lesions into a structured graph representation. Its node partitioning and edge connection design follow the clinical interpretation logic of dermatology, and uses a seven-point checklist as the semantic prior for the judgment criteria.

[0032] The multimodal heterogeneous graph topology construction module is used to construct a heterogeneous graph containing 17 nodes, including 8 clinical image nodes, 8 dermoscopy image nodes, and 1 metadata node. Specifically, as... Figure 3aAs shown, the 16 image nodes correspond to two visual modalities: clinical images and dermoscopy images. Each visual modality contains 7 seven-point checklist criteria nodes and 1 comprehensive diagnostic DIAG node. The metadata nodes are used to encode clinical information such as age, gender, and anatomical location of lesions to ensure that the semantics of the nodes are consistent with the clinical decision-making process.

[0033] For the multimodal heterogeneous graph topology construction module: Each image node uses the 512-dimensional visual feature vector output by the lesion perception significantly enhanced feature extraction module as its initial node feature; the clinical image nodes correspond one-to-one with the dermoscopy image nodes, corresponding to the 7 criteria nodes of the seven-point examination checklist and 1 comprehensive diagnostic DIAG node, respectively; the metadata nodes encode clinical information such as age, gender and lesion anatomical location into vectors of the same dimension through a two-layer fully connected network, thereby ensuring that the visual modality and metadata can interact under a unified feature dimension.

[0034] In terms of edge connection design, the module constructs three types of directed edges with clear semantic orientations to model information flow at different levels, including intra-modal criterion convergence edges, cross-modal supplementary edges, and metadata interaction edges. The construction rules for the three types of edges are as follows: Intramodal criterion convergence edge: The comprehensive diagnostic DIAG node of a single visual modality serves as the central node, establishing bidirectional connections with the seven seven-point checklist criterion nodes within the same modality. This star topology facilitates bidirectional information transfer between local visual criteria and the overall diagnostic objective, enabling the model to learn the structured reasoning relationship from criteria to diagnosis.

[0035] Cross-modal supplementary edge: All 8 nodes of the dermoscopy image establish unidirectional connections to corresponding semantically identical nodes in the clinical image. Given the advantages of dermoscopy images in microstructural representation, this edge is used to dynamically inject detailed information such as high-resolution textures, pigment networks, and vascular patterns into the macroscopic representation of the clinical image, compensating for the shortcomings of the clinical image in microtexture resolution.

[0036] Metadata interaction edge: Metadata nodes establish unidirectional connections to all 16 image nodes, incorporating clinical priors related to age, gender, and lesion anatomical location into the visual feature learning process, enhancing the model's sensitivity to discriminative lesions in high-risk populations or special anatomical locations.

[0037] The adjacency relationship of this topology is in Figure 3b This is intuitively reflected in the adjacency matrix, providing structural constraints and semantic guidance for subsequent graph attention networks, and supporting adaptive multimodal fusion under non-aligned conditions.

[0038] Traditional graph convolutional networks rely on a fixed adjacency structure during feature aggregation and apply an equal weighting strategy to all neighboring nodes, making it difficult to distinguish the differences in contributions of different nodes to the central node. This limitation is particularly pronounced in multimodal analysis of skin lesions: due to the inherent misalignment between clinical images and dermoscopic images in terms of spatial layout and semantic granularity, some node pairs have weak associations, such as the vascular pattern in a blurred clinical image and the corresponding node in a high-resolution dermoscopic image. Forcing equal fusion can easily introduce noise and weaken discriminative power.

[0039] To overcome the above problems, this embodiment uses Graph Attention Convolution (GAT) to achieve adaptive aggregation of node features. GAT dynamically calculates the association weights between nodes through a learnable self-attention mechanism and adjusts the fusion strength in real time based on feature similarity. It assigns high weights to highly related node pairs, while automatically suppressing the influence of weakly related or unrelated nodes, thus meeting the core requirement of selective fusion of non-aligned features in this task.

[0040] The dual-layer graph attention dynamic fusion module uses a two-layer multi-head graph attention network (GAT) to adaptively aggregate and fuse the initial features of multiple nodes. It does not require explicit spatial or semantic alignment between clinical images and dermoscopy images, and outputs 512-dimensional features of each node after fusion. GAT runs on the aforementioned 17-node graph topology, and its attention coefficient is calculated as follows: (5); In this mechanism, This represents the input feature vector of the GAT layer, where N=17 (containing 16 image modality nodes and 1 metadata node), and each , where F=512 is the initial feature dimension obtained after feature vectorization module and metadata processing. It is the attention coefficient from node v to node u. Let u be the set of neighbors of node u; Used to perform linear transformation on feature vectors, with F'=64 set in the first layer and F'=512 set in the second layer; It is the weight vector of a single-layer feedforward neural network in the attention mechanism, representing the feature concatenation operation. LeakyReLU introduces non-linear characteristics to enhance the expressive power of the model.

[0041] Based on the aforementioned attention coefficients, the node's updated feature vector (in This can be obtained by proportionally aggregating the features of neighboring nodes, as shown in formula (6): (6); here, The function is an exponential linear unit (ELU) nonlinear function, which can alleviate the gradient vanishing problem while preserving useful negative eigenvalue information. Furthermore, to improve model stability, we employ a multi-head attention mechanism, calculating multiple independent attention coefficients for each node and aggregating them through concatenation, as shown in formula (7): (7); For the m-th attention head, It is the normalized attention coefficient. These are the weights of the input linear transformation.

[0042] Attention coefficients between nodes are calculated using learnable weights, and high fusion weights are assigned to strongly correlated lesion nodes to automatically filter out unaligned weakly correlated noise nodes; after multi-head aggregation, the ELU activation function is used to alleviate gradient vanishing.

[0043] For the dual-layer graph attention dynamic fusion module: The first layer of GAT uses 8 attention heads, each of which outputs 8-dimensional features. The features from multiple heads are concatenated to obtain a 64-dimensional intermediate feature representation. The second layer of GAT uses a single attention head to map the 64-dimensional intermediate features back to the global fusion features with the same dimensions as the initial node features (i.e., 512 dimensions). This not only keeps the output dimensions consistent with the preceding feature extraction module, but also strengthens the global semantic association between multimodal information through high-level abstraction.

[0044] It is worth noting that the three types of edges each play a distinct role in the fusion process: intramodal criterion convergence edges strengthen the relationship between local criteria within a single modality and the overall diagnosis; cross-modal supplementary edges promote the complementarity between dermoscopic microscopic information and clinical macroscopic information; and metadata interaction edges dynamically inject clinical priors. The GAT mechanism can adaptively allocate weights based on node features and topological connectivity, effectively avoiding interference from invalid information. After two layers of GAT processing, each node ultimately represents the fusion of complementary information from clinical images, dermoscopic images, and patient metadata, providing more discriminative feature support for subsequent classification tasks.

[0045] The multi-task classification and prediction module completes the final classification based on the criterion-level fusion features output by the graph neural network. It concatenates the fusion features of clinical image nodes and dermoscopy image nodes under the same criterion semantics, inputs them into a parallel and independent two-layer fully connected classification head, and simultaneously outputs classification results for seven seven-point examination checklist criteria and one comprehensive diagnostic classification result for skin lesions.

[0046] The workflow of the multi-task classification prediction module is as follows: Step 1: Concatenate the fusion features of nodes of the same semantic category in clinical images and dermoscopy images to generate multiple sets of equal-length fusion feature vectors, and construct 8 sets of 1024-dimensional fusion vectors; Step 2: Set up 8 independent two-layer fully connected networks as classification heads. The first layer compresses the dimensions of the concatenated fused features and uses ReLU activation. The second layer outputs the probability distribution corresponding to the classification task. The classification framework uses 8 parallel two-layer fully connected networks. Each set of features is processed by the network: the first layer compresses the 1024-dimensional features to 512 dimensions (ReLU activation), and the second layer maps them to the target category dimension. By outputting the probability distribution, the network parameters of each set are independently learned to adapt the association between the features and the disease category.

[0047] Step 3: Combine the cross-entropy loss of all 7 sub-tasks of the seven-point checklist and the comprehensive diagnostic task as the total loss function, and train all network module parameters end-to-end. The model training aims to minimize the total loss and optimize the parameters, and the objective function is shown in formula (8): (8); Among them, the model parameters, dermoscopy images, clinical images, classification prediction probabilities, and true labels are defined according to the symbols in formula (8).

[0048] This design retains the diversity of multi-view discrimination signals while ensuring that the semantic branches work together to improve the overall diagnostic performance through joint optimization.

[0049] The overall workflow of the system is as follows: Step S1: Data preprocessing: The input clinical images and dermoscopy images are uniformly scaled and normalized, and the patient metadata is encoded and mapped to the dimension that matches the 512-dimensional visual feature vector; Step S2: Significantly enhanced lesion perception and feature extraction: The dual-branch ResNet-50 backbone network, combined with the embedded LASE module, processes clinical images and dermoscopic images respectively, and outputs 512-dimensional standardized visual feature vectors after global average pooling or feature vectorization; Step S3: Heterogeneous graph representation construction: Initialize 16 image nodes with standardized visual feature vectors, initialize metadata nodes with metadata encoding vectors, and construct heterogeneous graph topology according to intramodal criterion convergence edges, cross-modal supplementary edges, and metadata interaction edges; Step S4: Graph Attention Adaptive Fusion: A two-layer multi-head GAT network completes node message passing and feature aggregation, realizing the fusion of non-aligned multimodal information; Step S5: Multi-task classification output: Concatenate semantically identical bimodal node features, and output a seven-point checklist of pathological features and a comprehensive diagnosis of skin lesions through a parallel classification head; Step S6: Model training and inference: Optimize network parameters based on multi-task joint loss. After training, input patient multimodal data to complete automated classification and diagnosis of skin lesions.

[0050] It is worth mentioning that this embodiment demonstrates its effectiveness through experiments and results analysis: I. Experimental Setup: 1. Dataset Introduction: To comprehensively evaluate the performance of the proposed LASE-UGF model in multimodal skin lesion classification tasks, this embodiment's experiments are based on the publicly available Seven-Point Checklist (SPC) dataset. This dataset contains 1011 skin lesion cases, each providing two visual modalities: dermoscopic images and clinical images, along with structured metadata and basic patient information annotated by dermatologists according to the Seven-Point Checklist. As shown in Table 1, the official dataset is divided into a training set (614 cases), a validation set (197 cases), and a test set (200 cases).

[0051] As shown in Figure 4, this paper designs two types of diagnostic tasks based on a seven-point checklist: the first is the seven-point criterion classification, which consists of 7 sub-tasks; the second is the diagnostic classification, which divides cases into 5 categories, including basal cell carcinoma, nevus, seborrheic keratosis, melanoma, and other benign lesions.

[0052] Due to differences in imaging equipment, shooting angles, and areas of interest, the two types of images exhibit misalignment in spatial layout and semantic granularity. Dermoscopy images primarily capture the fine structures of lesions, such as irregular pigment networks, blue-white veils, or abnormal vascular morphology; while clinical images focus on presenting the overall outline of the lesion and its spatial relationship with the surrounding normal skin. Figure 4 As shown, dermoscopy images focus on local high-resolution details, while clinical images contain a wider range of background information, and the two lack a clear pixel-level or semantic-level correspondence.

[0053] Table 1. SPC Dataset Sample Division and Task Label Description .

[0054] 2. Evaluation Indicators: To comprehensively compare the proposed method with other studies based on the SPC dataset, this embodiment uses evaluation metrics consistent with existing literature, including accuracy (ACC), sensitivity (SEN), specificity (SPE), precision (PRE), and area under the receiver operating characteristic curve (AUC). Mean-accuracy, as the core metric for cross-method comparison, is defined as the arithmetic mean of the accuracy rates of the seven-point checklist subtasks and the comprehensive diagnostic task, used to comprehensively measure the model's overall performance in fine-grained criterion recognition and overall diagnostic decision-making. All metrics are calculated strictly according to common standards in the field of medical image analysis: accuracy reflects the proportion of correctly classified predictions; sensitivity measures the model's ability to identify positive samples; specificity characterizes the ability to discriminate negative samples; precision focuses on the proportion of true positives among predictions of positive; and AUC comprehensively evaluates the model's ranking and discrimination stability by integrating true positive and false positive rates at different thresholds. The above indicators together constitute a multi-dimensional evaluation system to ensure the comparability, robustness, and clinical reference value of the experimental results. The definitions of each indicator are as follows: (9); (10); (11); (12).

[0055] II. Ablation Experiment: To systematically evaluate the effectiveness of each core component in the proposed model, this embodiment designs a set of gradient ablation experiments, covering the performance comparison of different modal combinations and fusion strategies. Six configurations were set up in the experiments: a single-modal model using only clinical images (clin-only, CO), a single-modal model using only dermoscopic images (derm-only, DO), a fixed-feature concatenation model of clinical and dermoscopic features (clin-derm-concat, CDC), a fixed-feature concatenation model of three modalities with added metadata (clin-derm-meta-concat, CDMC), a dynamic fusion model of three modalities based on graph attention networks (clin-derm-meta-GAT, CDMG), and the complete model LASE-UGF (clin-derm-meta-GAT-LASE, CDMG-LASE).

[0056] Comparison of performance between single-modal and multi-modal fixed splicing: To systematically evaluate the performance gain of introducing multimodal information and reveal the limitations of fixed splicing fusion strategies, this embodiment compares the performance of unimodal and multimodal feature splicing models. As shown in Table 2, the DO model using only dermoscopic images outperforms the CO model using only clinical images, achieving an average accuracy of 73.7%. Its advantage is more pronounced on microstructure-dependent tasks such as PN and DaG, reflecting the important role of dermoscopic images in characterizing subsurface details of lesions.

[0057] Table 2. Comparison of classification accuracy between single-modal and multi-modal methods:

[0058] Further incorporating clinical images to construct a dual-modal fixed-stitch model improved the average accuracy to 75.3%, validating the complementary value of macroscopic and microscopic information. Specifically, the CDC model slightly outperformed the DO model on tasks such as BWV and VS, indicating that the overall morphology of clinical images helps dermatoscopes interpret lesions with blurred boundaries and alleviates the perspective bias of a single modality.

[0059] However, when patient metadata was added to the CDC to form the CDMC model, the average accuracy only increased slightly to 75.4%, with limited overall gain. Its accuracy on the DIAG task reached 76.5%, indicating that prior clinical information is helpful for comprehensive diagnosis; however, on tasks highly dependent on visual details such as PIG, the introduction of metadata actually introduced interference, and the accuracy of CDMC dropped to 63.8%, lower than CDC's 69.1% and even weaker than DO's 65.1%. This shows that the fixed splicing mechanism is difficult to dynamically adjust modal weights according to task differences.

[0060] In summary, simple concatenation strategies struggle to effectively coordinate multimodal and metadata information, exhibiting significant limitations, especially in scenarios with task heterogeneity and modal misalignment. Therefore, further analysis of the CDMG and CDMG-LASE models is necessary to verify the performance contributions of the graph attention dynamic fusion mechanism and the LASE feature enhancement module.

[0061] Heterogeneous graph fusion and the performance gains of the LASE module: To verify the contribution of the heterogeneous graph-based multimodal fusion module and the lesion perception enhancement module to the model performance, this section compares the classification performance of CDMC, CDMG, and CDMGD (i.e., the complete model LASE-UGF in this embodiment) on the SPC dataset. The results are shown in Table 3.

[0062] Table 3. Impact of Heterogeneous Graph Fusion Module and LASE Module on Classification Accuracy:

[0063] Experiments show that the CDMG model, which incorporates a graph attention network for dynamic fusion, achieves an average accuracy of 77.1%, a 1.7 percentage point improvement over the fixed-fragment CDMC model's 75.4%, and demonstrates positive gains across all eight tasks. Specifically, the accuracy for BWV increases from 87.1% to 90.6%, and for DIAG from 76.5% to 78.0%. This result indicates that graph attention networks help dynamically adjust modal weights based on task semantics: in microstructure-dependent tasks such as BWV, the model automatically enhances the fusion strength of dermoscopic features; and in the DIAG comprehensive diagnostic task, it strengthens the interaction between metadata and visual features.

[0064] After further introducing a lesion perception enhancement module, the average accuracy of the complete model CDMGD (i.e., LASE-UGF) was further improved to 78.4%, an increase of 1.3 percentage points compared to CDMG. All subtasks showed consistent positive gains: PN increased from 75.2% to 76.1%, PIG from 69.6% to 72.4%, RS from 78.5% to 81.9%, and DIAG from 78.0% to 79.8%. This indicates that the LASE module adapts to irregular lesion morphologies through deformable convolutional offset sampling and combines spatial-channel significant gating to suppress visual artifacts and background noise, providing more discriminative visual features for subsequent image fusion.

[0065] To visually demonstrate the feature optimization effects of LASE, Figure 5 Comparing the feature heatmaps of ResNet-50 and ResNet-50+LASE, it can be seen that the response region of the baseline model contains more non-lesion components, and the lesion boundaries are relatively blurred; while after adding the LASE module, the response region is more concentrated in the lesion-related area, making key structural features such as PN and VS clearer.

[0066] To further evaluate the model's comprehensive discriminative ability from multiple dimensions, Figure 6 The average performance of the three models on five commonly used medical image evaluation metrics is presented. The results show that CDMG-LASE outperforms the previous two generations of models in all metrics: the average ACC reaches 78.4%, an improvement of 3.0 percentage points compared to CDMC; the average AUC reaches 88.6%, an improvement of 5.0 percentage points compared to CDMC; the average SEN and SPE also improve simultaneously, indicating that the model has good overall performance in terms of sensitivity and specificity.

[0067] In summary, this embodiment addresses the challenges in multimodal skin lesion classification, such as lesion features being susceptible to background interference, irregular lesion morphology, spatial and semantic misalignment between clinical and dermoscopic images, and insufficient utilization of clinical metadata. It proposes the LASE-UGF model, which consists of a lesion perception enhancement module and a heterogeneous graph-based multimodal fusion module. The former enhances lesion-related visual representations and suppresses background and non-diagnostic interference through lesion perception channel filtering, deformable convolutional offset sampling, and spatial-channel saliency gating. The latter uses a seven-point checklist as a semantic prior, unifying clinical images, dermoscopic images, and patient metadata into heterogeneous graph nodes, and achieves criterion-level information interaction and dynamic fusion under misalignment conditions through a graph attention network. Experimental results on the SPC dataset show that LASE-UGF achieves an average accuracy of 78.4%, with the lesion perception enhancement module and the heterogeneous graph fusion module contributing performance gains of 1.3 and 1.7 percentage points, respectively, providing an effective solution for structured modeling of misaligned medical multimodal data.

[0068] It is worth mentioning that the technical features such as clinical images and dermoscopy images involved in this patent application should be regarded as prior art. The specific structure, working principle, and possible control methods and spatial arrangement of these technical features can be adopted using conventional choices in the field, and should not be regarded as the inventive point of this patent. This patent will not be further elaborated in detail.

[0069] For those skilled in the art, modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this invention should be included within the protection scope of this invention.

Claims

1. A non-aligned multimodal skin lesion classification system based on graph attention networks, characterized in that, It includes a lesion perception significantly enhanced feature extraction module, a multimodal heterogeneous graph topology construction module, a two-layer graph attention dynamic fusion module, and a multi-task classification prediction module, among which: The lesion perception significantly enhanced feature extraction module is configured with a dual-branch visual backbone network, which receives clinical images and dermoscopic images respectively. The LASE module is embedded after multiple convolution stages of the backbone network. The lesion perception channel is enhanced through lesion perception channel screening, deformable convolution offset sampling and spatial-channel significant gating to enhance the visual representation of lesions. The features of the two images are uniformly mapped into a 512-dimensional visual feature vector. The multimodal heterogeneous graph topology construction module is used to construct a heterogeneous graph containing multiple nodes, including A clinical image nodes, A dermoscopy image nodes, and B metadata nodes; The dual-layer graph attention dynamic fusion module uses a two-layer multi-head graph attention network to adaptively aggregate and fuse the initial features of multiple nodes, and outputs the fused 512-dimensional features of each node. The multi-task classification prediction module combines the fusion features of clinical image nodes and dermoscopy image nodes with the same semantic meaning, inputs them into a parallel independent two-layer fully connected classification head, and simultaneously outputs the classification results of the examination list sub-tasks and the comprehensive diagnostic classification results of skin lesions.

2. The non-aligned multimodal skin lesion classification system based on graph attention networks according to claim 1, characterized in that, The internal processing flow of LASE includes: The lesion perception channel screening unit performs channel compression and screening on the input features, reduces redundant channel responses, and retains key features related to lesion texture, pigment network, and vascular pattern. Deformable convolutional offset sampling units learn the spatial offset of convolutional sampling points and construct an adaptive sampling grid to match skin lesion areas with blurred boundaries and irregular shapes; The spatial-channel salient gating unit generates salient gating weights based on spatial location and channel response, and applies the salient gating weights to the original input features to enhance the response of the lesion area and suppress hair, oil, skin wrinkles and background noise.

3. The non-aligned multimodal skin lesion classification system based on graph attention networks according to claim 1, characterized in that, For the multimodal heterogeneous graph topology construction module: Each image node uses the 512-dimensional visual feature vector output by the lesion perception significantly enhanced feature extraction module as its initial node feature; the clinical image nodes correspond one-to-one with the dermoscopy image nodes, respectively corresponding to the pathological feature nodes and the comprehensive diagnosis DIAG nodes; the metadata nodes are used to encode clinical metadata such as patient age, gender, and lesion anatomical location.

4. The non-aligned multimodal skin lesion classification system based on graph attention networks according to claim 3, characterized in that, The multimodal heterogeneous graph topology construction module constructs three types of directed edges: intramodal criterion convergence edges, cross-modal supplementary edges, and metadata interaction edges. The construction rules for these three types of directed edges are as follows: Intramodal criterion convergence edge: The comprehensive diagnosis DIAG node of a single modality serves as the central node and establishes a bidirectional connection with the pathological feature nodes within this modality; Cross-modal supplementary edges: All nodes in the dermoscopy image establish unidirectional connections to corresponding nodes with the same semantics in the clinical image, passing on micro-texture detail information; Metadata interaction edge: Metadata nodes establish unidirectional connections to all image nodes, injecting epidemiological clinical prior knowledge into all visual feature nodes.

5. The non-aligned multimodal skin lesion classification system based on graph attention networks according to claim 1, characterized in that, For the dual-layer graph attention dynamic fusion module: The first layer of GAT uses 8 attention heads, each of which outputs 8-dimensional features. The features from multiple heads are concatenated to obtain a 64-dimensional intermediate feature representation. The second layer of GAT uses a single attention head to map the 64-dimensional intermediate features back to global fusion features of the same dimension as the initial node features; Attention coefficients between nodes are calculated using learnable weights, and high fusion weights are assigned to strongly correlated lesion nodes to automatically filter out unaligned weakly correlated noise nodes; after multi-head aggregation, the ELU activation function is used to alleviate gradient vanishing.

6. The non-aligned multimodal skin lesion classification system based on graph attention networks according to claim 5, characterized in that, The workflow of the multi-task classification prediction module is as follows: By concatenating the fusion features of nodes of the same semantic category in clinical images and dermoscopy images, multiple sets of equal-length fusion feature vectors are generated. Eight independent two-layer fully connected networks are set up as classification heads. The first layer compresses the dimension of the concatenated fused features and uses ReLU activation. The second layer outputs the probability distribution of the corresponding classification task. The cross-entropy loss of all checklist sub-tasks and comprehensive diagnostic tasks is used as the total loss function, and all network module parameters are trained end-to-end in a unified manner.

7. A non-aligned multimodal skin lesion classification system based on graph attention networks according to any one of claims 1-6, characterized in that, The overall workflow of the system is as follows: Step S1: Data preprocessing: The input clinical images and dermoscopy images are uniformly scaled and normalized, and the patient metadata is one-hot encoded and mapped to the dimension that matches the 512-dimensional visual feature vector; Step S2: Significantly enhanced lesion perception and feature extraction: The dual-branch visual backbone network, combined with the embedded module LASE, processes clinical images and dermoscopic images respectively, and outputs a 512-dimensional standardized visual feature vector after global average pooling or feature vectorization. Step S3: Heterogeneous graph representation construction: Initialize image nodes with standardized visual feature vectors and metadata nodes with metadata encoding vectors. Construct a dynamic graph topology according to three types of edges: intra-modal, cross-modal, and metadata. Step S4: Graph Attention Adaptive Fusion: A two-layer multi-head GAT network completes node message passing and feature aggregation, realizing the fusion of non-aligned multimodal information; Step S5: Multi-task classification output: Concatenate semantically bimodal node features, and output the seven pathological features of the checklist subtask and the comprehensive diagnosis results of skin lesions through the parallel classification head; Step S6: Model training and inference: Optimize network parameters based on multi-task joint loss. After training, input patient multimodal data to complete automated skin lesion classification and diagnosis.

Images