A Method and System for Diagnosing Skin Lesions Based on Detection Segmentation and Multimodal Alignment

Abstract

This invention discloses a method and system for diagnosing skin lesions based on detection, segmentation, and multimodal alignment, belonging to the field of medical artificial intelligence technology. The method includes: acquiring multimodal data to be diagnosed, comprising dermoscopic images, pathological and biochemical data, and microscopic tissue features; generating candidate lesion regions and a set of cue vectors by introducing a deformable convolutional detection-segmentation cascade model, driving a large segmentation model to output a lesion mask and a segmentation uncertainty heatmap; constructing a multimodal cross-attention fusion network, using biological semantic vectors as queries to weight and reorganize morphological features, and utilizing the uncertainty heatmap to perform local adaptive weighting on high-entropy edge regions; finally, outputting classification probability, risk score, and Grad-CAM saliency map via a multi-task prediction head. This invention effectively integrates morphological phenotype and molecular pathological information, improving the diagnostic accuracy of lesions with blurred boundaries while enhancing the interpretability of model decisions.

Description

Technical Field

[0001] This invention relates to the field of medical artificial intelligence technology, and more specifically, to a method and system for diagnosing skin lesions based on detection segmentation and multimodal alignment. Background Technology

[0002] Early and accurate diagnosis of skin lesions, especially melanoma and basal cell carcinoma (BCC), is crucial for improving patient survival. Currently, dermoscopy images are widely used clinically as the primary basis for non-invasive screening, with computer-aided diagnostic systems used for lesion segmentation, feature extraction, and classification. Existing technologies, such as deep learning-based image segmentation methods and object detection algorithms, have made some progress. However, dermoscopy images often suffer from blurred lesion edges, low contrast, hair occlusion, or the jagged, irregular boundaries characteristic of melanoma, making it difficult for single segmentation models to achieve the pixel-level accuracy required in clinical practice. Meanwhile, while large models possess powerful zero-shot segmentation capabilities, they are highly sensitive to the quality of prompts. When directly applied to skin lesion segmentation, they lack an effective automatic prompt generation mechanism, making it difficult to balance detection efficiency and segmentation precision.

[0003] On the other hand, the diagnosis of skin lesions is essentially a multimodal information fusion process. There is a close but non-linear correlation between imaging features and the patient's biochemical indicators, pathological slide information, medical history, and other data. However, most existing diagnostic systems rely on only a single image modality or use simple feature stitching for multimodal fusion, neglecting the deep semantic alignment of different modalities. This results in insufficient robustness of the algorithm when faced with "different images of the same disease" or "same images of different diseases," easily leading to misjudgments. Furthermore, existing segmentation models lack uncertainty perception and feedback adjustment capabilities when encountering regions with blurred boundaries, failing to proactively enhance highly uncertain regions, further limiting the accuracy and interpretability of the diagnostic system.

[0004] Therefore, there is an urgent need for a precise diagnostic method for skin lesions that can deeply integrate detection and segmentation tasks, make full use of multimodal information, and have uncertainty perception and feedback adjustment capabilities. Summary of the Invention

[0005] One objective of this invention is to provide a method and system for diagnosing skin lesions based on detection, segmentation, and multimodal alignment. By constructing a cascaded model deeply coupled with YOLO-SAM, combining a multimodal cross-attention fusion framework based on contrastive learning, and a local adaptive weighting mechanism guided by an uncertainty heatmap, this invention aims to solve key technical problems in existing technologies such as insufficient segmentation accuracy, lack of multimodal feature alignment, and weak processing capability for blurred boundary regions, thereby achieving efficient, accurate, and interpretable auxiliary diagnosis of skin lesions.

[0006] YOLO refers to You Only Look Once (v11 version), which, as the object detection branch, is responsible for quickly locating candidate lesion regions, outputting predicted bounding boxes, and generating a set of cue vectors to drive the segmentation model. SAM refers to Segment Anything Model (a SAM-like architecture segmentation base model), which is a cue-based large model segmentation branch that receives the cue vector set, performs zero-shot / few-shot fine segmentation, and outputs lesion masks and uncertainty heatmaps.

[0007] According to a first aspect of the present invention, a method for diagnosing skin lesions based on detection segmentation and multimodal alignment is provided, comprising the following steps:

[0008] Step S1: Obtain the multimodal data to be diagnosed, wherein the multimodal data includes at least morphological modal data and corresponding pathological / biochemical modal data;

[0009] Step S2: Input the morphological modality data into a pre-trained detection-segmentation cascade model, which includes an object detection branch and a cue-based large model segmentation branch;

[0010] Step S3: Extract candidate regions of lesions through the target detection branch, and dynamically generate a hint vector set based on the geometric features of the candidate regions. The hint vector set contains at least one center hint point and one boundary hint point.

[0011] Step S4: Input the cue vector set into the large model segmentation branch to drive it to perform zero-sample or small-sample segmentation on the morphological modality data, and generate lesion mask images and segmentation uncertainty heatmaps;

[0012] Step S5: Construct a multimodal cross-attention fusion network to align and fuse the morphological features extracted from the lesion mask image with the biological semantic vectors converted from the pathological / biochemical modal data in the latent space;

[0013] Step S6: Using the segmentation uncertainty heatmap, locally adaptively weight the feature maps in the multimodal cross-attention fusion network to generate weighted fusion features;

[0014] Step S7: Input the weighted fusion features into the multi-task prediction head and output the classification probability of skin lesions, risk score, and saliency map used to explain the model decision.

[0015] Optionally, in step S1, the multimodal data further includes microstructure modal data; the microstructure modal data is obtained through the following steps:

[0016] Pathological slide image data is acquired, and the microscopic tissue features of the pathological slide image data are extracted using a lightweight visual transformer. The microscopic tissue features include the nuclear heterogeneity index and the estimated dermal infiltration depth.

[0017] The micro-organism features are used as a third modality and are input into the multimodal cross-attention fusion network for fusion.

[0018] Optionally, in step S2, the target detection branch is a detection network based on the YOLOv11 architecture, and its neck network uses deformable convolutional layers to adaptively extract the boundary features of irregularly shaped lesions.

[0019] Optionally, the dynamic generation of the prompt vector set in step S3 specifically includes:

[0020] The geometric center of the predicted bounding box output by the target detection branch is used as the center cue point;

[0021] On the boundary of the prediction box, select N points that are closest to the lesion prediction boundary as the boundary prompt points, where N is an integer greater than or equal to 2.

[0022] Optionally, the step of generating the segmentation uncertainty heatmap in step S4 specifically includes:

[0023] In the mask decoder of the large model segmentation branch, an uncertainty estimation head is added in parallel. The uncertainty estimation head is used to calculate the entropy value of each pixel output by the decoder in terms of classification. The entropy values ​​of all pixels are combined to form the segmentation uncertainty heatmap.

[0024] Optionally, in step S5, the multimodal cross-attention fusion network is obtained through contrastive learning pre-training, and its feature alignment method during pre-training includes:

[0025] Construct positive sample pairs and negative sample pairs, wherein positive sample pairs are the morphological features and the bio-semantic vectors from the same patient, and negative sample pairs are the morphological features and the bio-semantic vectors from different patients;

[0026] By using the InfoNCE (Information Noise-Contrastive Estimation) loss function, the multimodal cross-attention fusion network is constrained to bring positive sample pairs closer together and push negative sample pairs further apart in the latent space.

[0027] Optionally, the step of aligning and blending in the latent space in step S5 specifically includes:

[0028] The biological semantic vector is used as a query, and the morphological features are used as keys and values, which are then input into the cross-attention layer of the multimodal cross-attention fusion network.

[0029] The cross-attention layer is used to calculate the correlation between the biological semantic vector and different spatial locations in the morphological features, and to perform weighted recombination of the morphological features based on the correlation to generate cross-modal enhanced features.

[0030] Optionally, the local adaptive weighting of the feature maps in the multimodal cross-attention fusion network in step S6 specifically includes:

[0031] Identify high uncertainty regions in the segmentation uncertainty heatmap where the entropy value exceeds a preset threshold;

[0032] In the multimodal cross-attention fusion network, a local adaptive attention mechanism is performed on the feature map patches corresponding to the high uncertainty regions.

[0033] Optionally, the method for generating the saliency map in step S7 is as follows:

[0034] Using Grad-CAM (Gradient-weighted Class Activation Mapping) technology, the gradient of the weighted fusion feature with respect to the classification probability is calculated in reverse, generating a pixel-level heatmap. The highlighted areas in the heatmap represent the key pixel regions where the model makes the current classification decision.

[0035] A second aspect of the present invention also provides a skin lesion diagnosis system based on detection segmentation and multimodal alignment, the system comprising a processor and a memory, the memory storing a computer program, the processor executing the computer program to implement the skin lesion diagnosis method based on detection segmentation and multimodal alignment described in the first aspect of the present invention.

[0036] The skin lesion diagnosis method and system based on detection segmentation and multimodal alignment disclosed herein have the following technical advantages:

[0037] This invention constructs a deeply coupled cascaded model of YOLO and SAM, utilizing YOLOv11 combined with deformable convolution to extract candidate lesion regions and automatically generating a dynamic cue vector set containing center and boundary points to drive SAM for zero-shot fine-grained segmentation. This effectively overcomes the segmentation failure problem of traditional single segmentation models. In particular, by introducing an uncertainty estimation head into the SAM mask decoder, the confidence level of the segmentation results can be quantified, improving segmentation accuracy to the pixel level.

[0038] This invention uses the InfoNCE loss function to force the alignment of image morphological features with pathological / biochemical semantic vectors in the latent space, solving the semantic bias problem caused by traditional simple feature splicing. During the inference phase, cross-attention calculation is performed using biochemical index vectors as queries and image features as keys and values. This allows the model to dynamically focus on image regions highly correlated with clinical indicators, effectively avoiding the risks of missed diagnoses and misdiagnoses caused by "different images for the same disease" or "same images for different diseases," thus improving diagnostic accuracy and robustness.

[0039] This invention utilizes a segmentation uncertainty heatmap to locally adaptively weight the feature maps of a multimodal fusion network: for high-uncertainty regions with entropy values ​​exceeding a threshold, the system automatically increases the weight coefficient of that region in cross-modal attention calculation and triggers secondary upsampling feature extraction. This closed-loop feedback mechanism enables the model to actively focus on difficult-to-distinguish boundary regions, thereby achieving refined analysis of key regions under limited computational resources and further improving the model's diagnostic ability for complex lesions.

[0040] This invention supports flexible integration of morphological modalities, pathological / biochemical modalities, and microscopic tissue modalities. Features such as nuclear heterogeneity and dermal invasion depth are extracted using a lightweight visual transformer and can be seamlessly integrated into a multimodal fusion network. This allows the system to adapt to the equipment conditions of different levels of medical institutions, operating not only in scenarios with only dermoscopy images but also demonstrating stronger multimodal diagnostic performance in central hospitals with pathological examination capabilities, exhibiting good scalability and universality.

[0041] Other features and advantages of the invention will become clear from the following detailed description of exemplary embodiments of the invention with reference to the accompanying drawings. Attached Figure Description

[0042] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments of the invention and, together with their description, serve to explain the principles of the invention.

[0043] Figure 1 This is a flowchart illustrating the skin lesion diagnosis method based on detection segmentation and multimodal alignment provided in an embodiment of the present invention.

[0044] Figure 2 This is a diagram of the multimodal cross-attention fusion network structure provided in an embodiment of the present invention. Detailed Implementation

[0045] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the invention.

[0046] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the invention or its application or use.

[0047] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0048] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0049] This invention proposes an embodiment of a skin lesion diagnosis method based on detection segmentation and multimodal alignment, specifically, as follows: Figure 1 As shown, it includes the following steps:

[0050] Step S1: Obtain multimodal data to be diagnosed. The multimodal data includes at least morphological modal data and corresponding pathological / biochemical modal data, wherein the morphological modal data includes dermoscopy image data. In this embodiment of the invention, in step S1, the multimodal data further includes microscopic tissue modal data. The microscopic tissue modal data is obtained through the following steps:

[0051] Pathological slide image data is acquired, and microscopic tissue features are extracted from the pathological slide image data using a lightweight visual transformer. These microscopic tissue features include the nuclear heterogeneity index and estimated dermal invasion depth. Specifically, a hematoxylin-eosin stained whole-slide image of the same lesion region as the dermoscopy image is acquired. Considering the color difference interference between different slide batches, this step introduces a color deconvolution normalization algorithm to decouple the RGB pathological image into a hematoxylin concentration channel and an eosin concentration channel, ensuring spatial consistency for subsequent feature extraction. A dual-branch lightweight pathological visual transformer is constructed. The normalized pathological image slide is input and mapped to sequence features through the patch embedding layer of the lightweight visual transformer. A graph cut attention pooling layer is added at the output of the Transformer encoder. This graph cut attention pooling layer constructs a graph structure with the nuclear centroid as nodes, and quantifies the internuclear distance and chromatin texture differences by calculating the attention coefficients between nodes. The final output nuclear heterogeneity index is defined as the weighted sum of the spectral energy and feature variance of the graph Laplacian matrix, used to quantify the degree of tumor cell polymorphism, rather than a simple count. A depth-aware bounding box regression head is introduced onto the deep feature map of the lightweight visual transformer. Specifically, a differentiable Hough voting layer is used to locate the epidermal-dermal junction, and then the feature pyramid structure is used to calculate the farthest Euclidean distance of tumor cell nests breaking through the basement membrane. The output estimate of dermal invasion depth is a continuous numerical variable with an uncertainty interval.

[0052] The microscopic tissue features are used as a third modality and are input into the multimodal cross-attention fusion network for fusion. Specifically, the nuclear heterogeneity index and the estimated dermal invasion depth are nonlinearly mapped and concatenated in the feature space to form a compact microscopic tissue modality representation vector. This vector is used as the third modality and is input into the multimodal cross-attention fusion network for fusion.

[0053] Step S2: Input the morphological modality data into a pre-trained detection-segmentation cascade model, which includes an object detection branch and a cue-based large model segmentation branch. The object detection branch is responsible for quickly locating the region of interest in the entire high-resolution dermoscopy image, filtering out a large amount of irrelevant background noise such as hair, bubbles, and scales, and outputting a bounding box containing the lesion. The cue-based large model segmentation branch receives the bounding box output from the first stage as a spatial prior cue. This segmentation branch is built on a large-scale pre-trained visual foundation model, possessing strong zero-shot generalization ability and capable of performing pixel-level fine contour tracing within the bounding box using the cue information from the detection box.

[0054] In this embodiment of the invention, in step S2, the target detection branch is a detection network based on the YOLOv11 architecture. Its neck network employs deformable convolutional layers to adaptively extract the boundary features of irregularly shaped lesions. The deformable convolutional layers adaptively extract the boundary features of irregularly shaped lesions by applying a learnable two-dimensional offset at the sampling position of the standard convolutional kernel. YOLOv11 is a single-stage target detection architecture. The classification head and regression head of YOLOv11 are decoupled, ensuring that the extracted lesion center position and lesion width and height features do not interfere with each other. In the feature fusion stage of the neck network, deformable convolution is introduced, i.e., at each sampling position p of the standard convolutional kernel... n Apply a learnable two-dimensional offset Δp to it. n For n=1, ..., N, the mathematical expression is:

[0055] Where x is the input feature map, y is the output feature map, w is the convolution weight, and Δp is the input feature map. n This represents the offset predicted by the network based on the curvature of the lesion edge. When irregular pigmentation extends from the lesion edge, the sampling points of the deformable convolution kernel actively bend and cluster in the direction of extension, thereby adaptively extracting the boundary features of irregularly shaped lesions in the multi-scale feature pyramid constructed by the neck network. The precise lesion contour extracted in step S2 is not only used for segmentation; its output precise lesion mask will serve as a spatial attention weight map, guiding the multimodal cross-attention fusion network in subsequent step S3 to focus on the core region of the lesion, preventing dermoscopy background noise from interfering with the alignment calculation of microscopic tissue modality features and biochemical modality features.

[0056] Step S3: Extract candidate regions of the lesion through the target detection branch, and dynamically generate a hint vector set based on the geometric features of the candidate regions. The hint vector set contains at least one central hint point and one boundary hint point. The candidate region specifically refers to the high-confidence predicted bounding box output by the target detection branch (YOLOv11). This bounding box is not the final diagnostic result, but rather serves as a preliminary spatial constraint on the lesion. It contains the main body of the lesion, and the edges may still contain a small amount of normal skin background. The coordinate information of this bounding box will serve as the sole geometric basis for generating subsequent hint points.

[0057] In this embodiment of the invention, the dynamic generation of the cue vector set in step S3 specifically includes: using the geometric center of the predicted bounding box output by the target detection branch as the center cue point; and selecting N points closest to the predicted boundary of the lesion on the boundary of the predicted bounding box as the boundary cue points, where N is an integer greater than or equal to 2. It should be noted that for the large model segmentation branch based on Transformer, the center cue point can activate the query vector of the core lesion region, preventing the model from diverting attention to irrelevant hair or bubble noise within the box. On the four boundary line segments of the predicted bounding box, points are not selected at equal intervals, but rather by calculating the gradient magnitude of the feature response map within the box, selecting N points closest to the actual predicted boundary of the lesion. N is preferably set to 4. That is, one point is selected at the midpoint of each of the four sides (top, bottom, left, and right), or the position is adaptively adjusted according to the lesion's eccentricity. The position of the boundary cue points in this step is dynamically changing. For example, if the lesion grows eccentrically and is close to the left side of the prediction box, the gradient weights of the left boundary cue points will be automatically enhanced to prompt the segmentation network to perform more refined edge correction on the left side. Furthermore, the generated center cue point and N boundary cue points are not directly input as two-dimensional pixel coordinates, but are mapped to high-dimensional embedding vectors with the same dimension as the word vectors of the segmentation model through a positional encoding layer.

[0058] Step S4: Input the cue vector set into the large model segmentation branch to drive it to perform zero-sample or small-sample segmentation on the morphological modality data, and generate lesion mask images and segmentation uncertainty heatmaps;

[0059] In this embodiment of the invention, the step of generating the segmentation uncertainty heatmap in step S4 specifically includes: adding an uncertainty estimation head in parallel to the mask decoder of the large model segmentation branch. The uncertainty estimation head is used to calculate the entropy value of each pixel output by the decoder in terms of classification, and combining the entropy values ​​of all pixels to form the segmentation uncertainty heatmap. The uncertainty estimation head is supervised during the training phase using a weighted binary cross-entropy loss, where pixels located in the neighborhood of the lesion boundary are given higher loss weights to guide the model to focus on the uncertainty quantification of edge regions that are crucial for clinical diagnosis. The large model segmentation branch is built based on a pre-trained visual foundation model on a large-scale natural or medical image dataset. Its backbone network is typically a variant of a visual transformer, capable of capturing long-distance global semantic dependencies through a self-attention mechanism. Zero-shot segmentation refers to the model's ability to infer the boundary contour between lesions and normal skin based solely on the positional information of center and boundary cues without having seen any labeled dermoscopic images.

[0060] Furthermore, at the output of the mask decoder in the large model segmentation branch, an uncertainty estimation head with the same structure but independent weights is set in parallel with the branch that generates the standard lesion mask. For each pixel location (i, j) in the morphological modality image, the mask decoder originally outputs a probability value p after Sigmoid activation. ij This represents the confidence level that the pixel belongs to the lesion category. The uncertainty estimation head calculates the Shannon entropy of the predicted probability distribution by performing multiple random forward propagations on the decoder features: ,

[0061] The entropy value H of all pixel locations ij Mapped to a pseudo-color space, a segmentation uncertainty heatmap is formed that is strictly aligned with the lesion mask image space. In step S4, when the large model segmentation branch performs fine edge delineation, the high-entropy regions in the segmentation uncertainty heatmap are more focused on clinically significant micro-invasive edges, rather than being dominated by clinically insignificant noise regions such as hair occlusion or bubble reflection. This improves the alignment accuracy of microscopic tissue modal features and morphological modal features under spatial attention weighting in the subsequent step S5.

[0062] In this embodiment of the invention, in step S5, the multimodal cross-attention fusion network is obtained through contrastive learning pre-training, specifically as follows: Figure 2 As shown, the multimodal input serves as the data source for this fusion network, encompassing three complementary medical features covering macroscopic morphology, molecular pathology, and microscopic tissue dimensions. These correspond to lesion mask images, pathological / biochemical modal data, and microscopic tissue features, respectively. The microscopic tissue features are extracted from pathological sections using a lightweight visual transformer and include the cell nuclear heterogeneity index and estimated dermal invasion depth. The contrastive learning pre-training part automatically constructs positive and negative sample pairs and uses the InfoNCE loss function for training. This forces the network to narrow the distance between positive samples and widen the distance between negative samples in the high-dimensional latent space, eliminating distribution differences between different modalities and addressing the industry pain point of semantic mismatch between image and pathological features. This provides a unified feature space for downstream attention fusion. The cross-attention layer uses a biological semantic vector as the query Q, morphological features as the key K and value V, and microscopic tissue features are simultaneously embedded to participate in feature calculation. It automatically calculates the correlation between molecular pathological semantics and the spatial location of lesions, generating an attention weight distribution. Background filtering is performed using a lesion mask, retaining only lesion region features for calculation and removing interference from skin, hair, and background noise. Cross-modal enhanced features are generated, simultaneously fusing spatial morphological details and molecular pathological semantics to solve the misdiagnosis problem of "different images for the same disease and the same image for different diseases." The feature output part outputs standardized cross-modal enhanced features, which are directly transmitted to the adaptive weighting part of the uncertain heatmap.

[0063] The feature alignment method in the pre-training of the multimodal cross-attention fusion network includes: constructing positive sample pairs and negative sample pairs, where the positive sample pairs consist of the morphological features and bio-semantic vectors from the same patient, and the negative sample pairs consist of the morphological features and bio-semantic vectors from different patients; and using the InfoNCE loss function, constraining the multimodal cross-attention fusion network to bring the positive sample pairs closer together and push the negative sample pairs further apart in the latent space. The InfoNCE loss function is the lower bound optimization objective for noise contrast estimation, and its mathematical expression is: ,in:

[0064] zi is the latent vector of the morphological features after being mapped by the projection head; The latent vector is the biological semantic vector corresponding to the same patient mapped by another projection head; is the implicit representation of the biological semantic vectors in all negative sample pairs within a batch; sim is the cosine similarity. This is a temperature coefficient used to adjust the sharpness of the distribution; a smaller value indicates a lower temperature coefficient. This makes the loss function more sensitive to negative samples with similarity, thereby encouraging the network to learn more discriminative cross-modal decision boundaries, which is particularly crucial for distinguishing skin tumors with similar clinical manifestations but different molecular subtypes.

[0065] It should be noted that biological semantic vectors typically have low dimensionality but high semantic concentration; for example, the embedding vector of the BRAF V600E mutation state directly indicates the tumor's driving pathway. Morphological feature maps are multi-channel spatial feature maps obtained by encoding dermoscopy images through convolutional networks or Transformers, with each spatial location corresponding to a high-dimensional local descriptor. These are used simultaneously as keys and values, allowing the network to determine how much local information to extract from the values ​​based on the degree of match between the query and the key. The query Q and the key vector K for each spatial location are then calculated. i The dot product similarity is calculated, and the attention weight distribution α is obtained by Softmax normalization. i :

[0066] Where Kj is the key vector at the j-th spatial location, and d is the feature dimension, i.e., the number of channels in the biological semantic vector and the morphological feature map. is the scaling factor, and N is the total number of spatial positions after the morphological feature map is flattened.

[0067] Using the above attention weights on the value vector V i We perform a weighted summation to obtain the cross-modal enhancement feature F. enhanced :

[0068] .

[0069] The operation steps in this embodiment of the invention effectively eliminate interference from the normal skin background, making the correlation calculation of the cross-attention layer more focused on the pathological changes of the lesion itself, thereby avoiding the dilution effect of background noise on the multimodal alignment accuracy.

[0070] In this embodiment of the invention, the step of alignment and fusion in the latent space in step S5 specifically includes: inputting the biological semantic vector as a query and the morphological features as keys and values ​​into the cross-attention layer of the multimodal cross-attention fusion network; the cross-attention layer is used to calculate the correlation between the biological semantic vector and different spatial locations in the morphological features, and to perform weighted recombination of the morphological features according to the correlation to generate cross-modal enhanced features. The design of "using the biological semantic vector as a query" in this step solves the common visual artifact interference problem in existing dermoscopy AI diagnosis. In clinical practice, some benign keratosis lesions are morphologically very similar to early malignant melanoma. In some specific embodiments, if the input biological semantic vector indicates a high level of inflammatory factors, the attention mechanism will force the network to focus on the red halo area and erosion edge in the image, even if this area is blurred in the image due to low contrast. The finally generated cross-modal enhanced features contain both the spatial details of the image and the semantic bias of the pathological state, greatly improving the interpretability and robustness of the diagnostic results.

[0071] Furthermore, the cross-attention layer also receives the lesion mask image as a spatial gating signal. Before calculating the correlation degree, the morphological features are spatially filtered using the lesion mask image, retaining only the morphological features within the lesion area for the calculation of keys and values.

[0072] Step S6: Using the segmentation uncertainty heatmap, locally adaptively weight the feature map in the multimodal cross-attention fusion network to generate weighted fusion features; through this mechanism, the attention intensity to different regions can be dynamically adjusted according to the clarity of the lesion edge, avoiding the introduction of erroneous features of high-noise and high-ambiguity regions into the final diagnostic decision.

[0073] In this embodiment of the invention, step S6, which involves locally adaptively weighting the feature maps in the multimodal cross-attention fusion network, specifically includes: identifying high-uncertainty regions in the segmentation uncertainty heatmap where the entropy value exceeds a preset threshold; and in the multimodal cross-attention fusion network, performing a local adaptive attention mechanism on the feature map blocks corresponding to the high-uncertainty regions, i.e., increasing the weight coefficient of the region in cross-modal attention calculation and performing secondary upsampling feature extraction. Specifically, in this embodiment, the threshold is set to 3.1, which is an optimal value determined based on the theoretical maximum value of entropy for binary classification segmentation tasks and empirical statistics. In this embodiment, the threshold of 3.1 corresponds to the 85th percentile of the pixel-level entropy distribution after Monte Carlo Dropout sampling, which can effectively filter out low-amplitude entropy increases caused by normal texture jitter and accurately capture structurally high-uncertainty regions caused by lesion edge infiltration or pigment network breakage. The secondary upsampling feature extraction utilizes the high sensitivity of shallow networks to gradient edges, enabling the final weighted fusion features to have strong semantic discriminative power in the main lesion region and high spatial resolution in the high-uncertainty boundary region, thus providing a set of robust representations with spatial granularity adaptation for the subsequent classifier.

[0074] Step S7: Input the weighted fusion features into the multi-task prediction head and output the classification probability, risk score, and saliency map for interpreting the model's decision for the skin lesion. In this embodiment of the invention, the method for generating the saliency map in step S7 is as follows: using Grad-CAM technology, the gradient of the weighted fusion features with respect to the classification probability is calculated in reverse to generate a pixel-level heatmap. The highlighted areas in the heatmap represent the key pixel areas where the model makes the current classification decision. It should be noted that the multi-task prediction head consists of three parallel sub-network branches that share the same input, namely the weighted fusion features, but each has independent parameter weights. The weighted fusion features include visual details of lesion morphology, semantic biases of biomolecules, and spatial correction information for edge uncertainty. The classification probability in the three outputs is a probability vector normalized by Softmax; the risk score is a continuous scalar value, ranging from 0 to 100, output by the regression branch after Sigmoid or linear activation; and the saliency map is a color heatmap spatially aligned with the original dermoscopic image. The specific steps involved in processing using Grad-CAM technology include:

[0075] Step S71: After the multi-task prediction head outputs the classification probability, the system selects the category corresponding to the highest probability and calculates the classification score y. c Relative to a certain intermediate layer feature map A in the weighted fusion feature map k gradient: ,

[0076] in, This represents the activation value of the k-th feature channel at spatial location (i, j).

[0077] Step S72: Perform global average pooling on the above gradients in the spatial dimension to obtain the importance weight of each feature channel to the target category. : ,

[0078] Where Z is the spatial size of the feature map.

[0079] Step S73: Perform a linear weighted combination of the feature maps from each channel according to their importance weights, and retain the regions that positively contribute to the target category using the ReLU activation function: ,

[0080] Step S74: The image is upsampled to the same size as the input dermoscopy image using bilinear interpolation and then superimposed on the original image to form the final saliency map.

[0081] In step S7 of this embodiment of the invention, the multimodal representations constructed in steps S1 to S6 are transformed into an integrated output encompassing classification diagnosis, risk quantification, and spatial interpretation through a multi-task prediction head. Specifically, the saliency map generated using Grad-CAM overcomes the "black box" nature of traditional deep learning models. Doctors can quickly assess the credibility and clinical acceptability of AI diagnostic conclusions by observing whether the highlighted areas in the heatmap match the classic diagnostic criteria of dermoscopy, thus forming a complete human-machine collaborative diagnostic closed loop.

[0082] The present invention also provides an embodiment of a skin lesion diagnosis system based on detection segmentation and multimodal alignment. The system includes a processor and a memory, the memory storing a computer program, and the processor executing the computer program to implement the skin lesion diagnosis method based on detection segmentation and multimodal alignment of the present invention.

[0083] This invention provides a skin lesion diagnosis method and system based on detection, segmentation, and multimodal alignment. It extracts candidate lesion regions by embedding a deformable convolutional YOLOv11 detection branch and dynamically generates center and boundary cue vector sets, driving a large segmentation model to output a lesion mask and an uncertainty heatmap based on pixel-level entropy. A multimodal cross-attention fusion network is constructed, using biological semantic vectors as queries to perform cross-modal weighted recombination of morphological features. Simultaneously, the uncertainty heatmap is used to apply local adaptive attention enhancement and secondary upsampling refinement to high-entropy edge regions. Finally, a multi-task prediction head synchronously outputs classification probabilities, risk scores, and Grad-CAM saliency maps. This invention improves the segmentation accuracy and cross-modal semantic alignment capability of lesions with blurred boundaries, strengthens the diagnostic robustness of infiltrative edges through uncertainty feedback loops, and enhances the clinical interpretability of model decisions through visualization maps, achieving efficient collaborative diagnosis of morphological phenotypes and molecular pathological information.

[0084] The above description of the structure, features, and effects of the present invention is based on the embodiments shown in the figures. However, the above are only preferred embodiments of the present invention. It should be noted that the technical features involved in the above embodiments and their preferred methods can be reasonably combined and matched by those skilled in the art to form a variety of equivalent solutions without departing from or changing the design concept and technical effects of the present invention. Therefore, the present invention is not limited to the scope of implementation shown in the figures. Any changes made in accordance with the concept of the present invention, or modifications to equivalent embodiments, that do not exceed the spirit covered by the specification and figures, should be within the protection scope of the present invention.

Claims

1. A method for diagnosing skin lesions based on detection segmentation and multimodal alignment, characterized in that, Includes the following steps: Step S1: Obtain the multimodal data to be diagnosed, wherein the multimodal data includes at least morphological modal data and corresponding pathological / biochemical modal data; Step S2: Input the morphological modality data into a pre-trained detection-segmentation cascade model, which includes an object detection branch and a cue-based large model segmentation branch; Step S3: Extract candidate regions of lesions through the target detection branch, and dynamically generate a hint vector set based on the geometric features of the candidate regions. The hint vector set contains at least one center hint point and one boundary hint point. Step S4: Input the cue vector set into the large model segmentation branch to drive it to perform zero-sample or small-sample segmentation on the morphological modality data, and generate lesion mask images and segmentation uncertainty heatmaps; Step S5: Construct a multimodal cross-attention fusion network to align and fuse the morphological features extracted from the lesion mask image with the biological semantic vectors converted from the pathological / biochemical modal data in the latent space; Step S6: Using the segmentation uncertainty heatmap, locally adaptively weight the feature maps in the multimodal cross-attention fusion network to generate weighted fusion features; Step S7: Input the weighted fusion features into the multi-task prediction head and output the classification probability of skin lesions, risk score, and saliency map used to explain the model decision.

2. The skin lesion diagnosis method based on detection segmentation and multimodal alignment according to claim 1, characterized in that, In step S1, the multimodal data further includes microstructure modal data, which is obtained through the following steps: Pathological slide image data is acquired, and the microscopic tissue features of the pathological slide image data are extracted using a lightweight visual transformer. The microscopic tissue features include the nuclear heterogeneity index and the estimated dermal infiltration depth. The micro-organism features are used as a third modality and are input into the multimodal cross-attention fusion network for fusion.

3. The skin lesion diagnosis method based on detection segmentation and multimodal alignment according to claim 1, characterized in that, In step S2, the target detection branch is a detection network based on the YOLOv11 architecture, and its neck network uses deformable convolutional layers to adaptively extract the boundary features of irregularly shaped lesions.

4. The skin lesion diagnosis method based on detection segmentation and multimodal alignment according to claim 1, characterized in that, The dynamic generation of the prompt vector set in step S3 specifically includes: The geometric center of the predicted bounding box output by the target detection branch is used as the center cue point; On the boundary of the prediction box, select N points that are closest to the lesion prediction boundary as the boundary prompt points, where N is an integer greater than or equal to 2.

5. The skin lesion diagnosis method based on detection segmentation and multimodal alignment according to claim 1, characterized in that, The step of generating the segmentation uncertainty heatmap in step S4 specifically includes: In the mask decoder of the large model segmentation branch, an uncertainty estimation head is added in parallel. The uncertainty estimation head is used to calculate the entropy value of each pixel output by the decoder in terms of classification. The entropy values ​​of all pixels are combined to form the segmentation uncertainty heatmap.

6. The skin lesion diagnosis method based on detection segmentation and multimodal alignment according to claim 1, characterized in that, In step S5, the multimodal cross-attention fusion network is obtained through contrastive learning pre-training, and its feature alignment methods during pre-training include: Construct positive sample pairs and negative sample pairs, wherein positive sample pairs are the morphological features and the bio-semantic vectors from the same patient, and negative sample pairs are the morphological features and the bio-semantic vectors from different patients; By using the InfoNCE loss function, the multimodal cross-attention fusion network is constrained to bring positive sample pairs closer together and push negative sample pairs further apart in the latent space.

7. The skin lesion diagnosis method based on detection segmentation and multimodal alignment according to claim 6, characterized in that, The step of aligning and blending in the latent space in step S5 specifically includes: The biological semantic vector is used as a query, and the morphological features are used as keys and values, which are then input into the cross-attention layer of the multimodal cross-attention fusion network. The cross-attention layer is used to calculate the correlation between the biological semantic vector and different spatial locations in the morphological features, and to perform weighted recombination of the morphological features based on the correlation to generate cross-modal enhanced features.

8. The skin lesion diagnosis method based on detection segmentation and multimodal alignment according to claim 5, characterized in that, The step S6 of performing local adaptive weighting on the feature maps in the multimodal cross-attention fusion network specifically includes: Identify high uncertainty regions in the segmentation uncertainty heatmap where the entropy value exceeds a preset threshold; In the multimodal cross-attention fusion network, a local adaptive attention mechanism is performed on the feature map patches corresponding to the high uncertainty regions.

9. The skin lesion diagnosis method based on detection segmentation and multimodal alignment according to claim 1, characterized in that, The method for generating the saliency map in step S7 is as follows: Using Grad-CAM technology, the gradient of the weighted fusion features with respect to the classification probability is calculated in reverse, generating a pixel-level heatmap. The highlighted areas in the heatmap represent the key pixel regions where the model makes the current classification decision.

10. A skin lesion diagnostic system based on detection segmentation and multimodal alignment, characterized in that, The system includes a processor and a memory, the memory storing a computer program, and the processor executing the computer program to implement the skin lesion diagnosis method based on detection segmentation and multimodal alignment as described in any one of claims 1-9.

Images