A medical image segmentation method based on a polarity fine-tuning large model migration method
By combining a polarity fine-tuning strategy with a Canny boundary map, a multi-scale mask guidance prompting module and a polarity decoder were constructed, which solved the problem of insufficient feature capture in medical image segmentation and achieved efficient skin lesion image segmentation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- NANTONG UNIV
- Filing Date
- 2026-01-20
- Publication Date
- 2026-06-05
AI Technical Summary
Existing general-purpose segmentation models lack semantic understanding in medical image segmentation, making it difficult to accurately identify skin lesion areas. Furthermore, traditional low-rank matrix factorization strategies fail to fully capture the potential complex features in the data when transferred to complex tasks, resulting in poor segmentation performance.
By combining a polarity fine-tuning strategy with a Canny boundary map, and decoupling features through a dual-path low-rank parameterized structure, a multi-scale mask guidance prompting module and a polarity decoder are constructed to achieve self-attention and cross-attention interaction of features, generating high-quality segmentation masks.
It improves the accuracy and reliability of skin lesion image segmentation, enhances noise suppression and feature extraction capabilities, and achieves efficient medical image segmentation.
Smart Images
Figure CN122156603A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of technology, and in particular to a medical image segmentation method based on a polarity fine-tuning large model transfer method. Background Technology
[0002] Image segmentation of skin lesions plays a crucial role in computer vision and medical image analysis. It involves dividing an image into different parts representing various skin elements. Histopathological image analysis of melanoma is a key step in the early diagnosis and prognostic assessment of skin lesions. Although this task has significant clinical importance, image segmentation of melanoma lesions still faces considerable challenges due to the high heterogeneity and complex tissue structure of the tissue samples. Designed segmentation methods often rely on the extraction of significant boundary features. However, melanoma lesions often exhibit blurred boundaries, irregular boundaries due to invasive growth, and uneven pigment distribution, making it difficult for models to accurately identify lesion areas. Furthermore, noise interference, such as hair, bubbles, and skin texture, often significantly impacts the segmentation process, thereby affecting the reliability of quantitative pathological analysis and clinical decision-making.
[0003] In recent years, the Segment Anything Model (SAM), a general-purpose segmentation model, has demonstrated excellent zero-shot generalization ability on natural images, ushering in a new era in skin lesion segmentation research. However, because SAM's basic architecture is based on natural image segmentation design, it often lacks semantic understanding when processing medical images. Without semantic understanding, SAM cannot automatically classify segmented regions into specific anatomical structures or pathologies. Introducing the LoRA method to fine-tune the SAM image encoder to adapt it to medical image tasks improves SAM's generalization ability to medical images. However, while this method relies on fine-tuning with large-scale labeled data to improve SAM's generalization ability to medical images, it also weakens SAM's zero-shot capability.
[0004] Meanwhile, in terms of fine-tuning strategies that use parameter-efficient learning in encoders and decoders, the adaptability of current fine-tuning methods is still limited when dealing with large-scale and diverse tasks. When transferring to medical image tasks, traditional low-rank matrix factorization strategies show discrete fitting effects when faced with high-dimensional inputs and complex images, and cannot fully capture the potential complex features in the data.
[0005] Therefore, the ability to effectively transfer large-scale models to more complex task scenarios at low cost and how to achieve better results in preserving image feature representation has become a research hotspot. Summary of the Invention
[0006] The purpose of this invention is to provide a medical image segmentation method based on a large model transfer method with polarity fine-tuning. This invention combines a large model transfer method with polarity fine-tuning strategy with self-suggestion flow of Canny boundary map to segment skin lesion images. It solves the technical problem that traditional low-rank matrix factorization strategies have discrete fitting effects and cannot fully capture the potential complex features in the data when facing high-dimensional input and complex images. Thus, it can effectively transfer large-scale models to more complex task scenarios at low cost, and achieve better results in maintaining image feature expression. This architecture effectively solves the limitations of traditional fine-tuning strategies and enhances the noise suppression and feature extraction capabilities of medical image classification models.
[0007] The inventive concept of this invention is as follows: a medical image segmentation method based on polarity fine-tuning large model transfer, which can accurately segment the lesion contour of skin lesion images and more accurately identify the lesion boundary. By using polarity fine-tuning strategy and Canny multi-scale self-suggestion flow fusion features, the segmentation accuracy is improved and the reliability of segmentation is effectively enhanced. This method can effectively help doctors diagnose skin lesions and allow patients to receive the best treatment period.
[0008] This invention is achieved through the following measures: a medical image depth segmentation method based on polarity fine-tuning large model transfer, comprising the following steps: S1: Expand and scale the medical image dataset, which contains N pairs of original color images and corresponding segmentation masks. Rotate and enhance the images clockwise with the center point of each image as the rotation center, and save the enhanced images with a step size of 20°, so that the dataset is expanded to 18×N pairs of original images and corresponding segmentation masks. Then, scale / crop the expanded original images and corresponding segmentation masks to 512×512 resolution to obtain standardized sample pairs for training. S2: Under the condition of freezing the weights of the backbone network of the pre-trained large model, the polarity fine-tuning strategy is injected into the attention layer and feedforward layer of the Transformer block; the features are decoupled and corrected by introducing a dual-path low-rank parameterized structure of positive and negative branches, and the polarity-enhanced image features are extracted without destroying the original feature expression structure, so as to obtain the feature map for subsequent segmentation. S3: Construct a multi-scale mask guidance prompt module MSGP, which aggregates and restores the feature map output from step S2 step by step, and integrates the Canny edge detection map and multi-scale semantic features to automatically generate mask prompts that match the target foreground region and dense and sparse tokens used to supplement the constraints of key parts, so as to replace the prompt information generated by manual point / box annotation; S4: Construct a polar decoder PoDecoder for feature interaction and spatial detail restoration; use a bidirectional Transformer architecture to perform self-attention and cross-attention interaction on the image features in step S2 and the cue tokens generated in step S3 to guide the model to focus on key segmentation regions; and restore the features to the input size step by step through deconvolution sequences or upsampling restoration modules to output the final segmentation mask. S5: Combine steps S2, S3, and S4 to construct the network model PoSAM, and train the model using a joint loss function, which includes foreground binary cross-entropy loss, weighted focus loss, and DSC loss; after training, accurate segmentation and boundary definition of skin lesion areas are achieved.
[0009] As a further optimization scheme for the medical image style transfer method based on polarity fine-tuning of large models provided by the present invention, the specific steps of step S2 are as follows: S2.1: For the image encoder (backbone network) of the arbitrary segmentation model SAM, the input skin lesion image is denoted as... ,in For image height, For image width, Number of channels; image The image is divided into several non-overlapping patches, and the side length of each patch is denoted as . And take The number of image patches in the vertical and horizontal directions are respectively , The total number of tokens is , No. Each image patch is flattened into a vector, and then mapped by a linear mapping matrix. Projected onto latent feature dimensions Simultaneously, learnable position embeddings are added. to the initial token ; All The initial feature sequence is obtained by splicing the components in a fixed order. This serves as the input for subsequent Transformer encoding.
[0010] in, This indicates the input of a medical image. , , These are the image height, width, and number of channels, respectively. The side length of the image patch (take) ), , These represent the number of image blocks in the vertical and horizontal directions, respectively. The total number of tokens; For the first Image patch flattening vector, It is a linear projection matrix. For latent feature dimensions, For learnable location embedding, For the corresponding token, For the initial token sequence; S2.2: Construct a Transformer multilayer encoder containing a polar low-rank adapter (PoLoRA), keeping the pre-trained weights of the SAM frozen and only updating the trainable parameters introduced by PoLoRA. Insert a dual-path polar low-rank branch in parallel into the attention layer linear mapping and the feedforward layer linear mapping of each Transformer block, with attention computation satisfying:
[0011] The input feature vector for any token is denoted as . The positive / negative polarity components and the power function are defined as follows:
[0012] Freeze the pre-trained weights of the linear layer Under these conditions, a dual-path polarity low-rank increment is introduced to obtain a polarity fine-tuning output. The calculation is as follows:
[0013] In the attention layer, the query, key, and value are obtained from the PoLoRA output of the corresponding projection layer: in For sequence The matrix output is obtained by calculating the feature vector of each token according to equation (3) and stacking them in order. The output of each attention head is:
[0014] The multi-head attention output is:
[0015] The output of the feedforward network is:
[0016] in Token-level input feature vector, , For linear layer input / output dimensions, For linear layer output, and These are the positive and negative polarity components, respectively. It is a zero vector. It is an element-wise exponentiation function. This indicates element-wise exponentiation. For exponential hyperparameters; For the frozen pre-trained weights, , , These are the frozen weights for the query, key, and value projection layers in the attention layer, respectively. Freeze weights in the projection layer for multi-head attention output. , The frozen weights are the two linear mappings of the feedforward network. It is a low-rank number. , These are positive and negative polarity gating matrices, respectively. , These are low-rank branch matrices with positive and negative polarities, respectively. , These are low-rank branch matrices with positive and negative polarities, respectively; , , These are the query, key, and value matrices for the attention mechanism. , , Indicates the first The corresponding blocks for each attention head. For single-head dimension, For transpose, Linearized Softmax operation, For the first Each attention head output, For the number of attention heads, This indicates a splicing operation. For multi-head attention output, For layer normalization, For activation function, This is the output of the feedforward network.
[0017] As a further optimization scheme for the medical image style transfer method based on polarity fine-tuning of large models provided by the present invention, the specific steps of step S3 are as follows: S3.1: To construct the multi-scale mask guidance and prompting module, the feature map output from step S2 is used as input, denoted as... Perform multi-scale semantic processing on it; set a scale index. Corresponding scaling factor For each scale branch, first adjust the spatial resolution to Then through Convolution with channel compression to , obtained the Semantic feature maps at various scales The calculation is as follows:
[0018] in, The feature map output by step S2, Its number of channels, and Its spatial dimensions; For scale indexing, is a scale factor and ; This means "upsample the input feature map to..." and through Convolution compresses the number of channels to The operation of ""; S3.2: To suppress redundant textures and low-level irrelevant semantics, semantic feature maps at each scale are processed. Further compression was performed, using... The convolutional kernel maps it to a single-channel response, and then a single-channel cue map is obtained by sigmoid activation. The calculation is as follows:
[0019] For the first Semantic feature maps at various scales for Convolution kernel, This represents a two-dimensional convolution operation. For the Sigmoid activation function, Single-channel prompt image S3.3: To achieve cross-scale alignment to the maximum resolution, the single-channel cue maps at each scale are uniformly aligned to the maximum resolution. The aligned tooltip is obtained by performing an upsampling operator on the scaled tooltip. The calculation is as follows:
[0020] in This indicates that the feature map is scaled up. Upsampling upsampling operator, This is the aligned prompt image; S3.4: Calculate and size-align the Canny edge map based on the input image. ,Will , , and The components are stitched together according to channel dimensions to form the final fused feature. The prompt token used for subsequent step S4 for prompt token construction and segmentation guidance is calculated as follows: in Use a Canny edge map aligned with the maximum resolution; This indicates a splicing operation. This is the feature map after fusing multi-scale cueing and edge constraints.
[0021] As a further optimization scheme for the medical image style transfer method based on polarity fine-tuning of large models provided by the present invention, the specific steps of step S4 are as follows: S4.1: Fuse the encoded feature map output from step S2 with the mask cue embedding output from step S3 to obtain the initial pixel sequence embedding. The calculation is as follows:
[0022] Wherein represents the image encoding feature map output from step S2. This indicates the dense embedding of the mask prompt output in step S3. This indicates a point-by-point addition operation. This represents the embedding of the initial image pixel sequence after fusion; In the bidirectional Transformer, self-attention modeling is performed on both the cue side and the image side, and bidirectional information interaction is achieved through cross-attention. Specifically, the sparse cue is first embedded and self-attention is used to obtain the query semantics. , and then For the query, cross-attention is performed using image sequences as keys, and the updated semantic hints are obtained through a feedforward network. :
[0023] in Sparse embedding of mask hints This indicates a self-attention operation. Indicates For query, For key, Cross-attention operation for values, , , These are the query matrix, key matrix, and value matrix, respectively. This represents a feedforward neural network. This represents the semantic features of the prompts after being updated by self-attention and cross-attention; Subsequently, the updated prompt semantics were used. Using the image-side intermediate semantics as the key, cross-attention is performed again to obtain the interacted image sequence. :
[0024] in Representing image sequence features after bidirectional interaction S4.2: The interactive image sequence Dimension reduction is performed to obtain the feature map. The spatial resolution is then restored step by step through deconvolution (or upsampling) sequences to obtain the restored high-resolution features. :
[0025] This represents the dimensionality transformation operation that restores sequence features to a two-dimensional feature map. This represents the decoded feature map after reconstruction. This represents a stepwise upsampling recovery operation consisting of interpolation upsampling and convolution. This represents a high-resolution feature map restored to the target spatial resolution.
[0026] S4.3: The high-resolution features obtained in step S4.2 Input into the prediction head and combine with the semantic hints from step S4.1 Pixel-by-pixel fusion to obtain the final segmentation mask : in This represents the prediction head of a multilayer perceptron. This indicates element-wise multiplication. This represents the final output binary segmentation mask for the lesion region.
[0027] As a further optimization scheme for the medical image style transfer method based on polarity fine-tuning of large models provided by the present invention, the specific steps of step S5 are as follows: S5.1: Divide the skin lesion image dataset into training and testing sets, and input them into the constructed PoSAM network model; S5.2: Combine the polarity fine-tuning strategy encoder module of step S2, the multi-scale mask guidance prompt module of step S3, and the polarity decoder of step S4 to construct a PoSAM network, so that the model forms a closed-loop processing structure of "encoding - self-generated prompting - interaction - decoding".
[0028] S5.3: End-to-end training of the model using a joint loss function: First, a binary cross-entropy loss is introduced into the foreground probability sequence output in step S3 for supervision, to constrain the generation quality of foreground cues. The loss function is:
[0029] Secondly, the final segmentation mask output in step S4 is subjected to supervision by a weighted combination of focus loss and DSC loss, resulting in the following segmentation loss:
[0030] And with the joint objective function Optimize network parameters.
[0031] in This represents the binary cross-entropy loss that supervises the foreground probability sequence output from step S3. Indicates the number of sequence elements used for supervision. Indicates the first Sum the elements of each sequence and take the mean. Indicates the relationship with the first The downsampled ground truth labels corresponding to each sequence element. Indicates the first Forecasting of sequence elements, This represents the Sigmoid function. Represents logarithmic operations; This represents the segmentation loss used to monitor the final segmentation mask. These are the controllable weighting coefficients for the two losses. For weighted focus loss, For DSC loss, This is the joint objective function used for training optimization.
[0032] Step S5.4: Iteratively update the trainable parameters using stochastic gradient descent (SGD) to minimize the joint objective function. To achieve the training objective, the PoSAM network was trained, and inference was performed on the test set to obtain the lesion region segmentation results.
[0033] To achieve the above-mentioned objectives, this invention also provides a medical image segmentation system based on a polarity fine-tuning large model transfer method, the system comprising: The medical image dataset module is configured to perform the following processes to augment and crop the medical image dataset; The pre-trained large model skeleton weight module is frozen and configured to perform the following process: under the condition of freezing the pre-trained large model skeleton weight, a polarity fine-tuning strategy is designed and injected into the attention layer and feedforward layer of the Transformer block. By introducing positive and negative dual-path low-rank branches, the features are decoupled and modified, and polarity-enhanced features are extracted without destroying the original representation structure. The Multi-Scale Mask Guidance and Prompt Module (MSGP) is configured to perform the following process: MSGP generates a self-prompt flow, adopts a step-by-step convergence and restoration strategy for the extracted feature maps, integrates Canny edge detection maps and multi-scale semantic features, and automatically generates and filters high-quality foreground region mask prompts (dense and sparse tokens) to replace manual point box annotation. The polar decoder PoDecoder module is constructed and configured to perform the following process: feature interaction and recovery on the polar decoder PoDecoder module; self-attention and cross-attention interaction between the output image features and the generated cue tokens using a bidirectional Transformer architecture to guide the model to focus on key parts; and feature recovery to the input size through deconvolution sequence upsampling to generate the final segmentation mask. The PoSAM network model is configured to perform the following process: using a joint loss function, including foreground binary cross-entropy loss, weighted focus loss, and DSC loss, the model is trained to achieve accurate segmentation and boundary definition of skin lesion areas.
[0034] Meanwhile, the present invention proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed, it implements the steps of the method described in the present invention.
[0035] Furthermore, the present invention proposes a computer-readable storage medium having a computer program stored thereon, the computer program being configured to implement the steps of the method described in the present invention when invoked by a processor.
[0036] Finally, the present invention proposes a computer program product, including a computer program / instructions, characterized in that the computer program / instructions, when executed by a processor, implement the steps of the method described in the present invention.
[0037] Compared with the prior art, the present invention, employing the above technical solution, has the following technical effects: 1. To address the issue that traditional low-rank matrix factorization strategies suffer from discrete fitting results and fail to fully capture the potential complex features in medical image tasks when transferring large models to high-dimensional inputs and complex images, this invention proposes a polar low-rank adaptation strategy. This strategy requires training only a very small number of low-rank matrices, significantly reducing training parameters and computational resource consumption, effectively balancing model performance and deployment costs, and achieving efficient transfer.
[0038] 2. To address the issue of susceptibility to noise interference when processing samples with blurred lesion boundaries and weak contrast, the designed PoLoRA module innovatively introduces dual low-rank branches for positive and negative samples. This enables explicit polarity-aware correction of fine-grained textures and high-level semantics, significantly improving the model's ability to accurately capture and suppress lesion boundaries.
[0039] 3. This invention proposes a Multi-Scale Mask Guided Prompt (MSGP) module, which achieves automatic generation of segmentation prompt tokens by fusing multi-scale features with Canny edge information. This module completely avoids the reliance on manual interactive prompts in the original large model framework, transforming the model from interactive segmentation to a fully automatic, end-to-end segmentation process, greatly improving practicality and automation. Attached Figure Description
[0040] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.
[0041] Figure 1 The diagram shown is a framework diagram of a medical image segmentation method based on polarity fine-tuning according to the present invention.
[0042] Figure 2 The diagram shows the structure of the polarity fine-tuning strategy in the Transformer layer in this invention.
[0043] Figure 3 The diagram shown is a structural diagram of the Canny multi-scale self-prompting flow module in this invention.
[0044] Figure 4 The diagram shows the structure of the decoder that fuses the encoder and the Canny multi-scale self-prompting stream in this invention.
[0045] Figure 5 The figure shown is a result of the fine-tuning strategy in this invention compared with other fine-tuning strategies.
[0046] Figure 6 The image shown is a result of segmenting a dataset of skin lesion images according to the present invention. Detailed Implementation
[0047] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. Of course, the specific embodiments described herein are merely illustrative and not intended to limit the invention. Example
[0048] See Figures 1 to 6This embodiment provides a medical image segmentation method based on polarity fine-tuning large model transfer, including the following steps: S1: Expand and crop the medical image dataset. The dataset contains N pairs of original color images and segmentation masks. Rotate each image clockwise around its center point, saving a new image every 20°, expanding to 18*N pairs of original images and corresponding segmentation masks. Then scale the original images to obtain 18*N pairs. Original image at the specified resolution and its corresponding segmentation mask; S2: Under the condition of freezing the pre-trained large model skeleton weights, a polarity fine-tuning strategy is designed and injected into the attention layer and feedforward layer of the Transformer block. By introducing positive and negative dual-path low-rank branches, the features are decoupled and modified, and polarity-enhanced features are extracted without destroying the original representation structure. S3: The multi-scale mask guidance prompt module MSGP is designed to generate a self-prompting stream. The feature map extracted by S2 adopts a step-by-step convergence and restoration strategy, integrates the Canny edge detection map and multi-scale semantic features, and automatically generates and filters out high-quality foreground region mask prompt dense and sparse tokens to replace manual point box annotation. S4: Construct a polar decoder PoDecoder to perform feature interaction and recovery. Use a bidirectional Transformer architecture to interact the image features output by S2 with the cue tokens generated by S3 through self-attention and cross-attention, guide the model to focus on key parts, and restore the features to the input size by upsampling the deconvolution sequence to generate the final segmentation mask. S5: Build the network model PoSAM described in S2, S3, and S4, and use a joint loss function, including foreground binary cross-entropy loss, weighted focus loss, and DSC loss, to train the model and achieve accurate segmentation and boundary definition of skin lesion areas.
[0049] Specifically, the steps of step S2 are as follows: S2.1: For the image encoder (backbone network) of the arbitrary segmentation model SAM, the input skin lesion image is denoted as... ,in For image height, For image width, Number of channels; image The image is divided into several non-overlapping patches, and the side length of each patch is denoted as . And take The number of image patches in the vertical and horizontal directions are respectively , The total number of tokens is , No. Each image patch is flattened into a vector. Through linear mapping matrix Projected onto latent feature dimensions Simultaneously, learnable position embeddings are added. to the initial token ; All The initial feature sequence is obtained by splicing the components in a fixed order. This serves as the input for subsequent Transformer encoding.
[0050] in, This indicates the input of a medical image. , , These are the image height, width, and number of channels, respectively. The side length of the image patch (take) ), , These represent the number of image blocks in the vertical and horizontal directions, respectively. The total number of tokens; For the first Image patch flattening vector, It is a linear projection matrix. For latent feature dimensions, For learnable location embedding, For the corresponding token, For the initial token sequence; S2.2: Construct a Transformer multilayer encoder containing a polar low-rank adapter (PoLoRA), keeping the pre-trained weights of the SAM frozen and only updating the trainable parameters introduced by PoLoRA. Insert a dual-path polar low-rank branch in parallel into the attention layer linear mapping and the feedforward layer linear mapping of each Transformer block, with attention computation satisfying:
[0051] The input feature vector for any token is denoted as . The positive / negative polarity components and the power function are defined as follows:
[0052] Freeze the pre-trained weights of the linear layer Under these conditions, a dual-path polarity low-rank increment is introduced to obtain a polarity fine-tuning output. The calculation is as follows:
[0053] In the attention layer, the query, key, and value are obtained from the PoLoRA output of the corresponding projection layer: in For sequence The matrix output is obtained by calculating the feature vector of each token according to equation (3) and stacking them in order. The output of each attention head is:
[0054] The multi-head attention output is:
[0055] The output of the feedforward network is:
[0056] in Token-level input feature vector, , For linear layer input / output dimensions, For linear layer output, and These are the positive and negative polarity components, respectively. It is a zero vector. It is an element-wise exponentiation function. This indicates element-wise exponentiation. For exponential hyperparameters; For the frozen pre-trained weights, , , These are the frozen weights for the query, key, and value projection layers in the attention layer, respectively. Freeze weights in the projection layer for multi-head attention output. , The frozen weights are the two linear mappings of the feedforward network. It is a low-rank number. , These are positive and negative polarity gating matrices, respectively. , These are low-rank branch matrices with positive and negative polarities, respectively. , These are low-rank branch matrices with positive and negative polarities, respectively; , , These are the query, key, and value matrices for the attention mechanism. , , Indicates the first The corresponding blocks for each attention head. For single-head dimension, For transpose, Linearized Softmax operation, For the first Each attention head output, For the number of attention heads, This indicates a splicing operation. For multi-head attention output, For layer normalization, For activation function, This is the output of the feedforward network.
[0057] Specifically, the steps of step S3 are as follows: S3.1: To construct the multi-scale mask guidance and prompting module, the feature map output from step S2 is used as input, denoted as... Perform multi-scale semantic processing on it; set a scale index. Corresponding scaling factor For each scale branch, first adjust the spatial resolution to Then through Convolution with channel compression to , obtained the Semantic feature maps at various scales The calculation is as follows:
[0058] in, The feature map output by step S2, Its number of channels, and Its spatial dimensions; For scale indexing, is a scale factor and ; This means "upsample the input feature map to..." and through Convolution compresses the number of channels to The operation of ""; S3.2: To suppress redundant textures and low-level irrelevant semantics, semantic feature maps at each scale are processed. Further compression was performed, using... The convolutional kernel maps it to a single-channel response, and then a single-channel cue map is obtained by sigmoid activation. The calculation is as follows:
[0059] For the first Semantic feature maps at various scales for Convolution kernel, This represents a two-dimensional convolution operation. For the Sigmoid activation function, Single-channel prompt image S3.3: To achieve cross-scale alignment to the maximum resolution, the single-channel cue maps at each scale are uniformly aligned to the maximum resolution. The aligned tooltip is obtained by performing an upsampling operator on the scaled tooltip. The calculation is as follows:
[0060] in This indicates that the feature map is scaled up. Upsampling upsampling operator, This is the aligned prompt image; S3.4: Calculate and size-align the Canny edge map based on the input image. ,Will , , and The components are stitched together according to channel dimensions to form the final fused feature. The prompt token used for subsequent step S4 for prompt token construction and segmentation guidance is calculated as follows:
[0061] in Use a Canny edge map aligned with the maximum resolution; This indicates a splicing operation. This is the feature map after fusing multi-scale cueing and edge constraints.
[0062] Specifically, the steps of step S4 are as follows: S4.1: Fuse the encoded feature map output from step S2 with the mask cue embedding output from step S3 to obtain the initial pixel sequence embedding. The calculation is as follows:
[0063] Wherein represents the image encoding feature map output from step S2. This indicates the dense embedding of the mask prompt output in step S3. This indicates a point-by-point addition operation. This represents the embedding of the initial image pixel sequence after fusion; In the bidirectional Transformer, self-attention modeling is performed on both the cue side and the image side, and bidirectional information interaction is achieved through cross-attention. Specifically, the sparse cue is first embedded and self-attention is used to obtain the query semantics. , and then For the query, cross-attention is performed using image sequences as keys, and the updated semantic hints are obtained through a feedforward network. :
[0064] in Sparse embedding of mask hints This indicates a self-attention operation. Indicates For query, For key, Cross-attention operation for values, , , These are the query matrix, key matrix, and value matrix, respectively. This represents a feedforward neural network. This represents the semantic features of the prompts after being updated by self-attention and cross-attention; Subsequently, the updated prompt semantics were used. Using the image-side intermediate semantics as the key, cross-attention is performed again to obtain the interacted image sequence. :
[0065] in Representing image sequence features after bidirectional interaction S4.2: The interactive image sequence Dimension reduction is performed to obtain the feature map. The spatial resolution is then restored step by step through deconvolution (or upsampling) sequences to obtain the restored high-resolution features. :
[0066] This represents the dimensionality transformation operation that restores sequence features to a two-dimensional feature map. This represents the decoded feature map after reconstruction. This represents a stepwise upsampling recovery operation consisting of interpolation upsampling and convolution. This represents a high-resolution feature map restored to the target spatial resolution.
[0067] S4.3: The high-resolution features obtained in step S4.2 Input into the prediction head and combine with the semantic hints from step S4.1 Pixel-by-pixel fusion to obtain the final segmentation mask : in This represents the prediction head of a multilayer perceptron. This indicates element-wise multiplication. This represents the final output binary segmentation mask for the lesion region.
[0068] Specifically, the steps of step S5 are as follows: S5.1: Divide the skin lesion image dataset into training and testing sets, and input them into the constructed PoSAM network model; S5.2: Combine the polarity fine-tuning strategy encoder module of step S2, the multi-scale mask guidance prompt module of step S3, and the polarity decoder of step S4 to construct a PoSAM network, so that the model forms a closed-loop processing structure of "encoding - self-generated prompting - interaction - decoding".
[0069] S5.3: End-to-end training of the model using a joint loss function: First, a binary cross-entropy loss is introduced into the foreground probability sequence output in step S3 for supervision, to constrain the generation quality of foreground cues. The loss function is:
[0070] Secondly, the final segmentation mask output in step S4 is subjected to supervision by a weighted combination of focus loss and DSC loss, resulting in the following segmentation loss:
[0071] And with the joint objective function Optimize network parameters.
[0072] in This represents the binary cross-entropy loss that supervises the foreground probability sequence output from step S3. This indicates the number of sequence elements used for supervision. Indicates the first Sum the elements of each sequence and take the mean. Indicates the relationship with the first The downsampled ground truth labels corresponding to each sequence element. Indicates the first Forecasting of sequence elements, This represents the Sigmoid function. Represents logarithmic operations; This represents the segmentation loss used to monitor the final segmentation mask. These are the controllable weighting coefficients for the two losses. For weighted focus loss, For DSC loss, This is the joint objective function used for training optimization.
[0073] Step S5.4: Iteratively update the trainable parameters using stochastic gradient descent (SGD) to minimize the joint objective function. To achieve the training objective, the PoSAM network was trained, and inference was performed on the test set to obtain the lesion region segmentation results.
[0074] The skin lesion dataset was expanded using image rotation, and the image size was standardized to 512×512 pixels. The dataset was then divided into training and validation sets in an 8:2 ratio. Input images were divided into 16 patch blocks and fed into a large model encoder infused with a polarity fine-tuning strategy for feature extraction. Simultaneously, the Canny multi-scale prompting module downsampled the decoupled feature maps to obtain multi-scale semantic information with sizes of 256×H×W, 64×2H×2W, and 16×4H×4W, respectively. Figure 1 As shown; when performing feature fusion, a polarity fine-tuning module is designed, such as... Figure 2 As shown, the polarity fine-tuning strategy is injected into the attention and MLP respectively to decouple the features; in the multi-scale cue stream, according to Figure 3 The method on the left first performs channel compression on feature maps of different scales, then fuses them with the Canny boundary map, and then uses... Figure 4 The decoder module decodes image features, sparse features, and dense features to obtain the final predicted image. During model training, a mini-batch stochastic gradient descent method is used to optimize the model. Each trial consists of 200 iterations, with a training batch size of 4, a learning rate of 6e−4, and a momentum parameter of 0.9. After these 200 iterations, the PoSAM model is evaluated using three segmentation metrics: DSC similarity coefficient, mean intersection-over-union ratio (MIU), and accuracy. The model achieves 91.1%, 84.9%, and 94.1% accuracy on the entire skin lesion dataset, respectively. The optimal model is then used to segment skin lesion images, and some segmentation results are shown below. Figure 6 As shown in the image.
[0075] Example 2: This example proposes a medical image segmentation system based on a polarity fine-tuning large model transfer method. The system includes: The medical image dataset module is configured to perform the following processes to augment and crop the medical image dataset; The pre-trained large model skeleton weight module is frozen and configured to perform the following process: under the condition of freezing the pre-trained large model skeleton weight, a polarity fine-tuning strategy is designed and injected into the attention layer and feedforward layer of the Transformer block. By introducing positive and negative dual-path low-rank branches, the features are decoupled and modified, and polarity-enhanced features are extracted without destroying the original representation structure. The Multi-Scale Mask Guidance and Prompt Module (MSGP) is configured to perform the following process: MSGP generates a self-prompt flow, adopts a step-by-step convergence and restoration strategy for the extracted feature maps, integrates Canny edge detection maps and multi-scale semantic features, and automatically generates and filters high-quality foreground region mask prompts (dense and sparse tokens) to replace manual point box annotation. The polar decoder PoDecoder module is constructed and configured to perform the following process: feature interaction and recovery on the polar decoder PoDecoder module; self-attention and cross-attention interaction between the output image features and the generated cue tokens using a bidirectional Transformer architecture to guide the model to focus on key parts; and feature recovery to the input size through deconvolution sequence upsampling to generate the final segmentation mask. The PoSAM network model is configured to perform the following process: using a joint loss function, including foreground binary cross-entropy loss, weighted focus loss, and DSC loss, the model is trained to achieve accurate segmentation and boundary definition of skin lesion areas.
[0076] Example 3: This example proposes an electronic system, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method steps of the present invention.
[0077] Example 4: This example proposes a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements the steps of the method described in this invention, which will not be repeated here.
[0078] Example 5: This example proposes a computer program product, including a computer program / instructions. When the computer program / instructions are executed by a processor, they implement the steps of the method described in this invention, which will not be repeated here.
[0079] It should be noted that the processing flow of embodiments 2-5 corresponds to the specific steps of the method provided in embodiment 1 of the present invention, and has the corresponding functional modules and beneficial effects of the method. Technical details not described in detail in this embodiment can be found in the method provided in embodiment 1 of the present invention.
[0080] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.
[0081] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A medical image segmentation method based on polarity fine-tuning large model transfer, characterized in that, Includes the following steps: S1: Expand and scale the medical image dataset, which contains N pairs of original color images and corresponding segmentation mask images; rotate and enhance the image clockwise with the center point of each image as the rotation center, and save the enhanced image with a step size of 20°, so that the dataset is expanded to 18×N pairs of original images and corresponding segmentation masks. The augmented original image and its corresponding segmentation mask are then scaled / cropped to 512×512 resolution to obtain standardized sample pairs for training. S2: Under the condition of freezing the weights of the backbone network of the pre-trained large model, the polarity fine-tuning strategy is injected into the attention layer and feedforward layer of the Transformer block; the features are decoupled and corrected by introducing a dual-path low-rank parameterized structure of positive and negative branches, and the polarity-enhanced image features are extracted without destroying the original feature expression structure, so as to obtain the feature map for subsequent segmentation. S3: Construct a multi-scale mask guidance prompt module MSGP, which aggregates and restores the feature map output from step S2 step by step, and integrates the Canny edge detection map and multi-scale semantic features to automatically generate mask prompts that match the target foreground region and dense and sparse tokens used to supplement the constraints of key parts, so as to replace the prompt information generated by manual point / box annotation; S4: Construct a polar decoder PoDecoder for feature interaction and spatial detail restoration; use a bidirectional Transformer architecture to perform self-attention and cross-attention interaction on the image features in step S2 and the cue tokens generated in step S3 to guide the model to focus on key segmentation regions; and restore the features to the input size step by step through deconvolution sequences or upsampling restoration modules to output the final segmentation mask. S5: Combine steps S2, S3, and S4 to construct the network model PoSAM, and train the model using a joint loss function, which includes foreground binary cross-entropy loss, weighted focus loss, and DSC loss; after training, accurate segmentation and boundary definition of skin lesion areas are achieved.
2. The medical image segmentation method based on polarity fine-tuning large model transfer method according to claim 1, characterized in that, S2 includes the following steps: S2.1: For the image encoder of an arbitrary segmentation model SAM, the input skin lesion image is denoted as... ,in For image height, For image width, The number of channels; the image The image is divided into several non-overlapping blocks, and the side length of each block is denoted as . , and take The number of image patches in the vertical and horizontal directions are respectively , The total number of tokens is , No. Each image patch is flattened into a vector. Through linear mapping matrix Projected onto latent feature dimensions Simultaneously, learnable position embeddings are added. to the initial token ; All The initial feature sequence is obtained by splicing the components in a fixed order. This serves as the input for subsequent Transformer encoding. in, This indicates the input of a medical image. , , These are the image height, width, and number of channels, respectively. Let the side length of the image patch be taken as... , , These represent the number of image blocks in the vertical and horizontal directions, respectively. The total number of tokens; For the first Image patch flattening vector, It is a linear projection matrix. For latent feature dimensions, For learnable location embedding, For the corresponding token, For the initial token sequence; S2.2: Construct a Transformer multilayer encoder containing a polar low-rank adapter (PoLoRA), keeping the pre-trained weights of the SAM frozen, updating the trainable parameters introduced by PoLoRA, and inserting a dual-path polar low-rank branch in parallel into the linear mapping of the attention layer and the linear mapping of the feedforward layer in each Transformer block. The attention computation satisfies: ； The input feature vector for any token is denoted as . The positive / negative polarity components and the power function are defined as follows: ； Freeze the pre-trained weights of the linear layer Under these conditions, a dual-path polarity low-rank increment is introduced to obtain a polarity fine-tuning output. The calculation is as follows: ； In the attention layer, the query, key, and value are obtained from the PoLoRA output of the corresponding projection layer: ； in For sequence The matrix output is obtained by calculating the feature vector of each token according to equation (3) and stacking them in order. The output of each attention head is: ； The multi-head attention output is: ； The output of the feedforward network is: ； in Token-level input feature vector, , For linear layer input / output dimensions, For linear layer output, and These are the positive and negative polarity components, respectively. It is a zero vector. It is an element-wise exponentiation function. This indicates element-wise exponentiation. For exponential hyperparameters; For the frozen pre-trained weights, , , These are the frozen weights for the query, key, and value projection layers in the attention layer, respectively. Freeze weights in the projection layer for multi-head attention output. , The frozen weights are the two linear mappings of the feedforward network. It is a low-rank number. , These are positive and negative polarity gating matrices, respectively. , These are low-rank branch matrices with positive and negative polarities, respectively. , These are low-rank branch matrices with positive and negative polarities, respectively; , , These are the query, key, and value matrices for the attention mechanism. , , Indicates the first The corresponding blocks for each attention head. For single-head dimension, For transpose, Linearized Softmax operation, For the first Each attention head output, For the number of attention heads, This indicates a splicing operation. For multi-head attention output, For layer normalization, For activation function, This is the output of the feedforward network.
3. The medical image segmentation method based on polarity fine-tuning large model transfer method according to claim 1, characterized in that, The specific steps of S3 are as follows: S3.1: To construct the multi-scale mask guidance and prompting module, the feature map output from step S2 is used as input, denoted as... Perform multi-scale semantic processing on it; set a scale index. Corresponding scaling factor For each scale branch, first adjust the spatial resolution to Then through Convolution with channel compression to , obtained the Semantic feature maps at various scales The calculation is as follows: ； in, The feature map output by step S2, Its number of channels, and Its spatial dimensions; For scale indexing, is a scale factor and ; This means "upsampling the input feature map to..." and through Convolution compresses the number of channels to The operation of ""; S3.2: To suppress redundant textures and low-level irrelevant semantics, semantic feature maps at each scale are processed. Further compression was performed, using... The convolutional kernel maps it to a single-channel response, and then a single-channel cue map is obtained by sigmoid activation. The calculation is as follows: ； For the first Semantic feature maps at various scales for Convolution kernel, This represents a two-dimensional convolution operation. For the Sigmoid activation function, This is a single-channel prompt image; S3.3: To achieve cross-scale alignment to the maximum resolution, the single-channel cue maps at each scale are uniformly aligned to the maximum resolution. The aligned tooltip is obtained by performing an upsampling operator on the scaled tooltip. The calculation is as follows: ； in This indicates that the feature map is scaled up. Upsampling upsampling operator, This is the aligned prompt image; S3.4: Calculate and size-align the Canny edge map based on the input image. ,Will , , and The components are stitched together according to channel dimensions to form the final fused feature. The prompt token used for subsequent step S4 for prompt token construction and segmentation guidance is calculated as follows: ； in Use a Canny edge map aligned with the maximum resolution; This indicates a splicing operation. This is the feature map after fusing multi-scale cueing and edge constraints.
4. The medical image segmentation method based on polarity fine-tuning large model transfer method according to claim 1, characterized in that, The specific steps of S4 are as follows: S4.1: Fuse the encoded feature map output from S2 with the mask cue embedding output from S3 to obtain the initial pixel sequence embedding. The calculation is as follows: ； Where, represents the image coding feature map output by S2, This indicates dense embedding of the mask hints output by S3. This indicates a point-by-point addition operation. This represents the embedding of the initial image pixel sequence after fusion; In the bidirectional Transformer, self-attention modeling is performed on both the cue side and the image side, and bidirectional information interaction is achieved through cross-attention. Specifically, the sparse cue is first embedded and self-attention is applied to obtain the query semantics. , and then For the query, cross-attention is performed using image sequences as keys, and the updated semantic hints are obtained through a feedforward network. : ； in Sparse embedding of mask hints This indicates a self-attention operation. Indicates For query, For key, Cross-attention operation for values, , , These are the query matrix, key matrix, and value matrix, respectively. This represents a feedforward neural network. This represents the semantic features of the prompts after being updated by self-attention and cross-attention; Subsequently, the updated prompt semantics were used. Using the image-side intermediate semantics as the key, cross-attention is performed again to obtain the interacted image sequence. : ； in Represents the image sequence features after two-way interaction; S4.2: The interactive image sequence Dimension reduction is performed to obtain the feature map. The spatial resolution is restored step by step through deconvolution sequences to obtain the restored high-resolution features. : ； ； This represents the dimensionality transformation operation that restores sequence features to a two-dimensional feature map. This represents the decoded feature map after reconstruction. This represents a stepwise upsampling recovery operation consisting of interpolation upsampling and convolution. This represents a high-resolution feature map restored to the target spatial resolution. S4.3: The high-resolution features obtained in S4.2 Input into the prediction head and combine with the cue semantics of S4.
1. Pixel-by-pixel fusion to obtain the final segmentation mask : ； in This represents the prediction head of a multilayer perceptron. This indicates element-wise multiplication. This represents the final output binary segmentation mask for the lesion region.
5. The medical image segmentation method based on polarity fine-tuning large model transfer method according to claim 1, characterized in that, The specific steps of S5 are as follows: S5.1: Divide the skin lesion image dataset into training and testing sets, and input them into the constructed PoSAM network model; S5.2: Combine the polarity fine-tuning strategy encoder module of S2, the multi-scale mask guidance prompt module of S3, and the polarity decoder of S4 to construct the PoSAM network, so that the model forms a closed-loop processing structure of "encoding - self-generated prompting - interaction - decoding". S5.3: End-to-end training of the model using a joint loss function: First, a binary cross-entropy loss is introduced into the foreground probability sequence output by S3 for supervision, to constrain the generation quality of foreground cues. The loss function is: ； Secondly, the final segmentation mask output by S4 is supervised by a weighted combination of focus loss and DSC loss, resulting in the segmentation loss as follows: ； And with the joint objective function Optimize network parameters; in This represents the binary cross-entropy loss supervised by the foreground probability sequence output by S3. Indicates the number of sequence elements used for supervision. Indicates the first Sum the elements of each sequence and take the mean. Indicates the relationship with the first The downsampled ground truth labels corresponding to each sequence element. Indicates the first Forecasting of sequence elements, This represents the Sigmoid function. Represents logarithmic operations; This represents the segmentation loss used to monitor the final segmentation mask. These are the controllable weighting coefficients for the two losses. For weighted focus loss, For DSC loss, This is the joint objective function used for training optimization; S5.4: Iteratively update the trainable parameters using stochastic gradient descent (SGD) to minimize the joint objective function. To achieve the training objective, the PoSAM network was trained, and inference was performed on the test set to obtain the lesion region segmentation results.
6. A medical image segmentation system based on a polarity fine-tuning large model transfer method, characterized in that, The system includes: The medical image dataset module is configured to perform the following processes to augment and crop the medical image dataset; The pre-trained large model skeleton weight module is frozen and configured to perform the following process: under the condition of freezing the pre-trained large model skeleton weight, a polarity fine-tuning strategy is designed and injected into the attention layer and feedforward layer of the Transformer block. By introducing positive and negative dual-path low-rank branches, the features are decoupled and modified, and polarity-enhanced features are extracted without destroying the original representation structure. The Multi-Scale Mask Guidance and Prompt Module (MSGP) is configured to perform the following process: MSGP generates a self-prompt flow, adopts a step-by-step convergence and restoration strategy for the extracted feature maps, integrates Canny edge detection maps and multi-scale semantic features, and automatically generates and filters high-quality foreground region mask prompts (dense and sparse tokens) to replace manual point box annotation. The polar decoder PoDecoder module is constructed and configured to perform the following process: feature interaction and recovery on the polar decoder PoDecoder module; self-attention and cross-attention interaction between the output image features and the generated cue tokens using a bidirectional Transformer architecture to guide the model to focus on key parts; and feature recovery to the input size through deconvolution sequence upsampling to generate the final segmentation mask. The PoSAM network model is configured to perform the following process: using a joint loss function, including foreground binary cross-entropy loss, weighted focus loss, and DSC loss, the model is trained to achieve accurate segmentation and boundary definition of skin lesion areas.
7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the computer program is executed, it implements the steps of the method as described in any one of claims 1 to 5.
8. A computer-readable storage medium having a computer program stored thereon, characterized in that, The computer program is configured to implement the steps of the method according to any one of claims 1 to 5 when invoked by a processor.
9. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 5.