An image segmentation method, system, computer device and storage medium

Abstract

The application relates to the technical field of computer vision, and discloses an image segmentation method, system, computer device and storage medium. The method comprises the following steps: extracting features through a local structure branch and a global semantic branch, and enhancing high-frequency details by using a boundary detail perception branch; guiding and aligning double-branch features through a cross-attention mechanism by using a shared learnable semantic query vector, obtaining consistent fusion features by fusing the double-branch features based on channel and spatial dimension joint weights; generating an edge intensity map and an edge perception weight based on gradient calculation, obtaining a channel-level global description and learning a global dependency relationship through global pooling, and modulating the consistent fusion features based on the edge perception weight and the global dependency relationship to output a segmentation result. The application realizes cross-branch semantic alignment and decoding stage structure constraint, and improves the segmentation precision and boundary integrity in a complex scene.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, specifically to an image segmentation method, system, computer device, and storage medium. Background Technology

[0002] In fields such as medical image analysis and natural image processing, image segmentation is a crucial preliminary step for achieving accurate diagnosis and intelligent understanding. With the development of deep learning technology, segmentation models based on encoder-decoder architectures have become mainstream. To balance local details with global contextual information, existing techniques often employ a dual-branch architecture combining convolutional neural networks and Transformers for feature extraction and fusion.

[0003] However, in practical applications, due to significant differences in receptive field, feature distribution, and semantic response patterns between local structural branches and global semantic branches, direct fusion often leads to semantic bias and information conflict between heterogeneous features, making it difficult to maintain consistency in the representation of the target region across different network stages. Furthermore, existing methods in the decoding and recovery stage often focus on simple superposition of semantic features or conventional attention enhancement, lacking explicit structural consistency constraints. This results in problems such as discontinuous target region boundaries, loss of local details, and missegmentation when facing complex scenes with blurred boundaries, significant scale changes, or artifact interference, severely impacting the accuracy and robustness of the segmentation results.

[0004] Therefore, there is an urgent need for an image segmentation scheme that can maintain the consistency of the target region representation structure throughout the feature fusion and spatial reconstruction process. Summary of the Invention

[0005] To address the issues of semantic bias due to heterogeneous features and insufficient structural constraints in the decoding stage that lead to discontinuous segmentation boundaries in existing technologies, this application proposes an image segmentation method and system that achieves structural consistency of target region representation throughout the entire process of cross-branch fusion and spatial reconstruction, thereby improving segmentation accuracy and boundary integrity in complex scenarios.

[0006] Firstly, an image segmentation method is provided, the method comprising: The image to be segmented is acquired, and multi-scale feature extraction is performed on the image to be segmented through local structure branch and global semantic branch to obtain local structure features and global semantic features respectively; wherein, the local structure branch captures feature representations of different receptive fields through multiple parallel convolution branches, and extracts gradient edge information using edge detection operators through boundary detail awareness branch to enhance high-frequency details. By utilizing shared learnable semantic query vectors, the local structural features and global semantic features are guided and aligned through a cross-attention mechanism to obtain local enhanced features and global enhanced features, respectively. Based on the local enhanced features and global enhanced features, the joint weights of the two branches are determined from the channel dimension and the spatial dimension, respectively, and the joint weights are used for weighted fusion to obtain consistent fusion features. An edge intensity map is generated based on the gradient calculation of the feature map in the horizontal and vertical directions, and an edge-aware weight is generated based on the edge intensity map. A channel-level global description is obtained by aggregating spatial information through global pooling, and the global dependency relationship between channels is learned based on the channel-level global description. The structural consistency modulation of the consistency fusion feature is performed using the edge-aware weight and the global dependency relationship, and the segmentation result of the image to be segmented is output.

[0007] The above scheme constructs a progressive processing link of "feature extraction - cross-branch alignment and fusion - structural consistency reconstruction", uses a shared semantic query vector as a unified anchor to eliminate semantic deviations between heterogeneous branches, and combines edge gradient information and global context to perform collaborative modulation of decoded features, thereby ensuring that the target region maintains structural continuity and regional integrity throughout the entire segmentation process.

[0008] In one embodiment, the step of guiding the alignment of the local structural features and the global semantic features through a cross-attention mechanism to obtain local enhanced features and global enhanced features respectively includes: The shared learnable semantic query vector is projected into a query matrix, and the local structural features and the global semantic features are flattened and projected into the corresponding key matrix and value matrix, respectively. Attention weights are calculated based on the query matrix and the key matrix, and the value matrix is ​​aggregated using the attention weights to obtain the local enhancement features and the global enhancement features, respectively.

[0009] In one embodiment, the shared learnable semantic query vectors include a preset number of query vectors and are initialized using a standard normal distribution; The aggregation of the value matrix using the attention weights to obtain local enhancement features and global enhancement features includes: The query aggregation features are obtained by averaging the attention aggregation outputs corresponding to each query vector. The query aggregate features are projected onto the feature space through linear mapping, and the spatial dimension is restored through expansion operations. Local modeling is then performed using convolutional layers to obtain enhanced features. The enhanced features are fused with the original branch features through residual connections to obtain the local enhanced features or the global enhanced features; The original branch feature is either the local structural feature or the global semantic feature.

[0010] In one embodiment, the step of determining the joint weights of the two branches from the channel dimension and the spatial dimension respectively based on the local enhancement features and the global enhancement features, and then using the joint weights to perform weighted fusion to obtain consistent fusion features includes: Global average pooling and global max pooling are performed on the local enhancement features and the global enhancement features respectively, and the resulting description vectors are concatenated to aggregate spatial information; The aggregated spatial features are passed to two one-dimensional convolutions to obtain the channel weights of two sets of branch features, and the joint channel weights are obtained by stacking and Softmax aggregation. The spatial joint weights are obtained by processing them in the same way in the spatial dimension. The joint channel weights are multiplied by the local enhancement features, and the joint spatial weights are multiplied by the global enhancement features and then summed to obtain the collaborative fusion features; The consistent fusion features are obtained by locally reconstructing and aligning the collaborative fusion features through convolution, batch normalization, and nonlinear activation.

[0011] In one embodiment, the step of generating an edge intensity map based on the gradients of the feature map in the horizontal and vertical directions, and generating edge-aware weights based on the edge intensity map, includes: The gradients of the feature map in the horizontal and vertical directions are calculated, and their absolute values ​​are summed to obtain the edge intensity map. The edge intensity map is compressed along the channel dimension to obtain single-channel edge features, and the channel dimension mean operation is performed on the decoded input features to obtain spatial average features. The single-channel edge features are concatenated with the spatial average features, and spatial attention weights are generated through convolutional layers and activation functions. The spatial attention weights are used to multiply the decoded input features element-wise to obtain edge-aware features; The decoding input feature is either the consistency fusion feature or the upsampling feature during the decoding process.

[0012] In one embodiment, the step of using the edge-aware weights and the global dependency to perform structural consistency modulation on the consistency fusion features and outputting the segmentation result of the image to be segmented includes: The edge-aware features are subjected to global average pooling, and spatial information is aggregated to obtain the channel-level global description; The channel-level global description is reduced in dimensionality by convolution and layer normalization, and then increased in dimensionality by nonlinear activation and convolution. The global dependencies between channels are learned to obtain the increased-dimensional global description. The upgraded global description and the edge-aware features are multiplied and added respectively to obtain the output features after structural consistency modulation; The spatial resolution is restored based on the output features, and the segmentation result is generated.

[0013] In one embodiment, the local structure branch captures feature representations of different receptive fields through multiple parallel convolutional branches, and extracts gradient edge information using an edge detection operator through a boundary detail-aware branch to enhance high-frequency details, including: In the local structure branch, feature representations of different receptive fields are captured and spliced ​​and fused through the multiple parallel convolution branches to obtain multi-scale fused features; The boundary detail perception branch uses edge detection operators to extract gradient edge information and combines it with local convolution to extract texture features, thereby obtaining high-frequency detail features. The local structural features are obtained by aggregating the multi-scale fusion features with the high-frequency detail features.

[0014] Secondly, an image segmentation system is provided, the system comprising: A multi-scale feature extraction module is used to acquire the image to be segmented. It performs multi-scale feature extraction on the image to be segmented through a local structure branch and a global semantic branch to obtain local structure features and global semantic features, respectively. The local structure branch captures feature representations of different receptive fields through multiple parallel convolution branches, and extracts gradient edge information using an edge detection operator through a boundary detail awareness branch to enhance high-frequency details. The feature alignment and fusion module is used to guide the alignment of the local structural features and the global semantic features through a cross-attention mechanism using a shared learnable semantic query vector, thereby obtaining local enhanced features and global enhanced features respectively; based on the local enhanced features and the global enhanced features, the joint weights of the two branches are determined from the channel dimension and the spatial dimension respectively, and the joint weights are used to perform weighted fusion to obtain consistent fusion features; The consistency modulation output module is used to generate an edge intensity map based on the gradients of the feature map in the horizontal and vertical directions, and generate edge-aware weights based on the edge intensity map; obtain a channel-level global description by aggregating spatial information through global pooling, and learn the global dependencies between channels based on the channel-level global description; perform structural consistency modulation on the consistency fusion features using the edge-aware weights and the global dependencies, and output the segmentation result of the image to be segmented.

[0015] Thirdly, a computer device is provided, including a memory and a processor, the memory being communicatively connected to the processor, and the memory storing a computer program executable on the processor, wherein the processor, when executing the computer program, implements the image segmentation method as described above.

[0016] Fourthly, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the image segmentation method described above.

[0017] The aforementioned image segmentation methods, systems, computer devices, and storage media introduce a shared, learnable semantic query vector as a unified semantic anchor point. This vector guides and aligns the features of both local structural branches and global semantic branches through cross-attention. This mechanism breaks down information barriers between heterogeneous branches, enabling local texture features and global contextual features to respond collaboratively around the same target region. This eliminates feature fusion conflicts caused by differences in receptive field and semantic level, improving the discriminative power and consistency of the fused features. Secondly, in the spatial structure restoration stage, explicit edge-aware weights are generated using gradient information from the feature maps to impose refined boundary constraints on the decoded features. Simultaneously, global contextual information is combined to enhance the semantic coherence within the region. This dual-constraint mechanism solves the problems of boundary breaks and missegmentation in complex interference scenarios encountered by traditional methods, ensuring high stability of the segmentation results in terms of contour continuity and regional integrity. Finally, by combining structure perception with multi-scale receptive field expansion and explicit boundary detail perception branches, the network's ability to capture high-frequency details and weak boundaries is enhanced. Combined with directional structure attention and random depth regularization strategies, the robustness and generalization ability of the model under complex conditions such as hair occlusion, low contrast and artifact interference are further improved, providing reliable technical support for high-precision image segmentation. Attached Figure Description

[0018] Figure 1 This is a segmentation diagram of a skin lesion. Figure 2 This is a flowchart of the overall algorithm for image segmentation. Figure 3 Diagram of DAMS module; Figure 4LGMF module diagram; Figure 5 LQT module diagram; Figure 6 EGA module diagram; Figure 7 Heatmap of characteristic responses before and after querying dual-branch shared lesions; Figure 8 Visualization comparison chart of the ISIC2018 dataset; Figure 9 Visualization comparison chart of the ISIC2017 dataset; Figure 10 A visualization comparison chart for the PH2 dataset. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.

[0021] Computer vision, as we know it, utilizes the combination of computers and image sensors to achieve the "observation" and "analysis" functions of the human eye. As an important branch of computer vision, medical image segmentation has very wide applications in today's smart healthcare era. Skin lesion segmentation, as a key technology within this field, has the core task of accurately and automatically delineating the boundaries of lesion areas from dermoscopic images, thus providing a basis for subsequent qualitative analysis. However, skin lesion segmentation is an extremely challenging task. In actual clinical scenarios, lesion areas often exhibit diverse shapes, varying scales, blurred boundaries, and low contrast with normal skin. Therefore, accurate and robust skin lesion segmentation technology is a key technology and major research content in computer-aided diagnostic systems, intelligent quality control in dermatology clinics, and digital medical decision-making systems. The effectiveness of skin lesion image segmentation is crucial. Figure 1 As shown.

[0022] In fields such as medical image analysis and natural image processing, image segmentation is a crucial preliminary step for achieving accurate diagnosis and intelligent understanding. With the continuous development of deep learning technology, segmentation models based on encoder-decoder architectures have become the mainstream paradigm. To balance local details and global contextual information, existing techniques often employ a dual-branch architecture combining Convolutional Neural Networks (CNNs) and Transformers for feature extraction and fusion. However, in practical applications, due to significant differences in receptive fields, feature distributions, and semantic response patterns between local structural branches and global semantic branches, direct fusion often leads to semantic biases and information conflicts between heterogeneous features, making it difficult to maintain consistency in the representation of the target region across different network stages. Furthermore, existing methods in the decoding and recovery stage often focus on simple superposition of semantic features or conventional attention enhancement, lacking explicit structural consistency constraints. This results in problems such as discontinuous target region boundaries, loss of local details, and missegmentation when facing complex scenes with blurred boundaries, significant scale changes, or artifact interference, severely impacting the accuracy and robustness of the segmentation results.

[0023] In the specific scenario of skin lesion image segmentation, the above problems are particularly prominent: dermoscopic images often suffer from complex interferences such as blurred lesion boundaries, hair occlusion, bubble artifacts, and uneven illumination. Existing segmentation methods typically perform independent optimizations at the stages of feature extraction, cross-branch fusion, and decoding reconstruction, lacking a unified modeling of the continuous evolution of lesion representation throughout the entire segmentation process. When lesion representations from different branches and scales are not consistently aligned before fusion, semantic response deviations easily arise between local structural information and global semantic information, leading to the simultaneous amplification of boundary responses and artifacts in the fused features. Furthermore, if explicit structural constraints are lacking in the decoding stage, the continuity of lesion contours during upsampling is difficult to maintain, resulting in segmentation results prone to boundary breaks, loss of local details, and missegmentation of regions. To address these issues, there is an urgent need for an image segmentation scheme that can maintain the structural consistency of the target region representation throughout the feature fusion and spatial reconstruction process. This embodiment provides an image segmentation method.

[0024] like Figure 2As shown, this application proposes a skin lesion image segmentation method for progressive lesion representation structure consistency modeling. The aim is to learn the mapping from local structural representation and global semantic representation to lesion masks, unifying this process into a progressive modeling process of lesion representation consistency across three sequentially dependent stages. The first stage improves the reliability of local structural representations in CNN branches through a Detail-aware Multi-scale Module (DAMS), providing boundary priors for cross-branch interactions. The second stage uses a Local-Global Merger Fusion Module (LGMF) to align the CNN local representation with the Transformer global representation using shared lesion responses as a reference, generating co-fused features. The third stage uses an Edge-guided Global Attention Module (EGA) to constrain the structural propagation of co-fused features during the decoding and reconstruction process, ensuring consistency between local edges and global semantic regions in the prediction space. See also... Figure 2 The overall flowchart of the algorithm shows that the information in the network goes through three states sequentially: the initial CNN features contain rich local details, but are easily affected by hair, low contrast, and blurred boundaries; the Transformer features have strong global semantic modeling capabilities, but lack fine spatial localization. DAMS first transforms the noisy local representation into a structurally reliable representation; LGMF further establishes lesion-related collaboration between this local representation and global semantics to form a lesion-consistent fused representation; finally, EGA maps the coordinated representation back to the prediction space under the joint constraints of edges and context, thereby maintaining the integrity of contours and regions. DAMS, LGMF, and EGA sequentially complete the progressive processing chain of "reliable representation generation - cross-branch representation alignment and fusion - structurally consistent and stable reconstruction".

[0025] The image segmentation method provided in this embodiment includes the following steps: The image to be segmented is acquired, and multi-scale feature extraction is performed on the image to be segmented through local structure branch and global semantic branch to obtain local structure features and global semantic features respectively. Among them, the local structure branch captures the feature representation of different receptive fields through multiple parallel convolution branches, and the boundary detail awareness branch uses edge detection operators to extract gradient edge information to enhance high-frequency details.

[0026] By utilizing shared learnable semantic query vectors, local structural features and global semantic features are guided and aligned through a cross-attention mechanism to obtain local and global enhanced features respectively. Based on the local and global enhanced features, the joint weights of the two branches are determined from the channel and spatial dimensions respectively, and the joint weights are used for weighted fusion to obtain consistent fusion features.

[0027] Edge intensity maps are generated based on gradient calculations in the horizontal and vertical directions of the feature maps, and edge-aware weights are generated based on the edge intensity maps. Spatial information is aggregated through global pooling to obtain channel-level global descriptions, and global dependencies between channels are learned based on the channel-level global descriptions. Structural consistency modulation is performed on the consistency fusion features using edge-aware weights and global dependencies, and the segmentation result of the image to be segmented is output.

[0028] Specifically, after acquiring the image to be segmented, it is first input into a dual-branch encoder structure. The local structure branch adopts a CNN architecture, composed of multiple DAMS modules and downsampling operations, used to extract local structural features containing rich edge texture details. The global semantic branch adopts a Swing Transformer architecture, extracting global semantic features with strong global context modeling capabilities through a hierarchical window attention mechanism. The two branches output local structural features at the same scale. and global semantic features Where C represents the number of channels, and H and W represent the height and width of the feature map, respectively.

[0029] After obtaining the features from both branches, the process proceeds to the cross-branch representation alignment and fusion stage. The core of this stage lies in utilizing a shared, learnable semantic query vector to enhance the lesion perception of both local structural features and global semantic features through a cross-attention mechanism, ensuring that the features from both branches produce consistent responses around the lesion region. Specifically, the shared query vector serves as a unified semantic anchor point, and cross-attention calculation is performed with the features from both the CNN branch and the Transformer branch to obtain locally enhanced features. and global enhancement features Then, the relative importance weights of the two branch features are calculated from both the channel and spatial dimensions, and these weights are used to... and We perform weighted fusion to obtain the consistent fusion feature G.

[0030] Finally, in the structural consistency reconstruction stage, the consistency fusion features are input into the stepwise decoder. During the decoding upsampling process, each EGA module explicitly models the local structural response of the lesion contour region through an edge-aware branch: calculating the gradients of the feature map in the horizontal and vertical directions, taking the absolute values ​​and summing them to obtain the edge intensity map; generating spatial attention weights based on the edge intensity map to enhance the edge guidance of the decoded features. Simultaneously, global average pooling (GAP) is used to aggregate spatial information to obtain a channel-level global description, learn the global dependencies between channels, and enhance the semantic consistency within the lesion region. Edge awareness and global context collaboratively constrain the decoding process, ensuring that the recovered lesion contour maintains continuity, integrity, and structural consistency.

[0031] It should be noted that the method for obtaining global semantic information is not limited to the Swin Transformer; it can also be replaced by layered visual Transformers (such as PVT, Pyramid Vision Transformer), pyramidal transformers, or Mamba, which are network structures with global modeling capabilities. The dilated convolutions in the DAMS module can be replaced with ordinary convolutions, depthwise separating convolutions, or deformable convolutions, and the branch dilation rate can be taken in different increasing combinations. Edge detection methods are not limited to the Sobel operator and finite difference computation; the Scharr operator or learnable gradient convolution kernels can also be used instead.

[0032] Based on the above steps, this embodiment constructs a progressive processing chain of "reliable representation generation - cross-branch representation alignment and fusion - structural consistency reconstruction", which enables the lesion representation to gradually evolve from local details into a consistent representation with unified semantics and complete structural constraints, thereby realizing the continuous transmission and gradual optimization of lesion structural information at different network stages.

[0033] In summary, the core concept of this application lies in: introducing a shared, learnable semantic query vector as a unified semantic anchor point to guide and align the features of local structural branches and global semantic branches through cross-attention, thereby eliminating semantic bias and information conflict between heterogeneous features; based on this, in the spatial structure restoration stage, gradient information is used to generate explicit edge-aware weights and combined with global context information for dual constraints, ensuring high stability of the segmentation results in terms of contour continuity and region integrity; simultaneously, through multi-scale receptive field expansion and explicit boundary detail-aware branches in the structure-aware representation learning stage, the network's ability to capture high-frequency details and weak boundaries is strengthened. These three stages are interconnected and support each other, jointly achieving the maintenance of "consistency of target region representation structure throughout the feature fusion and spatial reconstruction process," solving the problems of boundary breakage and region missegmentation in complex interference scenarios in traditional methods.

[0034] Specifically, in the feature extraction stage, reliable local structural priors are formed through detail-aware multi-scale modeling and boundary detail enhancement; in the cross-branch fusion stage, lesion consistency alignment across heterogeneous branches is achieved through shared query vectors, avoiding information conflicts caused by direct fusion; in the decoding and reconstruction stage, the continuity of lesion contours and the integrity of regional structures are maintained through edge constraints and global context collaborative modulation. These three stages are interconnected and mutually supportive, jointly improving the accuracy and stability of lesion segmentation results in complex skin lesion scenarios.

[0035] In cross-branch feature fusion, local structural features from CNN branches and global semantic features from Transformer branches naturally exhibit inconsistencies in feature distribution and semantic representation due to differences in network structure. If the features from the two branches are directly concatenated or weighted, lesion-related responses and background noise responses will be simultaneously mixed, leading to a decrease in the feature focusing ability of the lesion region in the fused features. Therefore, a cross-branch semantic alignment mechanism needs to be established before fusion to enhance the responses of the features from both branches under a unified lesion semantic reference. To address the above problem, in one embodiment, a cross-attention mechanism is used to guide the alignment of local structural features and global semantic features respectively, obtaining locally enhanced features and globally enhanced features, including: The shared learnable semantic query vectors are projected into a query matrix, and the local structural features and global semantic features are flattened and projected into the corresponding key matrix and value matrix, respectively. Attention weights are calculated based on the query matrix and key matrix. The value matrix is ​​then aggregated using these attention weights to obtain local and global enhancement features, respectively.

[0036] like Figure 5 As shown, the Lesion Query Transformer (LQT) module provided in this embodiment introduces a set of learnable lesion query vectors. Simultaneously apply lesion-detection guidance to the dual-branch features of CNN and Swin Transformer. See also Figure 5 The LQT module diagram shows that the query vector serves as a learnable parameter of the network, continuously updated through backpropagation during training. The query vector follows a standard normal distribution. Initialization is performed in a way that breaks the initial symmetry of parameters and ensures the diversity of initial representations, while also smoothing gradient propagation and improving the model's ability to model different lesion structures and the stability of training convergence. This can be viewed as a semantic probe, dynamically learning the relevant semantic patterns of lesions during training. Considering the characteristics of dermoscopic images—a small number of lesions per image, often dominated by a single main lesion, and the task being binary classification with sparse target distribution—this embodiment sets the query count to 8 to achieve a balance between lesion semantic coverage and computational complexity. It should be noted that the query count is not fixed at 8 and can be selected within the range of 1 to 32 according to actual task requirements. Sharing the lesion query vector allows the local structural features of the CNN and the global semantic features of the Transformer to form a unified representation, strengthening the collaborative perception capability of the two branches for lesion regions.

[0037] The features from the CNN encoder are represented as The feature representation from the Swin Transformer encoder is as follows X1 provides local details for lesion detection, while X2 provides global contextual information. First, X1 is flattened into a sequence X∈R^{N×C}, where N=H×W, representing the total number of spatial locations. Then, a shared learnable semantic query vector is used. The query matrix Q is constructed by projecting the input features X onto the feature space of the CNN. The keys K and values ​​V are obtained from the input features X through linear projection, calculated as follows: in Let be a learnable linear mapping matrix, used to project the query vector and input features onto the same feature space. In practice, these linear mappings can be implemented using fully connected layers, and the projected feature dimension is denoted as . That is, the dimension of each query vector.

[0038] Attention weights are calculated as follows: in The attention score matrix is ​​obtained by multiplying the query matrix by the transpose of the key matrix. The scaling factor is used to prevent the dot product result from becoming too large, causing the Softmax gradient to enter the saturation region. The Softmax function normalizes along the query dimension, ensuring that the sum of the attention weights for each query vector at all spatial locations is 1. The dimension of the attention weights A is... ,in This represents the number of query vectors (8 in this example), and N represents the number of spatial locations.

[0039] Then, the aggregation characteristics of the lesions are obtained. ,in , representing the lesion-related semantic vector aggregated from the input features for each query vector. Finally, the final output T1 is obtained through average aggregation of query dimensions and spatial feature mapping, the calculation process of which is as follows: in This represents the output corresponding to the i-th query vector, with dimension . First, the outputs of all query vectors are aggregated through an averaging operation to obtain the global semantics. This averaging operation integrates lesion patterns captured by multiple semantic probes, forming a more robust semantic description of the lesions. Subsequently, it is projected onto the feature space through a linear mapping Wz+b, and the spatial dimension is restored to C×H×W through an expansion operation. Finally, local modeling is performed using 3×3 convolution to obtain the enhanced feature u(z). This enhanced feature u(z) is then fused with the original feature X1 through residual connections, enhancing the lesion response while preserving the original local texture information. The method for determining T2 is the same as for T1; the most crucial aspect is that T1 and T2 share the same query vector. .

[0040] Based on the above mechanism, this embodiment achieves unified lesion perception enhancement of CNN local structural features and Transformer global semantic features by sharing a learnable lesion query vector. The query vector, as a learnable semantic probe, is dynamically updated during training, continuously adjusting its response to lesion patterns. Since T1 and T2 share the same query vector, they produce consistent enhanced responses in the lesion region, while the response is suppressed in the background region. This sharing mechanism enables local texture features and global semantic features to form a unified semantic reference around the lesion region, reducing the semantic bias between the two types of heterogeneous features. This provides a prerequisite for semantic alignment for subsequent weighted fusion, avoiding the problem of lesion information and background noise being simultaneously amplified due to direct fusion.

[0041] When using a cross-attention mechanism for lesion detection, the initialization method, quantity setting, and feature enhancement method after aggregation of query vectors directly affect the quality of lesion perception. Improper initialization of query vectors may lead to disordered attention distribution and slow convergence in the early stages of training; if the aggregation method is too simple, it will be difficult to fully integrate the complementary lesion information captured by multiple query vectors. Therefore, in one embodiment, the shared learnable semantic query vectors contain a preset number of query vectors and are initialized using a standard normal distribution. Attention weights are used to aggregate the value matrix to obtain local and global enhancement features, including: The query aggregation features are obtained by averaging the attention aggregation outputs corresponding to each query vector. The query aggregate features are projected onto the feature space through linear mapping, and the spatial dimension is restored through expansion operations. Local modeling is then performed using convolutional layers to obtain enhanced features. By fusing the enhanced features with the original branch features through residual connections, local enhanced features or global enhanced features can be obtained; Among them, the original branch features are either local structural features or global semantic features.

[0042] In this embodiment, the shared learnable semantic query vector includes A query vector, The value is 8. These query vectors follow a standard normal distribution. Initialization is performed. Standard normal distribution initialization is a common initialization strategy in deep learning, with a mean of 0 and a standard deviation of 1. The advantages of this initialization method are twofold: firstly, the initial values ​​are symmetrically distributed around zero, breaking the symmetry between neurons and allowing different query vectors to have different semantic response tendencies in the initial training phase; secondly, the gradient variance of the standard normal distribution is relatively stable, which helps the gradient propagation smoothly during training and avoids gradient explosion or vanishing problems caused by excessively large or small initial values. It should be noted that the number of query vectors can be flexibly selected within the range of 1 to 32 according to the actual task requirements. A smaller number results in higher computational efficiency, while a larger number provides stronger coverage of various lesion morphological patterns.

[0043] Given that dermoscopic images typically contain a small number of lesions and are often dominated by a single primary lesion, this embodiment sets the number of query vectors to 8 to achieve a good balance between semantic coverage of lesions and computational complexity.

[0044] After obtaining the attention aggregation output corresponding to each query vector Then, in this embodiment, the outputs of all query vectors are aggregated through an averaging operation. The specific calculation process of the averaging operation is as follows: For Each query vector corresponds to an output. For all query vectors in dimension Take the average above to get This averaging operation integrates lesion information captured from eight semantic probes from different perspectives, forming a more robust global semantic description of lesions. Compared with direct concatenation or weighted summation, the averaging operation reduces the computational overhead caused by excessively large feature dimensions while maintaining semantic dimensions, and is less prone to overfitting to the noisy response of a single query vector.

[0045] After obtaining the query aggregate feature z, it is projected onto the same channel space as the original branch features using a linear mapping Wz+b, where W is the learnable weight matrix and b is the bias term. The projected feature dimension is C (consistent with the number of channels in the original feature). Then, an expansion operation is used to restore the one-dimensional feature to a three-dimensional feature map with a spatial size of H×W, that is, the same channel features are copied at each spatial location. A 3×3 convolution is used to perform local spatial modeling on the expanded feature map to enhance the structural correlation between adjacent pixels, resulting in enhanced features. Finally, residual connections are used. Adding the original branch feature X1 to the final enhanced feature T1 = X1 + u(z) yields the final enhanced feature T1 = X1 + u(z). The residual connection preserves the texture details in the original feature while superimposing the lesion semantic response enhanced by query aggregation.

[0046] Based on the above approach, this embodiment achieves an effective transformation from multiple semantic probes to a single enhanced feature through three steps: initialization with a pre-defined number of query vectors using a normal distribution, averaging aggregation, and residual enhancement. During training, the eight query vectors learn different lesion semantic patterns. Averaging aggregation integrates the advantages of each probe, linear mapping and convolutional expansion feed the aggregated semantics back to the original feature space, and residual connections ensure the stability of information transmission. This entire process results in stronger and more consistent responses in the lesion region for both the final local and global enhanced features, providing a feature foundation for subsequent cross-branch weighted fusion.

[0047] After obtaining the local and global enhancement features, it is necessary to determine the relative importance of each feature branch before fusion. Different branch features have different discriminative capabilities in different channels and spatial locations: some channels may mainly encode boundary information, while others mainly encode regional semantics; some spatial locations (such as the lesion center) require more global semantic guidance, while others (such as the lesion edge) require more local detail support. If fixed weights are used for fusion, they cannot be dynamically adjusted according to the feature content, resulting in the fused features being interfered with by non-optimal branches in some regions or channels. Therefore, in one embodiment, based on the local and global enhancement features, the joint weights of the two branches are determined from the channel dimension and spatial dimension, respectively, and the joint weights are used for weighted fusion to obtain consistent fused features, including: Global average pooling and global max pooling are performed on local and global enhancement features respectively, and the resulting description vectors are concatenated to aggregate spatial information; The aggregated spatial features are passed to two one-dimensional convolutions to obtain the channel weights of two sets of branch features, and the joint channel weights are obtained by stacking and Softmax aggregation. The spatial joint weights are obtained by processing them in the same way in the spatial dimension. The joint channel weights are multiplied by the local enhancement features, and the joint spatial weights are multiplied by the global enhancement features and then summed to obtain the collaborative fusion features; Consistent fusion features are obtained by locally reconstructing and aligning the distribution of collaboratively fused features through convolution, batch normalization, and nonlinear activation.

[0048] like Figure 4 As shown, the LGMF module provided in this embodiment jointly models the relative importance of different feature branches from both channel and spatial perspectives. See also Figure 4 The LGMF module diagram shows that the module takes the local and global enhancement features output by LQT as input, calculates the joint weights through channel branching and spatial branching respectively, and finally outputs the collaborative fusion features.

[0049] In this embodiment, the local enhancement features and global enhancement features from the LQT module are denoted as follows: and In the channel branch, global average pooling (Avg) and global max pooling (Max) are first performed on T1 and T2 respectively, and the resulting description vectors are concatenated to aggregate spatial information. The calculation formula is as follows: It should be noted that, to avoid symbol confusion, the concatenated aggregated features will be denoted as […]. Its dimensions are In the formula below, C represents the number of channels, which are different physical quantities. Avg(T) and Max(T) represent the global mean and global maximum values ​​of the feature map T over the spatial dimension H×W, respectively. Global average pooling captures the average response level of each channel, reflecting the activation intensity of that channel across the overall lesion region; global max pooling captures the peak response of each channel, reflecting the activation intensity of that channel at the most prominent location within the lesion. Combining both provides a more comprehensive characterization of the characteristic response distribution of each channel. The concatenated result... It aggregates channel statistics from both branches.

[0050] Aggregated spatial features The inputs are passed to two one-dimensional convolutions, Conv1 and Conv2, to obtain the channel weights of the two sets of branch features, respectively. and : Where Conv1(·) and Conv2(·) represent 1×k one-dimensional convolutions, and the formula for the value of the convolution kernel width k is: γ=2, b=1. This adaptive convolutional kernel width design allows the calculation of channel attention weights to take into account the dependencies between adjacent channels, and the computational cost is logarithmic with the number of channels, reducing the number of parameters and computational overhead. and These represent the channel importance weights for branches T1 and T2, respectively.

[0051] Then aggregated via stacking and Softmax. and Obtain the joint weight of the channels Specifically, and Stack the tensors as 2×C along the channel dimension, and perform Softmax normalization along the branch dimension so that the sum of the weights of the two branches in each channel is 1, and each weight value is in the range (0,1). Thus, Each element in the table represents the relative importance percentage of the two branch features in the corresponding channel.

[0052] The spatial joint weights are obtained in the same way in the spatial branch. Specifically, global average pooling and global max pooling are performed on T1 and T2 respectively along the channel dimension. The resulting description vectors are concatenated and then passed through two independent convolutions to generate two sets of spatial attention maps. These maps are then stacked and normalized along the branch dimension using Softmax to obtain the joint spatial weights. The weight value at each spatial location represents the relative importance of the two branches at that location.

[0053] Once the joint channel weight W_C and the joint spatial weight W_S are determined, the calculation method for the collaborative fusion feature G can be expressed as follows: In the formula, "×" indicates element-wise multiplication. In the channel dimension, Channel weighting is applied to T1 to highlight channels with strong discriminative power in local enhancement features and suppress redundant channels; in the spatial dimension, Spatially weighting T2 enhances the spatial location of lesion regions within the global enhancement features while suppressing background regions. The two weighted features are then summed, achieving a synergistic complementarity between local and global enhancement features.

[0054] Since the obtained feature G still has limited ability to model local spatial structure and boundary continuity, a feature enhancement module consisting of 3×3 convolution, batch normalization (BN) and nonlinear activation function (ReLU) is used to locally reconstruct and align the distribution of the fused features, alleviate the feature scale fluctuations caused by the attention mechanism, and obtain the output, ultimately achieving the goal of unifying the representation space.

[0055] Based on the above mechanism, this embodiment dynamically calculates the relative importance weights of the two branch features in both channel and spatial dimensions, achieving adaptive fusion of local details and global semantics. In the channel dimension, a combination of global average pooling and max pooling is used to capture the response intensity and peak features of each channel, giving higher weights to channels with strong discriminative capabilities. In the spatial dimension, weights are assigned based on the response intensity of features at different locations, giving more attention to lesion regions. The use of joint channel and spatial weights ensures optimal complementarity between local structural information and global semantic information at different channels and spatial locations, reducing the impact of cross-branch semantic bias on fusion quality and providing consistent lesion fusion features for subsequent decoding and reconstruction.

[0056] During the decoding and recovery phase, the spatial resolution of the feature map is gradually restored after multiple upsampling operations. However, the upsampling operation itself cannot guarantee the continuity and integrity of the lesion contour. If edge structure information is not used to constrain the decoded features, the interpolation operation during the upsampling process can easily lead to edge blurring. Furthermore, in areas with blurred boundaries and low contrast, the lesion boundary response in the decoded features is weak, making it difficult to recover a clear contour. Therefore, in one embodiment, an edge intensity map is generated based on the gradients of the feature map in the horizontal and vertical directions, and edge-aware weights are generated based on the edge intensity map, including: Calculate the gradients of the feature map in the horizontal and vertical directions, take the absolute values ​​and sum them to obtain the edge intensity map; The edge intensity map is compressed along the channel dimension to obtain single-channel edge features, and the channel dimension mean operation is performed on the decoded input features to obtain spatial average features; Single-channel edge features are concatenated with spatial average features, and spatial attention weights are generated through convolutional layers and activation functions. The edge-aware features are obtained by multiplying the decoded input features element-wise using spatial attention weights. The decoding input features are either consistent fusion features or upsampling features during the decoding process.

[0057] like Figure 6 As shown, the EGA module provided in this embodiment explicitly models the local structural response of the lesion contour region through edge-aware branches. See also Figure 6The EGA module diagram shows that the module first extracts the edge intensity by calculating the gradient through finite difference, and then generates spatial attention weights to guide the decoding features through the edge structure.

[0058] In this embodiment, for the input features of the decoding stage (This feature can be either the consistent fusion feature output by LGMF or a feature from any upsampling stage during the decoding process.) First, the gradients of the feature map in the horizontal and vertical directions are approximated using finite difference to extract edge information. Specifically, the horizontal gradient... and vertical gradient Calculated as follows: for each spatial location in the feature map , (Put zeros at the boundaries). (Zeros are padded at the boundaries). This finite difference approximation preserves high-frequency variation information in the feature map, namely the boundary region corresponding to the lesion contour.

[0059] The edge intensity map is obtained by taking the absolute value of the gradient results and summing them. Each element in G represents the edge response intensity at its corresponding spatial location, with response values ​​at edge locations significantly higher than those in non-edge regions. The edge intensity map is then compressed along the channel dimension using mean compression to obtain single-channel edge features. This single-channel feature integrates the edge responses of all channels, reflecting the overall edge strength at each spatial location. Simultaneously, channel-dimensional mean operations are performed on the decoded input feature X to obtain the spatial average feature. This feature reflects the overall response strength of the input feature at each spatial location.

[0060] Single-channel edge features Spatial average characteristics By splicing along the channel dimension, we obtain... The joint features are fused through convolutional layers, and spatial attention weights are generated using a sigmoid activation function. The calculation formula is as follows: Where Avg(G) represents the single-channel edge feature obtained by averaging the edge intensity map G along the channel dimension, and Avg(X) represents the spatial average feature obtained by averaging the decoded input feature X along the channel dimension. Sigmoid activation maps the convolution output to the (0,1) interval, making... The value for each spatial location represents the importance of that location within a marginal or lesion region. Spatial attention weights. The edge-aware features are obtained by element-wise multiplication with the original input features X. : The core idea of ​​this design is to use edge structure information to constrain global context reconstruction, so that the decoded features can restore spatial resolution while maintaining the structural continuity and integrity of the lesion region. In an alternative embodiment, the method of obtaining edge information is not limited to the Sobel operator and finite difference calculation; the Scharr operator and learnable gradient convolution kernel can also be used instead.

[0061] Based on the above approach, this embodiment extracts gradient information from the feature map using finite difference approximation to generate an edge intensity map, which is then used as a structural constraint to generate spatial attention weights. These attention weights have high response values ​​at edge locations, enhancing the spatial location corresponding to the lesion contour in the decoded features. In non-edge regions (such as inside the lesion or background regions), the weight response is relatively smooth, ensuring that the semantic consistency within the region is not compromised. By combining edge intensity with the response intensity of the input features, the generation of attention weights simultaneously considers structural boundary information and feature response intensity, achieving adaptive modulation of the decoding process by local edge constraints and providing boundary-enhanced input features for subsequent global context modeling.

[0062] After obtaining edge-aware features, to further enhance the global semantic consistency within the lesion region and maintain boundary continuity, global contextual information needs to be introduced on top of edge constraints. Relying solely on edge constraints, pixels within the lesion region lack semantic relationships, easily leading to holes or inconsistent predictions within the region; relying solely on global context, however, makes it difficult to guarantee accurate boundary localization. Therefore, in one embodiment, edge-aware weights and global dependencies are used to perform structural consistency modulation on the consistency fusion features, outputting the segmentation result of the image to be segmented, including: Global average pooling is performed on edge-aware features to aggregate spatial information and obtain a channel-level global description; Dimensionality reduction is achieved by convolution and layer normalization of the channel-level global description, and then dimension increase is achieved by nonlinear activation and convolution. The global dependencies between channels are learned to obtain the dimension-upgraded global description. The upgraded global description and edge-aware features are multiplied and added respectively to obtain the output features after structural consistency modulation; Spatial resolution is restored based on output features to generate segmentation results.

[0063] In this embodiment, edge-aware features are used. Spatial information is aggregated by global average pooling to obtain a channel-level global description. The calculation formula is as follows: Here, GAP represents the global average pooling operation, performed independently for each channel. It compresses the spatial dimension H×W into a single numerical value, resulting in a channel description vector of length C. This channel-level global description integrates the overall response level of edge-aware features in each channel, reflecting the global spatial distribution of features across all channels. Through this operation, the spatial details of edge-aware features are compressed into statistical information at the channel dimension, providing input for subsequent learning of inter-channel dependencies.

[0064] After obtaining the channel-level global description V, the global dependencies between channels are learned through a series of transformations. Specifically, V first undergoes convolutional operations and layer normalization (LN) to reduce the dimensionality, compressing the number of channels from C to [the desired value]. (r is the dimensionality reduction ratio, which is taken as r=4 in this embodiment) to reduce computational overhead and extract the compact correlation between channels. Then, a nonlinear activation function (ReLU) is used to introduce nonlinear transformation capability, followed by convolution to achieve dimensionality increase, reducing the number of channels from... Restored to C. This dimensionality reduction-upgrading structure borrows from the lightweight channel attention design concept, reducing the number of parameters through a bottleneck structure while enabling the network to learn non-linear dependencies between channels. Global description after dimensionality upgrade. With edge-aware features Perform multiplication and addition operations respectively to obtain the output features after structural consistency modulation: In the formula, "·" indicates element-wise multiplication at the channel level, that is, the weight of the c-th channel in V2 is multiplied by... The c-th channel is multiplied by all spatial locations. The multiplication operation is performed using a channel-level global description. The characteristics of each channel are recalibrated to highlight channels with strong global response and suppress channels with weak global response; the addition operation preserves the original feature information and prevents feature distortion.

[0065] Finally, based on the output features modulated with structural consistency, the spatial resolution is gradually restored by the decoder. Specifically, the output features are sequentially fused with skip connection features through multiple upsampling layers (such as bilinear interpolation or transposed convolution) to gradually restore the spatial resolution to the original input image size. During each upsampling process, the EGA module re-performs edge awareness and global context modulation on the features at the current scale to ensure that the continuity of lesion contours and regional integrity are maintained at different scales. Finally, a classification layer (such as 1×1 convolution with Sigmoid activation) generates a segmentation mask with the same size as the original image, where each pixel value represents the probability that the location belongs to the lesion region, completing the segmentation result output of the image to be segmented.

[0066] Based on the above approach, this embodiment achieves structural consistency modulation through the synergistic effect of an edge-aware branch and a global context branch. The edge-aware branch utilizes gradient information to generate attention weights that produce strong responses at boundary locations, enhancing edge information in the decoded features. The global context branch aggregates channel-level global descriptions and learns inter-channel dependencies, improving semantic consistency within the lesion region. The result of their synergistic effect is that edge constraints play a dominant role in the lesion edge region, ensuring clear and continuous contours; while the global context plays a dominant role in the lesion interior region, maintaining semantic consistency within the region. This synergistic constraint of edge and context avoids the boundary breaks or region holes that may occur when relying on only a single constraint, ensuring both boundary accuracy and region integrity in the final segmentation result.

[0067] In skin lesion images, lesions vary greatly in scale, ranging from early melanomas a few millimeters in size to large moles several centimeters in size. A single receptive field convolutional kernel struggles to simultaneously capture the fine boundaries of small lesions and the overall contours of large lesions. Furthermore, dermoscopy images often contain hair occlusion and bubble artifacts; these high-frequency interferences are similar to the true lesion boundaries in the frequency domain, making it easy to misidentify artifacts as boundaries when relying solely on gradient edge extraction. Therefore, in one embodiment, a local structure branch captures feature representations of different receptive fields through multiple parallel convolutional branches, and a boundary detail perception branch uses edge detection operators to extract gradient edge information to enhance high-frequency details, including: In the local structural branch, feature representations of different receptive fields are captured by multiple parallel convolutional branches and then spliced ​​and fused to obtain multi-scale fused features; By using the boundary detail perception branch, gradient edge information is extracted using the edge detection operator, and texture features are extracted by combining local convolution to obtain high-frequency detail features; By aggregating multi-scale fusion features with high-frequency detail features, local structural features are obtained.

[0068] like Figure 3 As shown, the DAMS module provided in this embodiment first uses a multi-branch receptive field extension structure to extract lesion responses at different scales, and further combines boundary detail-aware branches to generate explicit structural priors. See also Figure 3 The DAMS module diagram shows that the module consists of four parts: multi-scale convolutional fusion path, detail-aware branch (DAB), channel and coordinate spatial attention (CCSA), and random depth regularization (DropPath).

[0069] In this embodiment, for input features First, feature representations of different receptive fields are captured through four parallel convolutional branches. The first branch uses a 1×1 convolution to maintain the current receptive field and preserve original spatial details; the second, third, and fourth branches use 3×3 dilated convolutions with dilation rates d=1, d=2, and d=3, respectively, to obtain receptive fields of different scales. The 3×3 dilated convolution with dilation rate d expands the receptive field from 3×3 to [a larger scale] without increasing the number of parameters. That is, the receptive field is 3×3 when d=1, 5×5 when d=2, and 7×7 when d=3. After parallel computation of the four branches, the number of output channels of each branch is 1. This means that the total number of channels is evenly distributed across the four branches along the channel dimension. It should be noted that dilated convolution can be replaced with ordinary convolution, depthwise splitting convolution, or deformable convolution, and the branch dilation rate can also take different increasing combinations.

[0070] The outputs of the four branches are concatenated along the channel dimension to obtain the result. F1 to F4 are the output features of the four branches, respectively. Subsequently, channel information is fused through 1×1 convolution, followed by batch normalization (BN) and ReLU activation to obtain multi-scale fused features. : Where Conv represents 1×1 convolution, used to mix the channel information of the four branches; BN represents batch normalization, used to accelerate training convergence and alleviate gradient vanishing; ReLU represents nonlinear activation function, introducing nonlinear expressive power.

[0071] Simultaneously, the input feature X is also fed into the boundary detail-aware branch (DAB) to extract high-frequency detail features F_d. This branch first uses a fixed Sobel operator to obtain gradient edge information, then fuses learnable local convolutional texture features and enhances high-frequency details, focusing on capturing edge contours and local texture information. Specifically, the edge detection branch uses the Sobel operator to extract edge responses in both the horizontal and vertical directions. The Sobel operator is a set of 3×3 discrete differential operators, with its horizontal convolution kernel being... The vertical convolution kernel is The input feature X is convolved with two Sobel kernels to obtain the horizontal and vertical edge responses. The absolute values ​​are then summed to obtain the edge response. Secondly, the texture modeling branch Convolutional sequences with small receptive fields (such as two 3×3 convolutions with ReLU activation) are used to capture local texture variations. Finally, and The high-frequency information is obtained by concatenating channels and performing 1×1 convolution. : This design provides the network with fine-grained spatial information independent of semantic features by explicitly modeling two key high-frequency features: edges and textures. This makes up for the shortcomings of standard convolution in preserving details and enhances the ability to accurately locate lesion boundaries.

[0072] Finally, the multi-scale fused feature F_m is modulated by the CCSA module and then residually aggregated with the high-frequency detail feature F_d and the module input feature X to obtain the Output. In the specific implementation, First, channel spatial attention modulation is performed using the CCSA module to obtain intermediate features. Then, DropPath is introduced as a random depth regularization strategy. The features are processed; finally, they are aggregated through three residual connections: Where p is the dropout probability and M is a mask variable following a Bernoulli distribution, the DropPath operation randomly discards some branch paths during training, forcing the network to be independent of responses from specific paths, thereby improving the model's generalization ability. DAMS is embedded in multiple scale stages, and as the network depth increases, DropPath alleviates feature overfitting and branch co-adaptation.

[0073] Based on the above mechanism, this embodiment expands the receptive field of the network through multi-scale parallel convolutional branches, enabling the network to simultaneously capture the fine boundaries of small lesions and the overall contours of large lesions. The boundary detail perception branch explicitly extracts edge information using the Sobel operator and captures texture changes through local convolution. The combination of these two forms high-frequency detail features, providing the network with a structural prior independent of semantic features. This design ensures that edge structure modeling is integrated throughout the feature extraction and feature enhancement process. Even under interference such as hair occlusion and bubble artifacts, the network can obtain stable boundary structure representations, providing a high-quality structural prior for subsequent cross-branch representation alignment and fusion. This avoids introducing additional interference by simultaneously amplifying the true boundaries and artifacts during the fusion process.

[0074] Building upon multi-scale fusion features, an attention modulation mechanism is needed to further enhance the structural response of the lesion region and suppress background noise and spurious responses. However, standard channel attention or spatial attention can only modulate in the channel or spatial dimension alone, making it difficult to achieve accurate structural enhancement in both dimensions simultaneously. Furthermore, DAMS modules are embedded across multiple scale stages. As network depth increases, without regularization constraints, the cascading of multiple DAMS modules can easily lead to feature overfitting and branch co-adaptation, meaning the outputs of each branch tend to become homogeneous, losing the diversity advantage of multi-scale feature extraction. Therefore, in one embodiment, multi-scale fusion features are aggregated with high-frequency detail features to obtain local structural features, including: The multi-scale fusion feature input direction structure attention module is subjected to channel spatial attention modulation; Channel attention weights are obtained by global average pooling and one-dimensional convolution, and channel weighting is performed on multi-scale fused features to obtain channel-weighted features, where the kernel width of the one-dimensional convolution is adaptively determined based on the number of channels. The channel-weighted features are averaged and pooled along the vertical and horizontal directions respectively to obtain two one-dimensional feature codes. After concatenating two one-dimensional feature codes to reduce dimensionality, spatial attention maps in the vertical and horizontal directions are generated. The channel-weighted features are triple-multiplied with the two spatial attention maps to obtain the intermediate features; A random depth regularization strategy is introduced to process intermediate features, and residual aggregation is performed with module input features and high-frequency detail features to obtain local structural features.

[0075] In this embodiment, the CCSA module employs a cascaded channel spatial attention mechanism to achieve adaptive feature representation. For input features... First, global average pooling is used to compress the spatial dimension to obtain the channel description vector. Then, a 1×k one-dimensional convolution is used to capture the dependencies between channels, where the kernel width k is adaptively determined according to the number of channels. Where C is the number of feature channels, and γ and b are hyperparameters. This indicates that the absolute value is rounded up. This adaptive kernel width design allows for the use of larger kernels with a larger number of channels to capture channel dependencies over longer distances, and smaller kernels with a smaller number of channels to reduce computational overhead, thus achieving a balance between computational efficiency and receptive field coverage.

[0076] Channel attention weights are obtained using the Sigmoid activation function. Then, channel-weighted features are obtained by channel-weighting the input features. Where "×" indicates channel-by-channel multiplication, that is, the weight of the c-th channel in S is multiplied by... The spatial positions of the c-th channel are multiplied together. This operation enhances channels with strong responses and suppresses channels with weak responses.

[0077] Then Average pooling is performed along both the vertical and horizontal directions to decompose the two-dimensional spatial information into two one-dimensional feature codes: in Global feature statistics along the vertical direction are encoded. Global feature statistics along the horizontal direction are encoded. This spatial decomposition method preserves precise location information. Unlike global average pooling, which compresses the entire space into a single value, the two one-dimensional codes after decomposition preserve the location information in the vertical and horizontal directions, respectively, enabling subsequent spatial attention to perceive the specific location of lesions in the image.

[0078] Features from two directions are concatenated along the spatial dimension and then subjected to 1×1 convolution, batch normalization, and ReLU activation to reduce dimensionality, computational cost, and cross-channel information interaction. Subsequently, the interacted features are segmented into vertical and horizontal parts along the spatial dimension, and a vertical spatial attention map is generated using two independent 1×1 convolutions and a sigmoid function. Spatial attention map in the horizontal direction Finally, the channel weighted values ​​are... The intermediate feature is obtained by triple multiplication with the spatial attention in both directions. .

[0079] Regarding random depth regularization, this embodiment introduces the DropPath strategy. Specifically, for intermediate features... Generate mask variables that follow a Bernoulli distribution. , where p is the preset drop probability (p=0.1 in this embodiment). The features after DropPath processing are: in The scaling factor is used to scale the preserved paths during training, ensuring consistent expected outputs between training and testing. Finally, features are aggregated through three residual connections: Where X represents the module input feature. These are high-frequency detail features.

[0080] Based on the above approach, this embodiment achieves joint attention modulation in both channel and spatial dimensions through the concatenated channel spatial attention mechanism of the CCSA module. Compared with parallel attention, the concatenated approach avoids competition between channel attention and spatial attention on the same feature map, allowing structure enhancement and noise suppression to proceed step by step without interference. The design of decomposed coordinate attention preserves precise location information, enabling spatial attention to perceive the specific location of lesions in the image, generating strong responses in boundary regions and weak responses in background regions. The random depth regularization of DropPath randomly discards parts of the path during training, forcing each branch to learn effective feature representations independently without relying on the co-adaptation of other branches. This alleviates the overfitting problem caused by the concatenation of multiple modules and improves the model's generalization ability and adaptability to different lesion structures.

[0081] In deep networks containing multiple DAMS modules, overfitting is prone to occur as the number of modules stacks, especially in medical image scenarios with limited training data. Furthermore, the cascading of multiple modules can easily lead to branch co-adaptation, where different branches gradually learn similar representations, losing the feature diversity originally designed for the multi-branch structure. Therefore, in one embodiment, a stochastic depth regularization strategy is introduced during feature extraction or feature fusion to mitigate overfitting and branch co-adaptation. Random depth regularization strategies include: Generate binary mask variables that follow a Bernoulli distribution; The scaling factor is calculated based on the binary mask variable and the preset drop probability; The intermediate features are discarded and scaled using a scaling factor.

[0082] In this embodiment, the random depth regularization strategy can be applied in both the output stage of the DAMS module and the feature fusion stage of the LGMF module. Taking the DAMS module as an example, after obtaining the intermediate feature F_t, a DropPath operation is introduced. First, a binary mask variable following a Bernoulli distribution is generated. , where p is the preset discard probability, which is a decimal between 0 and 1 (p=0.1 in this embodiment). This indicates that the value is 1 with probability 1-p and 0 with probability p. Mask variable M and intermediate features. They have the same feature map size and are sampled independently before training each sample.

[0083] Calculate the scaling factor based on the binary mask variable M and the drop probability p. When M=1, the scaling factor is The path is preserved and its feature values ​​are amplified; when M=0, the scaling factor is 0, and the path is completely discarded. The scaling factor is introduced to maintain consistency between the expected output during training and testing: the expected scaling factor during training is... The output is the same as the expected output during testing without dropping.

[0084] The intermediate features are processed using a scaling factor to obtain... The scaled features are compared with the module input features X and high-frequency detail features. Perform residual aggregation to obtain the final output. .

[0085] Based on the above approach, this embodiment employs the DropPath random depth regularization strategy to randomly discard some branch paths during training, ensuring that the network activates only a subset of paths for training in each iteration. This mechanism makes the network independent of the response of any specific path, forcing each branch to independently learn discriminative feature representations, thereby suppressing branch co-adaptation. Simultaneously, random discarding is equivalent to implicitly training an ensemble of multiple sub-networks during training, improving the model's robustness to training data noise, mitigating the overfitting problem of deep networks on limited medical image data, and enhancing the model's generalization performance on unknown data.

[0086] Secondly, this application also provides an image segmentation system, which includes: The multi-scale feature extraction module is used to acquire the image to be segmented. It performs multi-scale feature extraction on the image to be segmented through local structure branch and global semantic branch to obtain local structure features and global semantic features respectively. Among them, the local structure branch captures the feature representation of different receptive fields through multiple parallel convolution branches, and uses the boundary detail perception branch to extract gradient edge information using edge detection operators to enhance high-frequency details. The feature alignment and fusion module is used to guide the alignment of local structural features and global semantic features through a cross-attention mechanism using shared learnable semantic query vectors, thereby obtaining local enhanced features and global enhanced features respectively. Based on the local enhanced features and global enhanced features, the joint weights of the two branches are determined from the channel dimension and the spatial dimension respectively, and the joint weights are used for weighted fusion to obtain consistent fusion features. The consistency modulation output module is used to generate edge intensity maps based on the gradients of the feature maps in the horizontal and vertical directions, and generate edge-aware weights based on the edge intensity maps; it obtains channel-level global descriptions by aggregating spatial information through global pooling, and learns global dependencies between channels based on the channel-level global descriptions; it uses the edge-aware weights and global dependencies to perform structural consistency modulation on the consistency fusion features, and outputs the segmentation results of the image to be segmented.

[0087] For specific limitations regarding the image segmentation system, please refer to the limitations on image segmentation methods above, which will not be repeated here. Each module in the aforementioned image segmentation system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0088] Furthermore, this application also provides a computer device, including a memory and a processor. The memory is communicatively connected to the processor, and the memory stores a computer program that can run on the processor. When the processor executes the computer program, it implements the image segmentation method as described above. Moreover, the specific limitations of the computer device in implementing the image segmentation method can be found in the limitations of the image segmentation method described above, and will not be repeated here.

[0089] Furthermore, a computer-readable storage medium is provided that stores a computer program thereon, which, when executed by a processor, implements the image segmentation method as described above. Additionally, specific limitations on the implementation of the image segmentation method by this computer-readable storage medium can be found in the limitations on the image segmentation method described above, and will not be repeated here.

[0090] To verify the effectiveness of the progressive lesion representation structure consistency modeling method proposed in this application, this embodiment conducted a large number of comparative and ablation experiments on a publicly available skin lesion segmentation dataset, and provides a visualization effect analysis.

[0091] 1. Analysis of Transformer performance in querying dual-branch lesions The test uses five common segmentation network evaluation metrics, including Sensitivity (SE), Specificity (SP), Accuracy (ACC), Dice coefficient, and Intersection over Union (IoU). To verify the alignment effect of the shared query vector on the bi-branch features, this embodiment visualizes the feature response heatmaps before and after LQT module guidance, such as... Figure 7 As shown. Figure 7 The diagram shows a comparison of the original input image (a), the ground truth label (b), the pre-guided local branch features (X1), the pre-guided global branch features (X2), the post-guided local branch features (T1), and the post-guided global branch features (T2). Figure 7As shown, before guidance, features X1 from the local encoder and X2 from the global encoder, while capable of capturing some lesion information, exhibited scattered responses with significant activation around the lesions and in the background. This indicates that the two features had limited focus on the lesion region and differed in their focus areas. Features T1 and T2, guided by the lesion query, generated stronger responses in the lesion region, while activation in the background region was significantly suppressed. Comparison with the true label (b) reveals that the guided feature responses are more consistent with the lesion region, demonstrating that the dual-branch lesion query Transformer effectively filters lesion-related information and improves the discriminative power of feature representations. Furthermore, after guidance, the regions of focus for the local and global branches showed greater consistency.

[0092] 2. Ablation Effect Analysis To verify the effectiveness and synergistic effect of the three core modules DAMS, LGMF and EGA, this embodiment conducted ablation experiments on the ISIC2017 and ISIC2018 datasets, and the results are shown in Table 1.

[0093] Table 1 It can be seen that DAMS, LGMF, and EGA are not independent functional modules, but together constitute a complete technology chain of "reliable local structure generation—cross-branch lesion representation alignment and fusion—structural consistency decoding and reconstruction". Each link in this technology chain serves the same core goal—that is, maintaining the structural consistency of lesion representation throughout the feature fusion and spatial reconstruction process. SwinTransBlock is responsible for obtaining global contextual information of the lesion region. DAMS forms a reliable local structural representation through multi-scale structural enhancement and boundary detail compensation. LGMF uses the shared lesion query vector as a unified semantic reference to achieve consistent alignment between local structural features and global semantic features. EGA further utilizes edge and contextual information to constrain the decoding process, stably restoring the fused features to the prediction space. The absence of any link will affect the continuous transmission of lesion information at different network stages. Compared with the baseline, the complete network improves IoU and Dice by 3.02 and 2.41 percentage points, respectively, on ISIC2017, and by 4.68 and 3.39 percentage points, respectively, on ISIC2018. The above data confirms that the proposed solution is not a simple combination of multiple modules, but an organic synergy centered around the unified inventive concept of "maintaining structural consistency". Its performance improvement comes from the overall effect formed by the connection and mutual support of each link.

[0094] 3. Comparative analysis of segmentation performance with common segmentation networks To further verify the generalization ability of this application under conditions of large differences in lesion morphology, blurred boundaries, hair occlusion, and artifact interference, this embodiment selects the ISIC2017, ISIC2018, and PH2 datasets and compares the segmentation method proposed in this application with several commonly used segmentation algorithms (Unet, Unet++, AttentionUnet, FAT-Net, H2Former, GFANet, CSAP-Unet, I2U-Net, HTC-Net, PMFSNet, MPBA-Net, UCMV2-Net).

[0095] Table 2 shows the quantization comparison results of different methods on the ISIC2018 dataset, Table 3 shows the comparison results on the ISIC2017 dataset, and Table 4 shows the comparison results on the PH2 dataset.

[0096] Table 2 Table 3 Table 4 As can be seen from the data in Tables 2 to 4, this application achieved optimal results in the vast majority of evaluation indicators. Figure 8 This is a visualization comparison of the segmentation results of different methods on the ISIC2018 dataset. It can be seen that the method proposed in this application achieves the best segmentation effect on the ISIC2018 data. In the diagram, red represents the true labeled region, green represents the lesion region predicted by the proposed method, and yellow represents the overlapping area of ​​the true labeled region and the predicted region. The larger the yellow area, the better the segmentation effect. It can be seen that this method can accurately segment lesion regions, whether they are small or large.

[0097] Figure 9 The visualization compares the segmentation results of different methods on the ISIC2017 dataset. The visualization shows that the accuracy is further enhanced when processing images with blurred boundaries. However, some methods exhibit local boundary misjudgment when dealing with skin lesion images where the boundaries are almost integrated into the background. Compared to all the compared methods, the segmentation network in this application, based on the overall visualization results, maintains stable and high-performance segmentation capabilities even when dealing with small-sized lesions with blurred boundaries and medium-sized complex blurred boundary lesions. The outline of the yellow overlapping region is closest to the true label outline.

[0098] Figure 10This image presents a visualization comparison of segmentation results from different methods on the PH2 dataset. When processing skin lesion images obscured by hair, the state-of-the-art methods HTC-Net and UCMV2-Net exhibit stronger robustness, although subtle interference artifacts are still visible in areas with dense hair. The proposed method demonstrates optimal performance on all test samples, with the largest proportion of the yellow overlapping area reflecting the stability of the segmentation network.

[0099] The experimental results above show that the progressive lesion representation structure consistency modeling method proposed in this application can effectively adapt to complex skin lesion scenarios such as hair occlusion, bubble artifacts, illumination changes, and lesion scale changes, thereby improving the robustness and generalization ability of the model to complex lesion images.

[0100] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image segmentation method, characterized in that, The method includes: The image to be segmented is acquired, and multi-scale feature extraction is performed on the image to be segmented through local structure branch and global semantic branch to obtain local structure features and global semantic features respectively; wherein, the local structure branch captures feature representations of different receptive fields through multiple parallel convolution branches, and extracts gradient edge information using edge detection operators through boundary detail awareness branch to enhance high-frequency details. By utilizing shared learnable semantic query vectors, the local structural features and global semantic features are guided and aligned through a cross-attention mechanism to obtain local enhanced features and global enhanced features, respectively. Based on the local enhanced features and global enhanced features, the joint weights of the two branches are determined from the channel dimension and the spatial dimension, respectively, and the joint weights are used for weighted fusion to obtain consistent fusion features. An edge intensity map is generated based on the gradient calculation of the feature map in the horizontal and vertical directions, and an edge-aware weight is generated based on the edge intensity map. A channel-level global description is obtained by aggregating spatial information through global pooling, and the global dependency relationship between channels is learned based on the channel-level global description. The structural consistency modulation of the consistency fusion feature is performed using the edge-aware weight and the global dependency relationship, and the segmentation result of the image to be segmented is output.

2. The method according to claim 1, characterized in that, The step of guiding the alignment of the local structural features and the global semantic features through a cross-attention mechanism to obtain local enhanced features and global enhanced features respectively includes: The shared learnable semantic query vector is projected into a query matrix, and the local structural features and the global semantic features are flattened and projected into the corresponding key matrix and value matrix, respectively. Attention weights are calculated based on the query matrix and the key matrix, and the value matrix is ​​aggregated using the attention weights to obtain the local enhancement features and the global enhancement features, respectively.

3. The method according to claim 2, characterized in that, The shared learnable semantic query vectors contain a preset number of query vectors and are initialized using a standard normal distribution; The aggregation of the value matrix using the attention weights to obtain local enhancement features and global enhancement features includes: The query aggregation features are obtained by averaging the attention aggregation outputs corresponding to each query vector. The query aggregate features are projected onto the feature space through linear mapping, and the spatial dimension is restored through expansion operations. Local modeling is then performed using convolutional layers to obtain enhanced features. The enhanced features are fused with the original branch features through residual connections to obtain the local enhanced features or the global enhanced features; The original branch feature is either the local structural feature or the global semantic feature.

4. The method according to claim 1, characterized in that, The method involves determining the joint weights of the two branches based on the local enhancement features and the global enhancement features, respectively, from the channel dimension and the spatial dimension, and then using the joint weights to perform weighted fusion to obtain consistent fusion features, including: Global average pooling and global max pooling are performed on the local enhancement features and the global enhancement features respectively, and the resulting description vectors are concatenated to aggregate spatial information; The aggregated spatial features are passed to two one-dimensional convolutions to obtain the channel weights of two sets of branch features, and the joint channel weights are obtained by stacking and Softmax aggregation. The spatial joint weights are obtained by processing them in the same way in the spatial dimension. The joint channel weights are multiplied by the local enhancement features, and the joint spatial weights are multiplied by the global enhancement features and then summed to obtain the collaborative fusion features; The consistent fusion features are obtained by locally reconstructing and aligning the collaborative fusion features through convolution, batch normalization, and nonlinear activation.

5. The method according to claim 1, characterized in that, The step of generating an edge intensity map based on the gradient calculation of the feature map in the horizontal and vertical directions, and generating edge-aware weights based on the edge intensity map, includes: The gradients of the feature map in the horizontal and vertical directions are calculated, and their absolute values ​​are summed to obtain the edge intensity map. The edge intensity map is compressed along the channel dimension to obtain single-channel edge features, and the channel dimension mean operation is performed on the decoded input features to obtain spatial average features. The single-channel edge features are concatenated with the spatial average features, and spatial attention weights are generated through convolutional layers and activation functions. The spatial attention weights are used to multiply the decoded input features element-wise to obtain edge-aware features; The decoding input feature is either the consistency fusion feature or the upsampling feature during the decoding process.

6. The method according to claim 5, characterized in that, The step of using the edge-aware weights and the global dependency to perform structural consistency modulation on the consistency fusion features and outputting the segmentation result of the image to be segmented includes: The edge-aware features are subjected to global average pooling, and spatial information is aggregated to obtain the channel-level global description; The channel-level global description is reduced in dimensionality by convolution and layer normalization, and then increased in dimensionality by nonlinear activation and convolution. The global dependencies between channels are learned to obtain the increased-dimensional global description. The upgraded global description and the edge-aware features are multiplied and added respectively to obtain the output features after structural consistency modulation; The spatial resolution is restored based on the output features, and the segmentation result is generated.

7. The method according to claim 1, characterized in that, The local structure branch captures feature representations of different receptive fields through multiple parallel convolutional branches, and uses an edge detection operator to extract gradient edge information to enhance high-frequency details through a boundary detail-aware branch, including: In the local structure branch, feature representations of different receptive fields are captured and spliced ​​and fused through the multiple parallel convolution branches to obtain multi-scale fused features; The boundary detail perception branch uses edge detection operators to extract gradient edge information and combines it with local convolution to extract texture features, thereby obtaining high-frequency detail features. The local structural features are obtained by aggregating the multi-scale fusion features with the high-frequency detail features.

8. An image segmentation system, characterized in that, The system includes: A multi-scale feature extraction module is used to acquire the image to be segmented. It performs multi-scale feature extraction on the image to be segmented through a local structure branch and a global semantic branch to obtain local structure features and global semantic features, respectively. The local structure branch captures feature representations of different receptive fields through multiple parallel convolution branches, and extracts gradient edge information using an edge detection operator through a boundary detail awareness branch to enhance high-frequency details. The feature alignment and fusion module is used to guide the alignment of the local structural features and the global semantic features through a cross-attention mechanism using a shared learnable semantic query vector, thereby obtaining local enhanced features and global enhanced features respectively; based on the local enhanced features and the global enhanced features, the joint weights of the two branches are determined from the channel dimension and the spatial dimension respectively, and the joint weights are used to perform weighted fusion to obtain consistent fusion features; The consistency modulation output module is used to generate an edge intensity map based on the gradients of the feature map in the horizontal and vertical directions, and generate edge-aware weights based on the edge intensity map; obtain a channel-level global description by aggregating spatial information through global pooling, and learn the global dependencies between channels based on the channel-level global description; perform structural consistency modulation on the consistency fusion features using the edge-aware weights and the global dependencies, and output the segmentation result of the image to be segmented.

9. A computer device comprising a memory and a processor, the memory being communicatively connected to the processor, and the memory storing a computer program executable on the processor, characterized in that, When the processor executes the computer program, it implements the image segmentation method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the image segmentation method as described in any one of claims 1 to 7.

Images