Skin-scope image feature visualization method and device based on direction perception

Abstract

This application provides a method and apparatus for visualizing dermoscopic image features based on orientation awareness, comprising: processing a dermoscopic image through a backbone network of a segmentation network, wherein the spatial resolution of multiple hierarchical modules is progressively reduced and the number of channels is progressively increased; decoupling the orientation features of the dermoscopic image and hierarchically propagating the orientation prior through orientation-aware base blocks; extracting directional latent priors containing global response information in multiple directions from deep orientation features; fusing orientation features under the guidance of the directional latent priors through a decoder to obtain a segmentation feature map; progressively decoupling and enhancing multi-scale orientation information during feature extraction, and extracting compact directional latent priors from deep features; and using the orientation priors to guide dual attention computation within a local window, enabling the model to achieve a balance between structural consistency and detail accuracy, thereby achieving robust recovery of the continuous boundaries and topological integrity of lesions.

Description

Technical Field

[0001] This application relates to the field of computer vision, and more particularly to a method and apparatus for visualizing dermoscopic image features based on orientation awareness. Background Technology

[0002] Existing methods for skin lesion segmentation suffer from the following main drawbacks. First, while convolutional neural network (CNN)-based methods, such as U-Net and its derivatives, effectively fuse multi-scale features through local convolutions and skip connections, standard convolutional operations are inherently isotropic. Their receptive field expansion depends on network depth or dilated convolution strategies, making it difficult to fundamentally characterize the structural patterns of lesions extending continuously along specific directions. When dealing with lesions with blurred boundaries, complex morphologies, or obvious directional growth, these methods often exhibit problems such as discontinuous predictions or over-smoothing, resulting in insufficient structural integrity and topology preservation in the segmentation results.

[0003] Methods based on Transformer and its hybrid architectures, such as Swin-UNet and TransUNet, enhance long-range dependency modeling capabilities through self-attention mechanisms, capturing semantic relevance globally and thus improving overall segmentation consistency. However, these methods generally rely on global feature aggregation mechanisms based on image patches or feature tokens, which inevitably weakens local geometric continuity while improving semantic consistency. They are particularly difficult to characterize the process by which lesion structures gradually extend along specific directions to form the overall topology. Furthermore, self-attention mechanisms typically involve high computational complexity and are sensitive to the size of training data, limiting their practical application in clinical settings.

[0004] Structure-aware or connectivity-modeling methods, such as ConnNet, BiconNet, and SDCTrans, attempt to introduce structural constraints from the perspective of pixel connectivity or region relationships, thereby improving the continuity and boundary preservation capabilities of segmentation to some extent. However, most of these methods introduce structural information as auxiliary constraints or posterior regularizations into the model, and their core objective remains to optimize feature discrimination or attention allocation processes, rather than directly modeling how lesion structures are gradually generated and expanded in space. Therefore, when facing scenarios with blurred boundaries, complex morphologies, or cross-dataset generalization, their ability to model structural continuity remains limited, making it difficult to truly recover the overall topological structure of the lesions.

[0005] Existing attention mechanisms, whether channel attention, spatial attention, or dual attention, treat attention as a feature weighting or selection tool. Their core focus is on optimizing feature representation and information fusion, while paying less attention to how the structural semantics inherent in attention further participate in the generation and reconstruction of lesion morphology. Especially in scenarios where lesions exhibit obvious directional extension or hierarchical structural relationships, relying solely on feature-level attention selection is still insufficient to fully characterize the spatial propagation between local structures and the overall organization.

[0006] In summary, most existing skin lesion segmentation methods model the task as a pixel-by-pixel classification problem. Whether it's local feature extraction based on CNNs or global semantic modeling based on Transformers, their core improvements still focus on how features are selected and aggregated, rather than exploring how local structures gradually propagate, connect, and reconstruct the complete lesion morphology from the perspective of structure generation. These methods still have certain limitations in their ability to model structural continuity when facing scenarios with blurred boundaries, complex morphologies, or cross-dataset generalization. Summary of the Invention

[0007] The following is an overview of the topics described in detail in this article.

[0008] The purpose of this application is to at least partially solve one of the technical problems existing in the related technologies. The embodiments of this application provide a method and apparatus for visualizing dermoscopic image features based on orientation awareness.

[0009] An embodiment of the first aspect of this application provides a method for visualizing dermoscopic image features based on orientation awareness, comprising: Acquire dermoscopic images and input the dermoscopic images into the segmentation network; The dermoscopy image is processed by the backbone network of the segmentation network. The backbone network includes multiple hierarchical modules. The spatial resolution of the multiple hierarchical modules decreases layer by layer from the input direction to the output direction, while the number of channels increases layer by layer. Each hierarchical module decouples the directional features of the dermoscopy image and propagates the directional prior hierarchically through the directional awareness base block. Directional latent priors are extracted from the directional features output by the last-level hierarchical module. These directional latent priors contain global response information for multiple directions. Guided by the directional latent prior, the decoder fuses the directional features output by each level module to obtain a segmentation feature map. The segmentation feature map is predicted by the classification layer to obtain the segmentation probability map.

[0010] According to certain embodiments of the first aspect of this application, there are four hierarchical modules: the first hierarchical module has one directional grouping, the second hierarchical module has two directional groupings, the third hierarchical module has four directional groupings, and the fourth hierarchical module has eight directional groupings.

[0011] According to certain embodiments of the first aspect of this application, the hierarchical module is provided with a direction-aware basic block, through which direction features are decoupled and direction priors are propagated hierarchically; The decoupling of the directional features includes the following steps: dividing the input image into multiple directional groups in the channel dimension, calculating the first channel attention and the first spatial attention in parallel according to the features of the directional groups, and multiplying the features of the directional groups element-wise with the adaptive fusion results of the first channel attention and the first spatial attention to obtain the enhanced features of the directional groups; The hierarchical propagation of the directional prior includes the following steps: generating a directional prior map based on the directional features output by the deep hierarchical modules, and downsampling the directional prior map to obtain a selective enhancement modulation signal for guiding the features in different directions.

[0012] According to certain embodiments of the first aspect of this application, the extraction of directional latent priors from the directional features output from the last-layer hierarchical module includes: The directional features output from the last-level module are used to extract directional latent priors through global pooling and recoding operations.

[0013] According to certain embodiments of the first aspect of this application, the decoder is provided with a directional window attention fusion module; The directional window attention fusion module adjusts the number of channels in the input feature map so that the number of channels is divisible by the number of directional groups. For each spatial location on the feature map after adjusting the number of channels, a local window is extracted with the spatial location as the center. The features within the local window are divided into multiple grouped features in different directions along the channel dimension. Each grouped feature includes a center pixel feature and a neighborhood direction feature. The aggregated value of the domain direction feature and the center pixel feature are fused to obtain the local context feature; Extract the second spatial attention and the second channel attention based on the local context features; Enhanced modulation features are obtained by modulating the adaptive fusion values ​​of the second spatial attention and the second channel attention through directional prior. The enhanced modulation features in all directions are concatenated to obtain the concatenated features, which are then input into a lightweight convolutional layer to fuse the interactive information within a local window. The features output by the lightweight convolutional layers corresponding to all local windows are input into the global window attention layer to obtain global features; The global features are residually concatenated with the input feature map to obtain the segmentation feature map.

[0014] According to certain embodiments of the first aspect of this application, after obtaining a segmentation prediction map, a loss function is calculated based on the segmentation prediction map, and the parameters of the segmentation network are adjusted by the loss function, which includes size density loss, decoupling loss, connectivity constraint loss, and boundary pixel loss.

[0015] According to certain embodiments of the first aspect of this application, in the application phase, the method includes: Acquire real-time dermoscopy images; The real-time dermoscopy image is input into the trained segmentation network to obtain a segmentation probability map of the lesion region; Thresholding is performed on the segmentation probability map of the lesion region to obtain a binarized lesion segmentation mask; The segmentation mask is overlaid on the original real-time dermoscopy image and the lesion area is marked. The lesion area is visualized.

[0016] According to certain embodiments of the first aspect of this application, the visualization of the lesion area includes: The mean value of all channels in the features of the directional group is used as the spatial response heatmap of the directional group; The spatial response heatmaps grouped in different directions are stacked into a tensor, and the spatial response heatmaps are then normalized. A comprehensive visualization chart is generated based on the normalized spatial response heatmap. The comprehensive visualization chart includes the original dermoscopy image, spatial response heatmap, directional response statistical histogram, total directional response heatmap, dominant direction color-coded map, directional response correlation matrix, response spatial variation map, and directional response spectrum.

[0017] According to a second aspect of this application, an electronic device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the orientation-aware dermoscopy image feature visualization method as described in the first aspect of this application.

[0018] According to a third aspect of this application, a computer storage medium stores computer-executable instructions for performing the orientation-aware dermoscopy image feature visualization method as described in the first aspect of this application.

[0019] The above-mentioned scheme has at least the following beneficial effects: It redefines the skin lesion segmentation task from a traditional pixel-by-pixel classification problem into a direction-aware structural reconstruction problem. By explicitly modeling the extension trends of local regions along multiple directions and the generation process of the overall lesion topology, it improves the structural consistency and boundary integrity of the segmentation results. By designing a progressive direction-aware backbone network, multi-scale directional information is decoupled and enhanced layer by layer during feature extraction, and compact directional latent priors are extracted from deep features. Simultaneously, a directional window attention fusion module is constructed, using directional priors to guide dual attention calculations within local windows, enabling the model to achieve a balance between structural consistency and detail accuracy, thereby achieving robust recovery of the continuous boundaries and topological integrity of lesions. Furthermore, it provides a visualization method for directional response, transforming the internal direction-aware process of the model into an intuitive heatmap, enhancing the model's interpretability and providing quantitative references for clinical diagnosis. Attached Figure Description

[0020] The accompanying drawings are used to provide a further understanding of the technical solutions of this application and constitute a part of the specification. They are used together with the embodiments of this application to explain the technical solutions of this application and do not constitute a limitation on the technical solutions of this application.

[0021] Figure 1 This is a step-by-step diagram of a direction-aware dermoscopy image feature visualization method; Figure 2 This is a step-by-step diagram of the application of visualization methods in model prediction. Figure 3 It is a step-by-step diagram of the chart visualization process; Figure 4 This is a schematic diagram of the structure of a clinically practical system. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0023] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, or the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0024] The embodiments of this application provide a method and apparatus for visualizing dermoscopic image features based on orientation awareness. The core idea is to explicitly encode the complex boundary connectivity and directional association between the lesion and its surrounding skin tissue through a progressive orientation awareness modeling mechanism.

[0025] This approach elevates dermoscopic image segmentation from a traditional pixel-by-pixel classification problem to a direction-aware structural reconstruction problem. Lesions are viewed as being formed by the progressive assembly of multiple local structural fragments. Each local region contains not only texture and color information but also implicitly suggests a directional tendency extending into the surrounding space. The overall morphology of the lesion originates from the continuous propagation and orderly combination of these local structures in space. This fundamentally changes the modeling approach for segmentation tasks, enabling the model to understand lesion morphology from the perspective of structural generation, rather than simply performing pixel-level pattern matching.

[0026] The embodiments of this application will be further described below with reference to the accompanying drawings.

[0027] Reference Figure 1 A direction-aware dermoscopy image feature visualization method includes the following steps: Step S100: Obtain a dermoscopic image and input the dermoscopic image into the segmentation network; Step S200: The dermoscopy image is processed through the backbone network of the segmentation network; Step S300: Extract directional latent priors from the directional features output by the last layer's hierarchical module. The directional latent priors contain global response information for multiple directions. Step S400: The decoder fuses the directional features output by each level module under the guidance of the directional latent prior to obtain a segmentation feature map. Step S500: The segmentation feature map is predicted by the classification layer to obtain the segmentation probability map.

[0028] For step S100, multiple dermoscopic images are acquired, the multiple dermoscopic images are combined into a dataset, and the images are uniformly scaled to a preset size, for example, 224×320 pixels in this embodiment, but other pixels can be set in other embodiments.

[0029] Data augmentation operations are performed on the training images, including random rotation, random flipping, and brightness adjustment within the angular range of (-π / 6, π / 6), to increase the robustness and generalization ability of the model.

[0030] The dermoscopic image is input into the segmentation network. The main body of the segmentation network consists of two key modules working together: a progressive backbone network based on directional propagation, which is used to progressively decouple and enhance multi-scale directional information during feature extraction; and a Directional Window Attention Fusion Module (DWAF), which is responsible for context-aware fusion and refinement of multi-level and multi-directional features during the decoding stage, and finally outputs accurate lesion segmentation results.

[0031] For step S200, the preprocessed image is input into the progressive orientation-aware backbone network.

[0032] The backbone network consists of multiple hierarchical modules. From the input direction to the output direction, the spatial resolution of the multiple hierarchical modules decreases layer by layer, while the number of channels increases layer by layer. Each hierarchical module decouples the directional features of the dermoscopy image and propagates the directional prior hierarchically through the directional awareness base block.

[0033] The backbone network can adaptively and progressively adjust the granularity of orientation modeling based on the semantic hierarchy and spatial scale of the feature maps.

[0034] Specifically, the backbone network is based on DAResNet and contains four layer modules, labeled L1, L2, L3, and L4. The first layer module has one directional group size N1=1, the second layer module has two directional group sizes N2=2, the third layer module has four directional group sizes N3=4, and the fourth layer module has eight directional group sizes N4=8.

[0035] The hierarchical module is set up with orientation-aware basic blocks. Each orientation-aware basic block (DABasicBlock) contains the following two key operations: decoupling of orientation features and hierarchical propagation of orientation priors.

[0036] The decoupling of directional features includes the following steps: dividing the input image into multiple directional groups in the channel dimension; calculating the first channel attention and the first spatial attention in parallel based on the features of the directional groups; and multiplying the features of the directional groups element-wise with the adaptive fusion results of the first channel attention and the first spatial attention to obtain the enhanced features of the directional groups.

[0037] Specifically, for an input feature map X_in∈ ^(C×H×W) is first divided into N_l direction groups along the channel dimension, where N_l is the preset number of directions for this layer. The feature X_in^i ∈ […] of each direction group g_i. ^((C / N_l)×H×W) is fed into a parallel directional attention module. This module computes the channel attention A_c^i and the spatial attention A_s^i in parallel.

[0038] Channel attention is expressed by the formula: A_c^i = σ(MLP(GAP(X_in^i))); Spatial attention is expressed by the formula: A_s^i = σ(f_conv (X_in^i)); Where σ represents the Sigmoid activation function, GAP is global average pooling, and f_conv is lightweight convolution.

[0039] Subsequently, adaptive fusion is performed using learnable fusion weights α and β to generate enhanced features X for this directional group. ^i. Here, the weights α and β are normalized using Softmax.

[0040] The enhanced features of directional grouping are expressed by the formula: X ^i = X_in^i ⊙ (α·A_c^i + β·A_s^i); where ⊙ represents element-wise multiplication.

[0041] The hierarchical propagation of directional priors includes the following steps: generating a directional prior map based on the directional features output by the deep hierarchical modules, and downsampling the directional prior map to obtain a selective enhancement modulation signal used to guide the features in different directions.

[0042] Specifically, in deeper hierarchical modules (such as L4 layer), the orientation-aware block generates an orientation prior map P_dir∈ from the current feature X_out. ^(C_l×1×1), this prior map is downsampled and passed as a modulation signal to subsequent blocks or layers to guide the selective enhancement of features in different directions.

[0043] The forward process of the backbone network is represented as: F_l, P_l = B_l (F_(l-1), P_(l-1); θ_l), l=1,…,4; where B_l represents the sequence of direction-aware blocks contained in the l-th layer, θ_l is its parameter, P_0 is the initial prior, and P_0 is set to non-empty in this embodiment.

[0044] Through this progressive design, the network focuses on basic edge and texture orientations in shallow layers (L1, L2), and models more abstract, high-level directional patterns related to the overall morphology and extension trend of lesions in deeper layers (L3, L4). Based on the semantic hierarchy and spatial scale of the feature maps, different numbers of directional groups are set at different levels to achieve progressive directional perception from coarse to fine granular. Within each directional perception block, directional features are decoupled and enhanced through parallel computation and adaptive fusion of channel attention and spatial attention. A directional prior propagation mechanism is introduced to generate directional prior maps from deep features and pass them down to guide the directional selective enhancement of shallow features, forming a closed-loop flow of directional information.

[0045] For step S300, a compact directional latent prior D_prior∈ is extracted from the features of the deepest layer L4 of the backbone network through global pooling and recoding operations. ^(8×1×1). This prior contains global response information in 8 directions, which is used to guide the directional feature fusion in the subsequent decoding stage.

[0046] By leveraging compact directional priors extracted from deep features to modulate the multi-directional attention response during the decoding stage, the directional knowledge embedded in high-level semantic information can be fed back and guide the refined processing of local features. This prior-guided mechanism for local refinement of the set enables structurally coherent segmentation.

[0047] For step S400, the multi-scale directional features {F1, F2, F3, F4} output from each layer of the backbone network and the directional prior D_prior are input into the decoder. The decoder uses a directional window attention fusion module, which, guided by the directional prior, fuses and refines the skip connection features step by step to finally reconstruct a high-resolution segmentation feature map.

[0048] The directional window attention fusion module is used to replace the traditional decoding feature fusion operation. Its core is to perform dual attention calculation with directional separation within a sliding window and to modulate it through directional prior.

[0049] The directional window attention fusion module performs the following operations: Adjust the number of channels in the input feature map so that the number of channels is divisible by the number of directional groupings; For each spatial location on the feature map after adjusting the number of channels, a local window is extracted with the spatial location as the center. The features within the local window are divided into multiple grouped features in different directions along the channel dimension. Each grouped feature includes the center pixel feature and the neighborhood directional feature. The aggregated value of the domain orientation feature and the center pixel feature are fused to obtain the local context feature; Extract the second spatial attention and the second channel attention based on local contextual features; Enhanced modulation features are obtained by modulating the adaptive fusion values ​​of the second spatial attention and the second channel attention through directional prior. The enhanced modulation features from all directions are concatenated to obtain the concatenated features, which are then input into a lightweight convolutional layer to fuse the interactive information within the local window. The features output by the lightweight convolutional layers corresponding to all local windows are input into the global window attention layer to obtain global features; The global features are residually concatenated with the input feature map to obtain the segmentation feature map.

[0050] Specifically, given the high-level features F_h (upsampled output from the layer above the decoder) and the orientation prior D_prior, the directional window attention fusion module reconstructs a high-resolution segmentation feature map under the guidance of the orientation prior through the following processing steps.

[0051] Step 1: Channel adjustment.

[0052] Adjust the number of channels in F_h to C' so that it is divisible by the number of directions K=8, and obtain the adjusted feature F'.

[0053] Step 2: Sliding window processing.

[0054] For each spatial location on the feature map, a 3×3 local window is extracted centered on that location. The features within the window are uniformly divided into K directional groups along the channel dimension. For the k-th directional group, its window features W^k contain the center pixel features C^k and features N^k from 8 neighboring directions.

[0055] First, calculate the aggregate value N of the neighborhood features. ^k, and fused with the central feature to form a local context representation in that direction: L^k = C^k + γ·N ^k; where γ is the learnable weight coefficient. Step 3: Calculate dual attention within the window.

[0056] L^k and the corresponding directional prior slice p^k extracted from D_prior are fed into the dual attention submodule within the window. This submodule simultaneously computes the spatial attention Att_spatial^k and the channel attention Att_channel^k.

[0057] Spatial attention can be expressed by the formula: Att_spatial^k = PAM(L^k); Channel attention is expressed by the formula: Att_channel^k = CAM(L^k); PAM stands for Position Attention Module, and CAM stands for Channel Attention Module.

[0058] Step 4: Directional prior modulation.

[0059] Spatial attention and channel attention are adaptively fused and modulated using directional prior p^k, which can be expressed as: L_att^k = (η·Att_spatial^k + ζ·Att_channel^k) ⊙ σ(p^k); where η and ζ are learnable fusion weights and σ is the Sigmoid function.

[0060] Step 5: Feature recombination.

[0061] The enhanced features L_att^k of all K directional groups are re-concatenated, and the interactive information within the local window is further fused through a lightweight convolutional layer.

[0062] Step 6: Global window attention calculation.

[0063] After multiple sliding window processing steps, a global window attention layer is used to model the relationships between all spatial locations, capture long-range dependencies, and ensure global consistency of the segmentation decision.

[0064] Step 7: Residual connection.

[0065] Finally, after channel adjustment and residual connection, the directional window attention fusion module outputs refined features to obtain the segmentation feature map.

[0066] The segmentation feature map is expressed by the formula: F_refined = DWAF(F_h, D_prior) = G(S_1 (S_N(A(F_h), D_prior))) + F_h; Where A represents channel adjustment, S_i represents the i-th sliding window processing layer, and G represents the global window attention layer.

[0067] For example, in the directional window attention fusion module, the input feature F_h first passes through a channel adjustment layer to adjust the number of channels to a multiple of 8. The adjusted features are then fed into a multi-layer sliding window processing unit. Within each sliding window layer, a 3×3 local window is extracted at each location on the feature map, and the features within the window are divided into 8 directional groups along the channel dimension. For each directional group, spatial attention and channel attention are calculated separately and modulated and fused with the corresponding directional slices extracted from the directional prior D_prior. The enhanced features of all directional groups are re-stitched and fused through lightweight convolution. After multi-layer sliding window processing, the features enter the global window attention layer to capture long-range dependencies. Finally, the module input and the processed features are added through residual connections to output the refined feature F_refined.

[0068] By segmenting features into multiple directional groups along the channel dimension within a sliding window, local context modeling with directional separation is achieved; spatial attention and channel attention are computed in parallel for each directional group and adaptively fused; the attention response is modulated using compact directional priors extracted from the deep layers of the backbone network, enabling the model to focus on features consistent with the lesion extension direction; and long-range dependencies are captured through a global window attention layer to ensure global consistency of segmentation decisions.

[0069] For step S500, the segmentation feature map is predicted by the classification layer to obtain the segmentation probability map.

[0070] For example, in the overall architecture of the PuzzleNet model, the encoder on the left is a progressive orientation-aware backbone network. This backbone network contains four layers (L1 to L4). The input image first passes through the L1 layer (1 orientation), extracting basic edge features; then it passes through the L2 layer (2 orientations), L3 layer (4 orientations), and L4 layer (8 orientations) sequentially, progressively enhancing orientation awareness. Each layer consists of multiple stacked orientation-aware basic blocks, achieving decoupling and enhancement of orientation features. From the output of the L4 layer, a compact orientation prior D_prior (8×1×1 in size) is extracted through global pooling and recoding operations. The decoder on the right contains multiple directional window attention fusion modules. Skip connection features from each encoder layer are fused with the upsampled decoded features in the DWAF module, and directional features are refined under the guidance of the orientation prior D_prior. Finally, the lesion segmentation result is output after passing through the classification layer.

[0071] After obtaining the segmentation prediction map, a loss function is calculated based on the segmentation prediction map. The parameters of the segmentation network are then adjusted using the loss function, which includes size density loss, decoupling loss, connectivity constraint loss, and boundary pixel loss.

[0072] The total loss function is calculated as L_total = L_main + λ·L_prior, where L_main is the main output loss, L_prior is the auxiliary output loss used for supervised learning in the DWAF module, and λ is a hyperparameter, which is set to 0.3 in this embodiment.

[0073] L_main and L_prior have the same structure, both employing a combination of size density loss (SDL), bicon loss (BL), and boundary decoupling loss (EL) to address the uneven distribution of categories and optimize boundary localization accuracy.

[0074] Size density loss (SDL) is used to address the problem of large size differences in lesion areas.

[0075] Calculate the size density weight of the label: P_j (k) = {(1,k=0&-log(PDF_j (k)),k≠0)}; where PDF_j (k) is the size distribution probability density function of the j-th type of target.

[0076] SDL loss is defined as: L_sd = ∑_j^(class) P_j (k) (1-(2×∑(s×G_s )+ε) / (∑S+∑G_s+ε)); Where s is the predicted segmentation map, G_s is the true segmentation mask, and ε is the smoothing term.

[0077] Bicon loss (BL) includes decoupling loss and connectivity constraint loss.

[0078] The Bicon loss is expressed as: L_bicon = L_decouple + L_(con_const).

[0079] Boundary decoupling loss (EL) is optimized for boundary pixels.

[0080] L_edge=L_BEC(1-min{X _i (x,y)_(i=1)^8, G_s (x,y)}), (x,y)∈P_edge; where, _i is the i-th channel of the final output X, and P_edge is the set of edge pixels.

[0081] The loss functions of L_main and L_prior are combined as follows: L = L_bicon + L_sd + L_edge.

[0082] Reference Figure 2 In the application phase of model prediction, the visualization method includes the following steps: Step S610: Acquire real-time dermoscopy images; Step S620: Input the real-time dermoscopy image into the trained segmentation network to obtain the segmentation probability map of the lesion area; Step S630: Threshold the segmentation probability map of the lesion region to obtain a binarized lesion segmentation mask. Step S640: Overlay the segmentation mask with the original real-time dermoscopy image and mark the lesion area; Step S650: Visualize the lesion area.

[0083] Specifically, in step S610, RGB images of the patient's skin lesion area are acquired using a dermatoscope or other medical imaging equipment. The images are then uniformly scaled to the model input size, which in this embodiment is 224×320 pixels, and subjected to the same normalization processing as in the training phase, for example, using the mean and standard deviation of the ImageNet dataset for normalization.

[0084] For step S620, the pre-trained PuzzleNet model weight file is loaded to obtain the trained segmentation network.

[0085] The preprocessed image is input into the PuzzleNet model. The model performs forward propagation, passing sequentially through the progressive orientation perception backbone network, the DWAF module, and the classification layer, outputting a lesion region segmentation probability map with the same resolution as the input image.

[0086] In step S630, the segmentation probability map output by the model is thresholded to obtain a binarized lesion segmentation mask; in this embodiment, the threshold is 0.5. The segmentation mask is overlaid on the original image to mark the lesion area in the form of a highlighted outline or a semi-transparent color block.

[0087] For step S640, in application scenarios where enhanced model interpretability is required, the orientation visualization module can be activated. This module extracts the 8 orientation group feature responses of each sliding window layer in the DWAF module through pre-registered hook functions, and generates a comprehensive orientation analysis map according to the visualization method, intuitively displaying the directional features of lesions perceived by the model.

[0088] In step S650, the lesion area is visualized.

[0089] For video streams or batches of images, repeat the above steps to achieve continuous automated analysis.

[0090] The 8 directional feature groups extracted by the DWAF module inside the model are transformed into intuitive heatmaps through visualization methods, thereby enhancing the interpretability of the model.

[0091] Reference Figure 3 The chart visualization process includes the following steps: Step S710: The mean value of all channels in the features of the directional group is used as the spatial response heatmap of the directional group. Step S720: Stack the spatial response heatmaps grouped in different directions into a tensor and normalize the spatial response heatmaps. Step S730: Generate a comprehensive visualization chart based on the normalized spatial response heatmap.

[0092] Specifically, in step S710, during the model's forward inference, the feature map output by the DWAF module is obtained by registering with the sliding window layer through a hook function. Assume the output feature map shape is (B, C, H, W), where C is the number of channels and is divisible by the number of directions (8). The number of channels in each direction group is C / 8. For each direction group, the mean of all channels in that group is taken as the spatial response map for that direction. For direction indices i from 0 to 7, calculate: R_i = mean( output[i*(C / 8): (i+1)*(C / 8), ], dim=1 ); resulting in eight direction response maps R0 to R7 with shapes (1, H, W).

[0093] For step S720, the eight directional response maps are stacked into a tensor R∈ ^(8×H×W), calculate the global minimum value v_min and maximum value v_max, and linearly normalize each response map to the [0,1] interval. The normalization process is expressed as: R_i^norm = (R_i - v_min) / (v_max - v_min +ε); where ε is a small constant to prevent division by zero.

[0094] For step S730, the normalized directional response map is used to generate a comprehensive visualization chart according to a preset layout.

[0095] The comprehensive visualization charts include the original dermoscopy image, spatial response heatmap, directional response statistical histogram, total directional response heatmap, dominant direction color-coded map, directional response correlation matrix, response spatial variation map, and directional response spectrum.

[0096] Eight directional response heatmaps are generated, corresponding to the eight directions NW, N, NE, W, E, SW, S, and SE. Each heatmap uses thermal color mapping, with warmer colors indicating higher response intensity in that direction.

[0097] The directional response histogram displays the mean and standard deviation of the response in each direction.

[0098] The overall directional response heatmap is generated by superimposing and summing the responses of the eight directions, using "plasma" color mapping.

[0099] The dominant direction color coding map selects the direction index with the strongest response at each pixel position and uses the tab10 color palette for color coding.

[0100] The directional response correlation matrix is ​​used to calculate the Pearson correlation coefficients among the eight directional responses.

[0101] The response spatial variation map is used to calculate the coefficient of variation (standard deviation / mean) of the response in each direction, and to locate areas of drastic structural changes.

[0102] The directional response spectrum is presented as a histogram, showing the maximum, minimum, mean, and standard deviation for each direction.

[0103] The generated visualizations can be saved as high-resolution image files or displayed directly in the graphical user interface for user reference.

[0104] Reference Figure 4 By integrating the above-mentioned orientation-aware dermoscopy image feature visualization method into a complete clinical system, including an image acquisition unit, an image processing workstation, a display and interaction unit, a report output unit, and a data storage unit, an automated closed loop is formed from image acquisition, intelligent analysis, result visualization to report generation.

[0105] The image acquisition unit is a dermatoscope or digital camera, responsible for acquiring raw images.

[0106] The core image processing workstation is equipped with a high-performance GPU, which loads and runs the PuzzleNet model to generate segmented images of skin lesions.

[0107] The display and interaction unit is a high-resolution medical monitor with a graphical user interface. The image processing workstation transmits segmentation results and visualizations to the display and interaction unit. The display and interaction unit transmits user commands and parameter settings to the image processing workstation.

[0108] The data storage unit stores raw images, segmentation results, visualization reports, and patient information. The image processing workstation stores data in the data storage unit and retrieves historical data from it.

[0109] The report output unit is a printer or an electronic report generation module, which generates a visual report.

[0110] By achieving automated, high-precision segmentation and directional feature visualization of lesion regions, the system eliminates the need for manual delineation based on physician experience. It can process dermoscopic images in real-time in clinical settings, accurately delineating lesion boundaries, with particularly strong structural preservation capabilities for lesions with blurred edges and irregular shapes, significantly improving segmentation accuracy and consistency. Compared to traditional methods, the PuzzleNet system detects lesion regions with higher Dice and Jaccard coefficients, avoiding subjective differences from manual annotation and reducing repetitive work and potential human error for physicians. Simultaneously, the system possesses rapid inference capabilities, processing a single image in less than 30 milliseconds, providing real-time feedback on segmentation results and generating multi-dimensional directional response heatmaps to intuitively demonstrate the lesion extension trend perceived by the model, providing interpretable quantitative references for clinical diagnosis. System evaluations on multiple public datasets demonstrate excellent performance in both segmentation accuracy and cross-dataset generalization ability.

[0111] In terms of segmentation accuracy and structural integrity, a more precise modeling of lesion boundaries and internal structures is achieved through a progressive orientation-aware backbone network and an orientation-prior-guided attention fusion mechanism. On the ISIC 2018 dataset, the Dice coefficient reaches 0.8945 and the Jaccard index reaches 0.8265, exceeding DconnNet by approximately 0.41% and 0.54%, respectively, and SDCTrans by approximately 0.23% and 0.30%. Especially for lesions with blurred boundaries and complex morphologies, it effectively avoids segmentation fragmentation and over-smoothing, generating more complete structures with stronger topology preservation capabilities. Ablation experiments also confirm the synergistic effect of progressive orientation modeling and the prior-guided mechanism.

[0112] In terms of generalization ability, the robustness across datasets is significantly improved by modeling the essential directional growth patterns of lesions. In the PH²ISIC 2018 cross-dataset test, the Jaccard index reached 0.7629, which is about 0.9% higher than DconnNet and about 3.2% higher than SDCTrans, demonstrating that the model learns the general features of lesion structure rather than the superficial statistical characteristics of specific datasets, thus adapting to the diverse image distributions in clinical practice.

[0113] Regarding model interpretability, the directional response visualization module transforms the model's internal directional perception process into intuitive multidimensional heatmaps and statistical charts, revealing the extension trends and structural variation areas of lesions in different directions. This visualization functionality provides quantitative analysis tools, enhancing the interpretability of AI-assisted diagnosis.

[0114] It also has advantages in computational efficiency, with a single image inference time of only 27.78ms and a parameter count of 27.78M, which is lower than SDCTrans's 49.96ms and 50.10M, making it more suitable for real-time clinical applications.

[0115] Embodiments of this application provide an electronic device. The electronic device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the orientation-aware dermoscopy image feature visualization method described above.

[0116] This electronic device can be any smart terminal, including computers.

[0117] In general, for the hardware structure of electronic devices, the processor can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, to execute relevant programs and implement the technical solutions provided in the embodiments of this application.

[0118] The memory can be implemented in the form of read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory and is called and executed by the processor.

[0119] Input / output interfaces are used to implement information input and output.

[0120] The communication interface is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0121] The bus transmits information between various components of a device, such as the processor, memory, input / output interfaces, and communication interfaces. The processor, memory, input / output interfaces, and communication interfaces communicate with each other within the device via the bus.

[0122] Embodiments of this application provide a computer storage medium. The computer storage medium stores computer-executable instructions for performing the orientation-aware dermoscopy image feature visualization method described above.

[0123] It will be understood by those skilled in the art that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium. In the foregoing description of this specification, references to terms such as "one embodiment," "another embodiment," or "some embodiments," etc., indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of this application. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0124] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0125] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0126] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0127] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0128] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed between each other may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms. Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

[0129] The above is a detailed description of the preferred embodiments of this application, but this application is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this application, and these equivalent modifications or substitutions are all included within the scope defined by the claims of this application.

Claims

1. A method for visualizing dermoscopic image features based on orientation awareness, characterized in that, include: Acquire dermoscopic images and input the dermoscopic images into the segmentation network; The dermoscopy image is processed by the backbone network of the segmentation network. The backbone network includes multiple hierarchical modules. The spatial resolution of the multiple hierarchical modules decreases layer by layer from the input direction to the output direction, while the number of channels increases layer by layer. Each hierarchical module decouples the directional features of the dermoscopy image and propagates the directional prior hierarchically through the directional awareness base block. Directional latent priors are extracted from the directional features output by the last-level hierarchical module. These directional latent priors contain global response information for multiple directions. Guided by the directional latent prior, the decoder fuses the directional features output by each level module to obtain a segmentation feature map. The segmentation feature map is predicted by the classification layer to obtain the segmentation probability map.

2. The method for visualizing dermoscopic image features based on orientation awareness according to claim 1, characterized in that, There are four hierarchical modules. The first hierarchical module has one directional grouping limit, the second hierarchical module has two directional grouping limits, the third hierarchical module has four directional grouping limits, and the fourth hierarchical module has eight directional grouping limits.

3. The method for visualizing dermoscopic image features based on orientation awareness according to claim 2, characterized in that, The hierarchical module is equipped with a direction-aware basic block, through which direction features are decoupled and direction priors are propagated hierarchically. The decoupling of the directional features includes the following steps: dividing the input image into multiple directional groups in the channel dimension, calculating the first channel attention and the first spatial attention in parallel according to the features of the directional groups, and multiplying the features of the directional groups element-wise with the adaptive fusion results of the first channel attention and the first spatial attention to obtain the enhanced features of the directional groups; The hierarchical propagation of the directional prior includes the following steps: generating a directional prior map based on the directional features output by the deep hierarchical modules, and downsampling the directional prior map to obtain a selective enhancement modulation signal for guiding the features in different directions.

4. The method for visualizing dermoscopic image features based on orientation awareness according to claim 1, characterized in that, The extraction of directional latent priors from the directional features output from the last-layer hierarchical module includes: The directional features output from the last-level module are used to extract directional latent priors through global pooling and recoding operations.

5. The method for visualizing dermoscopic image features based on orientation awareness according to claim 1, characterized in that, The decoder is equipped with a directional window attention fusion module; The directional window attention fusion module adjusts the number of channels in the input feature map so that the number of channels is divisible by the number of directional groups. For each spatial location on the feature map after adjusting the number of channels, a local window is extracted with the spatial location as the center. The features within the local window are divided into multiple grouped features in different directions along the channel dimension. Each grouped feature includes a center pixel feature and a neighborhood direction feature. The aggregated value of the domain direction feature and the center pixel feature are fused to obtain the local context feature; Extract the second spatial attention and the second channel attention based on the local context features; Enhanced modulation features are obtained by modulating the adaptive fusion values ​​of the second spatial attention and the second channel attention through directional prior. The enhanced modulation features in all directions are concatenated to obtain the concatenated features, which are then input into a lightweight convolutional layer to fuse the interactive information within a local window. The features output by the lightweight convolutional layers corresponding to all local windows are input into the global window attention layer to obtain global features; The global features are residually concatenated with the input feature map to obtain the segmentation feature map.

6. The method for visualizing dermoscopic image features based on orientation awareness according to claim 1, characterized in that, After obtaining the segmentation prediction map, a loss function is calculated based on the segmentation prediction map, and the parameters of the segmentation network are adjusted by the loss function. The loss function includes size density loss, decoupling loss, connectivity constraint loss, and boundary pixel loss.

7. The method for visualizing dermoscopic image features based on orientation awareness according to claim 1, characterized in that, In the application phase, the method includes: Acquire real-time dermoscopy images; The real-time dermoscopy image is input into the trained segmentation network to obtain a segmentation probability map of the lesion region; Thresholding is performed on the segmentation probability map of the lesion region to obtain a binarized lesion segmentation mask; The segmentation mask is overlaid on the original real-time dermoscopy image and the lesion area is marked. The lesion area is visualized.

8. The method for visualizing dermoscopic image features based on orientation awareness according to claim 7, characterized in that, The visualization of the lesion area includes: The mean value of all channels in the features of the directional group is used as the spatial response heatmap of the directional group; The spatial response heatmaps grouped in different directions are stacked into a tensor, and the spatial response heatmaps are then normalized. A comprehensive visualization chart is generated based on the normalized spatial response heatmap. The comprehensive visualization chart includes the original dermoscopy image, spatial response heatmap, directional response statistical histogram, total directional response heatmap, dominant direction color-coded map, directional response correlation matrix, response spatial variation map, and directional response spectrum.

9. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the orientation-aware dermoscopic image feature visualization method as described in any one of claims 1 to 8.

10. A computer storage medium, characterized in that, The device stores computer-executable instructions for performing the orientation-aware dermoscopy image feature visualization method as described in any one of claims 1 to 8.

Images