Deep learning based lesion image processing system and method

Abstract

The application belongs to the technical field of image segmentation, and discloses a lesion image processing system and method based on deep learning, which comprises the following steps: acquiring a skin image, and extracting a multi-scale hierarchical feature map of the skin image through an encoder; inputting the multi-scale hierarchical feature map into a double-path processing module to output first-path output features and second-path output features; inputting the first-path output features and the second-path output features into a feature fusion module to generate a fused feature map through weighted fusion based on a cross-attention mechanism; and inputting the fused feature map into a segmentation decoder to output a segmentation region map of the milk coffee stain after multi-scale context aggregation. The application can significantly improve the accuracy, robustness and generalization ability of the milk coffee stain segmentation task in a complex skin background by respectively performing specific inhibition on skin color variation and skin wrinkle interference, combining the cross-attention mechanism to realize complementary fusion of features, and integrating multi-scale context information in the segmentation decoder.

Description

Technical Field

[0001] This invention relates to the field of image segmentation technology, specifically to a lesion image processing system and method based on deep learning. Background Technology

[0002] Café-au-lait spots are common pigmented skin lesions and are one of the important clinical criteria for diagnosing neurofibromatosis type 1 (NF1). Accurate automated segmentation of these spots is crucial for disease screening, disease assessment, and follow-up. However, achieving high-precision automated segmentation has always been a technical challenge due to the high similarity of café-au-lait spots in color, texture, and morphology to the surrounding normal skin, as well as the complex background interference they present in images.

[0003] Existing deep learning-based image segmentation methods, such as fully convolutional networks and their variants, have achieved significant success in general medical image segmentation. However, when these methods are directly applied to milk and coffee spot segmentation, their performance is often unsatisfactory, mainly due to the following reasons: Individual differences in skin color among patients can cause similar patches to exhibit significant color variations in different images. Traditional convolutional neural networks are not robust enough to such skin color changes and are prone to incorrectly generalizing features learned against a specific skin color background to other scenes, leading to oversegmentation or undersegmentation.

[0004] The interference from skin surface wrinkles is particularly prominent. The shadows and directional textures produced by wrinkles are extremely similar to the boundaries and internal textures of café au lait spots in low-level visual features. Common segmentation models struggle to distinguish these artifacts caused by physical deformation from true pathological features, often misclassifying wrinkles as lesion areas, severely affecting the specificity and accuracy of segmentation.

[0005] To complicate matters further, skin color and wrinkles often coexist and intertwine, such as wrinkles on dark skin near joints. Current technologies lack dedicated mechanisms to simultaneously and collaboratively address these two different types of interference factors. Some methods attempt to improve model robustness through data augmentation, but fail to achieve proactive decoupling and suppression of interference factors at the network structure level; others employing multi-branch networks have also failed to effectively model the anisotropic geometric characteristics of wrinkles.

[0006] Therefore, there is an urgent need in this field for a precise image segmentation method for café-au-lait spots that can effectively overcome the interference from both skin color variations and skin folds. Summary of the Invention

[0007] The purpose of this invention is to provide a method, system, computer program product, computer device, and computer-readable storage medium for processing lesion images based on deep learning, so as to solve the problems mentioned in the background art.

[0008] This invention provides a deep learning-based method for processing lesion images, comprising the following steps: Step S10: Obtain the skin image and extract its multi-scale hierarchical feature map through the encoder; Step S20: Input the multi-scale hierarchical feature map into the dual-path processing module, which includes: The first path is configured to perform primary color estimation and transformation processing on the multi-scale hierarchical feature map to generate the first path output feature; The second path is configured to extract the directional texture response from the multi-scale hierarchical feature map using a multi-directional deformable convolutional kernel, generate a wrinkle saliency mask based on the directional texture response, and perform a gating suppression operation on the multi-scale hierarchical feature map using the wrinkle saliency mask to generate the second path output feature. Step S30: Input the first path output features and the second path output features into the feature fusion module, perform weighted fusion based on the cross-attention mechanism, and generate a fused feature map; Step S40: Input the fused feature map into the segmentation decoder, and after multi-scale context aggregation, output the segmentation region map of the milk coffee spots.

[0009] The present invention also provides a deep learning-based lesion image processing system, comprising: The receiving unit is configured to: acquire a skin image and extract its multi-scale hierarchical feature map through an encoder; The multi-scale hierarchical unit is configured to input the multi-scale hierarchical feature map into a dual-path processing module, the dual-path processing module comprising: The first path is configured to perform primary color estimation and transformation processing on the multi-scale hierarchical feature map to generate the first path output feature; The second path is configured to extract the directional texture response from the multi-scale hierarchical feature map using a multi-directional deformable convolutional kernel, generate a wrinkle saliency mask based on the directional texture response, and perform a gating suppression operation on the multi-scale hierarchical feature map using the wrinkle saliency mask to generate the second path output feature. The fusion processing unit is configured to: input the output features of the first path and the output features of the second path into the feature fusion module, perform weighted fusion based on the cross attention mechanism, and generate a fused feature map; The segmentation processing unit is configured to input the fused feature map into the segmentation decoder, and after multi-scale context aggregation, output a segmentation region map of the milk coffee spots.

[0010] The present invention also provides a computer program product, including a computer program or instructions, which are executed by a processor to implement the method as described in any of the preceding claims.

[0011] The present invention also provides a computer device including a memory, a processor, and a computer program or instructions stored in the memory, the computer program or instructions being executed by the processor to implement the method as described in any of the preceding claims.

[0012] The present invention also provides a computer-readable storage medium storing a computer program or instructions which are executed by a processor to implement the method as described in any of the preceding claims.

[0013] This invention constructs a dual-path processing module to specifically suppress interference from skin color variations and skin wrinkles, and combines a cross-attention mechanism to achieve complementary feature fusion. Finally, multi-scale contextual information is integrated in the segmentation decoder, thereby significantly improving the accuracy, robustness, and generalization ability of the café-au-lait spot segmentation task in complex skin backgrounds. Attached Figure Description

[0014] Figure 1 This is a flowchart illustrating a deep learning-based lesion image processing method disclosed in an embodiment of the present invention. Figure 2 This is a functional architecture diagram of a deep learning-based lesion image processing method disclosed in an embodiment of the present invention; Figure 3 This is a schematic diagram illustrating the process of obtaining the primary color field through a primary color estimation network, as disclosed in an embodiment of the present invention. Figure 4 This is a schematic diagram of the structure of a deep learning-based lesion image processing system disclosed in an embodiment of the present invention. Detailed Implementation

[0015] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0016] Please see Figure 1 , Figure 2 This invention provides a deep learning-based lesion image processing method 100, comprising the following steps: Step S10: Obtain the skin image and extract its multi-scale hierarchical feature map through the encoder; In this step, an input skin image is received, which is a digital image containing the area of ​​the café au lait lesion to be detected. Of course, preprocessing operations such as denoising, cropping, and rotation can also be performed on the skin image, but these will not be elaborated upon here.

[0017] Next, a pre-trained deep convolutional neural network is used as the encoder, such as a backbone network like ResNet, VGG, or similar structures pre-trained on a large image dataset. This encoder performs a series of hierarchical convolution and pooling operations on the input skin image. During this process, different depth levels of the encoder output feature maps with different receptive fields and levels of semantic abstraction, which together constitute a multi-scale hierarchical feature map. The shallow feature maps have higher spatial resolution, preserving rich details such as subtle texture variations and edge contours; while the deep feature maps, although having lower spatial resolution, contain more discriminative high-level semantic information, capable of representing the overall morphology and category attributes of the lesion area.

[0018] Step S20: Input the multi-scale hierarchical feature map into the dual-path processing module, which includes: The first path is configured to perform primary color estimation and transformation processing on the multi-scale hierarchical feature map to generate the first path output feature; The second path is configured to extract the directional texture response from the multi-scale hierarchical feature map using a multi-directional deformable convolutional kernel, generate a wrinkle saliency mask based on the directional texture response, and perform a gating suppression operation on the multi-scale hierarchical feature map using the wrinkle saliency mask to generate the second path output feature. In this step, the present invention designs a parallel dual-path processing module to specifically and collaboratively process the two main sources of interference affecting the accurate segmentation of café au lait spots: skin color variations and skin wrinkles. The dual-path processing module specifically includes: The first path primarily addresses the impact of skin color variations as a source of interference. Its core principle involves primary color estimation and transformation. Specifically, this first path learns or infers primary color information representing the dominant color tone of the current skin image as a whole or in a localized area from the input multi-scale hierarchical feature map. Based on this primary color information, it adaptively transforms or corrects the multi-scale hierarchical feature map to normalize or reduce the impact of skin color differences between individuals and under different lighting conditions on the expression of lesion features. This improves the model's generalization ability and robustness in recognizing café-au-lait spots against different skin color backgrounds. The final output of this first path is the feature representation after skin color interference suppression or calibration, i.e., the first path output feature.

[0019] The second approach focuses on eliminating interference from skin folds. Its core principle is to identify and suppress linear textures and shadows caused by physical skin deformation that are easily confused with lesion features. Specifically, this second approach utilizes multi-directional deformable convolution kernels to perform convolution operations on multi-scale hierarchical feature maps. The deformable convolution can adaptively adjust the position of its sampling points, thereby better fitting fold structures of various directions and irregular shapes; combined with its multi-directional design, it can comprehensively capture texture responses from different angles.

[0020] The directional texture response output by the above operations can highlight directional structural features in the skin image. Based on these directional texture responses, the second path further generates a wrinkle saliency mask, which quantifies the probability of wrinkles existing at various spatial locations in the skin image.

[0021] Finally, a gated suppression operation is used to selectively suppress the original multi-scale hierarchical feature map, thereby effectively reducing the interference of wrinkle artifacts at the feature level. The gated suppression operation, for example, uses the mask value as a gate weight to dynamically attenuate the feature activation intensity of wrinkle-related regions in the feature map.

[0022] The final output of the second path is the feature representation after wrinkle interference suppression, i.e., the output feature of the second path.

[0023] Step S30: Input the first path output features and the second path output features into the feature fusion module, perform weighted fusion based on the cross-attention mechanism, and generate a fused feature map; In this step, the feature representations generated by the dual-path processing module, which focus on overcoming different interference sources, are deeply fused. Specifically, the feature fusion module adopts a cross-attention mechanism, which allows the output features of the first path and the output features of the second path to perform mutual information retrieval and importance assessment.

[0024] Specifically, the output features of one path are used as the query vector, while the output features of the other path are used as the key and value vectors. A set of attention weights is obtained by calculating the similarity between the query and the key. These attention weights identify the locations and channels where the features from the other path can provide more valuable complementary information during the fusion process. Then, these weights are used to perform a weighted summation of the value vectors, achieving feature recalibration and integration.

[0025] Through this cross-attention interaction, the features from the two paths can be adaptively fused to generate a fused feature map. This fused feature map combines features corrected by the first path that are insensitive to skin color changes, with features purified by the second path that are less affected by wrinkles, thus forming a unified feature representation that is more discriminative and robust against café-au-lait spots.

[0026] Step S40: Input the fused feature map into the segmentation decoder, and after multi-scale context aggregation, output the segmentation region map of the milk coffee spots.

[0027] In this step, the segmentation decoder is the decoder part in a U-Net architecture that is structurally symmetrical to the encoder and receives the fused feature map described above. The segmentation decoder gradually restores the spatial resolution of the fused feature map through a series of operations such as upsampling, deconvolution, or transposed convolution, so that its size eventually matches the input image.

[0028] During the sampling process described above, the segmentation decoder can fuse multi-scale feature maps from the corresponding levels of the encoder stage through skip connections and other methods to achieve multi-scale context aggregation. This aggregation mechanism can combine shallow features containing fine spatial information with deep features containing rich semantic information, ensuring that while restoring the detailed contours of the target, the overall understanding of the lesion area is preserved.

[0029] The segmentation decoder outputs a segmentation map of café-au-lait spots of the same size as the input skin image, through pixel-by-pixel classification layers (e.g., convolutional layers followed by Softmax or Sigmoid activation functions). This segmentation map, in the form of a probability map or a binary map, identifies all pixel regions identified as café-au-lait spots, thus completing the automatic segmentation process.

[0030] This invention constructs a dual-path processing module to specifically suppress interference from skin color variations and skin wrinkles, and combines a cross-attention mechanism to achieve complementary feature fusion. Finally, multi-scale contextual information is integrated in the segmentation decoder, thereby significantly improving the accuracy, robustness, and generalization ability of the café-au-lait spot segmentation task in complex skin backgrounds.

[0031] As an example, the first path performs primary color estimation and transformation processing on the multi-scale hierarchical feature map, and outputs the first path output features, including: Step S201: Perform multi-scale feature fusion on the multi-scale hierarchical feature map to obtain the fused basic feature map; In this step, multi-scale hierarchical feature maps, including high-resolution shallow features and low-resolution deep features, are fused using a feature pyramid network or a similar fusion structure. For example, deep features can be upsampled to align with the spatial dimensions of shallow features, and then concatenated or added element-wise along the channel dimension to generate a base feature map that contains both detailed and semantic information. Understandably, this fusion operation ensures that subsequent primary color estimation can simultaneously reference both local skin tone details and global skin tone distribution.

[0032] Step S202: The basic feature map is input into the primary color estimation network. The primary color estimation network aggregates contextual information through a spatial attention mechanism to regress the primary color field that is spatially aligned with the input skin image. The primary color field is used to characterize the smooth skin color distribution after excluding interference from lesions and wrinkles. In this step, the primary color estimation network is preferably composed of a lightweight convolutional neural network, which introduces a spatial attention mechanism that can automatically learn and weight the importance of different regions in the skin image for primary color estimation.

[0033] Specifically, the primary color estimation network focuses more on large, uniform areas of normal skin, while suppressing the weights of lesion areas, highlights, shadows, and wrinkles. Through this adaptive aggregation of contextual information, the primary color estimation network can regress a primary color field that is spatially aligned with the pixel-level coordinates of the input image.

[0034] It is understandable that the aforementioned primary color field is a dense color vector field, and the RGB or Lab color value predicted at each spatial location represents the smooth, intrinsic skin tone of the region where that point is located under ideal conditions (i.e., after excluding interference such as milk coffee spots, wrinkles, etc.).

[0035] Step S203: Calculate the channel-level affine transformation parameters based on the primary color field, including scaling factor and offset factor; use the scaling factor and offset factor to perform a channel-by-channel affine transformation on the multi-scale hierarchical feature map to achieve skin color normalization in the feature space, and output the first path output feature.

[0036] In this step, based on the aforementioned primary color field, a small neural network or fully connected layer is used to calculate the transformation parameters for each feature channel of the multi-scale hierarchical feature map, namely the scaling factor γ and the offset factor β. It can be understood that these transformation parameters are channel-level, representing that each feature channel independently learns a set of rules on how to adjust its feature response according to the primary color.

[0037] Next, a channel-by-channel affine transformation is performed on the original multi-scale hierarchical feature map using the aforementioned transformation parameters. This operation can be represented as linearly scaling and translating the activation values ​​of each channel of the feature map. Through this transformation, skin color normalization is achieved in the feature space, that is, the features learned under different skin color backgrounds are dynamically corrected to a relatively uniform color representation space, thereby significantly reducing the model's sensitivity to skin color changes and improving its generalization ability.

[0038] The final output of the first path features is the feature representation that is robust to skin color variations after this adaptive color correction.

[0039] In this implementation, a primary color estimation basis containing global and local information is obtained through multi-scale feature fusion. The primary color field representing the intrinsic skin color is accurately regressed by combining spatial attention mechanism. Based on this, adaptive skin color normalization of the feature space is achieved through channel-level affine transformation, thereby effectively eliminating the interference of individual skin color differences on feature expression and significantly improving the model's generalization ability and segmentation accuracy on different skin color populations.

[0040] As an example, please refer to Figure 3 The primary color estimation network uses a spatial attention mechanism to weighted aggregate contextual information to regress a primary color field spatially aligned with the input skin image, including: Step S2021: The primary color estimation network performs multi-layer convolutional encoding on the input basic feature map to generate a high-level semantic feature map; a spatial attention mechanism is applied to the high-level semantic feature map to generate a spatial attention weight map; wherein, the spatial attention weight map is used to highlight the feature responses of normal skin areas and suppress the feature responses of lesion and wrinkle areas; In this step, the basic feature map is input into an encoder consisting of multiple convolutional layers for further feature abstraction. Specifically, each convolutional layer expands its receptive field progressively through stacking, extracting a high-level semantic feature map capable of understanding the global content and regional relationships of the image. Understandably, this high-level semantic feature map encodes key semantic information used to distinguish between normal skin, lesion areas, and wrinkled areas.

[0041] A spatial attention mechanism is applied to the aforementioned high-level semantic feature map. Specifically, this spatial attention mechanism automatically learns and generates a spatial attention weight map by analyzing the feature activation intensity at each spatial location in the high-level semantic feature map. It can be understood that each pixel value in this spatial attention weight map reflects the importance of the corresponding spatial location to the primary color estimation; that is, the primary color estimation network learns to assign higher weights to large, uniform, normal skin areas, while significantly suppressing the weights of lesion and wrinkled areas. Through this process, the primary color estimation network can intelligently focus on effective skin color reference areas, eliminating interference from abnormal areas.

[0042] Step S2022: The high-level semantic feature map is weighted and fused using the spatial attention weight map to aggregate global context information and output context enhancement features; the context enhancement features are upsampled and convolutionally decoded to regress the primary color field with the same resolution as the input skin image.

[0043] In this step, the spatial attention weight map obtained above is multiplied element-wise with the high-level semantic feature map to achieve weighted fusion. This operation selectively aggregates global contextual information based on attention weights to strengthen the features of useful regions and weaken the features of interfering regions, thereby outputting a clean feature representation with enhanced contextual information, i.e., context-enhanced features.

[0044] Then, the context-enhanced features are input into the decoder, which consists of a series of upsampling layers (such as bilinear interpolation or transposed convolution) and convolutional layers. Its task is to progressively recover the spatial resolution of the feature map and map it to the color space. Through the process of upsampling and convolutional decoding, the primary color estimation network finally regresses a pixel-level primary color field that is perfectly aligned in spatial size with the original input skin image.

[0045] In this implementation, high-level semantic features with discriminative power are extracted through multi-layer convolutional coding. The spatial attention mechanism is combined to accurately focus on normal skin areas and suppress interference from lesions and wrinkles. Then, a high-precision primary color field is generated through weighted fusion and decoding reconstruction, thereby obtaining a more accurate spatial adaptive skin color reference that can be used for subsequent feature transformation.

[0046] As an example, a multi-directional deformable convolutional kernel is used to extract the directional texture response from the multi-scale hierarchical feature map. A wrinkle saliency mask is generated based on the directional texture response. The wrinkle saliency mask is then used to perform a gating suppression operation on the multi-scale hierarchical feature map to generate a second path output feature, including: Step S204: Anisotropic convolution operation is performed on the multi-scale hierarchical feature map using a multi-directional deformable convolution kernel to extract directional texture response feature map; In this step, the multi-directional deformable convolutional kernel is constructed by setting multiple initial directions (such as 5°, 90°, 135°, etc.) in parallel on the basis of deformable convolution. Unlike standard convolution with a fixed sampling grid, this deformable convolutional kernel can learn autonomously through the network and adaptively adjust the spatial position of each convolution sampling point, thereby flexibly conforming to irregular shapes such as bending and twisting of wrinkles; while the multi-directional design ensures comprehensive coverage of wrinkle textures at different angles, avoiding the omission of wrinkle features at specific angles by a single-directional convolutional kernel.

[0047] During operation, multi-directional deformable convolutional kernels perform convolution operations on the input multi-scale hierarchical feature maps containing high-resolution shallow detail features and low-resolution deep semantic features. Specifically: for shallow feature maps, fine texture details of folds are captured through flexible sampling (such as edge gray-level gradients and density distribution of fine folds); for deep feature maps, the overall extension direction and cluster distribution pattern of folds are captured by combining a large receptive field. The deformable convolution in each direction outputs a texture response feature map for the corresponding direction. Finally, all feature maps in all directions are concatenated along the channel dimension to form a directional texture response feature map.

[0048] It is understandable that each channel of the directional texture response feature map corresponds to a wrinkle texture response in one direction, and the feature activation intensity within the channel is positively correlated with the probability that there is a wrinkle in the corresponding direction at that location.

[0049] Step S205: Perform multi-scale feature aggregation and nonlinear activation on the directional texture response feature map to generate the wrinkle saliency mask; wherein, the values ​​in the wrinkle saliency mask are positively correlated with the probability of wrinkle existence; In this step, to address the scale differences in skin folds (such as fine, small folds versus wide, deep folds), a feature pyramid network or a similar multi-scale fusion structure is used to aggregate directional texture response feature maps at multiple scales. Specifically, the directional texture response feature maps are downsampled at multiple scales (to capture the global semantic information of large-scale folds) and upsampled at multiple scales (to preserve the local details of small-scale folds). After aligning the spatial resolution of feature maps at different scales through interpolation, they are concatenated along the channel dimension or summed element-wise with weights, thereby obtaining an aggregated feature map that combines the global distribution of large-scale folds with the local details of small-scale folds, avoiding missed or false positives caused by differences in fold scale.

[0050] Subsequently, the aggregated feature map is input into a lightweight convolutional network (e.g., composed of stacked 1×1 and 3×3 convolutions) to perform feature dimension compression and semantic enhancement, so as to highlight the feature differences between wrinkled and non-wrinkled regions. Then, the enhanced feature map is mapped through a non-linear activation function (such as the Sigmoid function) to output a wrinkle saliency mask that is perfectly aligned with the spatial resolution of the input skin image.

[0051] The wrinkle saliency mask is a single-channel feature map with pixel values ​​ranging from [0,1]. The value is positively correlated with the probability of wrinkles existing at the corresponding spatial location. That is, the closer the value is to 1, the higher the probability that the location is a wrinkled area; the closer the value is to 0, the higher the probability that the location is normal skin or a café-au-lait spot lesion area, thus providing a clear inhibition signal for subsequent gating inhibition operations.

[0052] Step S206: The wrinkle saliency mask is used as a gating signal and multiplied element-wise with the multi-scale hierarchical feature map to suppress the feature intensity of high wrinkle response regions and generate the second path output feature.

[0053] This step, based on a wrinkle saliency mask, selectively attenuates features from the original multi-scale hierarchical feature map, reducing interference from wrinkle pseudo-features while preserving the effective features of normal skin and lesion areas to the greatest extent.

[0054] Specifically, the wrinkle saliency mask is smoothed using Gaussian filtering and other methods to eliminate jitter noise at the mask edges and avoid local feature distortion. The smoothed wrinkle saliency mask is then used as a gating signal and multiplied element-wise with the original multi-scale hierarchical feature map. The essence of this operation is to dynamically adjust the activation intensity at each position in the feature map through the mask value. Its mathematical expression can be simplified as: Feature map activation value = Original activation value × (1 - Mask pixel value).

[0055] Through the above-described gating suppression operation, for high-probability wrinkle regions with mask values ​​close to 1, the activation intensity is significantly attenuated (e.g., when the mask value is 0.9, the feature activation value retains only 10% of the original value), thereby effectively suppressing wrinkle-related pseudo-features. For low-probability wrinkle regions (normal skin or lesion areas) with mask values ​​close to 0, their feature activation intensity is basically unaffected, ensuring that effective features are completely preserved.

[0056] The final output of the second path features is the pure feature representation after wrinkle pseudo-feature suppression.

[0057] In this embodiment, multi-directional deformable convolution kernels are used to accurately capture wrinkle texture features with arbitrary orientation. Multi-scale feature aggregation and nonlinear activation are combined to generate a high-precision wrinkle saliency mask. Furthermore, a gating inhibition mechanism is used to effectively attenuate the feature response of wrinkle regions, thereby significantly reducing the interference of skin wrinkles on lesion segmentation and improving the specificity and accuracy of café-au-lait spot segmentation.

[0058] As an example, anisotropic convolution operations are performed on the multi-scale hierarchical feature maps using multi-directional deformable convolution kernels to extract directional texture response feature maps, including: Step S2041: Based on the local context of the multi-scale hierarchical feature map, the flanking network of the deformable convolution module predicts the corresponding spatial offset field and modulation coefficient field for each main direction base convolution kernel in the multi-directional deformable convolution kernel group. In this step, although deformable convolution can achieve sampling point shifting, it lacks a specific design for skin folds, which have strong directional features. To address this, this invention employs a dual control mechanism of spatial offset field and modulation coefficient field, enabling the convolution kernel to not only adapt to the geometry of the folds but also dynamically adjust the receptive field weights according to the salience of the fold features, thereby achieving pixel-level precise response to skin folds of different angles, curvatures, and intensities.

[0059] Specifically, the flanking network employs a lightweight architecture consisting of three consecutive convolutional layers. The first layer uses a 5×5 convolutional kernel to expand the receptive field, while the latter two layers use 3×3 convolutions for fine-tuning. Using the multi-scale hierarchical feature maps extracted in the previous stage as input, this flanking network, through end-to-end learning, predicts the following two sets of key parameters for each spatial location using the principal orientation basis convolutional kernel: The spatial offset field is a vector field that predicts a two-dimensional offset vector (Δx, Δy) for each sampling point of the convolutional kernel at each spatial location. Its output channel count is 2N (N being the number of principal directions), with each principal direction corresponding to a two-dimensional offset vector field. This field uses the tanh activation function to control the offset range between [-1, 1], and then multiplies it by a preset maximum offset (e.g., 3 pixels) to achieve controllable deformation of the sampling points. For example, when processing curved wrinkles, the network predicts the offset that causes the sampling points to distribute along the wrinkle curve.

[0060] For example: Suppose there is a base convolution kernel with a principal direction of 0° (horizontal), whose standard 3x3 sampling grid (9 points) is a regular rectangle. If we need to process a downward-curving arc-shaped fold, using a fixed grid would cause most sampling points to fall on the top and bottom sides of the fold, failing to accurately capture the curved pixel lines that make up the fold.

[0061] When using a spatial offset field, for a pixel located on the curved fold in the image, the spatial offset field predicted by the flanking network will generate a set of offsets (Δx, Δy) for the convolutional kernel in the 0° direction. For sampling points at the top, a downward positive Δy offset may be predicted; for sampling points at the bottom, an upward negative Δy offset may be predicted. At the same time, some small horizontal Δx offsets will be used to adapt to the curvature.

[0062] After spatial offset, the nine sampling points are no longer positioned as a rectangular grid, but rather form a curve that matches the geometry of the curved fold. This allows the convolution operation to sample features directly from the fold itself, resulting in a strong response and accurately capturing the features of the curved fold.

[0063] The modulation coefficient field is a scalar field that predicts a weighted modulation coefficient in the range [0,1] for each sampling point of the convolutional kernel at each spatial location. It has N output channels, and the values ​​are constrained to the [0,1] interval by the sigmoid function. The physical meaning of this coefficient is that when a sampling point is located at a critical position on the edge of a fold, the network outputs a coefficient close to 1 to enhance the feature contribution of that point; when the sampling point falls in a noisy region, it outputs a coefficient close to 0 to suppress interference.

[0064] For example: the sampling points of the same dynamic convolution kernel in the 0° direction have been correctly distributed by the spatial offset field on a horizontal fold and the flat skin area next to it.

[0065] Sampling points A, B, and C: precisely fall on the clear upper edge of the horizontal fold.

[0066] Sampling points D, E, and F: Shaded areas falling inside the folds.

[0067] Sampling points G, H, and I: fell on flat, normal skin next to the folds.

[0068] For each of the above sampling points, the modulation coefficient field will be assigned a different coefficient m.

[0069] For points A, B, and C: because they are located at the critical edges defining the shape of the folds, they are assigned very high modulation coefficients, for example, m≈0.9-1.0. Consequently, the feature contributions of these sampling points are fully or nearly fully preserved.

[0070] For points D, E, and F: since they are located inside the folds, although they are related, their discriminative power is not as good as that of the edges, and they may be assigned a moderate coefficient, such as m≈0.5-0.7.

[0071] For points G, H, and I: since they are located on normal skin and represent interference information, they are assigned extremely low modulation coefficients, for example, m≈0.0-0.2. Consequently, the feature contributions of these sampling points are significantly suppressed and almost ignored.

[0072] Under the influence of the modulation coefficient field, the result of the convolution operation is mainly dominated by the key edge points (A,B,C), with the internal points (D,E,F) as auxiliary, while the influence of noise points (G,H,I) is minimized. This is equivalent to making the convolution kernel focus only on the key structure of the folds, while ignoring unimportant positions.

[0073] Step S2042: The sampling position of each principal direction base convolution kernel is dynamically offset using the spatial offset field, and the weight of each sampling position is adaptively modulated using the modulation coefficient field to generate a dynamic convolution kernel group that matches the local wrinkled texture geometry. In this step, taking a group of deformable convolutional kernels with 8 main directions as an example, each base convolutional kernel initially has 9 sampling points arranged in a 3×3 grid: For dynamic offset of sampling positions, specifically: based on the spatial offset field, each sampling point moves from its standard grid position (p0) to a new position (p0+Δp). It should be noted that eigenvalues ​​for non-integer positions are calculated using bilinear interpolation to ensure gradient backtracking. For example, when dealing with folds curving at 45°, the network automatically adjusts the sampling points to the optimal observation position along the fold's direction.

[0074] For weighted adaptive modulation, specifically: the original weight w of each sampling point is multiplied by the corresponding modulation coefficient λ to obtain the modulated weight w' = w × λ. This allows the convolution kernel to perform self-attention-based weight allocation based on the importance of local features. Specifically, sampling points at the edges of wrinkles are assigned λ≈1, while sampling points in flat skin regions are assigned λ≈0.2-0.5, achieving feature-selective convolution calculation.

[0075] Through the above two levels of adjustment, a dynamic convolution kernel that highly matches the local wrinkle texture is finally generated, and its effective receptive field can adaptively fit the wrinkle structure of different shapes.

[0076] Step S2044: The dynamic convolution kernel group is used to perform convolution calculations with the multi-scale hierarchical feature maps to generate multiple sets of directional response maps. Competitive activation and selective fusion of the channel dimensions are performed on the multiple sets of directional response maps to aggregate and generate the directional texture response feature map.

[0077] In this step, dynamic convolutional kernel groups are used to perform convolution calculations to generate feature response maps in N directions. Each response graph It is a two-dimensional feature matrix, where the numerical value represents the probability that a fold exists in a corresponding direction at that spatial location. For example, the response map in the 0° direction shows high activation values ​​at horizontal folds, while the response map in the 45° direction shows high activation values ​​at diagonal folds.

[0078] Among them, the competitive activation mechanism achieves feature selection through spatial-channel dual competition, specifically: Spatial domain competition: Perform global max pooling on the response map for each direction to obtain the maximum response value M for that direction across the entire map. i , Among them, M i The global significance of the fold in the i-th direction is represented by h, where h is the directional response map R. i In the spatial coordinate index along the height direction, w represents the directional response map R. i Spatial coordinate index in the width direction.

[0079] Channel domain competition: The maximum response values ​​in each direction are used to calculate the attention weights using the softmax function.

[0080] in, This is a temperature coefficient (e.g., set to 1.0) used to control the sharpness of the weight distribution.

[0081] Selective fusion employs a multi-layered strategy, as follows: The response maps for each direction are multiplied by their corresponding attention weights to enhance the dominant feature. The specific calculation formula is as follows:

[0082] Simultaneously, the average value of the response maps in each direction is calculated as a background feature to preserve detailed features. The specific calculation formula is as follows:

[0083] Finally, the weighted dominant features are integrated with the background features. The specific calculation formula is as follows:

[0084] in, To retain coefficients (e.g., 0.2-0.5), used to balance the importance of dominant features and background features.

[0085] In this implementation, the spatial offset field and modulation coefficient field are accurately predicted by the flanking network, enabling the convolution kernel to dynamically adapt to the geometric shape and feature intensity of different wrinkles. Then, the dominant directional features are highlighted by the competitive activation mechanism, and the multi-directional details are retained by selective fusion, thereby achieving pixel-level accurate representation of skin wrinkles and significantly improving the distinguishability and robustness of directional texture features.

[0086] Please see Figure 4 This invention also provides a deep learning-based lesion image processing system 200, comprising: The receiving unit 2001 is configured to: acquire a skin image and extract its multi-scale hierarchical feature map through an encoder; The multi-scale hierarchical unit 2002 is configured to input the multi-scale hierarchical feature map into a dual-path processing module, the dual-path processing module comprising: The first path is configured to perform primary color estimation and transformation processing on the multi-scale hierarchical feature map to generate the first path output feature; The second path is configured to extract the directional texture response from the multi-scale hierarchical feature map using a multi-directional deformable convolutional kernel, generate a wrinkle saliency mask based on the directional texture response, and perform a gating suppression operation on the multi-scale hierarchical feature map using the wrinkle saliency mask to generate the second path output feature. The fusion processing unit 2003 is configured to: input the output features of the first path and the output features of the second path into the feature fusion module, perform weighted fusion based on the cross attention mechanism, and generate a fused feature map; The segmentation processing unit 2004 is configured to input the fused feature map into the segmentation decoder, and after multi-scale context aggregation, output a segmentation region map of the milk coffee spots.

[0087] As an example, the multi-scale hierarchical unit 2002 is specifically configured as follows: Multi-scale feature fusion is performed on the multi-scale hierarchical feature map to obtain the fused basic feature map; The basic feature map is input into a primary color estimation network, which uses a spatial attention mechanism to weighted aggregate contextual information to regress a primary color field that is spatially aligned with the input skin image; wherein, the primary color field is used to characterize the smooth skin color distribution after excluding interference from lesions and wrinkles; Based on the primary color field, channel-level affine transformation parameters are calculated, including scaling factors and offset factors; using the scaling factors and offset factors, channel-by-channel affine transformation is performed on the multi-scale hierarchical feature map to achieve skin color normalization in the feature space, and the first path output features are output.

[0088] As an example, the multi-scale hierarchical unit 2002 is specifically configured as follows: The primary color estimation network performs multi-layer convolutional encoding on the input basic feature map to generate a high-level semantic feature map; a spatial attention mechanism is applied to the high-level semantic feature map to generate a spatial attention weight map; wherein, the spatial attention weight map is used to highlight the feature responses of normal skin areas and suppress the feature responses of lesion and wrinkle areas; The high-level semantic feature map is weighted and fused using the spatial attention weight map to aggregate global contextual information and output context-enhanced features; the context-enhanced features are upsampled and convolutionally decoded to regress the primary color field with the same resolution as the input skin image.

[0089] As an example, the multi-scale hierarchical unit 2002 is specifically configured as follows: Anisotropic convolution operations are performed on the multi-scale hierarchical feature maps using multi-directional deformable convolution kernels to extract directional texture response feature maps; Multi-scale feature aggregation and nonlinear activation are performed on the directional texture response feature map to generate the wrinkle saliency mask; wherein, the values ​​in the wrinkle saliency mask are positively correlated with the probability of wrinkle existence; The wrinkle saliency mask is used as a gating signal and multiplied element-wise with the multi-scale hierarchical feature map to suppress the feature intensity of high wrinkle response regions and generate the second path output feature.

[0090] As an example, the multi-scale hierarchical unit 2002 is specifically configured as follows: in step S2041, the flanking network of the deformable convolution module predicts the corresponding spatial offset field and modulation coefficient field for each main direction base convolution kernel in the multi-directional deformable convolution kernel group based on the local context of the multi-scale hierarchical feature map. The sampling positions of each principal direction base convolution kernel are dynamically offset using the spatial offset field, and the weights of each sampling position are adaptively modulated using the modulation coefficient field to generate dynamic convolution kernels that match the local wrinkled texture geometry. The dynamic convolution kernel group is used to perform convolution calculations with the multi-scale hierarchical feature map to generate multiple sets of directional response maps. Competitive activation and selective fusion of the channel dimensions are performed on the multiple sets of directional response maps to aggregate and generate the directional texture response feature map.

[0091] This invention also provides a computer program product, including a computer program or instructions, which are executed by a processor to implement the method described in any of the preceding embodiments.

[0092] This invention also provides a computer device including a memory, a processor, and a computer program or instructions stored in the memory, wherein the computer program or instructions are executed by the processor to implement the method as described in any of the preceding embodiments.

[0093] This invention also provides a computer-readable storage medium storing a computer program or instructions, which are executed by a processor to implement the method described in any of the preceding embodiments.

[0094] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A deep learning-based method for processing lesion images, characterized in that, The methods and steps include the following: Step S10: Obtain the skin image and extract its multi-scale hierarchical feature map through the encoder; Step S20: Input the multi-scale hierarchical feature map into the dual-path processing module, which includes: The first path is configured to perform primary color estimation and transformation processing on the multi-scale hierarchical feature map to generate the first path output feature; The second path is configured to extract the directional texture response from the multi-scale hierarchical feature map using a multi-directional deformable convolutional kernel, generate a wrinkle saliency mask based on the directional texture response, and perform a gating suppression operation on the multi-scale hierarchical feature map using the wrinkle saliency mask to generate the second path output feature. Step S30: Input the first path output features and the second path output features into the feature fusion module, perform weighted fusion based on the cross-attention mechanism, and generate a fused feature map; Step S40: Input the fused feature map into the segmentation decoder, and after multi-scale context aggregation, output the segmentation region map of the milk coffee spots; The first path performs primary color estimation and transformation processing on the multi-scale hierarchical feature map, and outputs the first path output features, including: Step S201: Perform multi-scale feature fusion on the multi-scale hierarchical feature map to obtain the fused basic feature map; Step S202: The basic feature map is input into the primary color estimation network. The primary color estimation network aggregates contextual information through a spatial attention mechanism to regress the primary color field that is spatially aligned with the input skin image. The primary color field is used to characterize the smooth skin color distribution after excluding interference from lesions and wrinkles. Step S203: Calculate the channel-level affine transformation parameters based on the primary color field, including scaling factor and offset factor; use the scaling factor and offset factor to perform a channel-by-channel affine transformation on the multi-scale hierarchical feature map to achieve skin color normalization in the feature space and output the first path output feature. This primary color estimation network uses a spatial attention mechanism to weighted aggregate contextual information to regress a primary color field spatially aligned with the input skin image, including: Step S2021: The primary color estimation network performs multi-layer convolutional encoding on the input basic feature map to generate a high-level semantic feature map; a spatial attention mechanism is applied to the high-level semantic feature map to generate a spatial attention weight map; wherein, the spatial attention weight map is used to highlight the feature responses of normal skin areas and suppress the feature responses of lesion and wrinkle areas; Step S2022: The high-level semantic feature map is weighted and fused using the spatial attention weight map to aggregate global context information and output context enhancement features; the context enhancement features are upsampled and convolutionally decoded to regress the primary color field with the same resolution as the input skin image.

2. The lesion image processing method based on deep learning according to claim 1, characterized in that: A multi-directional deformable convolutional kernel is used to extract the directional texture response from the multi-scale hierarchical feature map. A wrinkle saliency mask is generated based on the directional texture response. The wrinkle saliency mask is then used to perform gated suppression on the multi-scale hierarchical feature map to generate a second path output feature, including: Step S204: Anisotropic convolution operation is performed on the multi-scale hierarchical feature map using a multi-directional deformable convolution kernel to extract directional texture response feature map; Step S205: Perform multi-scale feature aggregation and nonlinear activation on the directional texture response feature map to generate the wrinkle saliency mask; wherein, the values ​​in the wrinkle saliency mask are positively correlated with the probability of wrinkle existence; Step S206: The wrinkle saliency mask is used as a gating signal and multiplied element-wise with the multi-scale hierarchical feature map to suppress the feature intensity of high wrinkle response regions and generate the second path output feature.

3. The lesion image processing method based on deep learning according to claim 2, characterized in that: Anisotropic convolution operations are performed on the multi-scale hierarchical feature maps using multi-directional deformable convolution kernels to extract directional texture response feature maps, including: Step S2041: Based on the local context of the multi-scale hierarchical feature map, the flanking network of the deformable convolution module predicts the corresponding spatial offset field and modulation coefficient field for each main direction base convolution kernel in the multi-directional deformable convolution kernel group. Step S2042: The sampling position of each principal direction base convolution kernel is dynamically offset using the spatial offset field, and the weight of each sampling position is adaptively modulated using the modulation coefficient field to generate a dynamic convolution kernel group that matches the local wrinkled texture geometry. Step S2043: The dynamic convolution kernel group is used to perform convolution calculations with the multi-scale hierarchical feature map to generate multiple sets of directional response maps. Competitive activation and selective fusion of the channel dimensions are performed on the multiple sets of directional response maps to aggregate and generate the directional texture response feature map.

4. A deep learning-based lesion image processing system for implementing the method of any one of claims 1-3, characterized in that: include: The receiving unit is configured to: acquire a skin image and extract its multi-scale hierarchical feature map through an encoder; The multi-scale hierarchical unit is configured to input the multi-scale hierarchical feature map into a dual-path processing module, the dual-path processing module comprising: The first path is configured to perform primary color estimation and transformation processing on the multi-scale hierarchical feature map to generate the first path output feature; The second path is configured to extract the directional texture response from the multi-scale hierarchical feature map using a multi-directional deformable convolutional kernel, generate a wrinkle saliency mask based on the directional texture response, and perform a gating suppression operation on the multi-scale hierarchical feature map using the wrinkle saliency mask to generate the second path output feature. The fusion processing unit is configured to: input the output features of the first path and the output features of the second path into the feature fusion module, perform weighted fusion based on the cross attention mechanism, and generate a fused feature map; The segmentation processing unit is configured to input the fused feature map into the segmentation decoder, and after multi-scale context aggregation, output a segmentation region map of the milk coffee spots.

5. A computer program product, comprising a computer program or instructions, characterized in that, The computer program or instructions are executed by a processor to implement the method described in any one of claims 1-3.

6. A computer device comprising a memory, a processor, and a computer program or instructions stored in the memory, characterized in that, The computer program or instructions are executed by a processor to implement the method described in any one of claims 1-3.

7. A computer-readable storage medium storing a computer program or instructions, characterized in that, The computer program or instructions are executed by a processor to implement the method described in any one of claims 1-3.

Images