A vitiligo auxiliary diagnosis method and device for multi-modal medical image collaborative segmentation and classification and a storage medium

Abstract

The embodiment of the application discloses a kind of vitiligo auxiliary diagnosis methods, devices and storage medium of multi-modal medical image collaborative segmentation and classification, wherein the method comprises: obtaining the multi-modal image pair of the clinical image and Wood lamp image of the same examinee;According to the imaging characteristics of two kinds of modalities, the image pair is preprocessed and modality-specific data enhancement;The multi-modal image pair after processing is input into the feature extraction network to obtain each modality feature, and spatial guidance information for subsequent segmentation is generated;Each modality feature is input into vector quantization fusion module for cross-modal feature fusion to obtain semantic consistent fusion feature;Based on the fusion feature, a segmentation branch and a classification branch are constructed to realize the joint output of lesion region segmentation and disease activity classification, and the collaborative effect of the two tasks is improved through inter-task interaction. By using the present application, end-to-end joint optimization of vitiligo lesion segmentation and disease activity classification can be realized, and the diagnostic performance is improved.

Description

Technical Field

[0001] This application relates to the fields of medical image processing, computer vision and artificial intelligence, specifically to a multimodal auxiliary diagnostic method and system for skin diseases, and more particularly to an auxiliary diagnostic method, device and storage medium for vitiligo using multimodal medical image collaborative segmentation and classification. Background Technology

[0002] Vitiligo is a common skin disease characterized primarily by the loss of skin pigmentation. Its clinical manifestations, lesion boundaries, and activity are significantly influenced by individual differences, lighting conditions, and background skin conditions. In clinical diagnosis and treatment, doctors typically combine clinical visible light imaging with Wood's lamp imaging for comprehensive assessment: clinical images provide information on the appearance, texture details, and morphological structure of the lesions, while Wood's lamp imaging, under specific lighting conditions, highlights pigmentation changes and enhances the contrast and visibility of lesion boundaries. By leveraging the complementary information from multimodal imaging, the limitations of single-modal diagnosis can be reduced to some extent, improving the reliability of locating the extent of vitiligo and assessing the disease's progression. This makes it valuable in follow-up evaluation, efficacy observation, and activity assessment.

[0003] Currently, deep learning-based methods for assisting in the diagnosis of vitiligo mainly focus on tasks such as lesion segmentation and disease state classification. Related multimodal fusion techniques can be broadly categorized into three paradigms: The first is the shallow direct fusion paradigm, which typically involves splicing or weighting features from different modalities before inputting them into the subsequent network. While simple to implement, it lacks an explicit cross-modal semantic alignment mechanism, easily leading to feature redundancy or weakening of key clues, making it difficult to fully leverage the advantages of Wood's lamp imaging in identifying boundaries and hidden lesions. The second is the attention-driven fusion paradigm, which establishes cross-modal dependencies through attention mechanisms to achieve alignment and complementary information. Information selection is an option, but such methods often have high computational costs, and training stability is significantly affected by the scale of data. Furthermore, when there are fluctuations in clinical image quality, noise interference, or significant modal differences, the fusion results are prone to semantic shifts. The third type is the multi-task joint learning paradigm, which achieves joint output of "localization + discrimination" by sharing an encoder and setting segmentation and classification branches. This can improve the overall diagnostic consistency, but in multimodal scenarios, fusion is often regarded as an independent link. There is a lack of fusion representation constraints oriented towards multi-task objectives. When the requirements of segmentation and classification for fusion features are inconsistent, problems such as mutual constraints between tasks and inconsistent optimization directions can easily occur.

[0004] Furthermore, multimodal vitiligo imaging presents challenges in actual acquisition, including significant differences in imaging mechanisms, alignment errors, background interference, and blurred boundaries. Existing methods largely rely on fusion and alignment within a continuous feature space, making it difficult to suppress semantic drift caused by modal differences at the representation level. This results in fused features failing to form stable and consistent semantic correspondences across different modalities, thus affecting the accuracy and repeatability of fine segmentation of lesion boundaries and disease activity assessment.

[0005] Therefore, how to fully utilize the complementary information of clinical images and Wood's lamp images, achieve semantic consistency fusion of cross-modal features, and form synergistic optimization with multi-task objectives such as segmentation and classification to improve the overall performance and stability of vitiligo lesion localization and activity discrimination remains a key technical problem that urgently needs to be solved in the field of multimodal auxiliary diagnosis of vitiligo. Summary of the Invention

[0006] In a first aspect, embodiments of this application provide a method, apparatus, and storage medium for auxiliary diagnosis of vitiligo using multimodal medical image collaborative segmentation and classification. The method includes: acquiring a multimodal image pair consisting of clinical images and Wood's lamp images of the same subject; preprocessing and modality-specific data augmentation of the image pair based on the imaging characteristics of the two modalities; inputting the processed multimodal image pair into a feature extraction network to obtain modal features and generating spatial guidance information for subsequent segmentation; performing cross-modal feature fusion on the input vector quantization and fusion module of each modal feature to obtain semantically consistent fusion features; constructing segmentation and classification branches based on the fusion features to achieve joint output of lesion region segmentation and disease activity classification, and improving the collaborative effect of the two tasks through inter-task interaction.

[0007] Secondly, embodiments of this application provide a multimodal auxiliary diagnostic method for vitiligo, the method comprising: The data acquisition module is used to acquire multimodal skin image data of the same subject, including clinical images and Wood's lamp images; The feature extraction module is used to input the multimodal skin image data into a pre-trained diagnostic model, perform image preprocessing and modality-specific data augmentation on the clinical images and Wood's lamp images respectively, and extract the initial features corresponding to each modality. The spatial guidance module is used to generate spatial guidance information for lesion localization based on the initial features, so as to highlight the suspected vitiligo area and suppress background interference, and provide regional priors for subsequent segmentation tasks; The vector quantization fusion module is used to input the clinical image features and Wood's lamp image features into the multimodal multi-task vector quantization fusion network. Through vector quantization representation and cross-modal feature interaction, a semantically consistent fusion representation is achieved, generating fusion features that simultaneously contain structural details and lesion discrimination clues. The multi-task diagnostic module is used to construct segmentation and classification branches based on the fusion features, output lesion region segmentation results and disease activity classification results, and improve the synergistic effect of segmentation and classification through inter-task information interaction to achieve auxiliary diagnosis of vitiligo.

[0008] Thirdly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the vitiligo multimodal auxiliary diagnosis method described in the first aspect.

[0009] Fourthly, embodiments of this application provide an electronic device hardware system, including: A multimodal image acquisition unit is used to acquire clinical images and Wood's lamp images of the same subject and generate paired multimodal image data. The multimodal image acquisition unit is used to perform object identification association or synchronous acquisition control on the two types of images to ensure a consistent correspondence between clinical images and Wood's lamp images in terms of the subject and acquisition time. The core computing unit, connected to the multimodal image acquisition unit, includes a central processing unit and a parallel processor. The parallel processor can be a graphics processing unit or a neural network accelerator, and is configured to efficiently perform image preprocessing, multimodal feature extraction, vector quantization fusion calculation, and multi-task joint inference to output lesion region segmentation results and disease activity classification results, and generate diagnostic information for display. The memory system, connected to the core computing unit, is used to store computer programs, pre-trained model parameters, multimodal image data, and data required for diagnostic result output; wherein, the data required for diagnostic result output includes at least segmentation results, classification results, and their corresponding record information; The communication unit, connected to the core computing unit, is used to realize image data transmission, diagnostic result reporting, or interaction with external terminals, including doctor workstations or mobile terminals.

[0010] By implementing the embodiments of this application, the overall accuracy and consistency of multimodal assisted diagnosis of vitiligo can be improved, especially in scenarios with fluctuating image quality, unclear boundaries, or strong background interference, where it exhibits more stable diagnostic performance. Its technical advantages are specifically reflected in the following two aspects: On the one hand, this method mitigates the impact of imaging differences between clinical images and Wood's lamp images on diagnostic representation. By introducing a vector quantization fusion mechanism, key representations from different modalities are mapped to a unified representation space, and cross-modal semantic correspondences are strengthened during the fusion process, resulting in a more consistent representation of the same lesion region by both modalities. This mechanism preserves the structural texture and morphological details of clinical images while fully utilizing the advantages of Wood's lamp images in highlighting pigment changes and lesion boundaries, thereby reducing semantic shifts caused by modal differences and improving the stability and usability of the fused features.

[0011] On the other hand, this method achieves synergistic optimization of "lesion localization" and "disease assessment." By constructing a multi-task joint learning framework for segmentation and classification, pixel-level lesion segmentation results provide more focused lesion evidence for classification. Simultaneously, the global semantic information learned by the classification task inversely constrains the regional consistency and boundary discrimination of the segmentation branches, reducing issues such as missed detections, missegmentation, and boundary fragmentation. Through inter-task information interaction and joint reasoning, this method can more fully exploit the complementary value of multimodal information, thereby simultaneously improving both lesion segmentation accuracy and the ability to classify disease activity.

[0012] In summary, the embodiments of this application, by making full use of the complementary information of clinical images and Wood's lamp images, have achieved consistent fusion of cross-modal representations and integrated output of multi-task diagnosis, which improves the reliability and interpretability of vitiligo auxiliary diagnosis and facilitates its use in clinical follow-up evaluation, efficacy comparison and decision support applications. Attached Figure Description

[0013] Figure 1 This is a schematic diagram of a method, device, and storage medium for the auxiliary diagnosis of vitiligo using multimodal medical image collaborative segmentation and classification, provided in an embodiment of this application.

[0014] Figure 2 This is a schematic diagram of the algorithm flow for a multimodal medical image collaborative segmentation and classification method, device and storage medium for the auxiliary diagnosis of vitiligo.

[0015] Figure 3 This is a schematic diagram of the algorithm flow for the data augmentation module, feature extraction module, spatial guidance module, vector quantization fusion module, and multi-task diagnostic module.

[0016] Figure 4 This is a schematic diagram of a multimodal medical image collaborative segmentation and classification method for the auxiliary diagnosis of vitiligo.

[0017] Figure 5 This is a schematic diagram of an electronic device hardware system for the auxiliary diagnosis of vitiligo using multimodal medical image collaborative segmentation and classification. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0019] With the increasing demand for skin imaging equipment and intelligent assisted diagnosis, clinical requirements for the accuracy and consistency of vitiligo diagnosis and follow-up assessment are constantly rising. Among these, lesion extent determination, boundary identification, and activity status assessment are crucial aspects affecting treatment plan selection and efficacy evaluation, directly impacting the reliability and repeatability of clinical decisions. However, due to factors such as differences in the examined site, skin color background, shooting angle, and lighting conditions, lesion boundaries in clinical images often appear blurred or lack sufficient contrast with surrounding skin. Furthermore, while Wood's lamp imaging can enhance the contrast between pigment changes and lesions, it can also be affected by imaging conditions, noise interference, and individual differences, leading to information discrepancies between modalities. Therefore, fully utilizing the complementary characteristics of clinical imaging and Wood's lamp imaging to construct a stable and reliable multimodal assisted diagnostic method, and achieving joint output of lesion segmentation and activity classification, has become an important direction for improving the quality of vitiligo diagnosis and clinical efficiency.

[0020] Currently, the interpretation of vitiligo images and the assessment of the condition still largely rely on doctors' experience and manual observation and comparison. While this method can identify typical white patches, it is prone to missed or misdiagnosis when the lesion boundaries are unclear, early lesions are hidden, or pigment changes are slight. The delineation of the lesion extent and follow-up comparison often rely on subjective estimation, making it difficult to achieve a precise, stable, and quantifiable assessment. At the same time, differences in doctors' experience, observation habits, and changes in data collection conditions may lead to inconsistent diagnostic conclusions, affecting the reliability of efficacy monitoring. In scenarios with a large number of outpatient visits or frequent follow-ups, the efficiency of manual interpretation is also difficult to maintain.

[0021] Based on this, embodiments of this application provide a method, device, and storage medium for auxiliary diagnosis of vitiligo using multimodal medical image collaborative segmentation and classification. By fusing complementary information from clinical images and Wood's lamp images, and employing multi-task joint reasoning to simultaneously output lesion segmentation results and disease activity classification results, automated and standardized auxiliary diagnosis of vitiligo is achieved, thereby improving the precision of lesion localization and the accuracy of disease assessment.

[0022] Please see Figure 1 , Figure 1 This is a schematic diagram of a method, device, and storage medium for the auxiliary diagnosis of vitiligo using multimodal medical image collaborative segmentation and classification, provided in an embodiment of this application.

[0023] S201, acquire a multimodal image pair consisting of clinical images and Wood's lamp images of the same subject.

[0024] Clinical images provide visible information such as skin appearance, texture, and details; Wood's lamp images highlight characteristic information such as pigmentation changes and lesion boundaries. By acquiring paired data of the same pair of 77 images, a consistent spatial and lesion correspondence is ensured between different modalities, facilitating subsequent effective fusion and joint diagnosis.

[0025] S202, the image pairs are preprocessed and modality-specific data augmented according to the imaging characteristics of the two modalities respectively.

[0026] The preprocessing is used to unify image size, quality, and input format, reducing interference caused by differences in acquisition conditions. Specifically, clinical images and Wood's lamp images can be uniformly adjusted to a preset resolution and channel format, and the images can be cropped, normalized, denoised, or have brightness and contrast corrected to reduce imaging differences caused by different devices, shooting distances, and exposure conditions. At the same time, object identification association and paired tissues can be performed on the two modal images, so that the images of the same subject in the two modalities maintain a corresponding relationship in terms of acquisition time and lesion area, which facilitates subsequent cross-modal feature alignment and fusion learning.

[0027] The modality-specific data augmentation adapts to the imaging differences between clinical images and Wood's lamp images, maintaining the structural and textural stability of clinical images while enhancing the recognizability of lesion boundaries and key regions in Wood's lamp images, thereby improving the model's adaptability to data acquired under different conditions. Specifically, data augmentation can include two categories: geometric augmentation and appearance augmentation. Geometric augmentation can employ random horizontal flipping, random cropping, rotation, affine transformation, and perspective transformation, and applies the same geometric transformation to both clinical images and Wood's lamp images synchronously in pairs to ensure that the correspondence between the location and extent of lesions in the two modalities is not disrupted. Secondly, appearance enhancement can be set according to modal differences. For clinical images, brightness / contrast perturbation, slight color jitter, blurring or noise perturbation can be used to simulate imaging changes under different lighting and skin color backgrounds, while limiting the enhancement amplitude to avoid damaging the texture and morphological structure of lesions. For Wood's lamp images, contrast enhancement, gamma correction or edge preservation enhancement can be used to highlight the pigmentation change area and lesion boundary information, and suppress the effects of background noise and artifacts.

[0028] By employing the aforementioned preprocessing and enhancement strategies, the training samples can cover a wider range of acquisition conditions and imaging fluctuations, thereby improving the stability and diagnostic consistency of the model in actual clinical scenarios.

[0029] S203, the processed multimodal image is input to the feature extraction network to obtain the features of each modality, and spatial guidance information for subsequent segmentation is generated.

[0030] Let clinical imaging be Wood lamp image is The two modal images are input into a feature extraction network to obtain their respective multi-scale features: in Indicates the first Feature maps at various scales.

[0031] In one implementation, the encoder consists of convolutional blocks, and its feature extraction can be represented as: in, This represents the convolution operation. For convolution parameters, (·) represents normalization. (·) represents a non-linear activation function. The clinical branch focuses on extracting local features of texture, structure, and morphological changes; the Wood's lamp branch focuses on extracting significant regional responses and boundary gradient cues related to pigment differences, which are reflected in the network representation as feature distributions with enhanced responses at the boundaries and higher contrast in lesion areas.

[0032] To generate spatial guidance information for subsequent segmentation, the two modalities can be combined in shallow and deep layers to output a guidance map.

[0033] in Indicates channel-dimensional splicing. For 1×1 convolution, For the Sigmoid function, This refers to a spatial guidance map, used to identify the probability / confidence of suspected lesion areas. 's' represents shallow layers and 'd' represents deep layers, which is used to simultaneously consider boundary details and semantic stability.

[0034] S204, the cross-modal feature fusion module quantizes and fuses the input vectors of each modality feature to obtain semantically consistent fused features.

[0035] In one implementation, shallow multi-scale features are initially fused using element-wise addition; for deep semantic features, they are first projected to a low-dimensional embedding space via convolution, and then vector quantization is performed using a learnable codebook to map different modal features to a unified discrete representation space; subsequently, the quantized modal features are concatenated, and convolutional interactions, attention enhancement, and residual connections are combined to generate the final fused features, thereby reducing the semantic differences between modalities and improving the discriminative power of the fused representation.

[0036] Step 1: Shallow Feature Fusion For shallow multi-scale features, preliminary fusion can be performed by directly adding elements one by one. Let the clinical image and the Wood's lamp image be at the [missing information - likely a date or time interval]. The shallow features of the layers are as follows The corresponding shallow fusion feature can be expressed as: in Indicates the first Multimodal fusion features of layers. This method enables rapid fusion of low-level texture and boundary structure information.

[0037] Step 2: Vector Quantization Matching For deep semantic features, let the deep features output by the clinical image and Wood's lamp image encoders be respectively... and To reduce the semantic differences between the two modalities, deep features are first mapped to a low-dimensional embedding space using 1×1 convolutions to obtain the corresponding embedding representations. and ; Step 3: Codebook Update The embedded features are discretized using a bidirectional vector quantization strategy. Let the learnable codebook be { }, for any input feature vector By calculating its distance to each codebook vector, the nearest codebook entry is selected for matching, and its index can be represented as: in, Representation and input features The closest codebook index. This is achieved by replacing the input features with the corresponding codebook vector. The quantized feature representations can be obtained separately. and This quantization process maps continuous features of different modalities to a unified discrete representation space, thereby mitigating the semantic shift caused by modal heterogeneity.

[0038] Step 4: Codebook Update To improve the stability of codebook training, an exponential moving average-based approach is used to update the codebook. The update method for the m-th codebook vector in the t-th iteration can be expressed as: in, Indicates the attenuation coefficient. Indicates allocation to the first The feature set of a codebook vector. The above update method can improve the stability of the codebook representation and avoid codebook collapse during training.

[0039] Step 5: Deep Fusion Output In obtaining quantitative characteristics Then, the two are concatenated by channels, and cross-modal interaction is performed through convolution operations. The fusion result is then enhanced by a channel attention mechanism. Simultaneously, residual connections of the original embedded features are introduced to preserve the original semantic information of each modality. The final deep fusion feature can be represented as: in, This indicates a channel splicing operation. Indicates attention weights. This represents element-wise multiplication. Through the above processing, semantic alignment, interactive fusion, and discriminative enhancement of cross-modal features can be achieved within a unified latent space, ultimately resulting in a semantically consistent fused feature representation.

[0040] S205, construct segmentation branches and classification branches based on the fusion features to achieve joint output of lesion region segmentation and disease activity classification, and improve the synergistic effect of the two tasks through inter-task interaction.

[0041] After obtaining the multimodal features obtained through vector quantization fusion, a cross-task interaction module is constructed, and a loss function is added to achieve collaborative learning between the lesion segmentation task and the disease activity classification task. This module completes joint diagnosis through processes such as multi-scale feature enhancement, spatial guided generation, boundary refinement, and global semantic aggregation.

[0042] Segmentation: Localization and Boundary Refinement of Lesion Region First, multi-scale context modeling is performed on the encoder output features. Let the shared features of the encoder output be... By enhancing feature representation capabilities through parallel multi-branch dilated convolution and spatial pyramid pooling operations, feature representations with multi-scale receptive fields are obtained.

[0043] Spatial guided weights are generated using a cascaded decoding structure. For the k-th layer features... Its spatial weights can be expressed as: in This represents the Sigmoid activation function. Indicates the first Spatial weights of layers.

[0044] Subsequently, enhanced features are generated through cross-scale gated fusion: in Indicates the characteristics after fusion. This represents element-wise multiplication. This indicates an upsampling operation; the features obtained after multi-scale fusion are concatenated through channel concatenation. This feature serves as the initial segmentation guide feature.

[0045] To further optimize lesion boundaries, a contextual attention module (CRA) is used to refine the segmentation features. Its inverse attention weights can be expressed as: in, Indicates the first The reverse attention weights of the layer This represents the segmentation features of the previous layer. This mechanism can suppress the response of non-boundary regions and highlight lesion boundary information.

[0046] Subsequently, the segmentation features are updated step by step through the spatial gated convolutional fusion module: The final result of lesion segmentation was obtained.

[0047] Classification: Disease activity and condition analysis In classification tasks, shared features are utilized. Disease activity classification is performed. First, multi-scale features are concatenated: Global semantic features are then obtained through global average pooling: in Global average pooling operation.

[0048] Finally, the global features are input into the classifier to obtain the disease activity prediction results.

[0049] loss function To achieve joint optimization of lesion segmentation and activity classification tasks, this paper designs a multi-task loss function. The classification branch adopts... To alleviate the class imbalance problem, the splitting branch uses a hybrid loss combining binary cross-entropy and DiceLoss. In addition, vector quantization loss is introduced. Regularization constraints are applied to the discrete feature embeddings. The overall loss function is defined as: in, These are the weighting coefficients for each loss term, used to balance the optimization contributions between different tasks. The balancing coefficients are determined to be... =1.0、 =1.0 and =0.1, to achieve the best balance between different vitiligo diagnostic tasks.

[0050] The focus loss for classification is defined as follows: in Represents the true class probability. Used for sample balancing A value of 2.0 strengthens the weighting of difficult samples. The mixed segmentation loss for S1 to S4 is defined as follows: in Let these represent the true value and the predicted value at the k-th scale, respectively. This is a smoothing parameter, often used in segmentation loss functions to prevent the denominator from being zero and to improve numerical stability. This loss function simultaneously guarantees pixel accuracy and overall shape overlap. The loss formula for vector quantization is as follows: in Represents encoder features, Indicates the most recent codebook embedding. This indicates that the gradient operator is stopped. Set it to 0.25. The first term makes the codebook equal to... Alignment, the second constraint encoder enters the target embedding space.

[0051] For ease of understanding, the following description, in conjunction with the specific algorithm processing flow, illustrates the method for auxiliary diagnosis of vitiligo using multimodal medical image collaborative segmentation and classification provided in this application. Please refer to [link to relevant documentation]. Figure 2 and Figure 3 , Figure 2 This is a schematic diagram of the algorithm flow for a multimodal medical image collaborative segmentation and classification method, device and storage medium for the auxiliary diagnosis of vitiligo. Figure 3This includes a data augmentation module, a feature extraction module, a spatial guidance module, a vector quantization fusion module, and a multi-task diagnostic module. For example... Figure 2 and Figure 3 As shown, the specific steps may include: ① Multimodal data input and augmentation The core objective of this module is to construct a robust and diverse multimodal data input stream, providing a high-quality foundation for subsequent feature learning. Specifically, this includes: 1. Definition of multimodal data source Clinical images of vitiligo: Images taken under normal clinical lighting conditions, mainly reflecting the true color, texture and morphological structure of the target, and are the main basis for doctors to make preliminary diagnoses.

[0052] Wood's lamp image: An image taken after a target is illuminated by a Wood's lamp. Certain skin lesions, such as vitiligo, will emit characteristic fluorescence under a Wood's lamp. Therefore, this image can provide key diagnostic information that cannot be captured by clinical images.

[0053] These two images describe the same target from different dimensions, forming a natural complementary relationship and together constituting a multimodal visual representation.

[0054] 2. Parallel Data Augmentation Strategies To address the challenges of scarce clinical data and complex acquisition environments, we performed the following augmentation operations in parallel on images from two modalities to improve the model's generalization ability. These strategies include the following data augmentation approaches: Geometric transformation enhancement: Horizontal Flip: Horizontally mirrors the image, increasing the mirror diversity of the data and simulating different shooting angles.

[0055] Perspective transformation: By changing the perspective relationship of an image, it simulates the effect of observing a target from different perspectives, enhancing the model's robustness to changes in perspective.

[0056] Affine transformation: performing combined transformations such as scaling, rotation, and shearing on images to further enrich geometric diversity.

[0057] Color perturbation enhancement: Color jitter: Randomly adjust the brightness, contrast, saturation, and hue of the image to simulate the differences in lighting intensity and camera exposure settings in different clinics, enhancing the model's tolerance to color changes.

[0058] Noise and local disturbance enhancement: Gaussian blur: Applying a Gaussian filter to an image to simulate the blurring effect produced during image acquisition and transmission.

[0059] Image sharpening: Enhances image edge details through high-pass filtering, complementing blurring operations.

[0060] Random erasure: Randomly select a rectangular region in the image and fill it with random or mean values, forcing the model to make a comprehensive judgment from multiple regions, thus improving its robustness to local occlusion.

[0061] Gaussian noise: Injecting Gaussian noise into an image to simulate noise interference during the image acquisition process.

[0062] 3. Multimodal data fusion combines independently enhanced clinical images and Wood's lamp images to form a unified multimodal data representation. This operation initially fuses information from the two modalities at the data level, enabling subsequent convolutional neural networks to simultaneously receive and process signals from both modalities, laying the foundation for cross-modal feature learning.

[0063] ② Feature extraction and vector quantization fusion This module aims to transform multimodal data into compact, robust, and information-rich feature representations, and enhance their expressive power through vector quantization techniques.

[0064] The multimodal data, after feature extraction and concatenation by the encoder, is first fed into an encoder consisting of multiple convolutional blocks. The encoder gradually transforms the original image spatial information into high-level semantic features through a series of convolution, normalization, and nonlinear activation operations. These feature maps not only contain the texture and edge information of the target but also encode its abstract category and structural attributes.

[0065] The continuous features output by the encoder in the core process of vector quantization fusion are fed into the vector quantization fusion module, and its processing flow is as follows: Codebook mapping: The encoder's feature map is flattened into vectors and distances are calculated between it and all embedded vectors in a predefined codebook. Through nearest neighbor lookup, each feature vector is replaced with the nearest vector in the codebook. This process discretizes the continuous feature space, effectively compressing the feature dimensionality and enhancing the robustness and generalization ability of the features by sharing codebook vectors.

[0066] Vector embedding and EMA update: The discretized codebook indexes are mapped back to the corresponding learnable embedding vectors. Meanwhile, to enable the codebook to adapt to constantly changing input data, we employ an exponential moving average (EMA) strategy to dynamically update the codebook vectors, ensuring their stability and adaptability.

[0067] Concatenation: The discrete embedding vectors retrieved from the codebook are concatenated with the original continuous encoder features through channels to form a hybrid feature that simultaneously contains both discrete and continuous information. This design combines the robustness of vector quantization with the fine-grained information of the original features.

[0068] Attention Enhancement and Residual Learning: Hybrid features are passed through a subnetwork containing convolutional blocks, random deactivation, and an attention mechanism. Random deactivation prevents some neurons from overfitting; the attention mechanism generates weights for each spatial location and channel of the feature map, guiding the model to focus on the regions most critical to the task. Subsequently, the attention weights are multiplied element-wise with the features to achieve feature weighting, and then fused with the original features through residual connections, preserving the original information and introducing new discriminative features.

[0069] Deep fusion feature output: Finally, the fused features are further processed by a convolutional block to generate the final deep fusion feature, which is then fed into the subsequent cross-task interaction module.

[0070] ③ Cross-task interaction and joint optimization This module achieves collaborative optimization of classification and segmentation tasks by sharing an encoder and task-specific decoders, thereby improving overall performance by leveraging complementary information between tasks.

[0071] Deeply fused features from the shared encoder are fed into the shared encoder, which consists of multiple convolutional or Transformer layers responsible for extracting higher-level contextual information. Its high-level feature representation is shared by both task branches, ensuring that the model can reason based on the same set of core semantic features when learning classification and segmentation tasks. This branch consists of a classification branch and a segmentation branch. Classification task branches Multi-scale feature aggregation: Feature maps p1, p2, and p3 output by the encoder at different levels are aggregated. These feature maps encode information at different scales, from local to global.

[0072] Feature fusion and dimensionality reduction: Multi-scale features are fused by adding elements one by one, then the spatial dimension is compressed by global average pooling, and finally converted into a one-dimensional feature vector by flattening.

[0073] Classification prediction: A one-dimensional vector is fed into the classification head, which is one or more fully connected layers, and is ultimately mapped to a probability distribution corresponding to different diagnostic categories, outputting the classification loss. .

[0074] Split task branches Decoder stacking and upsampling: Features from the shared encoder are fed into a stack of decoders consisting of multiple deconvolutional layers. The decoders progressively upsample the feature maps to restore spatial resolution.

[0075] Gated convolutional fusion: Gated convolutional fusion is employed at each layer of the decoder. This mechanism dynamically learns a gating weight to fuse low-level features containing rich spatial details from the corresponding layer of the encoder with upsampled features containing high-level semantics from the current layer, effectively mitigating the problem of detail loss during upsampling.

[0076] Resolution Restoration and Mask Generation: After multiple upsampling and fusion processes, the feature map resolution is restored to match that of the input image. Finally, a pixel-by-pixel segmentation mask is generated through a segmentation head, i.e., a 1x1 convolutional layer, and the segmentation loss is output. .

[0077] ④ Calculation of joint loss and optimization of total loss Classification loss Segmentation loss and vector quantization loss Weighted summation yields: in These are adjustable weight coefficients. Through backpropagation, the total loss is used to optimize all parameters of the entire network, enabling gradient flow and information sharing among multiple tasks, thereby achieving a synergistic improvement effect.

[0078] ⑤ Reasoning and Application During the inference phase, the model no longer performs gradient updates, but instead uses the trained parameters to predict new clinical images.

[0079] Image classification: The model receives a new multimodal image. After the above process, the category with the highest probability in the probability distribution output by the classification head is the model's diagnostic classification result for the target.

[0080] Image segmentation: The pixel-by-pixel mask output by the segmentation head clearly outlines the boundaries and extent of the lesion area, providing doctors with accurate lesion location information.

[0081] Through this multi-task collaborative learning approach, the model can fully utilize the complementary information of multimodal data, while improving the performance of classification and segmentation tasks, providing more comprehensive and accurate support for clinical diagnosis and analysis.

[0082] The vitiligo-aided diagnostic network based on multimodal data augmentation and cross-task interaction provided in this application effectively alleviates the challenges posed by the scarcity of clinical data and modal heterogeneity. By introducing multi-class data augmentation strategies, a robust and diverse multimodal data input stream is constructed. Simultaneously, through a vector quantization fusion module, complementary information from clinical images and Wood's lamp images is uniformly modeled in discrete and continuous spaces, mitigating the feature distribution gap caused by differences in imaging mechanisms at both the data and feature levels, providing a high-quality foundation for subsequent multi-task learning. Furthermore, it achieves collaborative optimization and deep interaction between classification and segmentation tasks. By constructing a cross-task interaction module, classification and segmentation heads are designed separately based on a shared encoder. Using techniques such as gated convolutional fusion, lesion category discrimination and pixel-level localization are performed synchronously in a unified feature space. This design not only fully utilizes the complementary information of multimodal data, but also realizes gradient flow and information sharing between tasks through a joint loss function, promoting a paradigm shift in vitiligo assisted diagnosis from "single task" to "multi-task collaboration". While maintaining low computational complexity, it significantly improves the model's diagnostic accuracy and generalization ability in complex clinical scenarios, providing an effective technical solution for the precision diagnosis and treatment of skin diseases.

[0083] Please see Figure 4 , Figure 4 This is a schematic diagram of a multimodal medical image collaborative segmentation and classification method for the auxiliary diagnosis of vitiligo. (Example:) Figure 4 As shown, this multimodal auxiliary diagnostic method for vitiligo may include: The data acquisition module 401 is used to acquire multi-source data of the object to be diagnosed. The data includes operating status data collected by sensors, monitoring image data or historical operating data. The acquired data is uniformly formatted to construct a dataset to be analyzed, providing basic data input for subsequent feature analysis.

[0084] The feature extraction module 402 is used to preprocess the data acquired by the data acquisition module and input the preprocessed data into the pre-trained feature extraction network to extract initial feature information that can characterize the target state. The initial features are used to describe the basic feature patterns of the object to be diagnosed under different operating states.

[0085] The spatial guidance module 403 is used to guide and enhance the initial features based on spatial structure information. By constructing spatial relationships between features, it strengthens the weights of key regions or key features, thereby obtaining spatially guided features containing spatial dependencies to improve the accuracy and effectiveness of feature expression.

[0086] The vector quantization fusion module 404 is used to input the spatially guided features into the vector quantization fusion network, discretize the features through the vector quantization mechanism, and fuse information from different feature dimensions to form a unified and stable fused feature representation, thereby improving the discriminative ability and robustness of the features.

[0087] The multi-task diagnostic module 405 is used to construct a multi-task diagnostic model based on the fused feature representation, perform multi-dimensional analysis on the object to be diagnosed, including tasks such as status recognition, fault type discrimination, and operation status assessment, and achieve multi-task collaborative diagnosis through joint learning to obtain the final diagnostic result.

[0088] Please see Figure 5 , Figure 5 This is a schematic diagram of an electronic device hardware system for multimodal medical image collaborative segmentation and classification in the auxiliary diagnosis of vitiligo. Figure 5 As shown, this multimodal auxiliary diagnostic method for vitiligo may include: S501: Multimodal image acquisition unit, including at least a visible light skin imaging device and a non-visible light skin detection device, such as an ultraviolet light imaging device, a near-infrared imaging sensor or a skin thermal infrared camera, for acquiring multimodal image data of the patient's skin lesion area to obtain multi-source data containing skin surface texture information, pigment distribution information and skin temperature or deep structure information.

[0089] The S502 core computing unit, including a central processing unit (CPU) and a graphics processing unit (GPU) with integrated parallel computing capabilities, is used to efficiently process acquired multimodal image data, perform computational processes such as feature extraction, spatially guided feature enhancement, vector quantization fusion, and multi-task diagnosis, and realize in-depth analysis and diagnostic reasoning of the characteristics of vitiligo lesion areas.

[0090] The S503 memory system is used to cache acquired multimodal image data, store pre-trained diagnostic model parameters, feature vector libraries, and historical diagnostic data to support data reading and result storage during model inference.

[0091] The S504 communication unit is used to enable data interaction between electronic devices and external terminals or medical information systems, including uploading diagnostic results, synchronizing patient data, and accessing remote medical systems.

[0092] It should be noted that, Figure 5 For details not mentioned in the corresponding embodiments and the specific implementation methods of each step, please refer to [link to relevant documentation]. Figure 1 , Figure 2 , Figure 3 and Figure 4 The embodiments shown and the foregoing content will not be repeated here.

[0093] This application also provides a computer-readable storage medium, such as, but not limited to, a disk, optical disk, solid-state drive, flash memory, ROM, RAM, or other non-transitory storage medium, on which a computer program is stored. When executed by a processor, this computer program causes the processor to perform actions such as... Figure 1 , Figure 2 and Figure 3 The steps performed in the illustrated method embodiment are shown. The program code in the storage medium can be loaded into memory and executed by a processor to implement a complete cross-modal target re-identification process.

[0094] The electronic device can be specifically implemented as a skin disease intelligent detection device, a hospital diagnostic terminal, an edge computing server, or a mobile medical terminal, etc. When the computer program stored in the memory system is executed by the core computing unit, the electronic device can realize the vitiligo multimodal assisted diagnosis method described in the first aspect above.

[0095] The steps in the method of this application embodiment can be adjusted, combined, or deleted according to actual needs.

[0096] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0097] The above-disclosed embodiments are merely some of the embodiments of this application, and should not be construed as limiting the scope of this application. Those skilled in the art can understand that implementing all or part of the above embodiments and making equivalent changes in accordance with the claims of this application are still within the scope of this application.

Claims

1. A method, device, and storage medium for auxiliary diagnosis of vitiligo using multimodal medical image collaborative segmentation and classification, characterized in that, The method includes: Acquire multimodal image pairs consisting of clinical images and Wood's lamp images of the same subject; perform preprocessing and modality-specific data augmentation on the image pairs according to the imaging characteristics of the two modalities respectively; The processed multimodal image is fed into a feature extraction network to obtain features for each modality, and spatial guidance information for subsequent segmentation is generated. The input vector quantization and fusion module of each modality feature is used to perform cross-modal feature fusion to obtain semantically consistent fused features; Based on the fusion features, segmentation and classification branches are constructed to achieve joint output of lesion region segmentation and disease activity classification, and the synergistic effect of the two tasks is improved through inter-task interaction.

2. The method as described in claim 1, characterized in that, The multimodal image pair consisting of clinical images and Wood's lamp images of the same subject includes: The clinical images provide visible information such as skin appearance, texture, and details; the Wood's lamp images highlight characteristic information such as pigmentation changes and lesion boundaries. By acquiring paired data of the same object, a consistent spatial and lesion correspondence is ensured between different modalities, facilitating effective subsequent fusion and joint diagnosis.

3. The method as described in claims 1 and 2, characterized in that, The image pairs are preprocessed and modality-specific data augmented based on the imaging characteristics of the two modalities, including: The preprocessing is used to unify image size, quality and input format, and reduce interference caused by differences in acquisition conditions; the modality-specific data enhancement, such as perspective transformation and affine transformation, is adapted to the imaging differences between clinical images and Wood's lamp images, which not only maintains the structural and texture stability of clinical images, but also enhances the recognizability of lesion boundaries and key areas in Wood's lamp images, thereby improving the model's adaptability to data under different acquisition conditions.

4. The method as described in claim 3, characterized in that, The processed multimodal image is fed into a feature extraction network to obtain features for each modality, and spatial guidance information for subsequent segmentation is generated, including: The clinical images and Wood's lamp images are respectively input into a feature extraction network for encoding, extracting shallow spatial features and deep semantic features corresponding to each modality; the shallow spatial features are integrated and enhanced to highlight the texture structure and boundary information related to vitiligo lesions; spatial guidance information is generated based on the enhanced features to identify the location distribution of suspected lesion areas; the spatial guidance information is fused with the corresponding modality features to guide the subsequent segmentation network to focus on the lesion area during feature decoding, thereby improving the lesion localization accuracy and the stability of the segmentation results.

5. The method as described in claim 4, characterized in that, The modal feature input vector quantization and fusion module performs cross-modal feature fusion to obtain semantically consistent fused features, including: The vector quantization fusion module maps continuous features from different modalities to a unified discrete representation space. It achieves cross-modal information correspondence and integration through shared or aligned representation units, thereby reducing the impact of differences in imaging mechanisms and representations between the two modalities on the fusion results. This fusion method preserves detailed information about clinical image structures while fully utilizing the highlighting effect of Wood's lamp images on lesion regions. The fused features simultaneously contain the key information required for localization and discrimination, providing a more stable and consistent input basis for subsequent segmentation and classification tasks.

6. The method as described in claim 5, characterized in that, Based on the fusion features, segmentation and classification branches are constructed to achieve joint output of lesion region segmentation and disease activity classification, and the synergistic effect of the two tasks is improved through inter-task interaction. The segmentation branch outputs pixel-level lesion region results to support precise localization and quantitative assessment of the extent, morphology, and boundaries of vitiligo. The classification branch outputs disease activity category results to support overall judgment and decision support for vitiligo condition status. The inter-task interaction enables information sharing and mutual constraints during training or inference, allowing the regional focus information obtained by the segmentation branch to provide more accurate lesion evidence for the classification branch, ensuring consistent segmentation results in overall semantics and reducing boundary fragmentation or missegmentation. Conversely, the global semantic information learned by the classification branch can, in turn, promote boundary discrimination and regional consistency of the segmentation branch, making the classification focus more on lesion-related regions and reducing background interference. Through this joint learning and collaborative mechanism, overall diagnostic performance and result consistency can be improved, better meeting the clinical need for integrated output of "lesion localization + condition assessment." 7. The method according to any one of claims 1-6, characterized in that, The device includes: A total loss function is constructed to simultaneously constrain the lesion segmentation task and the disease activity classification task. The total loss function includes a loss term for the segmentation task, a loss term for the classification task, and a quantization constraint loss term for vector quantization fusion. A joint optimization strategy was adopted to train the feature extraction network and the vector quantization fusion module end-to-end, so that clinical image features and Wood's lamp image features could achieve semantic alignment and complete fusion representation learning in a unified representation space. During training, by jointly optimizing the aforementioned loss terms, the ability to locate and delineate lesion areas and improve the ability to classify disease activity is enhanced, thereby improving the overall performance and output consistency of vitiligo auxiliary diagnosis.

8. A multimodal auxiliary diagnostic method for vitiligo, characterized in that, The method includes: The data acquisition module is used to acquire multimodal skin image data of the same subject, including clinical images and Wood's lamp images; The feature extraction module is used to perform image preprocessing and modality-specific data enhancement on the multimodal skin images, and input the processed image data into the feature extraction network to extract clinical image features and Wood's lamp image features; The spatial guidance module is used to generate spatial guidance information based on the clinical imaging features and Wood's lamp imaging features to indicate suspected lesion areas and provide localization priors for subsequent segmentation; The feature fusion module is used to input the clinical image features and Wood's lamp image features into the multi-task vector quantization fusion module, and to discretize and fuse the cross-modal features in a unified representation space to generate semantically consistent fused feature representations. The vector quantization fusion module is used to input the spatially guided features into the vector quantization fusion network. It discretizes the features using a vector quantization mechanism and fuses information from different feature dimensions to form a unified and stable fused feature representation. The multi-task diagnostic module is used to construct a multi-task diagnostic model based on the fused feature representation. It performs multi-dimensional analysis of the object to be diagnosed and achieves multi-task collaborative diagnosis through joint learning to obtain the final diagnostic result.

9. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1-7.

10. A hardware system for an electronic device, characterized in that, include: The multimodal image acquisition unit is used to acquire clinical images and Wood's lamp images, and generate paired multimodal image data of the same subject. The core computing unit, connected to the multimodal image acquisition unit, includes a central processing unit and a parallel processor. The parallel processor is configured to efficiently perform multimodal feature extraction, vector quantization fusion calculation, and multi-task joint inference. The memory system unit, connected to the core computing unit, is used to store computer programs, pre-trained model parameters, and data required for diagnostic result output. A communication unit, connected to the core computing unit, is used to realize image data transmission, diagnostic result reporting, or interaction with external terminals; When the computer program stored in the memory system is executed by the core computing unit, the electronic device is controlled to implement the vitiligo multimodal auxiliary diagnosis method as described in any one of claims 1-9.

Images