Skin state detection method and system based on image recognition

Abstract

This invention provides a skin condition detection method and system based on image recognition. The method includes acquiring a skin image to be analyzed and related textual metadata; inputting the preprocessed image into a trained skin lesion segmentation model and outputting the segmentation result of the lesion region; automatically extracting the size information and at least one morphological quantitative feature of the skin lesion based on the segmentation result, and integrating the size information and the morphological quantitative feature as supplementary features into the textual metadata to form enhanced metadata; assigning appropriate weights to the image features and the textual features based on the mutual information between the image features and the textual features, and dynamically weighting and concatenating them to form a fused feature vector; inputting the fused feature vector into a classifier and outputting the classification result of the skin condition, which significantly improves the model's ability to distinguish similar skin diseases, especially early malignant lesions, and improves the classification accuracy.

Description

Technical Field

[0001] This invention belongs to the field of image recognition technology, and specifically relates to a skin condition detection method and system based on image recognition. Background Technology

[0002] Skin diseases are among the most common health problems worldwide, and early detection and accurate diagnosis are crucial for treatment selection and prognosis. In recent years, with the rapid development of computer vision technology, image recognition-based automatic skin condition detection methods have become a research hotspot in the field of medical AI. These methods aim to assist doctors and even automate disease screening and classification by analyzing images of skin lesions.

[0003] Existing automatic skin detection methods typically follow this process: after acquiring skin images, they are preprocessed, then deep learning models (such as convolutional neural networks) are used for lesion segmentation or feature extraction, and finally, classification is performed based on the extracted features.

[0004] However, current technologies for diagnosing skin conditions rely not only on visual images but also on the patient's clinical metadata (such as age, medical history, and symptom descriptions). Most existing technologies focus solely on image features or simply concatenate image features with textual metadata at an early or late stage, failing to delve into the intrinsic connections between image and textual information and failing to create complementary and enhanced feature representations. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a skin condition detection method and system based on image recognition, which solves the technical problems in the prior art.

[0006] In a first aspect, the invention provides the following technical solution: a skin condition detection method based on image recognition, the method comprising: Obtain the skin image to be analyzed and related text metadata, and preprocess the skin image to remove interference noise from the image; The preprocessed image is input into the trained skin lesion segmentation model, and the segmentation result of the lesion area is output. Based on the segmentation results, the size information of the skin lesions and at least one morphological quantitative feature are automatically extracted. The size information and the morphological quantitative feature are then integrated into the text metadata as supplementary features to form enhanced metadata. Extract the image features of the skin image and the text features of the enhanced metadata. Based on the mutual information between the image features and the text features, assign appropriate weights to the two and perform dynamic weighted concatenation to form a fused feature vector. The fused feature vector is input into the classifier, which outputs the classification result of the skin condition.

[0007] Compared to existing technologies, the beneficial effects of this application are as follows: By automatically extracting the physical size and morphological quantitative features of lesions, the clinical diagnostic experience of doctors (such as asymmetry and boundaries in the ABCD rule) is transformed into calculable objective indicators, greatly enriching the feature dimensions used for diagnosis. Through a dynamic weighted fusion strategy based on mutual information, the correlation between image and text features is intelligently evaluated and weighted, achieving information complementarity and redundancy removal, generating a fusion feature representation with stronger discriminative power. The combination of these two approaches significantly improves the model's ability to distinguish similar skin diseases, especially early malignant lesions, and increases classification accuracy.

[0008] Furthermore, the training method for the skin lesion segmentation model includes: Obtain a dataset of skin disease images, which includes a small number of labeled samples and a large number of unlabeled samples; The unlabeled samples are clustered using a clustering algorithm. Based on a distance sampling strategy, some unlabeled samples are selected from each cluster and initial pseudo-labels are generated. The initial pseudo-labels are merged with the original labeled samples to form an expanded labeled training set; Based on the improved course learning strategy, the model is trained using the labeled training set and the remaining unlabeled samples; wherein, the improved course learning strategy dynamically adjusts the confidence threshold of the model when generating the initial pseudo-labels for unlabeled samples of different categories.

[0009] Furthermore, the calculation formula for dynamically adjusting the confidence threshold includes:

[0010] in, Indicates the first Each category during the training period The confidence threshold at that time This represents the initial confidence threshold. Represents the balance factor. Indicates the first Each category during the training period Historical average confidence level across clients Indicates the first Each category during the training period The weights of sample distribution across clients.

[0011] Furthermore, the step of outputting the classification result of skin condition includes: Output the initial classification result of the skin condition and the corresponding classification confidence score; When the classification confidence level is lower than a preset threshold, an interactive request is generated to guide the user to supplement specific clinical information, and the initial classification result is verified and updated based on the specific clinical information supplemented by the user. The specific clinical information includes dermoscopy images from specific angles, medical history texts, or skin oiliness detection data; the supplementary specific clinical information is extracted as new features, fused with the original fusion feature vector, and then input into the classifier again to obtain the updated final classification result.

[0012] Furthermore, the automatic extraction of the size information of the skin lesions specifically includes: Obtain the original clinical image containing the ruler reference and its corresponding segmentation results; The segmentation results are registered with the original clinical image to determine the precise location of the lesion outline in the original clinical image; In the original clinical image, the ruler scale edge and the lesion outline are extracted using an edge detection algorithm; Based on the extracted ruler scale edges, the number of pixels corresponding to a unit of actual physical length is determined to establish the conversion relationship between pixels and actual physical size. Measure the number of pixels occupied by the lesion outline in the image; Based on the conversion relationship, the number of pixels in the lesion outline is converted into the actual physical size.

[0013] Furthermore, the formula for calculating the actual physical dimensions is as follows: in: Indicates the actual physical length of the lesion. This represents the number of pixels occupied by the lesion outline in the image. To provide a reference for the number of pixels per unit length of the ruler in the image, The actual physical length of the reference ruler.

[0014] Furthermore, at least some feature extraction modules in the encoder of the skin lesion segmentation model are replaced with deep residual dilated convolution modules to expand the receptive field of the model; Among them, in the deep residual dilated convolution module, the first The size of the receptive field after layer-hole convolution is calculated using the following formula:

[0015] in: For the first The size of the receptive field of the layer For the first Layer step size, For the first The expansion rate of layer void convolution, For the first The size of the convolution kernel, For the first The size of the receptive field after convolution of the void layer.

[0016] Secondly, the invention provides the following technical solution: a skin condition detection system based on image recognition, the system comprising: The processing module is used to acquire the skin image to be analyzed and related text metadata, and to preprocess the skin image to remove interference noise from the image; The skin lesion segmentation module is used to input the preprocessed image into the trained skin lesion segmentation model and output the segmentation results of the lesion area; An integration module is used to automatically extract the size information of skin lesions and at least one morphological quantitative feature based on the segmentation results, and integrate the size information and the morphological quantitative feature as supplementary features into the text metadata to form enhanced metadata; The fusion module is used to extract image features of the skin image and text features of the enhanced metadata. Based on the mutual information between the image features and the text features, it assigns appropriate weights to the two and performs dynamic weighted concatenation to form a fused feature vector. The output module is used to input the fused feature vector into the classifier and output the classification result of the skin state.

[0017] Thirdly, the invention provides the following technical solution: a computer, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the skin condition detection method based on image recognition as described above.

[0018] Fourthly, the invention provides the following technical solution: a storage medium storing a computer program, which, when executed by a processor, implements the skin condition detection method based on image recognition as described above. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1A flowchart of the skin condition detection method based on image recognition provided in the first embodiment of the present invention; Figure 2 This is a structural block diagram of the skin condition detection system based on image recognition provided in the second embodiment of the present invention; Figure 3 This is a schematic diagram of the hardware structure of a computer provided in the third embodiment of the present invention.

[0021] The embodiments of the present invention will be further described below with reference to the accompanying drawings. Detailed Implementation

[0022] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain embodiments of the present invention, and should not be construed as limiting the present invention.

[0023] In the description of the embodiments of the present invention, it should be understood that the terms "length", "width", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing the embodiments of the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the present invention.

[0024] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of the present invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0025] Example 1 In the first embodiment of the present invention, please refer to Figure 1 A skin condition detection method based on image recognition includes the following steps S01 to S05: S01, acquire the skin image to be analyzed and related text metadata, and preprocess the skin image to remove interference noise from the image; In this embodiment, a skin image to be analyzed and related textual metadata are acquired. The skin image is preferably a dermoscopy image or a high-resolution clinical photograph. The textual metadata includes, but is not limited to, information such as the patient's age, gender, medical history, chief complaint, and lesion location.

[0026] Specifically, the preprocessing includes at least one of hair removal, bubble removal, and black frame removal.

[0027] In this embodiment, the preprocessing aims to remove interfering noise from the image that is irrelevant to diagnosis, ensuring the accuracy of subsequent analysis. The preprocessing operation may include one or more of the following processes, depending on the image quality: Hair removal treatments include: S11: To simplify the calculation, the acquired RGB skin image is first converted into a grayscale image.

[0028] S12: Perform a black hat transformation on the grayscale image. This operation effectively highlights hair structures that are darker than their surroundings. Its mathematical morphological implementation is: Black hat transformation result = Closure operation result of the original grayscale image. The closure operation involves first dilating the image using a linear structuring element (such as a line segment of 15 pixels in length, the direction of which can be adjusted according to common hair growth patterns), followed by an erosion operation. The purpose of the closure operation is to fill in small black holes inside objects and connect neighboring objects without significantly changing their area. Subtracting the closure operation result from the original image yields these filled-in small dark areas, i.e., the hair.

[0029] S13: Threshold the image after black hat transformation (e.g., using Otsu's adaptive thresholding method) to generate a clear binary mask for identifying hair regions. In this mask, white pixels represent hair and black pixels represent the background.

[0030] S14: Based on the generated binary mask, an image inpainting algorithm (e.g., a fast-moving inpainting algorithm or a hydrodynamic inpainting algorithm based on the Navier-Stokes equations) is used to fill and repair the hair region in the original dermatological image. This algorithm utilizes the texture and color information of the healthy skin region surrounding the hair to intelligently fill the area covered by hair, thereby obtaining a clear image without hair interference.

[0031] Bubble removal treatment includes: For bubble noise caused by protective gels, etc., which usually manifests as high-frequency, bright, isolated spots, this invention employs a nonlinear filtering method: Specifically, a median filter with a convolution kernel size of 5x5 is used to filter the skin disease image. Median filtering, by replacing the value of each pixel with the median value of all pixels in its neighborhood, can effectively smooth bubble noise in the image, while better preserving image edge information and avoiding edge blurring problems that may be caused by linear filtering such as Gaussian blur.

[0032] Black border removal processing includes The following steps are taken to handle invalid black borders caused by imaging equipment or obstructions: S15: Convert the dermatology image from the RGB color space to the HSL color space. The HSL color space is more in line with human perception of color and brightness.

[0033] S16: Extract the L channel, i.e., the luminance component, from the HSL image. The black-framed area represents an extremely low luminance value in this channel.

[0034] S17: Scan the brightness components row by row (from top to bottom, from bottom to top) and column by column (from left to right, from right to left). For each row or column, count the number of pixels whose pixel brightness value is lower than a preset threshold (e.g., brightness value <15, range 0-255), and calculate their proportion in the total pixels of that row / column.

[0035] S18: Set a preset ratio (e.g., 95%). When the proportion of low-brightness pixels in a row or column of the image edge region exceeds this ratio, the row / column is determined to be a black box region. The system automatically crops from the image boundary inwards until it encounters a row or column that does not meet the condition, thereby removing all black boxes.

[0036] S02, input the preprocessed image into the trained skin lesion segmentation model, and output the segmentation result of the lesion area; In the skin lesion segmentation model, at least some feature extraction modules in the encoder are replaced with deep residual dilated convolution modules to expand the model's receptive field. In the deep residual dilated convolution module, the first... The size of the receptive field after layer-hole convolution is calculated using the following formula:

[0037] in: For the first The size of the receptive field of the layer For the first Layer step size, For the first The expansion rate of layer void convolution, For the first The size of the convolution kernel, For the first The size of the receptive field after convolution of the void layer.

[0038] In this embodiment, the preprocessed image is input into the trained skin lesion segmentation model, and the model outputs a high-precision lesion region segmentation result (i.e., pixel-level mask).

[0039] The skin lesion segmentation model described is an improved model based on the U-Net++ architecture. Its core improvement lies in the encoder: at least some of the traditional feature extraction modules are replaced with a deep residual atrous convolution module.

[0040] The DRC module expands the model's receptive field by setting multiple dilated convolutional layers with different dilation rates (e.g., 1, 2, 4) in parallel. Specifically, in the deep residual dilated convolutional module, the first... The size of the receptive field after layer-hole convolution is calculated using the following formula:

[0041] in: For the first The size of the receptive field of the layer For the first Layer step size, For the first The expansion rate of layer void convolution, For the first The size of the convolution kernel, For the first The size of the receptive field after layered dilated convolution. Through this design, the model can capture a wider range of contextual information without increasing the number of parameters and computational cost, thus more accurately handling lesions with blurred boundaries.

[0042] Furthermore, the model integrates a full-scale feature fusion structure in the decoder. This structure uses dense skip connections to perform cross-scale stitching and fusion of feature maps from different levels on the encoder path and corresponding levels on the decoder path. Before fusion, a feature alignment module upsamples and spatially aligns the lower-level feature maps. This module sequentially employs bilinear interpolation and corner alignment algorithms to ensure the accuracy of feature map stitching. Specifically, the training method for the skin lesion segmentation model includes: Obtain a dataset of skin disease images, which includes a small number of labeled samples and a large number of unlabeled samples; The unlabeled samples are clustered using a clustering algorithm. Based on a distance sampling strategy, some unlabeled samples are selected from each cluster and initial pseudo-labels are generated. The initial pseudo-labels are merged with the original labeled samples to form an expanded labeled training set; Based on the improved course learning strategy, the model is trained using the labeled training set and the remaining unlabeled samples; wherein, the improved course learning strategy dynamically adjusts the confidence threshold of the model when generating the initial pseudo-labels for unlabeled samples of different categories.

[0043] More specifically, the formula for dynamically adjusting the confidence threshold includes:

[0044] in, Indicates the first Each category during the training period The confidence threshold at that time This represents the initial confidence threshold. Represents the balance factor. Indicates the first Each category during the training period Historical average confidence level across clients Indicates the first Each category during the training period The weights of sample distribution across clients.

[0045] In this embodiment, the training of the above-mentioned skin lesion segmentation model employs an improved semi-supervised learning method, specifically including: S21: Obtain a dataset of skin disease images, which contains a small number of labeled samples annotated by doctors and a large number of unlabeled samples.

[0046] S22: Use the K-Means clustering algorithm to cluster the unlabeled samples, assuming that the number of clusters is similar to the number of disease categories.

[0047] S23: Based on the root distance sampling strategy, select the top K unlabeled samples with the smallest Euclidean distance to the cluster center from each cluster, and generate initial pseudo-labels for these samples, whose labels are the categories of the clusters to which they belong.

[0048] S24: Merge the generated initial pseudo-labels with the original labeled samples to form an amplified labeled training set.

[0049] S25: Based on an improved course learning strategy, the skin lesion segmentation model is iteratively trained using an expanded labeled training set and the remaining unlabeled samples. The core of this strategy is to dynamically adjust the confidence threshold for generating pseudo-labels.

[0050] Specifically, for each category In the Set a dynamic and flexible threshold for each training cycle. The calculation formula is as follows:

[0051] in, Use an initial fixed confidence threshold (e.g., 0.9). Let be the learning performance coefficient of category m in the e-th training cycle, and its calculation method is as follows:

[0052] in: Representative model for the first The learning performance of a class of labeled samples is equal to the model's prediction accuracy for that class of labeled samples in the current training epoch. Representative model for the first The learning performance of unlabeled samples is inversely proportional to the difference between the number of unlabeled samples predicted by the model for that class and the number of samples for that class estimated by K-Means. This is a weighting factor (e.g., 0.7) used to balance the impact of labeled and unlabeled data.

[0053] For categories with poor learning outcomes ( (smaller value), dynamic threshold This will decrease accordingly, allowing more unlabeled samples of that category with slightly lower prediction confidence to participate in training, thus playing a role in "filling in the gaps" and balancing the learning progress of each category.

[0054] S03, Based on the segmentation results, automatically extract the size information of the skin lesions and at least one morphological quantitative feature, and integrate the size information and the morphological quantitative feature as supplementary features into the text metadata to form enhanced metadata; Specifically, the automatic extraction of size information for skin lesions includes: Obtain the original clinical image containing the ruler reference and its corresponding segmentation results; The segmentation results are registered with the original clinical image to determine the precise location of the lesion outline in the original clinical image; In the original clinical image, the ruler scale edge and the lesion outline are extracted using an edge detection algorithm; Based on the extracted ruler scale edges, the number of pixels corresponding to a unit of actual physical length is determined to establish the conversion relationship between pixels and actual physical size. Measure the number of pixels occupied by the lesion outline in the image; Based on the conversion relationship, the number of pixels in the lesion outline is converted into the actual physical size.

[0055] More specifically, the formula for calculating the actual physical dimensions is: in: Indicates the actual physical length of the lesion. This represents the number of pixels occupied by the lesion outline in the image. To provide a reference for the number of pixels per unit length of the ruler in the image, The actual physical length of the reference ruler.

[0056] In this embodiment, based on the lesion area segmentation results (i.e., the binary mask of the lesion area) output by the skin lesion segmentation model, this implementation method automatically extracts two major categories of key features: physical size information and morphological quantitative features, to accurately describe the clinical manifestation of the lesion in a digital way.

[0057] (1) Extraction of physical dimension information: This step aims to obtain the physical dimensions of the lesion in the real world, and the process is as follows: Image registration: First, the high-precision segmentation mask is registered with the original clinical image containing a ruler reference to determine the precise spatial location of the lesion outline in the original clinical image.

[0058] Edge and contour extraction: Using the Canny edge detection algorithm, the ruler scale edges and the true contours of lesions are extracted simultaneously from the registered original clinical images.

[0059] Size conversion model establishment: Based on the extracted ruler scale edges, calculate the number of pixels in the image corresponding to a unit of actual physical length (e.g., 1 cm). This establishes the conversion relationship between "pixels and physical size".

[0060] Physical size calculation: measuring the number of pixels occupied by the lesion outline in the image. The actual physical size of the lesion is calculated using the following formula. (e.g., maximum diameter or projected area): in: Indicates the actual physical length of the lesion. This represents the number of pixels occupied by the lesion outline in the image. To provide a reference for the number of pixels per unit length of the ruler in the image, The actual physical length of the reference ruler.

[0061] (2) Morphological quantitative feature extraction: This step directly calculates quantitative indicators of high interest in medical diagnosis based on the geometric and color properties of the segmentation mask: Asymmetry Index: Used to quantify the degree of asymmetry in the shape of lesions. In calculation, the lesion mask is divided into multiple sector regions along its major and minor axes passing through its centroid, and the ratio of the area difference between relative regions to the total area of ​​the lesion is calculated. A higher index value indicates a more irregular lesion shape and a potentially higher risk of malignancy.

[0062] Boundary irregularity index: Used to quantify the jaggedness or unevenness of lesion boundaries. It is obtained by calculating the ratio of the lesion's perimeter to the circumference of a circle of equal area. A perfect circle has a ratio of 1. The larger the ratio, the more irregular and uneven the boundary, which is a potential marker of malignancy.

[0063] Color uniformity index: Used to quantify the consistency of color distribution within a lesion. Within the lesion area of ​​the original RGB image, the standard deviations of the pixel values ​​for the R, G, and B color channels are calculated separately, and then the average is taken. The smaller the standard deviation, the more uniform the color within the lesion; the larger the value, the more chaotic the color distribution, the presence of multiple pigments, and an increased likelihood of malignancy.

[0064] (3) Enhance metadata formation: The automatically extracted physical size information (such as maximum diameter and area) and morphological quantitative features (asymmetry index, boundary irregularity index, and color uniformity index) are combined as a new set of objective digital features and integrated into the original textual metadata (such as patient age and medical history) to form enhanced metadata for subsequent intelligent diagnosis. This process transforms visual image information into quantifiable structured data, greatly enriching the model's judgment criteria.

[0065] S04, extract the image features of the skin image and extract the text features of the enhanced metadata. Based on the mutual information between the image features and the text features, assign appropriate weights to the two and perform dynamic weighted concatenation to form a fused feature vector. In this embodiment, an innovative dynamic weighted fusion mechanism is used to enable the model to adaptively determine under what circumstances it relies more on image information and under what circumstances it relies more on clinical metadata information, thereby making more accurate judgments.

[0066] 4.1 Feature extraction; Image feature extraction: An advanced deep learning model is used as the feature extraction backbone. In a preferred embodiment, the Swin Transformer network is employed. This network, through its hierarchical design and shifted window attention mechanism, is able to extract high-level semantic features with strong global awareness from the preprocessed skin image. Specifically, we extract the output feature map of the last stage of the Swin Transformer, and after global average pooling, obtain a fixed-dimensional image feature vector V. img .

[0067] Text feature extraction: The enhanced metadata obtained in the preceding steps (containing original patient text information and automatically extracted size and morphological quantification features) is used as input. First, numerical features are standardized, and categorical features are one-hot encoded to form a unified numerical vector. This vector is then input into a multilayer perceptron (MLP) consisting of multiple fully connected layers. The MLP learns and extracts a high-level text feature vector V from this structured data through nonlinear transformations. txt .

[0068] 4.2 Dynamically weighted feature fusion; Instead of simple feature concatenation, a dynamic weighting strategy based on mutual information is introduced, the core process of which is as follows: Mutual information estimation: First, calculate the image feature vector V. img With text feature vector V txt The mutual information (MI) between features. Mutual information measures the amount of information contained in one feature about another. In this implementation, we use a lightweight neural network estimator to approximate the MI. This estimator uses V... img and V txt The concatenated vector is taken as input, and after passing through several fully connected layers, a scalar is output, which is the estimated mutual information MI.

[0069] Dynamic weight allocation: Based on the calculated mutual information MI, appropriate fusion weights are dynamically assigned to image and text features. The weight assignment follows these principles: A high MI value indicates that the information expressed by image features and text features is highly consistent and mutually supportive. In this case, the model should utilize the information from both modalities more evenly, and may even appropriately increase the weight of text features because they provide clear clinical indications.

[0070] A low MI value indicates that the two modalities provide significantly different or highly complementary information. In this case, the model needs to determine which modality is more reliable. For example, when image features are blurry but text features (such as patient age and medical history) are very typical, the weight of text features should be appropriately increased; conversely, the weight of text features should also be increased.

[0071] We implement this strategy through a learnable weight assignment function. Specifically, the dynamic weights W of the image features... img Dynamic weights W of text features txt It is generated through a small neural network or a predefined smoothing function with MI as input, and then normalized by Softmax to ensure that W img + W txt = 1.

[0072] Weighted splicing: The dynamic weights are applied to the corresponding feature vectors and then concatenated to form the final fused feature vector V. fused The calculation formula is as follows:

[0073] in This represents a vector concatenation operation. This method generates... It is no longer a simple juxtaposition of two features, but a deeply fused representation after importance calibration.

[0074] 4.3 State Classification; The fusion feature vector obtained above The input is fed into a final classifier to output a classification result of the skin condition. This classifier can be a softmax classification layer to output the probability distribution of various skin diseases (e.g., melanoma, basal cell carcinoma, common moles, etc.); or it can be a support vector machine. The model ultimately selects the category with the highest probability as the diagnostic suggestion output.

[0075] S05, input the fused feature vector into the classifier and output the classification result of the skin state.

[0076] Specifically, the steps for outputting the classification results of skin condition include: Output the initial classification result of the skin condition and the corresponding classification confidence score; When the classification confidence level is lower than a preset threshold, an interactive request is generated to guide the user to supplement specific clinical information, and the initial classification result is verified and updated based on the specific clinical information supplemented by the user. The specific clinical information includes dermoscopy images from specific angles, medical history texts, or skin oiliness detection data; the supplementary specific clinical information is extracted as new features, fused with the original fusion feature vector, and then input into the classifier again to obtain the updated final classification result.

[0077] In this embodiment, an interactive diagnostic feedback mechanism is introduced to address the problem of insufficient confidence in the model when faced with difficult cases, thereby improving the practicality and reliability of the system in real clinical scenarios.

[0078] 5.1 Initial classification and confidence assessment; The fused feature vector obtained from the preceding steps is input into a fully connected layer classifier, which outputs a probability distribution corresponding to various skin conditions (e.g., healthy, seborrheic keratosis, basal cell carcinoma, melanoma, atypical nevi, etc.). We take the category with the highest probability as the initial classification result and its corresponding probability value as the classification confidence of the result.

[0079] 5.2 Triggering of the interactive feedback mechanism; The system presets a confidence threshold (e.g., 0.85). When the confidence of the initial classification result is lower than this threshold, it indicates that the model has low confidence in the current judgment, which may be due to image quality, atypical lesions, or insufficient information. At this time, the system will automatically activate the interactive diagnostic feedback mechanism to actively seek external information to assist in decision-making.

[0080] 5.3 Generating and responding to interactive requests; Request Generation: Instead of simply reporting "low confidence," the system generates specific, actionable interactive requests based on the initial classification results and uncertainties in the feature space, guiding users to supplement the most valuable clinical information. For example: If the model initially identifies "melanoma" but with low confidence, it may suggest: "The current diagnosis has low confidence and is suspected to be an atypical pigmented nevus. Please provide a dermoscopic polarized light image of the area to observe the deeper structures." If the lesion's morphology is between benign and malignant, it may prompt: "Please provide information on whether the patient has a history of rapid growth of this lesion recently?" or "Please provide information on the patient's immediate family history of skin cancer." The types of specific clinical information include, but are not limited to: dermoscopic images from specific imaging angles, key medical history texts, or skin physiological data measured by specialized equipment (such as skin oil content test data).

[0081] User Response: The user (doctor or patient) supplements the relevant clinical information according to the system prompts.

[0082] 5.4 Information Supplementation and Result Update; New Feature Extraction: The system inputs the information provided by the user into the corresponding feature extractor. For example, if the information provided is an image, the Swin Transformer is used to extract new image features; if the information provided is text, an MLP is used to extract new text features.

[0083] Secondary fusion: The extracted new feature vector is concatenated or weighted and summed with the original fused feature vector V_fused to form a fused feature vector with more comprehensive information and greater discriminative power.

[0084] Final classification: This fused feature vector is then fed back into the classifier for computation. Because of the introduction of crucial new evidence, the classifier is typically able to output a more reliable and confident final classification result, thus validating and updating the initial diagnosis.

[0085] In summary, a skin condition detection method based on image recognition has the following advantages: By integrating targeted preprocessing steps for hair removal, bubble removal, and black border removal, this invention effectively removes the most common interfering noise in clinical images. Compared to single or general filtering methods, this multimodal cleaning scheme significantly improves the quality of the input image, providing a clean and reliable data foundation for all subsequent analysis steps. This not only directly improves the accuracy of segmentation and classification but also enhances the robustness of the entire system under different acquisition devices and environments, reducing the risk of misdiagnosis due to image quality issues.

[0086] By replacing some feature extraction modules in the encoder with deep residual dilated convolution modules, the receptive field of the model is significantly expanded without increasing the number of model parameters or computational complexity. This allows the model to simultaneously capture the local details of lesions and their global context, thus achieving more accurate and robust segmentation of complex skin lesions of varying sizes, blurred boundaries, and irregular shapes. Accurate segmentation results provide a reliable guarantee for all subsequent quantitative analyses and diagnostic decisions.

[0087] Using semi-supervised federated learning as the training framework for the model yields two beneficial effects: In terms of data utilization efficiency: By using semi-supervised learning to generate pseudo-labels from a large amount of unlabeled data, the reliance on expensive and scarce medical expert-annotated data is greatly reduced, thus lowering the cost and threshold of model training.

[0088] In terms of data privacy and model performance: By training models locally in various medical institutions and aggregating only the parameters through federated learning, the privacy of sensitive medical data is fundamentally protected, breaking down "data silos". At the same time, it is possible to train high-performance models with stronger generalization ability and more comprehensiveness by utilizing more widely distributed data.

[0089] By automatically extracting the physical size and morphological quantitative features of lesions, the clinical diagnostic experience of doctors (such as asymmetry and boundaries in the ABCD rule) is transformed into calculable objective indicators, greatly enriching the feature dimensions used for diagnosis. Through a dynamic weighted fusion strategy based on mutual information, the correlation between image and text features is intelligently evaluated and weighted, achieving information complementarity and redundancy removal, generating a more discriminative fusion feature representation. The combination of these two approaches significantly improves the model's ability to distinguish similar skin diseases, especially early malignant lesions, and enhances classification accuracy.

[0090] By guiding users to supplement the most critical diagnostic information (such as specific images or medical history) and using this information for secondary decision-making, the diagnostic results were dynamically optimized and corrected.

[0091] Example 2 like Figure 2 As shown, in a second embodiment of the present invention, a skin condition detection system based on image recognition is provided, the system comprising: The processing module 10 is used to acquire the skin image to be analyzed and related text metadata, and to preprocess the skin image to remove interference noise from the image; The skin lesion segmentation module 20 is used to input the preprocessed image into the trained skin lesion segmentation model and output the segmentation result of the lesion area. The integration module 30 is used to automatically extract the size information of the skin lesions and at least one morphological quantitative feature based on the segmentation result, and integrate the size information and the morphological quantitative feature as supplementary features into the text metadata to form enhanced metadata; The fusion module 40 is used to extract the image features of the skin image and the text features of the enhanced metadata. Based on the mutual information between the image features and the text features, it assigns appropriate weights to the two and performs dynamic weighted concatenation to form a fused feature vector. The output module 50 is used to input the fused feature vector into the classifier and output the classification result of the skin state.

[0092] The skin condition detection system based on image recognition provided in this embodiment of the invention has the same implementation principle and technical effect as the aforementioned method embodiment. For the sake of brevity, any parts not mentioned in the system embodiment can be referred to the corresponding content in the aforementioned method embodiment.

[0093] Example 3 like Figure 3As shown, in the third embodiment of the present invention, the present invention provides the following technical solution: a computer, including a memory 202, a processor 201, and a computer program stored in the memory 202 and executable on the processor 201, wherein the processor 201 executes the computer program to implement the skin state detection method based on image recognition as described above.

[0094] Specifically, the processor 201 may include a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0095] The memory 202 may include a large-capacity memory for data or instructions. For example, and not limitingly, the memory 202 may include a hard disk drive (HDD), a floppy disk drive, a solid-state drive (SSD), flash memory, an optical disk drive, a magneto-optical disk drive, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 202 may include removable or non-removable (or fixed) media. Where appropriate, the memory 202 may be internal or external to a data processing device. In a particular embodiment, the memory 202 is non-volatile memory. In a particular embodiment, the memory 202 includes read-only memory (ROM) and random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable read-only memory (PROM), an erasable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), an electrically alterable read-only memory (EAROM), or flash memory, or a combination of two or more of these. Where appropriate, the RAM can be Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). DRAM can be Fast Page Mode Dynamic Random Access Memory (FPMDRAM), Extended Data Out Dynamic Random Access Memory (EDODRAM), Synchronous Dynamic Random Access Memory (SDRAM), etc.

[0096] The memory 202 can be used to store or cache various data files that need to be processed and / or communicated, as well as possible computer program instructions executed by the processor 201.

[0097] The processor 201 implements the above-described image recognition-based skin condition detection method by reading and executing computer program instructions stored in the memory 202.

[0098] In some embodiments, the computer may further include a communication interface 203 and a bus 200. For example, Figure 3 As shown, the processor 201, memory 202, and communication interface 203 are connected through bus 200 and complete communication with each other.

[0099] The communication interface 203 is used to enable communication between the various modules, devices, units, and / or equipment in the embodiments of this application. The communication interface 203 can also enable data communication with other components such as external devices, image / data acquisition devices, databases, external storage, and image / data processing workstations.

[0100] Bus 200 includes hardware, software, or both, that couples computer components together. Bus 200 includes, but is not limited to, at least one of the following: Data Bus, Address Bus, Control Bus, Expansion Bus, and Local Bus. For example, and not as a limitation, bus 200 may include Accelerated Graphics Port (AGP) or other graphics buses, Extended Industry Standard Architecture (EISA) buses, Front Side Bus (FSB), HyperTransport (HT) interconnects, Industry Standard Architecture (ISA) buses, InfiniBand interconnects, and Low Pin Count (LPC) interconnects. Bus 200 may include a bus, memory bus, MicroChannel Architecture (MCA) bus, Peripheral Component Interconnect (PCI) bus, PCI Express (PCIX) bus, Serial Advanced Technology Attachment (SATA) bus, Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses, or a combination of two or more of these. Where appropriate, bus 200 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, this application contemplates any suitable bus or interconnect.

[0101] Example 4 In the fourth embodiment of the present invention, in conjunction with the above-described image recognition-based skin condition detection method, the present invention provides the following technical solution: a storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the above-described image recognition-based skin condition detection method.

[0102] Those skilled in the art will understand that the data in the flowchart, or logic and / or steps otherwise described herein, for example, can be considered as a sequenced data table of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can mean any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0103] More specific examples of readable media (a non-exhaustive list of data) include: electrical connections (electronic devices) with one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0104] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0105] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0106] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A skin condition detection method based on image recognition, characterized in that, The method includes: Obtain the skin image to be analyzed and related text metadata, and preprocess the skin image to remove interference noise from the image; The preprocessed image is input into the trained skin lesion segmentation model, and the segmentation result of the lesion area is output. Based on the segmentation results, the size information of the skin lesions and at least one morphological quantitative feature are automatically extracted. The size information and the morphological quantitative feature are then integrated into the text metadata as supplementary features to form enhanced metadata. Extract the image features of the skin image and the text features of the enhanced metadata. Based on the mutual information between the image features and the text features, assign appropriate weights to the two and perform dynamic weighted concatenation to form a fused feature vector. The fused feature vector is input into the classifier, which outputs the classification result of the skin condition.

2. The skin condition detection method based on image recognition according to claim 1, characterized in that, The training method for the skin lesion segmentation model includes: Obtain a dataset of skin disease images, which includes a small number of labeled samples and a large number of unlabeled samples; The unlabeled samples are clustered using a clustering algorithm. Based on a distance sampling strategy, some unlabeled samples are selected from each cluster and initial pseudo-labels are generated. The initial pseudo-labels are merged with the original labeled samples to form an expanded labeled training set; Based on the improved course learning strategy, the model is trained using the labeled training set and the remaining unlabeled samples; wherein the improved course learning strategy dynamically adjusts the confidence threshold of the model when generating the initial pseudo-labels for unlabeled samples of different categories.

3. The skin condition detection method based on image recognition according to claim 2, characterized in that, The formula for dynamically adjusting the confidence threshold include: in, Indicates the first Each category during the training period The confidence threshold at that time This represents the initial confidence threshold. Represents the balance factor. Indicates the first Each category during the training period Historical average confidence level across clients Indicates the first Each category during the training period The weights of sample distribution across clients.

4. The skin condition detection method based on image recognition according to claim 1, characterized in that, The steps for outputting the classification results of skin condition include: Output the initial classification result of the skin condition and the corresponding classification confidence score; When the classification confidence level is lower than a preset threshold, an interactive request is generated to guide the user to supplement specific clinical information, and the initial classification result is verified and updated based on the specific clinical information supplemented by the user. The specific clinical information includes dermoscopy images from specific angles, medical history texts, or skin oiliness detection data; the supplementary specific clinical information is extracted as new features, fused with the original fusion feature vector, and then input into the classifier again to obtain the updated final classification result.

5. The skin condition detection method based on image recognition according to claim 1, characterized in that, The automatic extraction of the size information of skin lesions specifically includes: Obtain the original clinical image containing the ruler reference and its corresponding segmentation results; The segmentation results are registered with the original clinical image to determine the precise location of the lesion outline in the original clinical image; In the original clinical image, the ruler scale edge and the lesion outline are extracted using an edge detection algorithm; Based on the extracted ruler scale edges, the number of pixels corresponding to a unit of actual physical length is determined to establish the conversion relationship between pixels and actual physical size. Measure the number of pixels occupied by the lesion outline in the image; Based on the conversion relationship, the number of pixels in the lesion outline is converted into the actual physical size.

6. The skin condition detection method based on image recognition according to claim 5, characterized in that, The formula for calculating the actual physical dimensions is: in: Indicates the actual physical length of the lesion. This represents the number of pixels occupied by the lesion outline in the image. To provide a reference for the number of pixels per unit length of the ruler in the image, The actual physical length of the reference ruler.

7. The skin condition detection method based on image recognition according to claim 1, characterized in that, At least some feature extraction modules in the encoder of the skin lesion segmentation model are replaced with deep residual dilated convolution modules to expand the receptive field of the model. Among them, in the deep residual dilated convolution module, the first The size of the receptive field after layer-hole convolution is calculated using the following formula: in: For the first The size of the receptive field of the layer For the first Layer step size, For the first The expansion rate of layer void convolution, For the first The size of the convolution kernel, For the first The size of the receptive field after convolution of the void layer.

8. A skin condition detection system based on image recognition, characterized in that, The system includes: The processing module is used to acquire the skin image to be analyzed and related text metadata, and to preprocess the skin image to remove interference noise from the image; The skin lesion segmentation module is used to input the preprocessed image into the trained skin lesion segmentation model and output the segmentation results of the lesion area; An integration module is used to automatically extract the size information of skin lesions and at least one morphological quantitative feature based on the segmentation results, and integrate the size information and the morphological quantitative feature as supplementary features into the text metadata to form enhanced metadata; The fusion module is used to extract image features of the skin image and text features of the enhanced metadata. Based on the mutual information between the image features and the text features, it assigns appropriate weights to the two and performs dynamic weighted concatenation to form a fused feature vector. The output module is used to input the fused feature vector into the classifier and output the classification result of the skin state.

9. A computer comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the skin condition detection method based on image recognition as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the skin condition detection method based on image recognition as described in any one of claims 1 to 7.

Images