Content-based multi-modal skin disease searching method

Through the multimodal skin disease search method, the DCNN model is used to extract and fuse the skin disease images and metadata features, and the problem of inaccurate and timely diagnosis results in the prior art is solved, and more efficient diagnosis and classification of skin disease is achieved.

CN119006910APending Publication Date: 2024-11-22HEFEI QIANSHOU MEDICAL TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411090477.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-09
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

The existing technology cannot effectively process large-scale and diversified skin disease case data, resulting in inaccurate and timely diagnosis results.

Method used

Using content-based multimodal skin disease search method, image and metadata feature extraction and fusion are performed through deep convolutional neural network (DCNN) model, classification model of skin disease images is established, pre-trained model is used for fine-tuning, and data enhancement technology is combined , expand the training data set, improve the generalization ability and classification accuracy of the model.

Benefits of technology

It improves the accuracy and comprehensiveness of skin disease diagnosis, enhances the generalization ability of the model, and significantly improves the performance of skin disease classification and search.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119006910A_ABST
    Figure CN119006910A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of skin disease search, and particularly discloses a content-based multi-modal skin disease search method, which comprises the following steps of: collecting a medical data set containing skin disease images; the medical data set is preprocessed; feature extraction is carried out by using a deep convolutional neural network (DCNN) model, the feature extraction comprises image feature extraction and metadata feature extraction, and abstract representation of data is learned through convolution, pooling and a full connection layer; the feature extraction is input into a full-connection layer for classification training, and a classification model of the skin disease images is established. Through image and multi-modal data fusion, the accuracy and comprehensiveness of skin disease diagnosis are improved, data enhancement technologies such as rotation, zooming and overturning are utilized, a training data set is expanded, the generalization ability of the model is improved, and meanwhile, the classification model of the skin disease images is established. A pre-trained DCNN model is used for fine tuning in the field of medical images, model training is accelerated, and classification accuracy is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of skin disease search, and in particular to a content-based multimodal skin disease search method. Background Art

[0002] Dermatology case search utilizes modern technology and data resources to achieve rapid and accurate search and diagnosis of dermatology cases, so as to improve the efficiency and accuracy of medical diagnosis and provide better medical services for doctors and patients. The current dermatology case search has the following problems and shortcomings: a) data sources are limited and cannot cover data of various types and cases; b) data classification and annotation are not detailed enough, resulting in inaccurate and incomplete search results; c) image recognition and matching technology is not mature enough to meet the needs of identification and differentiation of complex cases; dermatology cases cover a variety of types and pathological characteristics, including but not limited to rashes, eczema, urticaria, skin cancer, etc., and their clinical manifestations and causes are diverse, making data collection and analysis challenging.

[0003] The root of the problem lies in the incompleteness and dispersion of the data, as well as the limitations of existing technologies. Traditional search methods cannot effectively handle large-scale and diverse skin disease case data, resulting in inaccurate and intime diagnosis results. To this end, we propose a content-based multimodal skin disease search method. Summary of the invention

[0004] The purpose of the present invention is to provide a content-based multimodal skin disease search method to solve the problem that the traditional search method proposed in the above background technology cannot effectively process large-scale and diverse skin disease case data, resulting in inaccurate and in-time diagnosis results.

[0005] To achieve the above-mentioned object, the present invention provides the following technical solutions: a content-based multimodal skin disease search method, collecting a medical data set containing skin disease images;

[0006] Preprocessing the medical data set;

[0007] Use the deep convolutional neural network (DCNN) model to extract features, including image feature extraction and metadata feature extraction, and learn the abstract representation of data through convolution, pooling and fully connected layers;

[0008] The feature extraction is input into the fully connected layer for classification training to establish a classification model for skin disease images.

[0009] Among them, the medical data set is standardized, including adjusting the image size and normalization, so that the format of all input images is consistent and suitable for the input requirements of the DCNN model;

[0010] Normalization formula:

[0011] I' = resize(I'(299,999));

[0012] I" = normalize(I');

[0013] Each skin disease image in the medical dataset has corresponding patient metadata, where the metadata includes information such as age, gender, and medical history;

[0014] The metadata feature extraction preprocessing formula is:

[0015] M'=normalize(encodeM).

[0016] Among them, feature extraction includes using pre-trained deep convolutional neural networks, such as the InceptionV3 model to extract image features. DCNN gradually extracts high-level features of the image through multiple layers of convolution, pooling, and fully connected layers;

[0017] The image feature extraction formula:

[0018]

[0019] in, is the output of the kth convolution kernel at position (i, j), is the weight of the kth convolution kernel, χ (i+m)(j+n) is the pixel value of the input image at position (i+m,j+n), b k is the bias term, σ is the activation function;

[0020] P ij =max(F ij );

[0021] Among them, (P_{ij}) is the output after pooling, which represents the maximum value selected from the pooling window;

[0022] The metadata feature extraction formula is:

[0023] z=W·x+b;

[0024] Among them, z is the output of the fully connected layer, W is the weight matrix, x is the input feature, and b is the bias term.

[0025] Among them, the feature fusion formula of the image feature and the metadata feature is:

[0026] F fusion =concat(F img ,F meta );

[0027] Among them, Ffusion is the fused feature, F img is the image feature, F meta It is a metadata feature;

[0028] The fused features are used to classify skin diseases and the probability distribution of each category is output;

[0029] The probability distribution formula:

[0030] y=softmax(FC(F fusion ));

[0031] Among them, y is the probability distribution of the classification result, FC is the fully connected layer, and softmax is the activation function.

[0032] Among them, the InceptionV3 model is trained using the training data set, and the extracted features are stored in the feature database. The image features in the feature database are measured for similarity with the features of the query image. The feature matching and similarity measurement formulas are:

[0033]

[0034] Where d is the distance between the query image and the database image, is the i-th eigenvalue of the query image, is the i-th eigenvalue of the database image;

[0035] Based on the similarity measurement results, the images most similar to the query image are retrieved and returned as retrieval results.

[0036] The present invention has at least the following beneficial effects:

[0037] The present invention improves the accuracy and comprehensiveness of skin disease diagnosis through image and multimodal data fusion, and uses data enhancement technology, such as rotation, scaling, flipping and other operations, to expand the training data set and increase the generalization ability of the model. At the same time, the pre-trained DCNN model is used for fine-tuning in the field of medical images to accelerate model training and improve classification accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a flowchart of the content-based multimodal skin disease search method of the present invention;

[0039] Figure 2 Schematic diagram of the skin disease search system architecture of the present invention. DETAILED DESCRIPTION

[0040] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] See also Figure 1-2 ,The present invention provides a technical solution: a content-based multimodal skin disease search method that collects a medical dataset containing skin disease images;

[0042] Preprocessing the medical data set;

[0043] Use the deep convolutional neural network (DCNN) model to extract features, including image feature extraction and metadata feature extraction, and learn the abstract representation of data through convolution, pooling and fully connected layers;

[0044] The feature extraction is input into the fully connected layer for classification training to establish a classification model for skin disease images;

[0045] This specification introduces a classification-driven multimodal skin disease search algorithm, which aims to retrieve images similar to the input image from a medical database. The algorithm adopts a deep convolutional neural network DCNN model to learn the abstract representation of data through convolution, pooling and fully connected layers, and separates the feature extraction and classification components, making full use of the powerful capabilities of DCNN in the visual system;

[0046] The present invention improves the accuracy and comprehensiveness of skin disease diagnosis through image and multimodal data fusion, and uses data enhancement technology, such as rotation, scaling, flipping and other operations, to expand the training data set and increase the generalization ability of the model. At the same time, the pre-trained DCNN model is used for fine-tuning in the field of medical images to accelerate model training and improve classification accuracy.

[0047] Medical dataset standardization, including image resizing and normalization, to make all input images consistent in format and suitable for the input requirements of the DCNN model;

[0048] Normalization formula:

[0049] I' = resize(I'(299,999));

[0050] I" = normalize(I');

[0051] Each skin disease image in the medical dataset has corresponding patient metadata, where the metadata includes information such as age, gender, and medical history;

[0052] The metadata feature extraction preprocessing formula is:

[0053] M' = normalize(encodeM);

[0054] Collect medical datasets containing skin disease images. Each image should have corresponding patient metadata, including age, gender, medical history and other information. Standardize the collected images, including preprocessing of image data, including denoising, standardization, cropping and other operations to ensure data quality and consistency, suitable for the input requirements of the DCNN model, and encode and normalize the patient's metadata to ensure data consistency and validity.

[0055] Feature extraction includes using pre-trained deep convolutional neural networks, such as the InceptionV3 model, to extract image features. DCNN gradually extracts high-level features of images through multiple layers of convolution, pooling, and fully connected layers.

[0056] The image feature extraction formula:

[0057]

[0058] in, is the output of the kth convolution kernel at position (i, j), is the weight of the kth convolution kernel, χ (i+m)(j+n) is the pixel value of the input image at position (i+m,j+n), b k is the bias term, σ is the activation function;

[0059] P ij =max(F ij );

[0060] Among them, (P_{ij}) is the output after pooling, which represents the maximum value selected from the pooling window;

[0061] The metadata feature extraction formula is:

[0062] z=W·x+b;

[0063] Among them, z is the output of the fully connected layer, W is the weight matrix, x is the input feature, and b is the bias term.

[0064] Image feature extraction uses the pre-trained deep convolutional neural network InceptionV3 model to extract image features. DCNN gradually extracts high-level features of the image through multiple layers of convolution, pooling, and fully connected layers. Metadata feature extraction uses a multi-layer perceptron (MLP) to process metadata. MLP gradually extracts high-level features of metadata through a series of fully connected layers and performs normalization.

[0065] The feature fusion formula of the image feature and the metadata feature is:

[0066] F fusion =concat(F img ,F meta );

[0067] Among them, F fusion is the fused feature, F img is the image feature, F meta It is a metadata feature;

[0068] The fused features are used to classify skin diseases and the probability distribution of each category is output;

[0069] The probability distribution formula:

[0070] y=softmax(FC(F fusion ));

[0071] Among them, y is the probability distribution of the classification result, FC is the fully connected layer, and softmax is the activation function.

[0072] The image features and metadata features are fused to make use of multimodal information for disease classification and search, which is convenient for feature extraction. The extracted features are input into the fully connected layer for classification training, and a classification model for skin disease images is established. The fused features are used to classify skin diseases. The classifier uses the fully connected layer to process the fused features and outputs the probability distribution of each category. In the training phase, the model is trained using the training data set, and the extracted features are stored in the feature database.

[0073] The InceptionV3 model is trained using a training data set and the extracted features are stored in a feature database. The image features in the feature database are similarly measured with the features of the query image. The feature matching and similarity measurement formulas are:

[0074]

[0075] Where d is the distance between the query image and the database image, is the i-th eigenvalue of the query image, is the i-th eigenvalue of the database image;

[0076] Based on the similarity measurement results, the images most similar to the query image are retrieved and returned as retrieval results.

[0077] During image retrieval, a query image is input, and the features of the query image are extracted using the trained DCNN model. Based on the similarity measurement results, the images most similar to the query image are retrieved and returned as retrieval results.

[0078] Skin disease search system architecture:

[0079] 1. Input layer:

[0080] Query Image: Input skin disease image;

[0081] Patient Metadata: Input patient metadata;

[0082] 2. DCNN feature extraction module:

[0083] Use deep convolutional neural networks to extract high-level features of images;

[0084] 3. MLP feature extraction module:

[0085] Use multi-layer perceptron to extract high-level features of metadata;

[0086] 4. Feature fusion module:

[0087] Combine image and metadata features to generate comprehensive feature representation;

[0088] 5. Classifier

[0089] Use the fully connected layer to classify the fused features and output the classification results;

[0090] 6. Feature Database

[0091] Store the features extracted from the training dataset;

[0092] 7. Similarity measurement module:

[0093] Calculate the similarity between the query image and the image features in the database;

[0094] 8. Retrieve images:

[0095] Returns the images most similar to the query image.

[0096] The multimodal skin disease search algorithm combines image and metadata information, and uses the powerful feature extraction capabilities of DCNN and the metadata processing capabilities of MLP to achieve efficient and accurate disease classification and search. The algorithm can be applied to medical image database management, clinical decision support systems, medical research and other fields to help doctors and researchers quickly and accurately retrieve and analyze medical images.

[0097] By adopting a classification-driven multimodal skin disease search algorithm, images similar to the input image can be effectively retrieved from the medical database, and combined with the patient's metadata information for more accurate disease classification and diagnosis. This algorithm utilizes the powerful feature extraction capabilities of DCNN and multimodal data fusion strategies, significantly improving the performance of skin disease classification and search.

[0098] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0099] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. Content-based multimodal skin disease search method: Collect medical datasets containing skin disease images; Preprocessing the medical data set; Use the deep convolutional neural network (DCNN) model to extract features, including image feature extraction and metadata feature extraction, and learn the abstract representation of data through convolution, pooling and fully connected layers; The feature extraction is input into the fully connected layer for classification training to establish a classification model for skin disease images.

2. The content-based multimodal skin disease search method according to claim 1, characterized in that: The medical dataset is standardized, including adjusting the image size and normalization, so that the format of all input images is consistent and suitable for the input requirements of the DCNN model; Normalization formula: I' = resize(I'(299,999)); I" = normalize(I'); Each skin disease image in the medical dataset has corresponding patient metadata, where the metadata includes information such as age, gender, and medical history; The metadata feature extraction preprocessing formula is: M'=normalize(encodeM).

3. The content-based multimodal skin disease search method according to claim 1, characterized in that: The feature extraction includes using a pre-trained deep convolutional neural network, such as the InceptionV3 model, to extract image features. DCNN gradually extracts high-level features of the image through multiple layers of convolution, pooling, and fully connected layers. The image feature extraction formula: in, is the output of the kth convolution kernel at position (i, j), is the weight of the kth convolution kernel, χ (i+m)(j+n) is the pixel value of the input image at position (i+m,j+n), b k is the bias term, σ is the activation function; P ij =max(F ij ); Among them, (P_{ij}) is the output after pooling, which represents the maximum value selected from the pooling window; The metadata feature extraction formula is: z=W·x+b; Among them, z is the output of the fully connected layer, W is the weight matrix, x is the input feature, and b is the bias term.

4. The content-based multimodal skin disease search method according to claim 1, characterized in that: The feature fusion formula of the image feature and the metadata feature is: F fusion =concat(F img ,F meta ); Among them, F fusion is the fused feature, F img is the image feature, F meta It is a metadata feature; The fused features are used to classify skin diseases and the probability distribution of each category is output; The probability distribution formula: y=softmax(FC(F fusion )); Among them, y is the probability distribution of the classification result, FC is the fully connected layer, and softmax is the activation function.

5. The content-based multimodal skin disease search method according to claim 3, characterized in that: The InceptionV3 model is trained using a training data set, and the extracted features are stored in a feature database. The image features in the feature database are similarly measured with the features of the query image. The feature matching and similarity measurement formulas are: Where d is the distance between the query image and the database image, is the i-th eigenvalue of the query image, is the i-th eigenvalue of the database image; Based on the similarity measurement results, the images most similar to the query image are retrieved and returned as retrieval results.