Androgenetic alopecia detection method based on multi-modal feature extraction

Abstract

The invention discloses an androgenetic alopecia detection method based on multi-modal feature extraction, and the method comprises the following steps: S1, collecting sample data: collecting two sample data of a plurality of patients with a difference of 180 days or more, each sample data being composed of hair image data and medical record data of a user; s2, constructing an image feature extraction model and a natural language processing model based on a fine tuning technology; s3, performing feature alignment on the feature data by the full connection layer, and inputting the feature data into a Mamba model for processing; and S4, calculating the correlation between the two modals of the image and the text by adopting a bidirectional attention mechanism to obtain joint feature data, and outputting the joint feature data to a classification layer after passing through a feature splicing layer. According to the invention, accurate detection of the hair state of the patient can be realized, reliable diagnosis data can be output, and the hair detection efficiency can be improved.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent medical image-assisted diagnosis and artificial intelligence technology, and particularly relates to a multimodal feature extraction method for androgenic alopecia detection. Background Art

[0002] Androgenic alopecia is a common hair disease that causes widespread distress to the majority of men and women. Existing methods for detecting androgenic alopecia are mainly based on manual diagnosis by doctors, which requires doctors to observe visually with their naked eyes, which consumes a lot of time and is inefficient. In addition, the uneven distribution of medical resources will further aggravate this problem. With the advancement of technology, some automatic detection methods based on neural networks have been proposed, but these methods often only use the patient's images to extract hair-related information from the image information, lack the integration of patient medical records, and the single input information may lead to low model prediction performance, thereby reducing the accuracy of the detection and failing to comprehensively and correctly judge the final correct state of the hair. There are also some methods that take medical record data into consideration when detecting hair loss, but they simply use medical record data through expert rules, decision trees and other methods, without in-depth and sufficient analysis and utilization of the characteristics of medical record data, resulting in low accuracy of the output results. Summary of the invention

[0003] The purpose of the present invention is to overcome the shortcomings of the prior art and provide an androgenic alopecia detection method that can accurately detect the patient's hair status, output reliable diagnostic data, and improve the efficiency of hair detection through multimodal feature extraction.

[0004] The object of the present invention is achieved through the following technical solution: a multimodal feature extraction androgenic alopecia detection method, comprising the following steps:

[0005] S1. Collect sample data: collect two sets of sample data from multiple patients with a difference of more than 180 days. Each set of sample data consists of the user's hair image data and medical record data.

[0006] S2, build an image feature extraction model and a natural language processing model based on fine-tuning technology;

[0007] The image feature extraction model consists of a ResNet-50 backbone network, in which there are four residual blocks. After passing through the four residual blocks, the feature map is input into a multi-level multi-scale feature network consisting of four multi-scale feature extraction modules, each of which contains four scale blocks. The feature maps output by the four residual blocks are respectively input into the scale blocks of the same level and one level deeper in the first multi-scale feature extraction module. In the multi-level multi-scale feature extraction module, the features output by the scale block of the previous layer are simultaneously input into the scale blocks of the same level and one level deeper in the next layer. Then, multiple feature maps are respectively passed through a 3*3 convolutional layer and then group normalized to obtain feature values ​​after scale-offset transformation. After obtaining the feature values, they are passed into the ReLu activation layer, and finally the feature data is transmitted to the pooling layer.

[0008] The loss function L involved in the image feature extraction model training process CE The formula is as follows:

[0009]

[0010] Among them, C is the number of categories, y i,c and are the actual and predicted probabilities of the ith sample for the cth class, respectively.

[0011] The natural language processing model based on fine-tuning technology consists of an embedding layer based on Llama 3 and a Transformer model;

[0012] The loss function L' involved in the text feature extraction model training process CE The formula is as follows:

[0013]

[0014] Where V is the vocabulary, y i (ω) and are the actual and predicted probabilities of the wth word in the ith sample, respectively.

[0015] A fully connected layer is connected after the Transformer model to fuse the data output by the image feature extraction model with the data output by the natural language processing model based on fine-tuning technology;

[0016] S3, the fully connected layer aligns the feature data and inputs it into the Mamba model for processing;

[0017] S4. Use the bidirectional attention mechanism to calculate the correlation between the two modalities of image and text, obtain the joint feature data, and output the joint feature data to the classification layer after passing it through a feature splicing layer.

[0018] The beneficial effects of the present invention are as follows: the present invention obtains the feature data of hair through the image feature extraction module, obtains the feature data of the patient's medical history through the text feature extraction module, and then calculates the relationship between the two modalities through the bidirectional attention mechanism, and outputs new joint feature data, and finally the joint feature data outputs detailed hair detection results, hair loss area trend prediction results and patient diagnosis reports through the classification layer. It can realize accurate detection of the patient's hair status, output reliable diagnostic data, and improve the efficiency of hair detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is an overall flow chart of the androgenic alopecia detection method of the present invention;

[0020] Figure 2 This is the structural diagram of the image feature extraction module;

[0021] Figure 3 This is the structural diagram of the text feature extraction module;

[0022] Figure 4 Schematic diagram of feature data fusion of bidirectional attention mechanism. DETAILED DESCRIPTION

[0023] The present invention proposes a method for detecting androgenic alopecia based on multimodal feature extraction, which mainly includes two modules: image feature extraction and text feature extraction. The image feature extraction module is mainly used for feature extraction of user hair pictures, and the text feature extraction module is mainly used for feature extraction of disease data. Finally, the feature data extracted from the image and the feature data extracted from the text are subjected to a series of operations such as feature alignment, feature fusion, joint representation and decision layer, effectively fusing the image information and the text information, generating a comprehensive feature representation containing multimodal information, and finally introducing the Mamba layer and the bidirectional attention mechanism to deepen the relationship between semantics, time series and features, thereby outputting the patient's predicted hair loss trend and shape and other information. Compared with the prediction information of LSTM, Mamba combines the advantages of memory enhancement, parallel processing and multi-branch attention mechanism in processing and learning the fused feature data, and performs well in processing long sequences, improving computational efficiency and model expression ability, overcoming some limitations of traditional RNN and LSTM. Finally, the joint feature data outputs detailed hair detection results, hair loss area trend prediction results and diagnosis reports for patients through the classification layer. The detailed process of the entire detection method is described as follows. The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0024] like Figure 1 As shown, a multimodal feature extraction androgenetic alopecia detection method comprises the following steps:

[0025] S1. Collect sample data: collect two sets of sample data from multiple patients with a difference of more than 180 days. Each set of sample data consists of the user's hair image data and medical record data.

[0026] S2, build an image feature extraction model and a natural language processing model based on fine-tuning technology;

[0027] The image feature extraction model is composed of a ResNet-50 backbone network. There are four residual blocks C1 to C4 distributed in the ResNet-50 backbone network. After passing through the four residual blocks, the feature maps obtained at each level are input into a multi-level multi-scale feature extraction network. After passing through the multi-level multi-scale feature network, the obtained data is transmitted to a module consisting of a 3*3 convolutional layer, group normalization, a linear rectifier unit (ReLU) and a pooling layer. Finally, the output data of the pooling layer is transmitted to a fully connected layer, waiting for feature dimension adjustment with the output data of the natural language processing model to match the dimensions of the two. The pooling layer effectively reduces the dimension of the data, enhances feature extraction, reduces computational complexity, and reduces overfitting, making the model more efficient and stable.

[0028] The four residual blocks will obtain feature maps with resolutions of 32*32, 64*64, 128*128, and 256*256 respectively. After obtaining the four feature maps, they are input into a multi-level multi-scale feature network composed of four multi-scale feature extraction modules L1~L4. Each multi-scale feature extraction module contains four scale blocks (Reshape). The scale blocks in L1~L4 are represented as P11~P14, P21~P24, P31~P34, and P41~P44 respectively. Figure 2As shown. The feature maps output by the four residual blocks are respectively input into the same-level and deeper-level scale blocks in the first multi-scale feature extraction module; in the multi-level multi-scale feature extraction module, the features output by the Reshape of the previous layer are simultaneously input into the same-level Reshape and the deeper-level Reshape of the next layer as the data source of the multi-level feature extraction, so as to realize the fine-grained extraction of features. The feature data output by the multi-level multi-scale feature fusion module is input into a fully connected layer to obtain the feature data of the hair. In the multi-level multi-scale feature network, the output of the C4 residual block of the backbone network is used as the input of the first layer P14 and P13 of the feature network, and the feature map extracted by P14 is used as the output of P24 and P23, forming a multi-level feature extraction network successively to realize the multi-fine-grained extraction of image features. The feature maps at multiple scales of multiple levels of features are obtained through the multi-level multi-scale feature network, and various information of the image at different resolutions and viewing angles is obtained. Then, the four features output by the multi-level multi-scale feature network are input into the 3*3 convolution layer to obtain the feature data of the feature map at each scaling scale. The feature data obtained are different due to different scaling scales, so the 3*3 convolution layer and group normalization processing method are introduced after the multi-level feature network. The feature data obtained from multiple feature maps are respectively passed through the 3*3 convolution layer and then group normalized to obtain the feature values ​​after scale-offset transformation. After obtaining the feature values, they are passed to the ReLu activation layer to form feature data with nonlinear relationships. Finally, the feature data is transmitted to the pooling layer to obtain a feature map of reduced size. The pooling layer effectively reduces the dimension of the data, enhances feature extraction, reduces computational complexity, and reduces overfitting, making the model more efficient and stable.

[0029] The loss function L involved in the image feature extraction model training process CE The formula is as follows:

[0030]

[0031] Among them, C is the number of categories, y i,c and are the actual and predicted probabilities of the ith sample for the cth class, respectively.

[0032] The natural language processing model based on fine-tuning technology consists of an embedding layer based on Llama 3 and a Transformer model; Figure 3As shown. The embedding layer based on Llama 3 obtains text information that can be used by the model by annotating and preprocessing the medical record information, including word segmentation, stop word removal, stem extraction and word form restoration, lowercase conversion, punctuation removal, etc. The text information obtained through preprocessing will generate a word embedding matrix in the embedding layer, so that each word is mapped to a vector in the word embedding matrix, and the text information in the medical record will be written as a two-dimensional matrix. When fine-tuning based on the Llama 3 model, select the instruction supervision mode and use a single GPU for training. The initial path of the model is " / your model path / Meta-Llama-3-8B-Instruct", and the training data is located in the "alpaca_zh" dataset under "LLaMA-Factory / data". We used the "llama3" template supported by Llama Factory and applied LoRA fine-tuning to the q_proj and v_proj modules of the model. The fine-tuned model will be saved to " / save output model / posttraining_checkpoint" and the existing cached data will be overwritten. During the training process, the batch size of each device is set to 2, and the training is optimized by 64 gradient accumulations. The learning rate scheduler selects the cosine scheduling method, the logging frequency is set to record once every 5 steps, and the model saving frequency is saved once every 100 steps. The initial learning rate is set to 5e-5, and the entire dataset is trained for one epoch. The fine-tuning type is LoRA, and the half-precision floating point training mode is turned on. The rank of LoRA is set to 4. After the fixed-dimensional dense vectors output by the embedding layer, these dense vector data are submitted to the Transformer model. The bidirectional attention mechanism of the Transformer model can effectively mine and capture global dependencies and important features in medical record text. The bidirectional attention mechanism can dynamically adjust the weight of each word and reallocate attention according to the context to better understand the text content.

[0033] The loss function L' involved in the text feature extraction model training process CE The formula is as follows:

[0034]

[0035] Where V is the vocabulary, y i (ω) and are the actual and predicted probabilities of the wth word in the ith sample, respectively.

[0036] At the same time, a fully connected layer is connected after the Transformer model to fuse the data output by the image feature extraction model with the data output by the Transformer model. After the above models are built, the feature data output by the two models are simultaneously transmitted to a fully connected layer, and the dimensions of the feature data are adjusted to make the dimensions of the feature data output by the two models the same. After obtaining data of the same dimension, the data is passed to Mamba to capture long-distance dependencies and strengthen the dependencies. Finally, the two feature data are used to use a bidirectional attention mechanism to calculate the attention weights from image to text and from text to image respectively. The weighted data obtained are used to form a new comprehensive feature representation through a feature splicing layer. The comprehensive features obtained by feature splicing are input into the classification layer, and the classification layer outputs the final hair detection results, predicted hair loss area trends, and diagnostic reports through comparison.

[0037] The sample data collected by S1 is used to train the image feature extraction model and the natural language processing model.

[0038] Input the image data into the image feature extraction model, and input the medical record data into the natural language processing model. Before training the model, you need to set the initialization value used by the model during training. Set the learning rate of the Adam optimizer of the image feature extraction model to 0.001, and Beta1=0.9 and Beta2=0.999 as the initial values ​​of the optimization parameters during training. Set the initial value of weight decay to 0.0001. During the training process, dynamically adjust the weight decay value based on the loss function curve and the complexity of model training. The smoothing coefficient in the loss function is 0.1. Set the batch size to 64, and set an initial epoch number. Dynamically draw the training and validation loss curves during training, observe their changes in real time during training, and adjust the number of training epochs appropriately. When it is observed that the model validation loss curve is no longer changing, it means that the validation loss of the model is no longer improving, and training can be stopped at this time.

[0039] In the natural language processing model, in addition to the different epoch numbers and batch sizes, a dropout rate of 0.1 is required when setting initialization parameters. During the training process, the values ​​are adjusted accordingly with the training situation to make the extracted data features more reliable. In the natural language processing model, due to the large semantic analysis, the batch number of the model is adjusted to 256. At the same time, the epoch number of the model is set to twice the initial value of the epoch number of the image feature extraction model, which improves the matching rate of the natural language model and enables it to quickly improve the accuracy.

[0040] After the initialization parameters of the model are prepared, the sample data prepared in step S1 is divided into training set data and validation set data in a ratio of 7:3. After the preparation is completed, the image data in the training set data and the corresponding medical record data are input into the image feature extraction model and the natural language processing model one by one.

[0041] During the training process, continuous classification and regression are used to improve the learning rate and correctness of the model. At the same time, the changes in the verification loss curves of each model and the results of the entire method are observed to judge the maturity of the model. When the verification loss curves of the two models no longer improve and the output hair detection results and diagnostic reports meet the corresponding characteristics, it indicates that a mature detection method has been obtained and the training is terminated. The final image feature extraction model and the natural language processing model based on fine-tuning technology are obtained.

[0042] S3, the fully connected layer aligns the feature data and then inputs it into the Mamba model for processing to capture long-distance dependencies and retain long-term dependency information.

[0043] Mamba is an existing model (https: / / zhuanlan.zhihu.com / p / 689215356): MAMBA (Memory-Augmented Multi-Branch Attention) has the following advantages compared to traditional RNN, LSTM and Transformer:

[0044] Long sequence memory capability: MAMBA introduces a memory enhancement mechanism that effectively captures and stores long sequence information through multiple memory modules, solving the problem of memory loss in RNN and LSTM when processing long sequences.

[0045] Parallel computing efficiency: Similar to Transformer, MAMBA uses a multi-head attention mechanism to process sequence data in parallel, instead of relying on step-by-step processing of sequences like RNN and LSTM, thereby improving computing efficiency.

[0046] Multi-branch structure: MAMBA’s multi-branch structure allows different branches to focus on different aspects of the sequence, further improving the expressiveness and performance of the model by parallel computing and fusing multiple features.

[0047] Adaptive weight adjustment: MAMBA’s multi-head attention mechanism can adaptively adjust the weights of different parts of information, making the model more flexible and accurate when processing complex sequence data.

[0048] Reducing the vanishing gradient problem: RNN and LSTM are prone to the vanishing gradient problem in long sequence training. MAMBA effectively alleviates this problem by introducing attention mechanisms and memory modules, thereby improving the stability and effectiveness of model training.

[0049] In summary, MAMBA combines the advantages of memory enhancement, parallel processing, and multi-branch attention mechanism, performs well in processing long sequences, improving computational efficiency and model expressiveness, overcomes some limitations of traditional RNN and LSTM, and is superior to Transformer in some aspects.

[0050] S4. A bidirectional attention mechanism is used to calculate the correlation between the two modalities of image and text, so as to obtain more effective joint feature data. The joint feature data is output to the classification layer after passing through a feature splicing layer, thereby outputting an automatic prediction result of the health status of the hair, the trend of hair loss area, and a diagnosis report.

[0051] When using the bidirectional attention mechanism for cross-modal feature data fusion, the image features and text features are converted into key vectors. The key generation layer formula is:

[0052] K img =W k ×F img ,K text =W k ×F text

[0053] Among them, W k To set the weight of the feature on the key, F img and F text Represent the feature data of image and text respectively;

[0054] By converting image features and text features into query vectors, the query generation layer formula is:

[0055] Q img =W q ×F img ,Q text =W q ×F text

[0056] Among them, W q To set the weight of the feature in the query;

[0057] By converting image features and text features into value vectors, the value generation layer formula is:

[0058] V img =W v ×F img ,Vtext =W v ×F text

[0059] Among them, W v To set the weight of the feature in the value;

[0060] After the above calculation operations, the attention weight matrices from image features to text features and from text features to image features are calculated in the attention weight calculation layer. The calculation formulas involved are as follows:

[0061]

[0062] d k is a vector dimension for the key vector, the scaling factor Used to prevent the inner product value from being too large, thereby stabilizing the gradient;

[0063] Generate a new context vector based on the attention weight, and obtain a new feature representation through weighted summation. The formula involved is as follows:

[0064] F i ' mg =A img-to-text V text

[0065] F te ' xt =A text-to-img V img

[0066] The structure of the bidirectional attention mechanism is as follows Figure 4 As shown in the figure, Modeling is responsible for capturing and integrating bidirectional information, while softmax is responsible for converting this information into a normalized weight distribution for actual feature weighting and information selection. The combination of the two achieves efficient semantic understanding and information processing. In the feature map after bidirectional attention processing, the feature vector of each word has integrated global information.

[0067] In getting F i ' mg and F te ' xt After the new feature representation is completed, the new feature representation is transmitted to a feature concatenation layer to represent the new feature F i ' mg and F te ' xt Spliced ​​together to form a comprehensive feature vector F fused , and finally the comprehensive feature vector F fused The data is transmitted to the classification layer to obtain the final results such as hair loss trend prediction, hair detection, and diagnosis report.

[0068] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. A multimodal feature extraction androgenic alopecia detection method, characterized in that: The steps include: S1. Collect sample data: collect two sets of sample data from multiple patients with a difference of more than 180 days. Each set of sample data consists of the user's hair image data and medical record data. S2, build an image feature extraction model and a natural language processing model based on fine-tuning technology; The image feature extraction model consists of a ResNet-50 backbone network, in which there are four residual blocks. After passing through the four residual blocks, the feature map is input into a multi-level multi-scale feature network consisting of four multi-scale feature extraction modules, each of which contains four scale blocks. The feature maps output by the four residual blocks are respectively input into the scale blocks of the same level and one level deeper in the first multi-scale feature extraction module. In the multi-level multi-scale feature extraction module, the features output by the scale block of the previous layer are simultaneously input into the scale blocks of the same level and one level deeper in the next layer. Then, multiple feature maps are respectively passed through a 3*3 convolutional layer and then group normalized to obtain feature values ​​after scale-offset transformation. After obtaining the feature values, they are passed into the ReLu activation layer, and finally the feature data is transmitted to the pooling layer. The loss function L involved in the image feature extraction model training process CE The formula is as follows: Among them, C is the number of categories, y i,c and are the actual and predicted probabilities of the ith sample for the cth class, respectively. The natural language processing model based on fine-tuning technology consists of an embedding layer based on Llama 3 and a Transformer model; The loss function L' involved in the text feature extraction model training process CE The formula is as follows: Where V is the vocabulary, y i (ω) and are the actual and predicted probabilities of the wth word in the ith sample, respectively. A fully connected layer is connected after the Transformer model to fuse the data output by the image feature extraction model with the data output by the natural language processing model based on fine-tuning technology; S3, the fully connected layer aligns the feature data and inputs it into the Mamba model for processing; S4. Use the bidirectional attention mechanism to calculate the correlation between the two modalities of image and text, obtain the joint feature data, and output the joint feature data to the classification layer after passing it through a feature splicing layer.

2. The method for detecting androgenic alopecia by multimodal feature extraction according to claim 1, characterized in that: When using the bidirectional attention mechanism for cross-modal feature data fusion, the image features and text features are converted into key vectors. The key generation layer formula is: K img =W k ×F img ,K text =W k ×F text Among them, W k To set the weight of the feature on the key, F img and F text Represent the feature data of image and text respectively; By converting image features and text features into query vectors, the query generation layer formula is: Q img =W q ×F img ,Q text =W q ×F text Among them, W q To set the weight of the feature in the query; By converting image features and text features into value vectors, the value generation layer formula is: V img =W v ×F img ,V text =W v ×F text Among them, W v To set the weight of the feature in the value; After the above calculation operations, the attention weight matrices from image features to text features and from text features to image features are calculated in the attention weight calculation layer. The calculation formulas involved are as follows: d k is a vector dimension for the key vector, the scaling factor Used to prevent the inner product value from being too large, thereby stabilizing the gradient; Generate a new context vector based on the attention weight, and obtain a new feature representation through weighted summation. The formula involved is as follows: F i ′ mg =A img-to-text V text F te ′ xt =A text-to-img V img In getting F i ' mg and F te ' xt After the new feature representation is obtained, the new feature representation is transmitted to a feature concatenation layer, and the new feature representation F i ' mg and F te ' xt Spliced ​​together to form a comprehensive feature vector F fused , and finally the comprehensive feature vector F fused Passed to the classification layer.

Images