A nail disease image processing system
By enhancing data and extracting features of the A disease image, combining the classification loss function and the balanced mixed agent loss function, the model is optimized, and the problem of inaccurate classification of A disease image is solved, and classification accuracy and feature representation quality are improved.
Patent Information
- Application Number
- CN202411125379.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-16
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2044-08-16
AI Technical Summary
Existing A-deficiency image processing technology is difficult to accurately capture the global context information of the image. Due to the problem of category imbalance, the model is prone to bias towards high-frequency lesions when classified, and ignores rare but important lesion types, resulting in inaccurate classification.
By augmenting the target disease A image data, the first sub-image and the second sub-image are generated, the feature extraction model is used to extract the mark token and features, the classification loss function and balanced mixed agent loss function are constructed, and the feature extraction model and classification model are optimized. , to improve classification accuracy.
The classification accuracy of disease A images is improved, especially in the data set of disease A data set with small data volumes, which enhances the model's ability to describe different image information, forms high-quality and separate feature representations, and improves prediction accuracy.
Smart Images

Figure CN118658008B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of disease image processing, and in particular to a nail disease image processing system. Background Art
[0002] With the continuous advancement of information technology, medical data processing and analysis technology has shown great application potential in many fields. Especially in the field of medical image analysis, accurate and efficient medical image data processing technology can help medical experts better understand and analyze image data, thereby promoting the development of medical diagnosis and treatment methods. As a common skin-related disease, nail disease has image features such as changes in the color, shape and texture of the nail plate, which are of great significance for the early detection and tracking of the disease. Therefore, it is very important for the medical field to develop technology that can accurately analyze nail disease images.
[0003] However, existing image data processing techniques often encounter a series of challenges when dealing with complex medical images, especially when processing image data related to nail diseases. The lesion features of nail diseases show a high degree of subtle differences in medical images, which requires data processing methods to not only capture local detail changes, but also understand the significance of these changes in the global context. First, although traditional image data processing methods, such as convolutional neural networks (CNNs), perform well in extracting local image features, they often have difficulty capturing global context information that is critical to nail disease data. This is because the image manifestations of nail diseases may cover a variety of forms, ranging from subtle color changes to significant structural abnormalities, and the correct identification of these features depends on the model's ability to comprehensively understand the global and local information of the image. Second, the class imbalance problem of nail disease image data poses an additional challenge to model training. In real-world datasets, some types of nail diseases may be more common than others, resulting in the trained model being biased towards classifying images of high-frequency lesions, while ignoring those rarer but equally important lesion types, and failing to provide accurately classified image data. This bias can seriously affect the accuracy and reliability of models that require nail disease image data in clinical applications. This shows that there is a problem of inaccurate classification of nail disease images. Summary of the invention
[0004] The present application provides a nail disease image processing system, which can solve the problem of inaccurate classification of nail disease images.
[0005] The present application embodiment provides a nail disease image processing system, the nail disease image processing system comprising:
[0006] A data enhancement module is used to obtain multiple target nail disease images and perform data enhancement on each target nail disease image to obtain a first sub-image and a second sub-image corresponding to each target nail disease image;
[0007] A feature extraction module is used to perform feature extraction on each first sub-image using a feature extraction model to obtain a tag token and first sub-image features of each target nail disease image, and to perform feature extraction on each second sub-image using a feature extraction model to obtain second sub-image features of each target nail disease image; the tag token is used to describe global information of the first sub-image;
[0008] A first classification module is used to classify each first sub-image feature using a first classification model to obtain a first image category probability distribution of each target nail disease image, and to classify each marked token using a second classification model to obtain a second image category probability distribution of each target nail disease image; the first image category probability distribution is used to describe the probability that the target nail disease image belongs to each nail disease category according to the first sub-image feature, and the second image category probability distribution is used to describe the probability that the target nail disease image belongs to each nail disease category according to the marked token;
[0009] A first construction module is used to construct a classification loss function based on all first image category probability distributions and all second image category probability distributions; the classification loss function is used to describe the accuracy of all first image category probability distributions and all second image category probability distributions;
[0010] The second construction module is used to obtain multiple proxy vectors of each nail disease category based on all target nail disease images, and to construct a balanced hybrid proxy loss function based on all proxy vectors, all first sub-image features, and all second sub-image features; the proxy vector is used to describe the true information of the corresponding nail disease category, and the balanced hybrid proxy loss function is used to describe the accuracy of the similarity between the proxy vector, the first sub-image feature, and the second sub-image feature;
[0011] A final loss function construction module is used to construct a final loss function according to the classification loss function and the balanced hybrid proxy loss function, and use the final loss function to optimize the feature extraction model, the first classification model, and the second classification model to obtain an optimized feature extraction model, an optimized first classification model, and an optimized second classification model;
[0012] The second classification module is used to classify the nail disease image to be predicted using the optimized feature extraction model, the optimized first classification model, and the optimized second classification model to obtain the first final image category probability distribution and the second final image category probability distribution of the nail disease image to be predicted, and perform weighted addition of the first final image category probability distribution and the second final image category probability distribution to obtain the final image category probability distribution. The nail disease image to be predicted is an image of the nail disease category corresponding to the probability with the largest median value of the final image category probability distribution.
[0013] Optionally, each first sub-image feature is classified using a first classification model to obtain a first image category probability distribution of each target nail disease image, including:
[0014] For each first sub-image feature, perform the following steps:
[0015] Mapping each feature vector in the first sub-image feature to a low-dimensional space to obtain a plurality of embedded feature vectors;
[0016] Calculate the weight of each embedded feature vector, and calculate the probability that the target nail disease image corresponding to the first sub-image feature belongs to each nail disease category according to all weights and all embedded feature vectors;
[0017] All probabilities are integrated to obtain the first image category probability distribution corresponding to the first sub-image feature.
[0018] Optionally, each feature vector in the first sub-image feature is mapped to a low-dimensional space to obtain multiple embedded feature vectors, including:
[0019] By formula:
[0020]
[0021] Calculate the Embedding feature vector ;
[0022] in, Indicates the first sub-image feature feature vectors, represents the weight matrix, , represents the total number of feature vectors in the first sub-image feature, represents the activation function, Indicates normalization.
[0023] Optionally, calculate the weight of each embedded feature vector, including:
[0024] By formula:
[0025]
[0026] Calculate the The weights of the embedded feature vectors ;
[0027] in, and are weight matrices, represents the activation function;
[0028] The probability that the target nail disease image corresponding to the first sub-image feature belongs to each nail disease category is calculated based on all weights and all embedded feature vectors, including:
[0029] By formula:
[0030]
[0031]
[0032] Calculate the probability matrix ;
[0033] Among them, the probability matrix The elements in are the probabilities that the target nail disease image corresponding to the first sub-image feature belongs to each nail disease category. represents the weight matrix, , represents the number of disease categories, represents the dimension of the first sub-image feature, Represents the aggregate feature representation.
[0034] Optionally, the classification loss function is:
[0035]
[0036] in, represents the value of the classification loss function, represents the hyperparameter, Indicates The weight of each nail disease category, , represents the number of disease categories, Indicates Whether the target nail disease image actually belongs to The image of the nail disease category has a value of 1 or 0. Indicates The probability distribution of the second image category corresponding to the target nail disease image is The probability of an image belonging to the nail disease category, Indicates The probability distribution of the first image category corresponding to the target nail disease image is The probability of an image belonging to the nail disease category, , Indicates the number of target nail disease images.
[0037] Optionally, multiple proxy vectors for each nail disease category are obtained based on all target nail disease images, including:
[0038] By formula:
[0039]
[0040] Calculate the The number of proxy vectors for each disease category ;
[0041] in, Indicates the maximum value of the number of target nail disease images that actually correspond to all nail disease categories. Indicates that the truth belongs to The number of target nail disease images for the images of the nail disease category, , Indicates the number of nail disease categories;
[0042] Generate a corresponding number of proxy vectors for each nail disease category.
[0043] Optionally, a balanced hybrid proxy loss function is constructed based on all proxy vectors, all first sub-image features, and all second sub-image features, including:
[0044] For each first sub-image feature, further feature extraction is performed on the first sub-image feature to obtain a first depth feature;
[0045] For each second sub-image feature, further feature extraction is performed on the second sub-image feature to obtain a second depth feature;
[0046] A balanced hybrid proxy loss function is constructed based on all proxy vectors, all first deep features, and all second deep features.
[0047] Optionally, the balanced hybrid proxy loss function is:
[0048]
[0049]
[0050] in, represents the value of the balanced hybrid proxy loss function, represents the number of target nail disease images, represents the number of disease categories, Represents the first parameter, represents the set of all first depth features and all second depth features, represents the set of all proxy vectors, Indicates that the truth belongs to The number of target nail disease images for the images of the nail disease category, Represents the second parameter, represents the set of all first depth features and all second depth features whose nail disease category corresponding to the maximum probability value is the same as the nail disease category corresponding to the first parameter in all first image category probability distributions and all second image category probability distributions, represents the set of all proxy vectors with the same disease category as the first parameter, represents the temperature parameter, represents the intermediate parameter, Represents the third parameter, Indicates that among all the probability distributions of the first image category and all the probability distributions of the second image category, the disease category corresponding to the maximum probability value is The set of all first depth features and all second depth features of the nail disease category, Indicates The set of all proxy vectors corresponding to the disease category, Indicates The number of proxy vectors for each disease category.
[0051] Optionally, the final loss function is:
[0052]
[0053] in, represents the value of the final loss function, represents the value of the classification loss function, represents the value of the balanced hybrid proxy loss function, and All represent hyperparameters.
[0054] Optionally, the feature extraction model, the first classification model, and the second classification model are optimized using the final loss function to obtain an optimized feature extraction model, an optimized first classification model, and an optimized second classification model, including:
[0055] Determine whether the final loss function value is less than the preset loss function value;
[0056] If yes, the feature extraction model is used as the optimized feature extraction model, the first classification model is used as the optimized first classification model, and the second classification model is used as the optimized second classification model;
[0057] Otherwise, adjust the parameters of the feature extraction model, the first classification model, and the second classification model, activate the feature extraction module, return to the step of using the feature extraction model to perform feature extraction on each first sub-image to obtain the marking token and first sub-image features of each target nail disease image, and use the feature extraction model to perform feature extraction on each second sub-image to obtain the second sub-image features of each target nail disease image.
[0058] The above solution of the present application has the following beneficial effects:
[0059] In an embodiment of the present application, by performing data enhancement on each target nail disease image, a first sub-image and a second sub-image corresponding to each target nail disease image are obtained, which can provide two different images of the target nail disease image and increase the data of the target nail disease image; the first sub-image feature and the marking token are used to obtain the first image category probability distribution and the second image category probability distribution, which can analyze the global information and feature information of the first sub-image to obtain two aspects of image category probability distribution, thereby improving the classification accuracy of the target nail disease image; multiple proxy vectors for each nail disease category are obtained based on all target nail disease images, so that the number of proxy vectors is related to the number of corresponding target nail disease images, so as to realize different degrees of consideration of the nail disease categories according to the imbalance between the nail disease categories, and construct a balanced hybrid proxy loss function based on the proxy vector, the first sub-image feature and the second sub-image feature, so that the balanced hybrid proxy loss function The rational loss function takes into account three aspects of information: nail disease category, first sub-image, and second sub-image. The addition of the second image features provides image information different from the first sub-image features, improves the balanced hybrid proxy loss function's description of different image information, and promotes the model to learn higher quality and more separated feature representations in the embedding space, especially in nail disease datasets with smaller data volumes, where richer relationships help to form high-quality feature distributions. The final loss function is constructed based on the classification loss function and the balanced hybrid proxy loss function, so that the final loss function can describe the prediction accuracy and feature extraction accuracy, thereby improving the accuracy of the final loss function. The feature extraction model, the first classification model, and the second classification model are optimized using the high-accuracy final loss function, so that the feature extraction model, the first classification model, and the second classification model achieve the expected performance, thereby improving the accuracy of the classification of nail disease images.
[0060] Other beneficial effects of the present application will be described in detail in the subsequent specific implementation section. BRIEF DESCRIPTION OF THE DRAWINGS
[0061] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0062] Figure 1 A schematic diagram of the structure of a nail disease image processing system provided in one embodiment of the present application;
[0063] Figure 2 A flowchart of the use of a nail disease image processing system provided in one embodiment of the present application;
[0064] Figure 3 A schematic diagram of the structure of an improved ViT network provided in an embodiment of the present application;
[0065] Figure 4 A schematic diagram of the working framework of a nail disease image processing system provided in one embodiment of the present application;
[0066] Figure 5 A schematic diagram of a classifier structure provided in an embodiment of the present application;
[0067] Figure 6 A schematic diagram of inference results provided in one embodiment of the present application. DETAILED DESCRIPTION
[0068] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.
[0069] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.
[0070] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.
[0071] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.
[0072] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.
[0073] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.
[0074] In response to the problem of inaccurate classification of existing nail disease images, an embodiment of the present application provides a nail disease image processing system. The nail disease image processing system performs data enhancement on each target nail disease image to obtain a first sub-image and a second sub-image corresponding to each target nail disease image, and can provide two different images of the target nail disease image to increase the data of the target nail disease image; uses the first sub-image feature and the marking token to obtain the first image category probability distribution and the second image category probability distribution, and can analyze the global information and feature information of the first sub-image to obtain two aspects of image category probability distribution, thereby improving the classification accuracy of the target nail disease image; obtains multiple proxy vectors for each nail disease category based on all target nail disease images, so that the number of proxy vectors is related to the number of corresponding target nail disease images, so as to realize different degrees of consideration of nail disease categories according to the imbalance between nail disease categories, and constructs a first sub-image category based on the proxy vector, the first sub-image feature, and the second sub-image feature. A balanced hybrid proxy loss function is constructed, so that the balanced hybrid proxy loss function takes into account three aspects of information: nail disease category, first sub-image, and second sub-image. The addition of the second image features provides image information different from the first sub-image features, improves the balanced hybrid proxy loss function's description of different image information, and promotes the model to learn higher quality and more separated feature representations in the embedding space, especially in nail disease datasets with smaller data volumes, where richer relationships help to form high-quality feature distributions; a final loss function is constructed based on the classification loss function and the balanced hybrid proxy loss function, so that the final loss function can describe the prediction accuracy and feature extraction accuracy, thereby improving the accuracy of the final loss function; the feature extraction model, the first classification model, and the second classification model are optimized using the high-accuracy final loss function, so that the feature extraction model, the first classification model, and the second classification model achieve the expected performance, thereby improving the accuracy of the classification of nail disease images.
[0075] like Figure 1 As shown, the nail disease image processing system 100 provided by the present application includes:
[0076] The data enhancement module 101 is used to obtain a plurality of target nail disease images and perform data enhancement on each target nail disease image to obtain a first sub-image and a second sub-image corresponding to each target nail disease image;
[0077] The feature extraction module 102 is used to perform feature extraction on each first sub-image using the feature extraction model to obtain a tag token and a first sub-image feature of each target nail disease image, and to perform feature extraction on each second sub-image using the feature extraction model to obtain a second sub-image feature of each target nail disease image; the tag token is used to describe the global information of the first sub-image;
[0078] The first classification module 103 is used to classify each first sub-image feature using the first classification model to obtain a first image category probability distribution of each target nail disease image, and to classify each marked token using the second classification model to obtain a second image category probability distribution of each target nail disease image; the first image category probability distribution is used to describe the probability that the target nail disease image belongs to each nail disease category according to the first sub-image feature, and the second image category probability distribution is used to describe the probability that the target nail disease image belongs to each nail disease category according to the marked token;
[0079] A first construction module 104 is used to construct a classification loss function based on all first image category probability distributions and all second image category probability distributions; the classification loss function is used to describe the accuracy of all first image category probability distributions and all second image category probability distributions;
[0080] The second construction module 105 is used to obtain multiple proxy vectors for each nail disease category based on all target nail disease images, and to construct a balanced hybrid proxy loss function based on all proxy vectors, all first sub-image features, and all second sub-image features; the proxy vector is used to describe the real information of the corresponding nail disease category, and the balanced hybrid proxy loss function is used to describe the accuracy of the similarity between the proxy vector, the first sub-image feature, and the second sub-image feature;
[0081] A final loss function construction module 106 is used to construct a final loss function according to the classification loss function and the balanced hybrid proxy loss function, and optimize the feature extraction model, the first classification model, and the second classification model using the final loss function to obtain an optimized feature extraction model, an optimized first classification model, and an optimized second classification model;
[0082] The second classification module 107 is used to classify the nail disease image to be predicted using the optimized feature extraction model, the optimized first classification model, and the optimized second classification model to obtain a first final image category probability distribution and a second final image category probability distribution for the nail disease image to be predicted, and perform weighted addition of the first final image category probability distribution and the second final image category probability distribution to obtain a final image category probability distribution, where the nail disease image to be predicted is an image of the nail disease category corresponding to the maximum probability of the median value of the final image category probability distribution.
[0083] like Figure 2 As shown, the nail disease image processing system provided by the present application includes the following steps during use:
[0084] Step 21, obtaining multiple target nail disease images, and performing data enhancement on each target nail disease image to obtain a first sub-image and a second sub-image corresponding to each target nail disease image.
[0085] The target nail disease images are nail disease images with real nail disease categories, such as images of onychomycosis, images of nail matrix nevus, and images of nail black lines.
[0086] It should be noted that when performing data enhancement on the target nail disease image, two different data enhancement methods can be arbitrarily selected, such as geometric transformation, pixel transformation, etc., to obtain the first sub-image and the second sub-image corresponding to the target nail disease image. Depending on the data enhancement method, the obtained first sub-image and the second sub-image are different. For example, the first sub-image is a mirrored target nail disease image, and the second sub-image is a target nail disease image with enhanced brightness.
[0087] Exemplarily, computer tools such as the pytorch library and the opencv library in Python can be used to perform data enhancement to obtain a first sub-image and a second sub-image corresponding to the target nail disease image.
[0088] It is worth mentioning that data enhancement of the target nail disease image can obtain different sub-images containing the target nail disease image, so that various image states of the target nail disease image can be simulated, thereby increasing the data of the target nail disease image.
[0089] Step 22, using the feature extraction model to perform feature extraction on each first sub-image to obtain the marking token and first sub-image features of each target nail disease image, and using the feature extraction model to perform feature extraction on each second sub-image to obtain the second sub-image features of each target nail disease image.
[0090] The above-mentioned marking token is used to describe the global information of the first sub-image. The above-mentioned first sub-image feature is used to describe the local information of the first sub-image, and the above-mentioned second sub-image feature is used to describe the local information of the second sub-image.
[0091] In some embodiments of the present application, the steps of extracting features from each first sub-image using a feature extraction model to obtain a marker token and first sub-image features of each target nail disease image, and extracting features from each second sub-image using a feature extraction model to obtain second sub-image features of each target nail disease image are specifically as follows:
[0092] In the first step, the feature extraction model is used to extract features from each first sub-image to obtain the marking token and first sub-image features of each target nail disease image.
[0093] In the second step, the feature extraction model is used to extract features from each second sub-image to obtain the second sub-image features of each target nail disease image.
[0094] Exemplarily, the feature extraction model may be an improved visual self-attention (ViT, Vision Tansformer) network, the improved ViT network is a ViT network in which a prediction layer is inserted on the residual link between the norm layer and the addition layer of each encoder module, the prediction layer includes a first convolution unit, a first activation unit, a second convolution unit, a second activation unit, and a third convolution unit connected in sequence, the input end of the first convolution unit is the input end of the prediction layer, the output end of the third convolution unit is the output end of the prediction module, and the first convolution unit and the third convolution unit are both 1×1 convolutions, the second convolution unit is a 3×3 convolution, and the first activation unit and the second activation unit are both Gaussian Error Linear Unit (GELU) activation functions. When the improved ViT model extracts features from an image, it generates image tag tokens and image features. In this step, only the tag token of the first sub-image is retained.
[0095] It is worth mentioning that adding an estimation layer to the ViT network can capture the local features of the image and enhance the ViT network's ability to process spatial information, thereby improving the accuracy of the obtained first sub-image features and second sub-image features.
[0096] The improved ViT network is described below with reference to a specific example.
[0097] like Figure 3 As shown, the input of the encoder module of the ViT network is first divided into image blocks, and then input into the encoder module together with the identification token for feature extraction. The encoder module with the estimation layer added is composed of the first norm layer, multi-head attention layer, first addition layer, second norm layer, multi-layer perception layer, second addition layer, and first estimation layer and second estimation layer connected in sequence. The identification token is input to the first addition layer and the first norm layer of the encoder module, the output end of the second addition layer is the output end of the encoder module, the input end of the first estimation layer is connected to the output end of the first norm layer, the output end of the first estimation layer is connected to the input end of the first addition layer, the output end of the first addition layer is connected to the input end of the second addition layer, the output end of the second norm layer is connected to the input end of the second estimation layer, and the output end of the second estimation layer is connected to the input end of the second addition layer. The first estimation layer and the second estimation layer have the same structure, both of which are composed of the first 1×1 convolution, the first GELU activation function, the 3×3 convolution, the second GELU activation function and the second 1×1 convolution connected in sequence. In the figure, + represents the addition in the first addition layer or the second addition layer.
[0098] Step 23, using the first classification model to classify each first sub-image feature to obtain the first image category probability distribution of each target nail disease image, and using the second classification model to classify each marking token to obtain the second image category probability distribution of each target nail disease image.
[0099] The first image category probability distribution is used to describe the probability that the target nail disease image belongs to each nail disease category according to the first sub-image feature, and the second image category probability distribution is used to describe the probability that the target nail disease image belongs to each nail disease category according to the marking token.
[0100] In some embodiments of the present application, the first classification model may be a multiple instance learning model (MIL), and the second classification model may be a multi-layer perceptron (MLP). The steps of using the first classification model to classify each first sub-image feature to obtain a first image category probability distribution of each target nail disease image, and using the second classification model to classify each marked token to obtain a second image category probability distribution of each target nail disease image are specifically as follows:
[0101] In the first step, each first sub-image feature is classified using the first classification model to obtain the first image category probability distribution of each target nail disease image.
[0102] For each first sub-image feature, perform the following steps:
[0103] First, each feature vector in the first sub-image feature is mapped to a low-dimensional space to obtain a plurality of embedded feature vectors.
[0104] Specifically, through the formula:
[0105]
[0106] Calculate the Embedding feature vector .
[0107] in, Indicates the first sub-image feature feature vectors, represents the weight matrix, , represents the total number of feature vectors in the first sub-image feature, represents the activation function, Indicates normalization.
[0108] Then, the weight of each embedded feature vector is calculated, and the probability that the target nail disease image corresponding to the first sub-image feature belongs to each nail disease category is calculated based on all weights and all embedded feature vectors.
[0109] Specifically, through the formula:
[0110]
[0111] Calculate the The weights of the embedded feature vectors .
[0112] in, and are weight matrices, Represents the activation function.
[0113] By formula:
[0114]
[0115]
[0116] Calculate the probability matrix .
[0117] Among them, the probability matrix The elements in are the probabilities that the target nail disease image corresponding to the first sub-image feature belongs to each nail disease category. represents the weight matrix, , represents the number of disease categories, represents the dimension of the first sub-image feature, Represents the aggregate feature representation.
[0118] Finally, all probabilities are integrated to obtain the first image category probability distribution corresponding to the first sub-image feature.
[0119] In the second step, each labeled token is classified using the second classification model to obtain the second image category probability distribution of each target nail disease image.
[0120] Exemplarily, a second classification model, such as a multi-layer perceptron, can be run using computer software such as Python to obtain a second image category probability distribution for each target nail disease image.
[0121] It is worth mentioning that by obtaining the first image category probability distribution and the second image category probability distribution, category prediction can be performed from both global information and local information of the target nail disease image, thereby improving the comprehensiveness of the analysis of the target nail disease image.
[0122] Step 24: construct a classification loss function based on all first image category probability distributions and all second image category probability distributions.
[0123] The above classification loss function is used to describe the accuracy of all first image category probability distributions and all second image category probability distributions.
[0124] Specifically, the above classification loss function is:
[0125]
[0126] in, represents the value of the classification loss function, represents the hyperparameter, Indicates The weight of each nail disease category, , represents the number of disease categories, Indicates Whether the target nail disease image actually belongs to The image of the nail disease category is 1 or 0. The target nail disease image actually belongs to If there are images of the nail disease category, the value is 1. The target nail disease image does not actually belong to If the image is of the nail disease category, the value is 0. Indicates The probability distribution of the second image category corresponding to the target nail disease image is The probability of an image belonging to the nail disease category, Indicates The probability distribution of the first image category corresponding to the target nail disease image is The probability of an image belonging to the nail disease category, , Indicates the number of target nail disease images.
[0127] For example, a classification loss function can be constructed using computer software for mathematical operations such as Python.
[0128] It is worth mentioning that by constructing the classification loss function, the accuracy of the probability distribution of the first image category and the probability distribution of the second image category can be described, thereby reflecting the performance of the first classification model and the second classification model.
[0129] Step 25, obtaining multiple proxy vectors for each nail disease category based on all target nail disease images, and constructing a balanced hybrid proxy loss function based on all proxy vectors, all first sub-image features, and all second sub-image features.
[0130] The above-mentioned proxy vector is used to describe the real information of the corresponding nail disease category, and the balanced mixed proxy loss function is used to describe the accuracy of the similarity between the proxy vector, the first sub-image feature and the second sub-image feature.
[0131] In some embodiments of the present application, the steps of obtaining multiple proxy vectors for each nail disease category based on all target nail disease images, and constructing a balanced hybrid proxy loss function based on all proxy vectors, all first sub-image features, and all second sub-image features are specifically as follows:
[0132] In the first step, multiple proxy vectors for each nail disease category are obtained based on all target nail disease images.
[0133] First, through the formula:
[0134]
[0135] Calculate the The number of proxy vectors for each disease category .
[0136] in, Indicates the maximum value of the number of target nail disease images that actually correspond to all nail disease categories. Indicates that the truth belongs to The number of target nail disease images for the images of the nail disease category, , Represents the number of nail disease categories.
[0137] Then, a corresponding number of proxy vectors are generated for each nail disease category.
[0138] For example, a proxy vector can be generated using a model based on a multi-layer perceptron. The proxy vector physically represents the center or cluster point of the nail disease category in the high-dimensional feature space, reflecting the real information of the nail disease category. Among all the target nail disease images, there are 8 target nail disease images that are actually melanonychia images, and 3 target nail disease images that are actually nevus maculae. The value is 8.
[0139] In the second step, a balanced hybrid proxy loss function is constructed based on all proxy vectors, all first sub-image features, and all second sub-image features.
[0140] Firstly, for each first sub-image feature, further feature extraction is performed on the first sub-image feature to obtain a first depth feature.
[0141] Exemplarily, a multi-layer perceptron may be used to obtain the first depth feature.
[0142] Then, for each second sub-image feature, further feature extraction is performed on the second sub-image feature to obtain a second depth feature.
[0143] Exemplarily, a multi-layer perceptron may be used to obtain the second depth feature.
[0144] Finally, a balanced hybrid proxy loss function is constructed based on all proxy vectors, all first deep features, and all second deep features.
[0145] Specifically, the balanced hybrid proxy loss function is:
[0146]
[0147]
[0148] in, represents the value of the balanced hybrid proxy loss function, represents the number of target nail disease images, represents the number of disease categories, Represents the first parameter, represents the set of all first depth features and all second depth features, represents the set of all proxy vectors, Indicates that the truth belongs to The number of target nail disease images for the images of the nail disease category, Represents the second parameter, represents the set of all first depth features and all second depth features whose nail disease category corresponding to the maximum probability value is the same as the nail disease category corresponding to the first parameter in all first image category probability distributions and all second image category probability distributions, represents the set of all proxy vectors with the same disease category as the first parameter, represents the temperature parameter, represents the intermediate parameter, Represents the third parameter, Indicates that among all the probability distributions of the first image category and all the probability distributions of the second image category, the disease category corresponding to the maximum probability value is The set of all first depth features and all second depth features of the nail disease category, Indicates The set of all proxy vectors corresponding to the disease category, Indicates The number of proxy vectors for each disease category.
[0149] Exemplarily, a balanced hybrid proxy loss function may be constructed using mathematical computing computer software such as Python.
[0150] It is worth mentioning that by constructing a balanced hybrid proxy loss function, the accuracy of the first sub-image features and the second sub-image features can be described, thereby reflecting the performance of the feature extraction model. At the same time, by calculating the number of proxy vectors for nail disease types, more proxy vectors can be allocated to nail disease categories with fewer corresponding target nail disease images, ensuring attention and learning for each nail disease category.
[0151] Step 26, construct a final loss function based on the classification loss function and the balanced hybrid proxy loss function, and use the final loss function to optimize the feature extraction model, the first classification model, and the second classification model to obtain an optimized feature extraction model, an optimized first classification model, and an optimized second classification model.
[0152] Specifically, the above final loss function is:
[0153]
[0154] in, represents the value of the final loss function, represents the value of the classification loss function, represents the value of the balanced hybrid proxy loss function, and All represent hyperparameters.
[0155] In some embodiments of the present application, the above-mentioned steps of optimizing the feature extraction model, the first classification model, and the second classification model using the final loss function to obtain the optimized feature extraction model, the optimized first classification model, and the optimized second classification model are specifically as follows:
[0156] Determine whether the final loss function value is less than the preset loss function value.
[0157] If so, the feature extraction model is used as the optimized feature extraction model, the first classification model is used as the optimized first classification model, and the second classification model is used as the optimized second classification model.
[0158] Otherwise, adjust the parameters of the feature extraction model, the first classification model, and the second classification model, activate the feature extraction module, return to the step of using the feature extraction model to perform feature extraction on each first sub-image to obtain the marking token and first sub-image features of each target nail disease image, and use the feature extraction model to perform feature extraction on each second sub-image to obtain the second sub-image features of each target nail disease image.
[0159] Exemplarily, the final loss function value is 2.6, which is greater than the preset value of the loss function of 1.5, indicating that the performance of the feature extraction model, the first classification model, and the second classification model at this time does not meet expectations, then the parameters of the feature extraction model, the first classification model, and the second classification model are adjusted, and the steps of using the feature extraction model to extract features from each first sub-image to obtain the marking token and first sub-image features of each target nail disease image, and using the feature extraction model to extract features from each second sub-image to obtain the second sub-image features of each target nail disease image are returned. At this time, the final loss function value is 1.3, which is less than the preset value of the loss function, indicating that the performance of the feature extraction model, the first classification model, and the second classification model at this time all meet expectations, and they are the optimized feature extraction model, the optimized first classification model, and the optimized second classification model.
[0160] It is worth mentioning that using the final loss function to optimize the feature extraction model, the first classification model, and the second classification model can improve the performance of the feature extraction model, the first classification model, and the second classification model, thereby increasing the accuracy of the calculation.
[0161] Step 27, use the optimized feature extraction model, the optimized first classification model, and the optimized second classification model to classify the nail disease image to be predicted, obtain the first final image category probability distribution and the second final image category probability distribution of the nail disease image to be predicted, and perform weighted addition on the first final image category probability distribution and the second final image category probability distribution to obtain the final image category probability distribution. The nail disease image to be predicted is an image of the nail disease category corresponding to the probability with the largest median value of the final image category probability distribution.
[0162] It should be noted that the above-mentioned nail disease images to be predicted are nail disease images that do not have a real nail disease category and need to be predicted for the nail disease category.
[0163] Specifically, data enhancement is performed on the nail disease image to be predicted to obtain a first image to be predicted and a second image to be predicted, and features are extracted from the first image to be predicted and the second image to be predicted using an optimized feature extraction model to obtain a final marking token, a first final image feature and a second final image feature, and then the first final feature is classified using the optimized first classification model to obtain a first final image category probability distribution, and the second final feature is classified using the optimized second classification model to obtain a second final image category probability distribution, and the first final image category probability distribution and the second final image category probability distribution are weightedly added to obtain a final image category probability distribution of the nail disease image to be predicted, and the nail disease image to be predicted is an image of the nail disease category corresponding to the probability with the largest median value of the final image category probability distribution.
[0164] Exemplarily, the probability of the black line on the nail in the first final image category probability distribution is 30%, and the probability of the nail matrix nevus is 45%. The probability of the black line on the nail in the second final image category probability distribution is 67%, and the probability of the nail matrix nevus is 39%. After weighted addition, the probability of the black line on the nail is 97%, and the probability of the nail matrix nevus is 84%. Then the nail disease image to be predicted is an image of the black line on the nail.
[0165] It is worth mentioning that using a high-performance feature extraction model, a first classification model, and a second classification model to obtain the final image probability distribution of the nail disease image to be predicted can improve the accuracy of the final image probability distribution and improve the accuracy of the classification of the nail disease image.
[0166] In addition, by accurately classifying nail disease images, accurate image data of nail disease categories can be provided, and therefore can be applied to research and analysis in related fields. For example, if research on black nail disease is required, the nail disease image processing system of the present application can accurately identify multiple black nail disease images from a large number of nail disease images, thereby providing accurate data for the research on black nail disease.
[0167] The nail disease image processing system of the present application is exemplarily described below with reference to a specific example.
[0168] The working framework of the nail disease image processing system of this application is as follows Figure 4 As shown in the figure, the clinical image (i.e. the target nail disease image mentioned above) is obtained after data enhancement. and Two sub-images, and Through the feature extraction model, we get The corresponding identification token cls token and the first sub-image feature ,as well as The corresponding second sub-image features , in the classification branch, the identification token and All of them are input into the classifier (i.e., the first classification model and the second classification model mentioned above) to obtain the first image category probability distribution and the second image category probability distribution. After weighted summation, the image category probability distribution of the clinical image is obtained. , and calculate the value of the classification loss function ,at the same time Input to the multi-layer perceptron (MLP) to get the first deep feature , in the contrastive learning branch, The second deep feature is obtained in the multi-layer perceptron , the hybrid proxy model generates multiple proxy vectors, and based on all proxy vectors, , Calculate the value of the balanced hybrid proxy loss function .
[0169] The structure of the above classifier is as follows Figure 5 As shown, the identification token is input into the multi-task classification model, the first sub-image feature is input into the multi-layer perceptron, + is addition, and the classifier outputs the predicted image of the disease category (i.e., the image category probability distribution in the above text) ), the multilayer perceptron includes an attention layer, an embedding layer, and a classification layer of the multilayer perceptron. The embedding layer of the multilayer perceptron includes a linear layer, a norm layer, and an activation function layer connected in sequence. The attention layer of the multilayer perceptron includes a first linear layer, a norm layer, an activation function layer, a regularization layer, and a second linear layer connected in sequence.
[0170] The nail disease image processing system of the present application is exemplarily described below with reference to an experiment.
[0171] The nail disease image processing system, ViT model, and residual neural network (ResNet50) of the present application are used to process the nail disease images in the nail disease image dataset, and the processing result of one nail disease image is as follows: Figure 6 As shown, the nail disease image in the figure is a nail black line image, and the inference models are ResNet50, ViT and the nail disease image processing system (Ours) of this application. The inference result of ResNet50 is a nail fungus image, which is inconsistent with the actual situation, the inference result of ViT is a nail mother nevus image, which is inconsistent with the actual situation, and the inference result of the nail disease image processing system of this application is a nail black line image, which is consistent with the actual situation. The results of the index evaluation of all classification results are shown in Table 1.
[0172] Table 1
[0173]
[0174] Among them, Ours is the nail disease image processing system of this application, Accuracy indicates accuracy, Precision indicates precision, Sensitivity indicates sensitivity, F1-score indicates the harmonic mean of precision and recall, AUC indicates the area under the ROC curve, and the ROC curve is a curve of true positive rate (sensitivity) and false positive rate drawn by changing the decision threshold:
[0175]
[0176]
[0177]
[0178]
[0179]
[0180] in, represents true positives, which is the number of nail disease images in the nail disease image dataset that the model correctly predicts as positive (i.e., images classified as specific nail disease categories). represents true negatives, which is the number of nail disease images in the nail disease image dataset that the model correctly predicts as negative (i.e., images classified as healthy nails). represents false positives, which is the number of nail disease images in the nail disease image dataset that the model incorrectly predicts as positive. is the false negative, which is the number of nail disease images in the nail disease image dataset that the model incorrectly predicts as negative. represents the decision threshold, represents the true positive rate.
[0181] It can be seen that the nail disease image processing system of the present application is 1.41 percentage points higher than ResNet50 in accuracy and 2.03 percentage points higher than ViT; in terms of F1 score, the nail disease image processing system of the present application is 4.58 percentage points higher than ResNet50, which emphasizes that the nail disease image processing system of the present application has achieved a better balance between accuracy and recall when identifying real cases. In addition, the nail disease image processing system of the present application also showed good performance in AUC, proving that it maintains consistent discrimination under different decision thresholds. These results fully demonstrate that the nail disease image processing system of the present application can pay more attention to the lesion part of the nail disease image data, judge the subtle differences between different cases, and provide more accurate information.
[0182] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0183] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0184] The above is a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.
Claims
1. A nail disease image processing system, characterized in that: include: A data enhancement module is used to obtain a plurality of target nail disease images, and perform data enhancement on each of the target nail disease images to obtain a first sub-image and a second sub-image corresponding to each of the target nail disease images; the first sub-image and the second sub-image are obtained by performing data enhancement on the target nail disease images in two different data enhancement methods; A feature extraction module, used to perform feature extraction on each first sub-image using a feature extraction model to obtain a tag token and first sub-image features of each target nail disease image, and to perform feature extraction on each second sub-image using the feature extraction model to obtain second sub-image features of each target nail disease image; the tag token is used to describe the global information of the first sub-image; A first classification module is used to classify each first sub-image feature using a first classification model to obtain a first image category probability distribution of each target nail disease image, and to classify each marking token using a second classification model to obtain a second image category probability distribution of each target nail disease image; the first image category probability distribution is used to describe the probability that the target nail disease image belongs to each nail disease category according to the first sub-image feature, and the second image category probability distribution is used to describe the probability that the target nail disease image belongs to each nail disease category according to the marking token; A first construction module is used to construct a classification loss function based on all first image category probability distributions and all second image category probability distributions; the classification loss function is used to describe the accuracy of all first image category probability distributions and all second image category probability distributions; A second construction module is used to obtain multiple proxy vectors of each nail disease category based on all target nail disease images, and to construct a balanced hybrid proxy loss function based on all proxy vectors, all first sub-image features, and all second sub-image features; the proxy vector is used to describe the real information of the corresponding nail disease category, the balanced hybrid proxy loss function is used to describe the accuracy of the similarity between the proxy vector, the first sub-image feature, and the second sub-image feature, and the proxy vector physically represents the center or aggregation point of the nail disease category in the high-dimensional feature space; A final loss function construction module, used to construct a final loss function according to the classification loss function and the balanced hybrid proxy loss function, and optimize the feature extraction model, the first classification model, and the second classification model using the final loss function to obtain an optimized feature extraction model, an optimized first classification model, and an optimized second classification model; The second classification module is used to classify the nail disease image to be predicted using the optimized feature extraction model, the optimized first classification model, and the optimized second classification model to obtain a first final image category probability distribution and a second final image category probability distribution of the nail disease image to be predicted, and perform weighted addition of the first final image category probability distribution and the second final image category probability distribution to obtain a final image category probability distribution. The nail disease image to be predicted is an image of the nail disease category corresponding to the probability with the largest median value of the final image category probability distribution.
2. The nail disease image processing system according to claim 1, characterized in that: The first classification model is used to classify each first sub-image feature to obtain a first image category probability distribution of each target nail disease image, including: For each first sub-image feature, perform the following steps: Mapping each feature vector in the first sub-image feature to a low-dimensional space to obtain a plurality of embedded feature vectors; Calculating the weight of each embedded feature vector, and calculating the probability that the target nail disease image corresponding to the first sub-image feature belongs to each nail disease category according to all weights and all embedded feature vectors; All probabilities are integrated to obtain the first image category probability distribution corresponding to the first sub-image feature.
3. The nail disease image processing system according to claim 2, characterized in that: The step of mapping each feature vector in the first sub-image feature to a low-dimensional space to obtain a plurality of embedded feature vectors includes: By formula: Calculate the Embedding feature vector ; in, Indicates the first sub-image feature feature vectors, represents the weight matrix, , represents the total number of feature vectors in the first sub-image feature, represents the activation function, Indicates normalization.
4. The nail disease image processing system according to claim 3, characterized in that: The calculation of the weight of each embedded feature vector includes: By formula: Calculate the The weights of the embedded feature vectors ; in, and are weight matrices, represents the activation function; The calculating, based on all weights and all embedded feature vectors, the probability that the target nail disease image corresponding to the first sub-image feature belongs to each nail disease category includes: By formula: Calculate the probability matrix ; Among them, the probability matrix The elements in are the probabilities that the target nail disease image corresponding to the first sub-image feature belongs to each nail disease category. represents the weight matrix, , represents the number of disease categories, represents the dimension of the first sub-image feature, Represents the aggregate feature representation.
5. The nail disease image processing system according to claim 1, characterized in that: The classification loss function is: in, represents the value of the classification loss function, represents the hyperparameter, Indicates The weight of each nail disease category, , represents the number of disease categories, Indicates Whether the target nail disease image actually belongs to The image of the nail disease category has a value of 1 or 0. Indicates the The probability distribution of the second image category corresponding to the target nail disease image is The probability of an image belonging to the nail disease category, Indicates the The probability distribution of the first image category corresponding to the target nail disease image is The probability of an image belonging to the nail disease category, , Indicates the number of target nail disease images.
6. The nail disease image processing system according to claim 1, characterized in that: The method of obtaining multiple proxy vectors for each nail disease category based on all target nail disease images includes: By formula: Calculate the The number of proxy vectors for each disease category ; in, Indicates the maximum value of the number of target nail disease images that actually correspond to all nail disease categories. Indicates that the truth belongs to the The number of target nail disease images for the images of the nail disease category, , Indicates the number of nail disease categories; Generate a corresponding number of proxy vectors for each nail disease category.
7. The nail disease image processing system according to claim 1, characterized in that: The constructing a balanced hybrid proxy loss function based on all proxy vectors, all first sub-image features, and all second sub-image features includes: For each first sub-image feature, further feature extraction is performed on the first sub-image feature to obtain a first depth feature; For each second sub-image feature, further feature extraction is performed on the second sub-image feature to obtain a second depth feature; A balanced hybrid proxy loss function is constructed based on all proxy vectors, all first deep features, and all second deep features.
8. The nail disease image processing system according to claim 7, characterized in that: The balanced hybrid proxy loss function is: in, represents the value of the balanced hybrid proxy loss function, represents the number of target nail disease images, represents the number of disease categories, Represents the first parameter, represents the set of all first depth features and all second depth features, represents the set of all proxy vectors, Indicates that the truth belongs to The number of target nail disease images for the images of the nail disease category, Represents the second parameter, represents a set of all first depth features and all second depth features whose nail disease category corresponding to the maximum probability value is the same as the nail disease category corresponding to the first parameter in all first image category probability distributions and all second image category probability distributions, represents the set of all proxy vectors with the same disease category as that corresponding to the first parameter, represents the temperature parameter, represents the intermediate parameter, Represents the third parameter, Indicates that among all the probability distributions of the first image category and all the probability distributions of the second image category, the disease category corresponding to the maximum probability value is The set of all first depth features and all second depth features of the nail disease category, Indicates the The set of all proxy vectors corresponding to the disease category, Indicates The number of proxy vectors for each disease category.
9. The nail disease image processing system according to claim 1, characterized in that: The final loss function is: in, represents the value of the final loss function, represents the value of the classification loss function, represents the value of the balanced hybrid proxy loss function, and All represent hyperparameters.
10. The nail disease image processing system according to claim 9, characterized in that: The method of optimizing the feature extraction model, the first classification model, and the second classification model by using the final loss function to obtain an optimized feature extraction model, an optimized first classification model, and an optimized second classification model includes: Determine whether the value of the final loss function is less than a preset value of the loss function; If yes, the feature extraction model is used as the optimized feature extraction model, the first classification model is used as the optimized first classification model, and the second classification model is used as the optimized second classification model; Otherwise, adjust the parameters of the feature extraction model, the first classification model, and the second classification model, activate the feature extraction module, return to the step of using the feature extraction model to perform feature extraction on each first sub-image to obtain the marking token and first sub-image features of each target nail disease image, and use the feature extraction model to perform feature extraction on each second sub-image to obtain the second sub-image features of each target nail disease image.
