Fungus sample microscopic recognition method and system based on machine learning

Abstract

The invention relates to the technical field of medical image processing, in particular to a fungus sample microscopic recognition method and system based on machine learning, and the method comprises the following steps: clinically collecting dermatoscope pictures for nail fungus infection recognition; performing infection classification and infection position marking on the dermatoscope picture to obtain a nail fungus infection classification and a nail fungus infection position of the dermatoscope picture; a deep neural network is used for training based on a data set formed by the dermatoscope picture, the nail fungal infection classification and the nail fungal infection position, and a nail fungal infection recognition model used for recognizing nail fungal infection and positioning the infection position according to the dermatoscope picture is constructed. According to the invention, the deep neural network is utilized to construct the nail fungal infection identification model, nail fungal infection is identified and the infection position is positioned according to the dermatoscope image, standard and objective nail fungal infection identification is achieved, and the nail fungal infection identification precision and efficiency are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a method and system for microscopic identification of fungal samples based on machine learning. Background Art

[0002] Fungal nail infection refers to a disease caused by dermatophytes, yeasts and non-dermatophyte fungi (referred to as fungi) infecting the nail plate or under the nail. At present, the identification of fungal nail infection is mostly determined by clinical diagnosis, fluorescence microscopy and fungal culture. Among them, the clinical diagnosis of doctors is subjective, and the accuracy of identification depends on the experience and level of the physician. It is more limited in the judgment of small-scale infections and is prone to missed detection. The judgment methods of fluorescence microscopy and fungal culture are time-consuming, complicated to operate, and have insufficient diagnostic efficiency, and cannot determine the location of the infection.

[0003] Therefore, it is difficult to achieve efficient and accurate determination of onychomycosis infection and location of infection in the prior art, and it is unable to assist in the medical diagnosis of onychomycosis infection. Summary of the invention

[0004] The purpose of the present invention is to provide a method and system for microscopic identification of fungal samples based on machine learning, so as to solve the technical problem that it is difficult to achieve efficient and accurate judgment of nail fungal infection and location of infection in the prior art.

[0005] In order to solve the above technical problems, the present invention specifically provides the following technical solutions:

[0006] A method for microscopic identification of fungal samples based on machine learning, characterized by comprising the following steps:

[0007] Dermatoscopic images collected clinically for identification of nail fungal infection;

[0008] Performing infection classification and marking of infection locations on the dermoscopic image to obtain the nail fungus infection classification and nail fungus infection location of the dermoscopic image;

[0009] A deep neural network is used to train the dermoscopic images and the data set consisting of the onychomycosis classification and the onychomycosis infection location, so as to construct an onychomycosis infection recognition model for identifying onychomycosis infection and locating the infection location based on dermoscopic images.

[0010] As a preferred embodiment of the present invention, the dermoscopic image is acquired by using a polarization mode and an immersion mode.

[0011] As a preferred embodiment of the present invention, the nail fungus infection classification includes a nail fungus infection category and a nail fungus non-infection category, wherein:

[0012] According to the clinical diagnosis results, fluorescence microscopy results and fungal culture results of the subjects of the dermatoscopic images, the infection classification of the dermatoscopic images is annotated, including:

[0013] When the clinical diagnosis result of the subject of the dermoscopic image is nail fungus infection, the fluorescent microscopy result is positive, and the fungal culture result is positive, the dermoscopic image is marked as nail fungus infection;

[0014] When the clinical diagnosis result of the subject of the dermoscopic image is non-nail fungal infection, or the fluorescent microscopy result is negative, or the fungal culture result is negative, the dermoscopic image is marked as non-nail fungal infection;

[0015] Methods for confirming fungal culture results include:

[0016] Morphological identification of fungal cultures of subjects belonging to dermoscopic images was performed to determine the fungal culture results as positive, negative, or pending;

[0017] Molecular sequencing was performed on fungal cultures with pending fungal culture results to confirm whether the fungal culture results were positive or negative.

[0018] As a preferred solution of the present invention, the network structure of the deep neural network includes a feature extraction network, a feature processing network and a region proposal network, wherein:

[0019] The feature extraction network is used to extract features from the dermoscopic image to obtain a feature map of the dermoscopic image;

[0020] The feature processing network is used to perform multi-scale convolution, multi-scale enhancement and multi-scale fusion processing on the feature map of the dermoscopic image to obtain a target semantically enhanced fused feature map of the dermoscopic image;

[0021] The region proposal network is used to output the nail fungus infection classification and nail fungus infection location of the dermoscopic image according to the target semantic enhanced fusion feature map.

[0022] As a preferred solution of the present invention, the feature processing network includes a target semantic recognition layer, a target semantic enhancement layer and a target semantic fusion layer. The target semantic recognition layer is used to perform multi-scale convolution of target semantics on the feature map of the dermoscopic image to obtain a target semantic feature map. The target semantic enhancement layer is used to perform multi-scale enhancement of target semantics on the target semantic feature map to obtain a target semantic enhanced feature map. The target semantic fusion layer is used to perform multi-scale fusion of target semantics on the target semantic enhanced feature map to obtain a target semantic enhanced fused feature map.

[0023] The method for the target semantic recognition layer to perform multi-scale convolution of target semantics on the feature map of the dermoscopic image includes:

[0024] Determine the multi-scale convolution kernel;

[0025] The feature maps of the dermoscopic images are sequentially convolved using multi-scale convolution kernels to obtain multi-scale feature maps;

[0026] Using the multi-scale feature map as the target semantic feature map;

[0027] Multi-scale convolution can be expressed as:

[0028]

[0029] in, is the feature value at position (x, y, z) on the target semantic feature map obtained by convolution of the i-th scale convolution kernel on the j-th channel, H i , W i and D i is the height, width and depth of the i-th scale convolution kernel, f is the activation function of the i-th scale convolution, is the weight parameter connected to the convolution kernel position (h, w, d) corresponding to the target semantic feature map in the m-th channel of the i-1-th scale convolution kernel, is the feature value at position (x+h, y+w, z+d) on the target semantic feature map in the mth channel of the i-1th scale convolution kernel, b ij It is the deviation of the jth channel corresponding to the i-th scale convolution kernel.

[0030] As a preferred solution of the present invention, the method in which the target semantic enhancement layer performs multi-scale enhancement of the target semantics on the target semantic feature map comprises:

[0031] Locate the position with the highest feature value in the target semantic feature map in each channel of each scale convolution kernel (x max ,y max , z max ), and (x max ,y max , z max ) is the center, a feature matrix is ​​extracted from each channel feature map of each scale convolution kernel, and then multiplied with a two-dimensional Gaussian distribution matrix of the same size to obtain the target semantic enhancement feature map;

[0032] Multi-scale enhancement can be expressed as:

[0033]

[0034]

[0035]

[0036] In the formula, is the feature value at position (x, y, z) on the target semantic enhancement feature map in the jth channel of the i-th scale convolution kernel, is the eigenvalue at position (x, y, z) on the target semantic feature map obtained by convolution of the i-th scale convolution kernel on the j-th channel, k*k is the size of the feature matrix, (x max ,y max , z max ) is the position coordinate with the highest feature value in the target semantic feature map, (x, y, z) is the position coordinate on the feature map, stl is constraint 1, and st2 is constraint 2.

[0037] As a preferred solution of the present invention, the method for the target semantic fusion layer to perform multi-scale fusion of target semantics on the target semantic enhancement feature map includes:

[0038] The maximum pooling process is performed on the target semantic enhancement feature map to obtain the maximum pooling feature map of the target semantic enhancement. The maximum pooling process can be expressed as:

[0039]

[0040] In the formula, is the feature value at position (x, y, z) on the maximum pooled feature map of the target semantic enhancement in the jth channel of the i-th scale convolution kernel, F*F*F is the size of the pooling window, p is the pooling step size, ma x is the maximum value identifier, r, s, t are indexes that vary in the range [0, F-1], is the feature value at position (x*p+r, y*p+s, z*p+t) on the target semantic enhancement feature map in the j-th channel of the i-th scale convolution kernel;

[0041] The target semantic enhancement feature map is subjected to average pooling processing to obtain the target semantic enhancement average pooling feature map. The average pooling processing can be expressed as:

[0042]

[0043] In the formula, is the feature value at position (x, y, z) on the maximum pooled feature map of the target semantic enhancement in the jth channel of the i-th scale convolution kernel, F*F*F is the size of the pooling window, p is the pooling step size, r, s, t are indexes that vary in the range of [0, F-1], is the feature value at position (x*p+r,y*p+s,z*p+t) on the target semantic enhancement feature map in the j-th channel of the i-th scale convolution kernel;

[0044] The maximum pooling feature map of target semantic enhancement is fused with the average pooling feature map of target semantic enhancement to obtain the target semantic enhancement fused feature map;

[0045] Multi-scale fusion can be expressed as:

[0046]

[0047] In the formula, is the feature value at position (x, y, z) on the target semantic enhancement fusion feature map, N is the total number of convolution kernel scales, and M is the total number of channels.

[0048] As a preferred solution of the present invention, the loss function for training the deep neural network is the classification loss and regression loss of the region proposal network, and the evaluation indicators for evaluating the deep neural network are the recall rate and the average precision.

[0049] As a preferred embodiment of the present invention, the method for training a deep neural network includes:

[0050] Dividing the data set consisting of the dermoscopic images, the classification of nail fungus infection and the location of nail fungus infection into a training set and a test set;

[0051] In the training set, based on the loss function, the dermoscopic image is used as input, and the nail fungus infection classification and the nail fungus infection location are used as output, and a deep neural network is trained to obtain the nail fungus infection recognition model;

[0052] In the test set, the performance of the nail fungus infection recognition model was evaluated based on the evaluation indicators.

[0053] As a preferred embodiment of the present invention, the present invention provides a fungus sample microscopic identification system based on machine learning, which is applied to a fungus sample microscopic identification method based on machine learning. The system includes:

[0054] A data collection unit for clinical collection of dermoscopic images for identification of nail fungal infection;

[0055] A data processing unit, used to mark the infection category and infection location of the dermoscopic image, and obtain the nail fungus infection category and nail fungus infection location of the dermoscopic image;

[0056] A deep learning unit, for training using a deep neural network based on a data set consisting of the dermoscopic image, the onychomycosis infection category, and the onychomycosis infection location, to construct an onychomycosis infection recognition model for identifying onychomycosis infection and locating the infection location based on the dermoscopic image;

[0057] The recognition output unit is used to identify nail fungus infection and locate the infection position in the dermatoscope image using the nail fungus infection recognition model.

[0058] Compared with the prior art, the present invention has the following beneficial effects:

[0059] The present invention utilizes a deep neural network to construct a nail fungus infection recognition model, thereby realizing the recognition of nail fungus infection and locating the infection site based on dermoscopic images, achieving standard and objective nail fungus infection recognition, and improving the recognition accuracy and efficiency of nail fungus infection. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.

[0061] Figure 1 A flow chart of a method for microscopic identification of fungal samples based on machine learning provided in an embodiment of the present invention;

[0062] Figure 2 A block diagram of a fungal sample microscopic identification system based on machine learning provided in an embodiment of the present invention;

[0063] Figure 3 A dermatoscopic image provided by an embodiment of the present invention;

[0064] Figure 4 The recognition result of the nail fungus infection recognition model of the dermoscopic image provided by the embodiment of the present invention;

[0065] Figure 5 The target semantic feature map provided by the embodiment of the present invention;

[0066] Figure 6 The target semantic enhancement feature map provided by the embodiment of the present invention;

[0067] Figure 7 The target semantic enhancement fusion feature map provided by the embodiment of the present invention. DETAILED DESCRIPTION

[0068] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0069] like Figure 1As shown, the present invention provides a method for microscopic identification of fungal samples based on machine learning, characterized in that it includes the following steps:

[0070] Dermatoscopic images collected clinically for identification of nail fungal infection;

[0071] Classify the infection and mark the infection location of the dermoscopic images to obtain the classification and location of nail fungus infection in the dermoscopic images;

[0072] A deep neural network was used to train a dataset consisting of dermoscopic images, onychomycosis classification, and onychomycosis location, to build a onychomycosis recognition model for identifying onychomycosis and locating infection locations based on dermoscopic images.

[0073] In order to improve the standardization and efficiency of the identification of nail fungal infection, the present invention constructs a mapping relationship between dermoscopic images and the classification and location of nail fungal infection by training a neural network, so that it is possible to know whether the nail fungus is infected and the location of the infection based on the dermoscopic image. The standardized identification of nail fungal infection is achieved by using the model, without relying on the expert experience of clinical diagnosis, and has strong objectivity. In addition, the identification of nail fungal infection is performed only by using images, without involving complex operations such as fungal culture and microscopy, which is time-saving and has high diagnostic efficiency.

[0074] The present invention is based on the onychomycosis identification model for efficiently identifying onychomycosis. In order to improve the recognition accuracy of the onychomycosis infection model for onychomycosis, especially the recognition accuracy of onychomycosis infection in small areas, a feature processing network is set when training the deep neural network, including a target semantic recognition layer, a target semantic enhancement layer and a target semantic fusion layer. The target semantic recognition layer is used to perform multi-scale convolution of the target semantics on the feature map of the dermoscopic image to obtain a target semantic feature map, that is, all semantic information related to the recognition of onychomycosis infection is extracted at multiple scales in the feature map of the dermoscopic image, so as to fully capture the context information and reduce the loss of information related to the recognition of onychomycosis infection. The target semantic enhancement layer is used to perform multi-scale enhancement of the target semantics on the target semantic feature map to obtain a target semantic enhanced feature map, that is, to enhance the semantic information related to the recognition and positioning of onychomycosis infection, and retain the information related to the onychomycosis infection. The semantic information that is strongly related to the identification and positioning of fungal infection is extracted, and the semantic information that is weakly related to the identification and positioning of onychomycosis infection is eliminated, so that the subsequent region proposal network can highlight the important semantic information, so that the region proposal network can accurately and quickly obtain the important semantic information to analyze the classification and location of onychomycosis infection, especially in the identification and positioning of small-sized onychomycosis targets. The target semantic fusion layer is used to perform multi-scale fusion of the target semantics on the target semantic enhancement feature map to obtain the target semantic enhancement fusion feature map, that is, to aggregate the important information at multiple scales, and map the important information at each scale with the target size in a stable negative correlation, balance the feature brightness of the important information at each scale, reduce the mutual interference between the important information, and ensure the orderly fusion of the important information at multiple scales, which can play the greatest role in the region proposal network, rather than competing for attention from each other, resulting in a decrease in the recognition and positioning accuracy.

[0075] Specifically, in order to extract all semantic information related to the identification of nail fungal infection, fully capture contextual information, and reduce the loss of information related to the identification of nail fungal infection, the target semantic recognition layer performs convolution operations on the feature map of the dermoscopic image with convolution kernels of multiple scales, thereby obtaining feature maps of multiple scales.

[0076] In order to enhance the semantic information related to the identification and positioning of onychomycosis infection, the target semantic enhancement layer locates the position with the highest eigenvalue (highest feature brightness) in the target semantic feature map in each channel of each scale convolution kernel, which can be regarded as the center point of the onychomycosis infection area, indicating that the feature matrix around the center point is semantic information strongly related to the identification and positioning of onychomycosis infection. Then, using Gaussian filtering, the feature matrix around the center point can be enhanced in the original feature map, that is, the semantic information strongly related to the identification and positioning of onychomycosis infection is enhanced in the original feature map, the feature brightness is improved, and it is easier to be noticed by the region proposal network.

[0077] It is particularly beneficial for the recognition and positioning of small-sized nail fungus targets. The small-sized nail fungus infection area itself contains less semantic information and is easily distracted by large-sized targets. After being enhanced by semantic information, it will grab attention from large-sized targets and be more easily noticed, which correspondingly improves the accuracy of recognition and positioning of small-sized nail fungus targets.

[0078] In order to evenly aggregate important information at multiple scales, the target semantic fusion layer uses a combination of maximum pooling and average pooling to achieve a stable negative correlation mapping between the important information at each scale and the target size, retaining the characteristic brightness of the important information of each onychomycosis target, especially the characteristic brightness of small-sized onychomycosis targets, avoiding brightness equalization by large-sized onychomycosis targets, and grabbing attention in the region proposal network to ensure the recognition and positioning accuracy of small-sized onychomycosis targets.

[0079] Dermoscopic images were acquired using polarization mode and immersion mode.

[0080] The classification of nail fungus infection includes nail fungus infection category and nail fungus non-infection category, among which:

[0081] According to the clinical diagnosis results, fluorescence microscopy results and fungal culture results of the subjects of the dermatoscopic images, the infection classification of the dermatoscopic images is annotated, including:

[0082] When the clinical diagnosis result of the subject of the dermoscopic image is nail fungus infection, the fluorescent microscopy result is positive, and the fungal culture result is positive, the dermoscopic image is marked as nail fungus infection;

[0083] When the clinical diagnosis result of the subject of the dermoscopic image is non-nail fungal infection, or the fluorescent microscopy result is negative, or the fungal culture result is negative, the dermoscopic image is marked as non-nail fungal infection;

[0084] Methods for confirming fungal culture results include:

[0085] Morphological identification of fungal cultures of subjects belonging to dermoscopic images was performed to determine the fungal culture results as positive, negative, or pending;

[0086] Molecular sequencing was performed on fungal cultures with pending fungal culture results to confirm whether the fungal culture results were positive or negative.

[0087] The infected location of the dermoscopic image is annotated, the location of the nail fungus infection in the dermoscopic image is framed, and the framed nail fungus infection area is used as the true value box and as the true value of the regression loss in the loss function of the region proposal network.

[0088] The network structure of the deep neural network includes feature extraction network, feature processing network and region proposal network, among which,

[0089] The feature extraction network is used to extract features from the dermoscopic image to obtain a feature map of the dermoscopic image;

[0090] The feature processing network is used to perform multi-scale convolution, multi-scale enhancement and multi-scale fusion processing on the feature map of the dermoscopic image to obtain the target semantic enhancement fusion feature map of the dermoscopic image;

[0091] The region proposal network is used to output the onychomycosis classification and onychomycosis location of dermoscopic images based on the target semantics-enhanced fused feature map.

[0092] The feature processing network includes a target semantic recognition layer, a target semantic enhancement layer and a target semantic fusion layer. The target semantic recognition layer is used to perform multi-scale convolution of target semantics on the feature map of the dermoscopic image to obtain a target semantic feature map. The target semantic enhancement layer is used to perform multi-scale enhancement of the target semantic feature map to obtain a target semantic enhanced feature map. The target semantic fusion layer is used to perform multi-scale fusion of target semantics on the target semantic enhanced feature map to obtain a target semantic enhanced fused feature map.

[0093] In order to extract all semantic information related to the identification of nail fungal infection, fully capture contextual information, and reduce the loss of information related to the identification of nail fungal infection, the target semantic recognition layer performs convolution operations on the feature map of the dermoscopic image with convolution kernels of multiple scales to obtain feature maps of multiple scales, as follows:

[0094] The method of performing multi-scale convolution of target semantics on the feature map of the dermoscopic image in the target semantic recognition layer includes:

[0095] Determine the multi-scale convolution kernel;

[0096] The feature maps of the dermoscopic images are sequentially convolved using multi-scale convolution kernels to obtain multi-scale feature maps;

[0097] The multi-scale feature map is used as the target semantic feature map, such as Figure 5 As shown, three feature maps obtained by multi-scale convolution kernels and three target semantic feature maps are selected;

[0098] Multi-scale convolution can be expressed as:

[0099]

[0100] in, is the feature value at position (x, y, z) on the target semantic feature map obtained by convolution of the i-th scale convolution kernel on the j-th channel, H i , W i and D iis the height, width and depth of the i-th scale convolution kernel, f is the activation function of the i-th scale convolution, is the weight parameter connected to the convolution kernel position (h, w, d) corresponding to the target semantic feature map in the m-th channel of the i-1-th scale convolution kernel, is the feature value at position (x+h, y+w, z+d) on the target semantic feature map in the mth channel of the i-1th scale convolution kernel, b ij It is the deviation of the jth channel corresponding to the i-th scale convolution kernel.

[0101] Multi-scale convolution can first extract multi-scale information: features can be extracted in networks at different levels to obtain multi-scale information. In the identification and location of nail fungus infections, targets may appear in dermatoscope images at different sizes. By detecting and locating at different scales, targets of different sizes can be better identified.

[0102] Secondly, it can capture contextual information: By extracting features at different levels, multi-scale convolution can capture a wider range of contextual information. Features at deeper levels usually have a better grasp of global semantic information, while features at shallower levels can provide more local detail information. Through multi-scale convolution, the algorithm can simultaneously utilize global and local information at different levels to better understand the content of dermoscopic images.

[0103] Finally, it can reduce information loss: In deep neural networks, as the number of network layers increases, the size of the feature map gradually decreases, which may lead to information loss, especially for smaller targets or details. By fusing features at multiple scales, the impact of information loss can be reduced and network performance can be improved.

[0104] In order to enhance the semantic information related to the identification and positioning of nail fungus infection, the target semantic enhancement layer locates the position with the highest eigenvalue (highest feature brightness) in the target semantic feature map in each channel of each scale convolution kernel, which can be regarded as the center point of the nail fungus infection area, indicating that the feature matrix around the center point is semantic information strongly related to the identification and positioning of nail fungus infection. Then, using Gaussian filtering, the feature matrix around the center point can be enhanced in the original feature map, that is, the semantic information strongly related to the identification and positioning of nail fungus infection is enhanced in the original feature map, the feature brightness is improved, and it is easier to be noticed by the region proposal network, as follows:

[0105] The method of performing multi-scale enhancement of the target semantics on the target semantic feature map by the target semantic enhancement layer includes:

[0106] Locate the position with the highest feature value in the target semantic feature map in each channel of each scale convolution kernel (x max ,y max,z max ), and (x max ,y max ,z max ) is the center, a feature matrix is ​​extracted from each channel feature map of each scale convolution kernel, and then multiplied with the two-dimensional Gaussian distribution matrix of the same size to obtain the target semantic enhancement feature map, such as Figure 6 As shown, three target semantic enhancement feature maps corresponding to the three target semantic feature maps are enhanced;

[0107] Multi-scale enhancement can be expressed as:

[0108]

[0109]

[0110]

[0111] In the formula, is the feature value at position (x, y, z) on the target semantic enhancement feature map in the jth channel of the i-th scale convolution kernel, is the eigenvalue at position (x, y, z) on the target semantic feature map obtained by convolution of the i-th scale convolution kernel on the j-th channel, k*k is the size of the feature matrix, (x max ,y max ,z max ) is the position coordinate with the highest feature value in the target semantic feature map, (x, y, z) is the position coordinate on the feature map, st1 is constraint 1, and st2 is constraint 2.

[0112] It is particularly beneficial for the recognition and positioning of small-sized nail fungus targets. The small-sized nail fungus infection area itself contains less semantic information and is easily distracted by large-sized targets. After being enhanced by semantic information, it will grab attention from large-sized targets and be more easily noticed, which correspondingly improves the accuracy of recognition and positioning of small-sized nail fungus targets.

[0113] In order to evenly aggregate important information at multiple scales, the target semantic fusion layer uses a combination of maximum pooling and average pooling to make a stable negative correlation mapping between important information at each scale and the target size, retaining the characteristic brightness of important information of each nail fungus target, especially the characteristic brightness of small-sized nail fungus targets, avoiding brightness equalization by large-sized nail fungus targets, and grabbing attention in the region proposal network to ensure the recognition and positioning accuracy of small-sized nail fungus targets, as follows:

[0114] The method of performing multi-scale fusion of target semantics on the target semantic enhancement feature map by the target semantic fusion layer includes:

[0115] The target semantic enhancement feature map is subjected to maximum pooling processing to obtain the maximum pooling feature map of target semantic enhancement. The maximum pooling processing can be expressed as:

[0116]

[0117] In the formula, is the feature value at position (x, y, z) on the maximum pooled feature map of the target semantic enhancement in the jth channel of the i-th scale convolution kernel, F*F*F is the size of the pooling window, p is the pooling step size, max is the maximum value identifier, r, s, t are indexes that change in the range of [0, F-1], is the feature value at position (x*p+r,y*p+s,z*p+t) on the target semantic enhancement feature map in the j-th channel of the i-th scale convolution kernel;

[0118] The target semantic enhancement feature map is averaged and pooled to obtain the target semantic enhancement averaged pooled feature map. The average pooling process can be expressed as:

[0119]

[0120] In the formula, is the feature value at position (x, y, z) on the maximum pooled feature map of the target semantic enhancement in the jth channel of the i-th scale convolution kernel, F*F*F is the size of the pooling window, p is the pooling step size, r, s, t are indexes that vary in the range of [0, F-1], is the feature value at position (x*p+r,y*p+s,z*p+t) on the target semantic enhancement feature map in the j-th channel of the i-th scale convolution kernel;

[0121] The maximum pooled feature map of target semantic enhancement is fused with the average pooled feature map of target semantic enhancement to obtain the target semantic enhancement fused feature map, such as Figure 7 As shown, the target semantic enhancement fusion feature map is obtained by fusing three target semantic enhancement feature maps;

[0122] Multi-scale fusion can be expressed as:

[0123]

[0124] In the formula, is the feature value at position (x, y, z) on the target semantic enhancement fusion feature map, N is the total number of convolution kernel scales, and M is the total number of channels.

[0125] Specifically, for onychomycosis targets with similar brightness in the feature map, the larger the target size, the smaller the The larger the target size, the smaller the The smaller, therefore and target size. However, the brightness of small-sized nail fungus targets is higher. It may still be larger than the large-sized onychomycosis target with smaller brightness, which results in the characteristic brightness of the small-sized onychomycosis target being equalized by the characteristic brightness of the large-sized onychomycosis target during fusion. In order to reduce the assimilation weakening of the brightness of the small-sized target, the The difference is in this case, and Negative correlation, the larger the target size The smaller the size, the brighter the target. Larger, brighter, smaller large-sized nail fungus targets Smaller, making small size targets Always larger than the large size target, establishing In other words, the relatively stable negative correlation mapping between the fused feature map and the target size allows the characteristic brightness of the small-sized onychomycosis target to be retained, and the brightness of the small-sized onychomycosis target with larger brightness is still greater than that of the large-sized onychomycosis target with smaller brightness after fusion.

[0126] The loss function for training the deep neural network is the classification loss and regression loss of the region proposal network, and the evaluation indicators for evaluating the deep neural network are recall rate and average precision.

[0127] The classification loss is the softmax function loss, and the regression loss is the SmoothL1 function loss.

[0128] The softmax function loss is: In the formula, s i is the i-th vector value of the fully connected layer in the classifier, and n is the total number of classification categories;

[0129] The loss of SmoothL1 function is: In the formula, P is the predicted value, P r is the true value, α is the smoothing area control parameter, α=3.0.

[0130] In order to improve the identification standardization and efficiency of nail fungal infection, the present invention constructs a mapping relationship between dermoscopic images and nail fungal infection classification and nail fungal infection location by training a neural network, so that it is possible to know whether the nail fungal infection is present and the infection location based on the dermoscopic images, as follows:

[0131] Methods for training deep neural networks include:

[0132] The dataset consisting of dermoscopic images, nail fungus infection classification and nail fungus infection location was divided into a training set and a test set;

[0133] In the training set, based on the loss function, the dermoscopic images were used as input, and the nail fungus infection classification and nail fungus infection location were used as output to train the deep neural network and obtain the nail fungus infection recognition model;

[0134] In the test set, the performance of the nail fungus infection recognition model was evaluated based on the evaluation indicators.

[0135] like Figure 2 As shown, the present invention provides a fungus sample microscopic identification system based on machine learning, which is applied to a fungus sample microscopic identification method based on machine learning. The system includes:

[0136] Data collection unit, used for clinical collection of dermatoscopic images for identification of nail fungal infection (such as Figure 3 shown);

[0137] A data processing unit, used to mark the infection category and infection location of the dermoscopic image, and obtain the nail fungus infection category and nail fungus infection location of the dermoscopic image;

[0138] A deep learning unit, for training a deep neural network based on a data set consisting of dermoscopic images, onychomycosis infection categories, and onychomycosis infection locations, to construct an onychomycosis infection recognition model for identifying onychomycosis infection and locating infection locations based on dermoscopic images;

[0139] The recognition output unit is used to identify the nail fungus infection and locate the infection position in the dermoscopic image using the nail fungus infection recognition model (such as Figure 4 As shown, Figure 4 The location selected in the middle box is the infection location).

[0140] The present invention utilizes a deep neural network to construct a nail fungus infection recognition model, thereby realizing the recognition of nail fungus infection and locating the infection site based on dermoscopic images, achieving standard and objective nail fungus infection recognition, and improving the recognition accuracy and efficiency of nail fungus infection.

[0141] The above embodiments are only exemplary embodiments of the present application and are not intended to limit the present application. The protection scope of the present application is defined by the claims. Those skilled in the art may make various modifications or equivalent substitutions to the present application within the essence and protection scope of the present application, and such modifications or equivalent substitutions shall also be deemed to fall within the protection scope of the present application.

Claims

1. A method for microscopic identification of fungal samples based on machine learning, characterized in that: The following steps are involved: Dermatoscopic images collected clinically for identification of nail fungal infection; Performing infection classification and marking of infection locations on the dermoscopic image to obtain the nail fungus infection classification and nail fungus infection location of the dermoscopic image; A deep neural network is used to train the dermoscopic images and the data set consisting of the onychomycosis classification and the onychomycosis infection location, so as to construct an onychomycosis infection recognition model for identifying onychomycosis infection and locating the infection location based on dermoscopic images.

2. The method for microscopic identification of fungal samples based on machine learning according to claim 1, characterized in that: The dermoscopic image is acquired by using polarization mode and infiltration mode.

3. The method for microscopic identification of fungal samples based on machine learning according to claim 2, characterized in that: The onychomycosis infection classification includes an onychomycosis infection category and an onychomycosis non-infection category, wherein: According to the clinical diagnosis results, fluorescence microscopy results and fungal culture results of the subjects of the dermatoscopic images, the infection classification of the dermatoscopic images is annotated, including: When the clinical diagnosis result of the subject of the dermoscopic image is nail fungus infection, the fluorescent microscopy result is positive, and the fungal culture result is positive, the dermoscopic image is marked as nail fungus infection; When the clinical diagnosis result of the subject of the dermoscopic image is non-nail fungal infection, or the fluorescent microscopy result is negative, or the fungal culture result is negative, the dermoscopic image is marked as non-nail fungal infection; Methods for confirming fungal culture results include: Morphological identification of fungal cultures of subjects belonging to dermoscopic images was performed to determine the fungal culture results as positive, negative, or pending; Molecular sequencing was performed on fungal cultures with pending fungal culture results to confirm whether the fungal culture results were positive or negative.

4. The method for microscopic identification of fungal samples based on machine learning according to claim 3, characterized in that: The network structure of the deep neural network includes a feature extraction network, a feature processing network and a region proposal network, wherein: The feature extraction network is used to extract features from the dermoscopic image to obtain a feature map of the dermoscopic image; The feature processing network is used to perform multi-scale convolution, multi-scale enhancement and multi-scale fusion processing on the feature map of the dermoscopic image to obtain a target semantically enhanced fused feature map of the dermoscopic image; The region proposal network is used to output the nail fungus infection classification and nail fungus infection location of the dermoscopic image according to the target semantic enhanced fusion feature map.

5. The method for microscopic identification of fungal samples based on machine learning according to claim 4, characterized in that: The feature processing network includes a target semantic recognition layer, a target semantic enhancement layer and a target semantic fusion layer. The target semantic recognition layer is used to perform multi-scale convolution of target semantics on the feature map of the dermoscopic image to obtain a target semantic feature map. The target semantic enhancement layer is used to perform multi-scale enhancement of target semantics on the target semantic feature map to obtain a target semantic enhanced feature map. The target semantic fusion layer is used to perform multi-scale fusion of target semantics on the target semantic enhanced feature map to obtain a target semantic enhanced fused feature map. The method for the target semantic recognition layer to perform multi-scale convolution of target semantics on the feature map of the dermoscopic image includes: Determine the multi-scale convolution kernel; The feature maps of the dermoscopic images are sequentially convolved using multi-scale convolution kernels to obtain multi-scale feature maps; Using the multi-scale feature map as the target semantic feature map; Multi-scale convolution can be expressed as: in, is the feature value at position (x, y, z) on the target semantic feature map obtained by convolution of the i-th scale convolution kernel on the j-th channel, H i , W i and D i is the height, width and depth of the i-th scale convolution kernel, f is the activation function of the i-th scale convolution, is the weight parameter connected to the convolution kernel position (h, w, d) corresponding to the target semantic feature map in the m-th channel of the i-1-th scale convolution kernel, is the feature value at position (x+h, y+w, z+d) on the target semantic feature map in the mth channel of the i-1th scale convolution kernel, b ij It is the deviation of the jth channel corresponding to the i-th scale convolution kernel.

6. The method for microscopic identification of fungal samples based on machine learning according to claim 5, characterized in that: The method for the target semantic enhancement layer to perform multi-scale enhancement of the target semantics on the target semantic feature map includes: Locate the position with the highest feature value in the target semantic feature map in each channel of each scale convolution kernel (x max ,y max ,z max ), and (x max ,y max ,z max ) as the center, extract a feature matrix from each channel feature map of each scale convolution kernel, and then compare it with the same size Multiplying the two-dimensional Gaussian distribution matrix to obtain the target semantic enhancement feature map; Multi-scale enhancement can be expressed as: In the formula, is the feature value at position (x, y, z) on the target semantic enhancement feature map in the jth channel of the i-th scale convolution kernel, is the eigenvalue at position (x, y, z) on the target semantic feature map obtained by convolution of the i-th scale convolution kernel on the j-th channel, k*k is the size of the feature matrix, (x max ,y max ,z max ) is the position coordinate with the highest feature value in the target semantic feature map, (x, y, z) is the position coordinate on the feature map, st1 is constraint 1, and st2 is constraint 2.

7. The method for microscopic identification of fungal samples based on machine learning according to claim 6, characterized in that: The method for the target semantic fusion layer to perform multi-scale fusion of target semantics on the target semantic enhancement feature map includes: The maximum pooling process is performed on the target semantic enhancement feature map to obtain the maximum pooling feature map of the target semantic enhancement. The maximum pooling process can be expressed as: In the formula, is the feature value at position (x, y, z) on the maximum pooled feature map of the target semantic enhancement in the jth channel of the i-th scale convolution kernel, F*F*F is the size of the pooling window, p is the pooling step size, max is the maximum value identifier, r, s, t are indexes that change in the range of [0, F-1], is the feature value at position (x*p+r,y*p+s,z*p+t) on the target semantic enhancement feature map in the j-th channel of the i-th scale convolution kernel; The target semantic enhancement feature map is subjected to average pooling processing to obtain the target semantic enhancement average pooling feature map. The average pooling processing can be expressed as: In the formula, is the feature value at position (x, y, z) on the maximum pooled feature map of the target semantic enhancement in the jth channel of the i-th scale convolution kernel, F*F*F is the size of the pooling window, p is the pooling step size, r, s, t are indexes that vary in the range of [0, F-1], is the feature value at position (x*p+r,y*p+s,z*p+t) on the target semantic enhancement feature map in the j-th channel of the i-th scale convolution kernel; The maximum pooling feature map of target semantic enhancement is fused with the average pooling feature map of target semantic enhancement to obtain the target semantic enhancement fused feature map; Multi-scale fusion can be expressed as: In the formula, is the feature value at position (x, y, z) on the target semantic enhancement fusion feature map, N is the total number of convolution kernel scales, and M is the total number of channels.

8. The method for microscopic identification of fungal samples based on machine learning according to claim 7, characterized in that: The loss function for training the deep neural network is the classification loss and regression loss of the region proposal network, and the evaluation indicators for evaluating the deep neural network are recall rate and average precision.

9. The method for microscopic identification of fungal samples based on machine learning according to claim 8, characterized in that: Methods for training deep neural networks include: Dividing the data set consisting of the dermoscopic images, the classification of nail fungus infection and the location of nail fungus infection into a training set and a test set; In the training set, based on the loss function, the dermoscopic image is used as input, and the nail fungus infection classification and the nail fungus infection location are used as output, and a deep neural network is trained to obtain the nail fungus infection recognition model; In the test set, the performance of the nail fungus infection recognition model was evaluated based on the evaluation indicators.

10. A fungal sample microscopic identification system based on machine learning, characterized in that: A method for microscopic identification of fungal samples based on machine learning as described in any one of claims 1 to 9, the system comprising: A data collection unit for clinical collection of dermoscopic images for identification of nail fungal infection; A data processing unit, used to mark the infection category and infection location of the dermoscopic image, and obtain the nail fungus infection category and nail fungus infection location of the dermoscopic image; A deep learning unit, for training using a deep neural network based on a data set consisting of the dermoscopic image, the onychomycosis infection category, and the onychomycosis infection location, to construct an onychomycosis infection recognition model for identifying onychomycosis infection and locating the infection location based on the dermoscopic image; The recognition output unit is used to identify nail fungus infection and locate the infection position in the dermatoscope image using the nail fungus infection recognition model.

Images