Deep learning-based eardrum perforation recognition method

Abstract

The present application belongs to the technical field of medical image analysis based on deep learning, and provides a tympanic membrane perforation recognition method based on deep learning, which comprises: acquiring a tympanic membrane image and constructing a tympanic membrane training data set; constructing a deep learning-based tympanic membrane perforation recognition model, the deep learning-based tympanic membrane perforation recognition model comprising a lightweight backbone network, a multi-scale channel and spatial attention module, a feature fusion network and a network output layer; training the deep learning-based tympanic membrane perforation recognition model to obtain a target tympanic membrane perforation recognition model; and obtaining a recognition result of the tympanic membrane image based on the target tympanic membrane perforation recognition model. The present application introduces multi-scale channel attention and multi-scale depth separable convolution spatial attention to cope with the problems of difficult identification of small perforations and various perforation forms, aggregates context path semantic information and bottom-up path information, and realizes automatic high-precision recognition of tympanic membrane perforation.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of medical image analysis based on deep learning, especially to the technical field of tympanic membrane image segmentation and feature extraction, and particularly to a tympanic membrane perforation recognition method based on deep learning. BACKGROUND

[0002] In middle ear diseases, the recognition of tympanic membrane perforation plays a crucial role. It is not only a key basis for judging the degree of middle ear infection, inflammation or trauma, but also directly affects the choice of treatment plan and prognosis evaluation.

[0003] In Chinese Patent Application Publication No. CN118470315A, an ear-nose-throat examination image segmentation method and system are disclosed, which includes constructing a first recognition model to recognize the characteristics of the tympanic membrane, obtaining the shooting distance and shooting angle, and screening the video data according to the shooting distance and shooting angle to obtain image data; a two-dimensional coordinate system is established on the image, and the intersection of the two-dimensional coordinate system and the tympanic membrane boundary is identified by a second recognition model to obtain a set of intersection points; a fitting ellipse is obtained according to the set of intersection points, a segmentation ellipse is obtained according to the segmentation threshold and the fitting ellipse, the image is segmented according to the segmentation ellipse to obtain a segmented image; a tympanic membrane perforation recognition model is constructed to recognize the segmented image to obtain a perforation data set, and a perforation coefficient and a perforation risk coefficient are calculated from the perforation data set.

[0004] In Chinese Patent Application Publication No. CN115316921A, an inner ear canal intelligent self-checking method and system are disclosed, which belong to the technical field of individual health medical examination, and include: obtaining inner ear canal detection image information of a person being examined; identifying an inner ear canal recognition position point from the inner ear canal detection image information according to a preset inner ear canal recognition point feature; analyzing and obtaining standard recognition image information from the inner ear canal detection image information according to the inner ear canal recognition position point, preset reference image information, and recognition reference points; according to the comparison result of the standard recognition image information and the preset normal inner ear canal detection image information, analyzing and outputting the normal image similarity rate and specific difference point labeling information of the standard recognition image information and the normal inner ear canal detection image information; taking the normal image similarity rate and specific difference point labeling information as the final detection result, and outputting the final detection result to the terminal held by the person being examined.

[0005] However, the existing tympanic membrane perforation recognition method still has some problems. The traditional diagnostic method mainly relies on otoscopy and the clinical experience of doctors, which is not only subjective but also prone to missed diagnosis when the perforation is small or hidden. In addition, although imaging techniques such as acoustic immittance testing and temporal bone CT examination can provide more detailed information about the middle ear structure, these methods often require professional equipment and complex operation procedures, and in some cases it is difficult to achieve rapid and accurate diagnosis. In recent years, with the development of artificial intelligence technology, image recognition methods based on deep learning provide a new idea for the automatic high-precision recognition of tympanic membrane perforation. By using advanced algorithms such as convolutional neural networks (CNN), otoscopic images can be automatically analyzed to achieve rapid detection and classification of tympanic membrane perforation. This method not only reduces the interference of human factors and improves the accuracy and efficiency of diagnosis, but also has broad application prospects and is expected to provide clinicians with a more powerful auxiliary tool to promote the progress of middle ear disease diagnosis technology.

[0006] Therefore, in order to realize the automatic detection and classification of tympanic membrane perforation, it is very urgent and necessary to seek a deep learning-based tympanic membrane perforation recognition method to improve the accuracy and efficiency of tympanic membrane perforation recognition. SUMMARY

[0007] The present application aims at the defects in the prior art and proposes a deep learning-based tympanic membrane perforation recognition method. The method includes obtaining tympanic membrane images and constructing a tympanic membrane training data set; constructing a deep learning-based tympanic membrane perforation recognition model, which includes a lightweight backbone network, a multi-scale channel and spatial attention module, a feature fusion network, and a network output layer; training the deep learning-based tympanic membrane perforation recognition model to obtain a target tympanic membrane perforation recognition model; and obtaining the recognition result of the tympanic membrane image based on the target tympanic membrane perforation recognition model. To address the problem of small perforation being difficult to identify and the variety of perforation morphology, the present application introduces multi-scale channel attention and multi-scale depth separable convolution spatial attention, aggregates context path semantic information and bottom-up path information, realizes automatic high-precision recognition of tympanic membrane perforation, improves accuracy and efficiency, and has significant application value.

[0008] The present application provides a deep learning-based tympanic membrane perforation recognition method, which includes the following steps:

[0009] S1, obtaining tympanic membrane images and constructing a tympanic membrane training data set;

[0010] S2, constructing a deep learning-based tympanic membrane perforation recognition model: the tympanic membrane perforation recognition model includes a lightweight backbone network, a multi-scale channel and spatial attention module, a feature fusion network, and a network output layer;

[0011] S21, a lightweight backbone network for extracting tympanic membrane image perforation feature information is constructed, and first, second and third perforation features are output, the lightweight backbone network comprising an initial convolution, a first layer network, a second layer network, a third layer network, a fourth layer network, a fifth layer network and a spatial pyramid pooling module connected in series;

[0012] S22, a multi-scale channel and spatial attention module is constructed, and spatial features are output, the multi-scale channel and spatial attention module comprising a multi-scale channel attention module and a multi-scale depth separation convolution spatial attention module;

[0013] S23, a feature fusion network is constructed, and first, second and third tympanic membrane fine-grained features are output, the feature fusion network being used for integrating multi-scale feature information extracted by the network, and the feature fusion network comprising a feature pyramid network and a path aggregation network;

[0014] S24, a network output layer is constructed, which is used for outputting tympanic membrane image recognition results, the tympanic membrane image recognition results comprising a number of categories, a confidence corresponding to each category, a predicted box height, a predicted box width, a predicted box center point coordinate and a mask;

[0015] S3, a deep learning-based tympanic membrane perforation recognition model is trained to obtain a target tympanic membrane perforation recognition model;

[0016] S4, based on the target tympanic membrane perforation recognition model, an identification result of a tympanic membrane image is obtained.

[0017] Preferably, S1 specifically comprises the following sub-steps:

[0018] S11, a tympanic membrane label image database is constructed: based on an ear endoscope, a plurality of tympanic membrane images are obtained, multi-class labeling is performed to obtain corresponding label images, a tympanic membrane label image database is constructed, and the multi-class labeling comprises a tympanic membrane region and a perforation region;

[0019] S12, data preprocessing: based on the tympanic membrane label image database, data augmentation technology is used for expansion and normalization processing, the size and format are unified, and a tympanic membrane training data set is obtained;

[0020] S13, dividing training data set and test data set: based on the tympanic membrane training data set, the training data set and the test data set are obtained by dividing according to a first proportion.

[0021] Preferably, the initial convolution in S21 comprises a first convolution, the first layer network comprises a second convolution and a third convolution, the second layer network comprises a fourth convolution and a fifth convolution, the third layer network comprises a first general inverted bottle neck module, the fourth layer network comprises a second general inverted bottle neck module, and the fifth layer network comprises a sixth convolution and a seventh convolution; the first general inverted bottle neck module comprises a first additional depth convolution, a first inverted bottle neck module and a feature module connected in series, and the second general inverted bottle neck module comprises a second additional depth convolution, a third additional depth convolution and a second inverted bottle neck module connected in series; the first inverted bottle neck module comprises a first depth convolution, the second inverted bottle neck module comprises a second depth convolution, and the feature module comprises a fourth additional depth convolution; the input of the initial convolution is the input of the lightweight backbone network, and the outputs of the third layer network, the fourth layer network and the spatial pyramid pooling module are the first perforated feature, the second perforated feature and the third perforated feature respectively.

[0022] Preferably, in S22, the third perforated feature is input into the multi-scale channel attention module to obtain a channel attention map, and the channel attention map is element-wise multiplied with the third perforated feature to obtain a channel prior; the channel prior is input into the multi-scale depth separation convolution spatial attention module to obtain a spatial attention map, and the spatial attention map is element-wise multiplied with the channel prior to obtain a spatial feature; in S24, the input of the network output layer is the first tympanic membrane fine-grained feature, the second tympanic membrane fine-grained feature and the third tympanic membrane fine-grained feature, and the first segmentation feature map, the second segmentation feature map and the third segmentation feature map are obtained in sequence, and the first segmentation feature map, the second segmentation feature map and the third segmentation feature map are sequentially subjected to grid prediction, and the prediction box and the mask with the highest confidence are screened and retained, thereby obtaining the tympanic membrane image recognition result.

[0023] Preferably, in S23, the feature pyramid network comprises a third feature compression convolution, an eighth convolution and a fourth feature compression convolution, and the path aggregation network comprises a ninth convolution, a first down-sampling, a tenth convolution, a second down-sampling and an eleventh convolution; the input of the feature pyramid network is the first perforated feature, the second perforated feature and the spatial feature, and the output is the first semantic feature, the second semantic feature and the third semantic feature; the first semantic feature, the second semantic feature and the third semantic feature are input into the path aggregation network, and the output of the path aggregation network is the first tympanic membrane fine-grained feature, the second tympanic membrane fine-grained feature and the third tympanic membrane fine-grained feature.

[0024] Preferably, S3 specifically comprises the following sub-steps:

[0025] S31, setting a loss function CIoU;

[0026] S32, setting model parameters and evaluation indexes, and the evaluation indexes comprise precision, recall, accuracy and F1 score;

[0027] S33, training based on the eardrum training dataset: based on the training dataset and the test dataset, the eardrum image and the corresponding label image in the training dataset are input into the eardrum perforation recognition model based on deep learning for training.

[0028] Preferably, the spatial pyramid pooling module in S21 comprises a first feature compression convolution, a second feature compression convolution, and a first maximum pooling unit, a second maximum pooling unit and a third maximum pooling unit connected in series, the input of the first maximum pooling unit is the output of the first feature compression convolution, the output of the first feature compression convolution and the outputs of the first maximum pooling unit, the second maximum pooling unit and the third maximum pooling unit are input to the second feature compression convolution through a channel splicing operation, the input of the first feature compression convolution is the input of the spatial pyramid pooling module, and the output of the second feature compression convolution is the output of the spatial pyramid pooling module.

[0029] Preferably, CIoU in S31 is represented as:

[0030] ;

[0031] wherein, represents the Euclidean distance between the center points of the predicted box and the real box; represents the diagonal distance of the smallest closed region that can contain the predicted box and the real box; represents the intersection over union; represents a balance factor for balancing the loss caused by the difference in aspect ratio and the difference in area, and v represents the normalized value of the difference in aspect ratio between the predicted box and the real box.

[0032] Preferably, in S31, and v are respectively represented as:

[0033] ;

[0034] ;

[0035] wherein, represents the width of the real box; represents the height of the real box; represents the width of the predicted box; represents the height of the predicted box; represents the tangent value.

[0036] Preferably, S33 specifically comprises the following sub-steps:

[0037] Input the tympanic membrane image in the training data set and the corresponding label into the deep learning-based tympanic membrane perforation recognition model, output the tympanic membrane image recognition result, and obtain the network weight parameters of the deep learning-based tymanic membrane perforation recognition model; repeat the execution until all training generations are completed, obtain the weight parameters of the deep learning-based tympanic membrane perforation recognition model corresponding to the optimal evaluation index of the test data set in all training generations, and obtain the target tympanic membrane perforation recognition model.

[0038] Compared with the prior art, the technical effect of the present application is:

[0039] 1. The deep learning-based tympanic membrane perforation recognition method designed by the present application introduces a multi-scale channel and a multi-scale depth separable convolution spatial attention to cope with the problems of difficult identification of small perforations and various perforation forms, dynamically allocates weights in the channel and spatial dimensions, captures the spatial features of the perforation position and perforation size and the multi-scale features of the perforation category, effectively supplements the feature extraction capability of the lightweight backbone network, and can better capture the details and structures in the tympanic membrane image, thereby improving the image segmentation precision.

[0040] 2. The deep learning-based tympanic membrane perforation recognition method designed by the present application, the feature pyramid network transmits the top layer semantic information to the bottom layer semantic information step by step through the context path, the path aggregation network obtains different scale feature maps through the introduction of horizontal connection and bottom-up path information transmission, and realizes information fusion between different resolution feature maps through path aggregation; at the same time, the feature fusion network with a learnable down-sampling module is adopted, the feature extraction loss is reduced through the learnable mechanism, better feature integration is realized, the parameter amount is reduced, and the image segmentation precision is improved.

[0041] 3. The deep learning-based tympanic membrane perforation recognition method designed by the present application realizes automatic high-precision recognition of tympanic membrane perforation by constructing a multi-class labeled tympanic membrane image benchmark library and a deep learning-based tympanic membrane perforation recognition model, improves accuracy and efficiency by using objective medical image analysis, and also provides a reliable basis for legal identification, which has significant application value; at the same time, based on the tympanic membrane perforation segmentation result, the pixel area size of the perforation part can be calculated, so that the tympanic membrane perforation recognition is better. BRIEF DESCRIPTION OF DRAWINGS

[0042] Other features, objects and advantages of the present application will become more apparent from the following detailed description of non-limiting embodiments made with reference to the accompanying drawings.

[0043] Figure 1 is a flow chart of the deep learning-based tympanic membrane perforation recognition method of the present application;

[0044] Figure 2 is a deep learning-based tympanic membrane perforation recognition model structure schematic diagram of the present application;

[0045] Figure 3 is a structural schematic diagram of a lightweight backbone network of the present application;

[0046] Figure 4 is a structural schematic diagram of a multi-scale channel and spatial attention module of the present application;

[0047] Figure 5 is a structural schematic diagram of a feature fusion network of the present application;

[0048] Figure 6 is a normal tympanic membrane binary image in one specific embodiment of the present application;

[0049] Figure 7 is a healed tympanic membrane binary image in one specific embodiment of the present application;

[0050] Figure 8 is a perforated tympanic membrane image in one specific embodiment of the present application;

[0051] Figure 9 is a perforated tympanic membrane binary image in one specific embodiment of the present application;

[0052] Figure 10 is a perforated tympanic membrane perforation part binary image in one specific embodiment of the present application. DETAILED DESCRIPTION

[0053] The present application will be further described below in conjunction with the drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the related application, and not to limit the application. In addition, it should be noted that only the parts related to the application are shown in the drawings for ease of description. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the drawings and in conjunction with the embodiments.

[0054] Figure 1 The tympanic membrane perforation recognition method based on deep learning of the present application comprises the following steps:

[0055] S1, obtaining tympanic membrane images and constructing a tympanic membrane training data set: obtaining a plurality of tympanic membrane images and performing multi-class labeling, constructing a tympanic membrane label image database, and dividing the training data set and the test data set after data preprocessing.

[0056] S11, constructing a tympanic membrane label image database: obtaining a plurality of tympanic membrane images based on an otoscope, performing multi-class labeling to obtain corresponding label images, constructing a tympanic membrane label image database, and multi-class labeling including tympanic membrane region and perforation region.

[0057] S12, data preprocessing: based on the tympanic membrane label image database, the data set for tympanic membrane training is obtained through data enhancement technology expansion and normalization processing, unified size and format, the data enhancement technology includes rotation, flip, occlusion, splicing (Mosaic), copy-paste (copy-paste) and the like, and the diversity of the data set is improved.

[0058] with the total standard deviation and the mean m, the normalization processing is performed:

[0059] (1);

[0060] wherein, and respectively represent the pixel value of the pixel point in the tympanic membrane image before and after the normalization processing.

[0061] The unified size is 640*640*3.

[0062] S13, dividing the training data set and the test data set: based on the data set for tympanic membrane training, the training data set and the test data set are obtained by dividing according to the first proportion.

[0063] S2, constructing a deep learning-based tympanic membrane perforation recognition model: as shown in Figure 2 The deep learning-based tympanic membrane perforation recognition model includes a lightweight backbone network, a multi-scale channel and spatial attention module CPMS, a feature fusion network and a network output layer. The tympanic membrane image is sequentially obtained through the lightweight backbone network, the multi-scale channel and spatial attention module CPMS, the feature fusion network to obtain the perforation feature image, the spatial feature image, the multi-scale feature image and the tympanic membrane image recognition result. The lightweight backbone network is used to extract the perforation feature information of the tympanic membrane image, the multi-scale channel and spatial attention module is used to improve the extraction ability of the model to the spatial feature information, the feature fusion network is used to process and integrate the multi-scale feature information extracted by the network, and the network output layer is used to output the tympanic membrane image recognition result.

[0064] S21, each network module in the lightweight backbone network adopts channel separable convolution and SE channel attention mechanism and is connected through residual connection, introduces a universal inverted bottleneck (UIB) search block, combines an inverted bottleneck (IB), a ConvNext, a feedforward network (FFN) and a new additional depth convolution variant.

[0065] As shown in Figure 3 ​As shown, the lightweight backbone network comprises an initial convolution, a first layer network, a second layer network, a third layer network, a fourth layer network, a fifth layer network and a spatial pyramid pooling module SPPF, the initial convolution, the first layer network, the second layer network, the third layer network, the fourth layer network, the fifth layer network and the spatial pyramid pooling module SPPF are connected in series, and the input of the initial convolution is the input of the lightweight backbone network; the initial convolution comprises a first convolution, the first layer network comprises second and third convolutions, the second layer network comprises fourth and fifth convolutions, the third layer network comprises a first general inverted bottle neck module, the fourth layer network comprises a second general inverted bottle neck module, and the fifth layer network comprises sixth and seventh convolutions; the first general inverted bottle neck module comprises a first additional depth convolution, a first inverted bottle neck module and a feature module, the second general inverted bottle neck module comprises a second additional depth convolution, a third additional depth convolution and a second inverted bottle neck module; the first inverted bottle neck module comprises a first depth convolution, the second inverted bottle neck module comprises a second depth convolution, and the feature module comprises a fourth additional depth convolution. The first additional depth convolution, the first inverted bottle neck module and the feature module are connected in series, and the second additional depth convolution, the third additional depth convolution and the second inverted bottle neck module are connected in series. The outputs of the third layer network, the fourth layer network and the spatial pyramid pooling module SPPF are respectively a first perforated feature, a second perforated feature and a third perforated feature, which are taken as the output of the lightweight backbone network.

[0066] The convolution kernel of the first convolution, the second convolution and the fourth convolution is 3x3 and the step is 2, the convolution kernel of the third convolution, the fifth convolution, the sixth convolution and the seventh convolution is 1x1 and the step is 1, the convolution kernel of the first additional depth convolution is 5x5 and the step is 2, the convolution kernel of the second additional depth convolution is 3x3 and the step is 2, the convolution kernel of the third additional depth convolution is 5x5 and the step is 1, the convolution kernel of the fourth additional depth convolution is 3x3 and the step is 1, and the convolution kernel of the first depth convolution and the second depth convolution is 3x3 and the step is 1; the output channel numbers of the first convolution, the second convolution, the third convolution, the fourth convolution, the fifth convolution, the sixth convolution and the seventh convolution are 32, 32, 32, 64, 64, 960 and 1280 respectively, the output channel numbers of the first additional depth convolution, the second additional depth convolution, the third additional depth convolution and the fourth additional depth convolution are 96, 128, 128 and 96 respectively, and the output channel numbers of the first depth convolution and the second depth convolution are 96 and 128 respectively.

[0067] In a preferred embodiment, the first inverted bottle neck module and the second inverted bottle neck module each have four and are connected in series.

[0068] The spatial pyramid pooling module SPPF includes a first feature compression convolution CBS, a first max pooling unit, a second max pooling unit, a third max pooling unit, and a second feature compression convolution CBS, the first max pooling unit, the second max pooling unit, and the third max pooling unit are connected in series, the input of the first max pooling unit is the output of the first feature compression convolution CBS, the output of the first feature compression convolution CBS and the outputs of the first max pooling unit, the second max pooling unit, and the third max pooling unit are input to the second feature compression convolution CBS through a channel splicing operation, the input of the first feature compression convolution CBS is the input of the spatial pyramid pooling module SPPF, and the output of the second feature compression convolution CBS is the output of the spatial pyramid pooling module SPPF, that is, the third perforated feature. The convolution kernel of the first max pooling unit, the second max pooling unit, and the third max pooling unit is 5*5, and the step is 1, the convolution kernel of the first feature compression convolution and the second feature compression convolution CBS is 1*1, and the step is 1, and the output channel numbers of the first feature compression convolution and the second feature compression convolution CBS are 512 and 1024 respectively.

[0069] In S22, the multi-scale channel and spatial attention module encodes context information, effectively extracts spatial features, retains channel prior knowledge, has the ability to focus on channel attention and spatial attention, and dynamically allocates weights in the channel and spatial dimensions through parallel multi-scale feature extraction channels and spatial attention of depth separable convolution.

[0070] As shown in Figure 4 , the multi-scale channel and spatial attention module CPMS includes a multi-scale channel attention module and a multi-scale depth separable convolution spatial attention module, and the overall structure includes sequential placement of channel attention and spatial attention. The input of the multi-scale channel and spatial attention module CPMS is the third perforated feature, and the channel attention map is obtained through the multi-scale channel attention module, the third perforated feature and the channel attention map are multiplied to obtain channel prior, the channel prior is obtained through the multi-scale depth separable convolution spatial attention module to obtain the spatial attention map, and the channel prior and the spatial attention map are multiplied to obtain spatial features, which are output as the multi-scale channel and spatial attention module CPMS.

[0071] The multi-scale channel attention module is provided with an average pooling unit, a max pooling unit, and a shared multi-layer perception unit, and the multi-scale depth separable convolution spatial attention module is provided with a depth convolution unit. The output channel number of the multi-scale channel and spatial attention module CPMS is 1024.

[0072] In S23, as Figure 5As shown, the feature fusion network includes a feature pyramid network FPN and a path aggregation network PAN, the feature pyramid network FPN includes a third feature compression convolution CBS, an eighth convolution and a fourth feature compression convolution CBS, the path aggregation network PAN includes a ninth convolution, a first down-sampling, a tenth convolution, a second down-sampling and an eleventh convolution; the input of the feature pyramid network FPN is the first perforated feature, the second perforated feature and the spatial feature, the spatial feature is subjected to the third feature compression convolution CBS to obtain a third semantic feature, the third semantic feature is subjected to an up-sampling operation and then is channel spliced with the second perforated feature as the input of the eighth convolution, the output of the eighth convolution is subjected to the fourth feature compression convolution CBS to obtain a second semantic feature, the second semantic feature is subjected to an up-sampling operation and then is channel spliced with the first perforated feature to obtain a first semantic feature; the input of the path aggregation network PAN is the first semantic feature, the second semantic feature and the third semantic feature, the first semantic feature is subjected to the ninth convolution to obtain a first eardrum fine-grained feature, the first eardrum fine-grained feature is subjected to the first down-sampling and then is channel spliced with the second semantic feature as the input of the tenth convolution, the output of the tenth convolution is subjected to the second down-sampling and then is channel spliced with the third semantic feature as the input of the eleventh convolution, the outputs of the tenth convolution and the eleventh convolution are a second eardrum fine-grained feature and a third eardrum fine-grained feature respectively; the first eardrum fine-grained feature, the second eardrum fine-grained feature and the third eardrum fine-grained feature are the outputs of the feature fusion network, so as to sufficiently extract the features of eardrum similar edge refinement and texture refinement, ensure the fine granularity of the final output result and improve the model recognition accuracy.

[0073] The output channel numbers of the third feature compression convolution and the fourth feature compression convolution CBS are 512 and 256 respectively, the output channel numbers of the eighth convolution, the ninth convolution, the tenth convolution and the eleventh convolution are 512, 256, 512 and 1024 respectively, and the output channel numbers of the first and second down-samplings are 256 and 512 respectively; the parameters of the first and second down-samplings can be learned.

[0074] The feature fusion network is combined by the feature pyramid network FPN and the path aggregation network PAN, the feature pyramid network FPN transmits the top layer semantic information to the bottom layer semantic information step by step through the context path, the path aggregation network PAN obtains the feature maps of different scales through the introduction of the lateral connection and the bottom-up path information transmission, and realizes the information fusion between the feature maps of different resolutions through the path aggregation; the down-sampling Adown with learning ability is introduced to reduce the parameter quantity and better preserve the image features.

[0075] S24, the network output layer comprises a twelfth convolution, a thirteenth convolution and a fourteenth convolution, the input of the network output layer is the first eardrum fine-grained feature, the second eardrum fine-grained feature and the third eardrum fine-grained feature, the first eardrum fine-grained feature is subjected to the twelfth convolution to obtain a first segmentation feature map, the second eardrum fine-grained feature is subjected to the thirteenth convolution to obtain a second segmentation feature map, and the third eardrum fine-grained feature is subjected to the fourteenth convolution to obtain a third segmentation feature map; the output channel numbers of the twelfth convolution, the thirteenth convolution and the fourteenth convolution are all 255.

[0076] The sizes of the first segmentation feature map, the second segmentation feature map and the third segmentation feature map are 255x80x80, 255x40x40 and 255x20x20 respectively. The first segmentation feature map is responsible for detecting small targets, the second segmentation feature map is responsible for detecting medium targets, and the third segmentation feature map is responsible for detecting large targets; grid prediction is sequentially performed on the first segmentation feature map, the second segmentation feature map and the third segmentation feature map to obtain a feature map division network and a mask after detector convolution, and further to obtain a prediction box and a mask; based on the confidence score and the overlap degree of the prediction box and the mask, the prediction box and the mask are screened through a non-maximum suppression (NMS) algorithm, and the prediction box and the mask with the highest confidence are retained, that is, an eardrum image recognition result is obtained, which includes six kinds of information, i.e., the number of categories, the confidence (nc) corresponding to each category, the prediction box height, the prediction box width, the prediction box center point coordinate and the mask.

[0077] S3, training a deep learning-based eardrum perforation identification model to obtain a target eardrum perforation identification model: setting a loss function, setting model parameters and evaluation indexes, training the deep learning-based eardrum perforation identification model based on a training data set and a test data set, obtaining the network weight parameters of the deep learning-based eardrum perforation identification model corresponding to the best evaluation index of the test data set, and obtaining the target eardrum perforation identification model.

[0078] S31, setting a loss function CIoU;

[0079] (2) ;

[0080] wherein represents the Euclidean distance between the center points of the prediction box and the real box; represents the diagonal distance of the smallest closed region capable of containing the prediction box and the real box; represents the intersection over union; represents a balance factor for balancing the loss caused by the difference in aspect ratio and the difference in area, v represents the normalized value of the aspect ratio difference between the prediction box and the real box, and has:

[0081] (3) ;

[0082] (4);

[0083] wherein, represents the width of the real frame; represents the height of the real frame; represents the width of the predicted frame; represents the height of the predicted frame; represents the tangent value.

[0084] S32, set model parameters and evaluation indexes, the evaluation indexes include precision, recall, accuracy and F1 score, respectively represented as:

[0085] (5);

[0086] (6);

[0087] (7);

[0088] (8);

[0089] wherein, TP represents true positive; FP represents false positive; TN represents true negative; and FN represents false negative.

[0090] S33, training based on the tympanic membrane training data set: based on the training data set and the test data set, the tympanic membrane image and the corresponding label image in the training data set are input into the deep learning based tympanic membrane perforation recognition model for training.

[0091] S331, the tympanic membrane image and the corresponding label in the training data set are taken as the input of the deep learning based tympanic membrane perforation recognition model, and the first perforation feature, the second perforation feature and the third perforation feature of the tympanic membrane image are extracted through the lightweight backbone network.

[0092] S332, the third perforation feature is extracted through the multi-scale channel and spatial attention module CPMS to obtain channel attention and spatial attention, encode context information, and obtain spatial features.

[0093] S333, the first perforation feature, the second perforation feature and the spatial feature are input into the feature fusion network to obtain the first tympanic membrane fine-grained feature, the second tympanic membrane fine-grained feature and the third tympanic membrane fine-grained feature.

[0094] S334, the first eardrum, the second eardrum and the third eardrum fine-grained feature input network output layer, to obtain the first segmentation feature map, the second segmentation feature map and the third segmentation feature map, to obtain the eardrum image recognition result, to obtain the network weight parameter of the eardrum perforation recognition model based on deep learning.

[0095] S335, repeat S331 to S334 until all training generations are completed, obtain the evaluation index of the test data set in all training generations, and obtain the weight parameter of the target eardrum perforation recognition model based on deep learning.

[0096] S4, based on the target eardrum perforation recognition model, obtaining the recognition result of the eardrum image: based on the target eardrum perforation recognition model, the eardrum image after normalization is taken as the input of the target eardrum perforation recognition model, and the output of the target eardrum perforation recognition model is obtained, that is, the recognition result of the eardrum image.

[0097] Based on the segmentation mask of the recognition result of the eardrum image, the segmentation mask of the perforation is binarized, the mask pixels are traversed, the pixel number of different category segmentation parts is counted, and the statistical result is updated to the output result, to obtain the perforation area size.

[0098] As shown in Figure 6-8 and Table 1, examples of eardrum image recognition results under different conditions are shown. For normal eardrum, the outline of the eardrum and the confidence level of recognition are displayed; for healing eardrum, the outline of the eardrum, the part of the healing perforation and the confidence level of recognition are displayed; for perforated eardrum, the outline of the eardrum, the part of the perforation, the area of the eardrum outline, the area of the perforation part and the confidence level of recognition are displayed, and the highest recognition confidence level of the eardrum image is up to 0.93, the reliability is good, and the overall recognition accuracy reaches 95.7%, with high recognition accuracy. In a preferred embodiment, Figure 8 The perforated eardrum image in FIG. 8 is a result schematic diagram obtained by network prediction and recognition, wherein the most central frame represents the outer rectangle of the perforation part, and the adjacent outer frame represents the outer rectangle of the eardrum outline.

[0099] Figure 9 Examples of binary images of the segmentation results of the perforation part and the non-perforation part of the eardrum perforation image are shown, the central black area is the binary representation of the perforation part, and the white area is the binary representation of the eardrum outline, so as to more intuitively display the eardrum part.

[0100] Figure 10 The white area in FIG. 9 shows a binary image example of the perforation part of the eardrum perforation image.

[0101] In one specific embodiment, as shown in Table 1, the perforated portion area of the perforated eardrum is represented by 1425.5 pixel points, the eardrum image area is represented by 9108 pixel points, the proportion of the perforated portion is calculated according to the ratio of the number of pixel points, the ratio of the image area of the perforated portion to the image area of the eardrum is represented, and further image reference and help are provided for the recognition of the eardrum condition.

[0102] Table 1

[0103]

[0104] The present application proposes a deep learning-based eardrum perforation recognition method. To address the problems of difficult identification of small perforations and diverse perforation patterns, multi-scale channels and multi-scale depth separable convolution spatial attention are introduced to dynamically allocate weights in the channel and spatial dimensions to capture the spatial features of the perforation position and perforation size and the multi-scale features of the perforation category, effectively supplementing the feature extraction capability of the lightweight backbone network, better capturing the details and structures in the eardrum image, and thus improving the image segmentation accuracy; the feature pyramid network transmits the top-level semantic information to the bottom-level semantics level by level through the context path, the path aggregation network obtains feature maps of different scales by introducing horizontal connection and bottom-up path information transmission, and realizes information fusion between feature maps of different resolutions through path aggregation; at the same time, a feature fusion network with a learnable downsampling module is used to reduce feature extraction loss through a learnable mechanism, better feature integration is realized, the parameter amount is reduced, and the image segmentation accuracy is improved; by constructing a multi-class labeled eardrum image benchmark library and a deep learning-based eardrum perforation recognition model, automatic high-precision recognition of eardrum perforation is realized, objective medical image analysis is used to improve accuracy and efficiency, and reliable basis is provided for legal identification, which has significant clinical application value; based on the eardrum perforation segmentation result, the pixel area size of the perforated portion can be calculated to provide image reference for eardrum condition recognition.

[0105] Finally, it should be noted that the above embodiments are only illustrative and not limiting the technical solutions of the present application. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that the present application can still be modified or replaced equivalently without departing from the spirit and scope of the present application. Any modification or partial replacement should be covered in the scope of the claims of the present application.

Claims

1. A deep learning-based method for identifying tympanic membrane perforations, characterized in that: It includes the following steps: S1. Obtain tympanic membrane images and construct a dataset for tympanic membrane training; S2. Construct a deep learning-based tympanic membrane perforation identification model: The tympanic membrane perforation identification model includes a lightweight backbone network, a multi-scale channel and spatial attention module, a feature fusion network, and a network output layer; S21. Construct a lightweight backbone network for extracting perforation feature information of tympanic membrane images, and output the first perforation feature, the second perforation feature and the third perforation feature. The lightweight backbone network includes an initial convolution, a first layer network, a second layer network, a third layer network, a fourth layer network, a fifth layer network and a spatial pyramid pooling module connected in series. S22. Construct a multi-scale channel and spatial attention module to output spatial features. The multi-scale channel and spatial attention module includes a multi-scale channel attention module and a multi-scale depth separation convolutional spatial attention module. In S22, the third perforation feature is input into the multi-scale channel attention module to obtain the channel attention map. The channel attention map and the third perforation feature are multiplied element-wise to obtain the channel prior. The channel prior is input into the multi-scale depth separation convolution spatial attention module to obtain the spatial attention map. The spatial attention map and the channel prior are multiplied element-wise to obtain the spatial feature. S23. Construct a feature fusion network to output the first tympanic membrane fine-grained feature, the second tympanic membrane fine-grained feature, and the third tympanic membrane fine-grained feature. The feature fusion network is used to integrate the multi-scale feature information extracted by the network. The feature fusion network includes a feature pyramid network and a path aggregation network. In S23, the feature pyramid network includes the third feature compression convolution, the eighth convolution, and the fourth feature compression convolution, while the path aggregation network includes the ninth convolution, the first downsampling, the tenth convolution, the second downsampling, and the eleventh convolution. The inputs of the feature pyramid network are the first perforation feature, the second perforation feature, and the spatial feature, and the outputs are the first semantic feature, the second semantic feature, and the third semantic feature. The first semantic feature, the second semantic feature, and the third semantic feature serve as the inputs of the path aggregation network, and the outputs of the path aggregation network are the first tympanic membrane fine-grained feature, the second tympanic membrane fine-grained feature, and the third tympanic membrane fine-grained feature. S24. Construct the network output layer to output the tympanic membrane image recognition results. The tympanic membrane image recognition results include the number of categories, the confidence score for each category, the predicted bounding box height, the predicted bounding box width, the coordinates of the predicted bounding box center point, and the mask. In S24, the input to the network output layer is the first tympanic membrane fine-grained feature, the second tympanic membrane fine-grained feature, and the third tympanic membrane fine-grained feature, which sequentially obtain the first segmentation feature map, the second segmentation feature map, and the third segmentation feature map. The first segmentation feature map, the second segmentation feature map, and the third segmentation feature map are sequentially subjected to gridded prediction, and the prediction box and mask with the highest confidence are selected and retained, thus obtaining the tympanic membrane image recognition result. S3. Train a deep learning-based tympanic membrane perforation recognition model to obtain the target tympanic membrane perforation recognition model; S4. Based on the target tympanic membrane perforation recognition model, obtain the recognition results of the tympanic membrane image.

2. The deep learning-based tympanic membrane perforation identification method according to claim 1, characterized in that: S1 specifically includes the following sub-steps: S11. Construct a tympanic membrane label image database: Based on an otoscope, obtain several tympanic membrane images, perform multi-category annotation to obtain corresponding label images, and construct a tympanic membrane label image database. The multi-category annotation includes the tympanic membrane region and the perforation region. S12. Data Preprocessing: Based on the tympanic membrane tag image database, the data is expanded and normalized using data augmentation techniques to unify the size and format, thus obtaining a dataset for tympanic membrane training. S13. Divide the training dataset and the test dataset: Based on the tympanic membrane training dataset, divide it according to the first ratio to obtain the training dataset and the test dataset.

3. The deep learning-based tympanic membrane perforation identification method according to claim 1, characterized in that: In S21, the initial convolution includes a first convolution, the first layer includes a second and a third convolution, the second layer includes a fourth and a fifth convolution, the third layer includes a first general inverted bottleneck module, the fourth layer includes a second general inverted bottleneck module, and the fifth layer includes a sixth and a seventh convolution. The first general inverted bottleneck module includes a first additional depth convolution, a first inverted bottleneck module, and a feature module connected in series. The second general inverted bottleneck module includes a second additional depth convolution, a third additional depth convolution, and a second inverted bottleneck module connected in series. The first inverted bottleneck module includes a first depth convolution, the second inverted bottleneck module includes a second depth convolution, and the feature module includes a fourth additional depth convolution. The input of the initial convolution is the input of the lightweight backbone network, and the outputs of the third layer, the fourth layer, and the spatial pyramid pooling module are the first perforation feature, the second perforation feature, and the third perforation feature, respectively.

4. The deep learning-based tympanic membrane perforation identification method according to claim 2, characterized in that: S3 specifically includes the following sub-steps: S31. Define the loss function CIoU; S32. Set model parameters and evaluation metrics, including precision, recall, accuracy, and F1 score. S33. Training based on the tympanic membrane training dataset: Based on the training dataset and the test dataset, the tympanic membrane images and corresponding label images in the training dataset are input into the deep learning-based tympanic membrane perforation recognition model for training.

5. The deep learning-based tympanic membrane perforation identification method according to claim 1, characterized in that: The spatial pyramid pooling module in S21 includes a first feature compression convolution, a second feature compression convolution, and a first max pooling unit, a second max pooling unit, and a third max pooling unit connected in series. The input of the first max pooling unit is the output of the first feature compression convolution. The output of the first feature compression convolution and the outputs of the first max pooling unit, the second max pooling unit, and the third max pooling unit are concatenated through a channel concatenation operation and input to the second feature compression convolution. The input of the first feature compression convolution is the input of the spatial pyramid pooling module, and the output of the second feature compression convolution is the output of the spatial pyramid pooling module.

6. The deep learning-based tympanic membrane perforation identification method according to claim 4, characterized in that: In S31, CIoU is represented as: ； in, This represents the Euclidean distance between the center points of the predicted bounding box and the ground truth bounding box; It represents the diagonal distance of the smallest closure region that can simultaneously contain both the predicted bounding box and the ground truth bounding box; Indicates intersection, union, and ratio; represents the balancing factor used to weigh the losses caused by aspect ratio differences and regional differences, and v represents the normalized value of the aspect ratio difference between the predicted box and the ground truth box.

7. The deep learning-based tympanic membrane perforation identification method according to claim 6, characterized in that: S31 and v are represented as: ； ； in, Indicates the width of the actual bounding box; Indicates the height of the actual bounding box; Indicates the width of the prediction box; Indicates the height of the prediction box; This represents the tangent value.

8. The deep learning-based tympanic membrane perforation identification method according to claim 4, characterized in that: S33 specifically includes the following sub-steps: Input the tympanic membrane images and corresponding labels from the training dataset into the deep learning-based tympanic membrane perforation recognition model, output the tympanic membrane image recognition results, and obtain the network weight parameters of the deep learning-based tympanic membrane perforation recognition model; repeat the process until all training generations are completed, and obtain the weight parameters of the deep learning-based tympanic membrane perforation recognition model when the evaluation index of the test dataset is optimal among all training generations, thus obtaining the target tympanic membrane perforation recognition model.

Images