Tympanic membrane perforation identification method based on deep learning

Abstract

The invention belongs to the technical field of medical image analysis based on deep learning, and provides a tympanic membrane perforation recognition method based on deep learning, which comprises the following steps: acquiring a tympanic membrane image and constructing a data set for tympanic membrane training; constructing a tympanic membrane perforation recognition model based on deep learning, wherein the tympanic membrane perforation recognition model based on deep learning comprises a lightweight backbone network, a multi-scale channel and space attention module, a feature fusion network and a network output layer; training a tympanic membrane perforation recognition model based on deep learning to obtain a target tympanic membrane perforation recognition model; and obtaining an identification result of the tympanic membrane image based on the target tympanic membrane perforation identification model. In order to solve the problems that micro perforation is difficult to recognize and perforation forms are diversified, multi-scale channel attention and multi-scale depth separable convolution space attention are introduced, context path semantic information and path information from bottom to top are aggregated, and automatic high-precision recognition of tympanic membrane perforation is achieved.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image analysis based on deep learning, and in particular to the technical field of tympanic membrane image segmentation and feature extraction, and more particularly to a tympanic membrane perforation recognition method based on deep learning. Background Art

[0002] Identifying a perforated tympanic membrane plays a crucial role in middle ear diseases. It is not only a key factor in determining the severity of middle ear infection, inflammation, or trauma, but also directly influences treatment options and prognosis assessment.

[0003] Chinese patent application publication number CN118470315A discloses an ENT examination image segmentation method and system, which includes constructing a first recognition model to identify tympanic membrane features, obtaining a shooting distance and a shooting angle, filtering the captured video data according to the shooting distance and the shooting angle, and obtaining image data; establishing a two-dimensional coordinate system on the image, identifying the intersection of the two-dimensional coordinate system and the tympanic membrane boundary through a second recognition model, and obtaining a set of intersection points; obtaining a fitted ellipse according to the intersection point set, obtaining a segmentation ellipse according to a segmentation threshold and the fitted ellipse, and segmenting the image according to the segmentation ellipse to obtain a segmented image; constructing a tympanic membrane perforation recognition model to identify the segmented image, and obtain a perforation data set, and calculating a perforation coefficient and a perforation risk coefficient through the perforation data set.

[0004] Disclosed in Chinese patent application publication number CN115316921A is an intelligent self-inspection method, system and storage medium for the inner auditory canal, which relates to the field of personal care medical examination technology, and includes: obtaining inner auditory canal detection image information of the inspected person; identifying inner auditory canal identification position points from the inner auditory canal detection image information according to preset inner auditory canal identification point features; analyzing and obtaining standard identification image information from the inner auditory canal detection image information according to the inner auditory canal identification position points, preset reference image information and identification reference points; analyzing and outputting the normal image similarity rate and specific distinguishing point annotation information between the standard identification image information and the normal inner auditory canal detection image information based on the comparison result between the standard identification image information and the preset normal inner auditory canal detection image information; using the normal image similarity rate and the specific distinguishing point annotation information as the final detection result, and outputting the final detection result to the terminal held by the inspected person.

[0005] However, existing methods for identifying tympanic membrane perforations still present several challenges. Traditional diagnostic methods primarily rely on otoendoscopic examination and the physician's clinical experience, which is not only highly subjective but also prone to misdiagnosis when the perforation is small or hidden. Furthermore, while imaging techniques such as acoustic immittance testing and temporal bone CT can provide more detailed information about middle ear structures, these methods often require specialized equipment and complex procedures, and in some cases, struggle to achieve rapid and accurate diagnosis. In recent years, with the advancement of artificial intelligence (AI), deep learning-based image recognition methods have provided new approaches for automated, high-precision identification of tympanic membrane perforations. By utilizing advanced algorithms such as convolutional neural networks (CNNs), otendoscopic images can be automatically analyzed, enabling rapid detection and classification of tympanic membrane perforations. This approach not only reduces human interference and improves diagnostic accuracy and efficiency, but also has broad application prospects, potentially providing clinicians with a more powerful auxiliary tool and promoting advancements in middle ear disease diagnostics.

[0006] Therefore, in order to realize the automatic detection and classification of tympanic membrane perforation, it is urgent and necessary to seek a tympanic membrane perforation recognition method based on deep learning to improve the accuracy and efficiency of tympanic membrane perforation recognition. Summary of the Invention

[0007] In response to the above-mentioned defects in the prior art, the present invention proposes a method for identifying tympanic membrane perforations based on deep learning. The method includes acquiring a tympanic membrane image and constructing a tympanic membrane training dataset; constructing a tympanic membrane perforation recognition model based on deep learning, the tympanic membrane perforation recognition model based on deep learning including a lightweight backbone network, a multi-scale channel and spatial attention module, a feature fusion network and a network output layer; training the tympanic membrane perforation recognition model based on deep learning to obtain a target tympanic membrane perforation recognition model; and obtaining the recognition result of the tympanic membrane image based on the target tympanic membrane perforation recognition model. In order to address the problem that tiny perforations are difficult to identify and the perforation morphology is diverse, the present invention introduces multi-scale channel attention and multi-scale depth-separable convolutional spatial attention, aggregates contextual path semantic information and bottom-up path information, realizes automated high-precision recognition of tympanic membrane perforations, improves accuracy and efficiency, and has significant application value.

[0008] The present invention provides a method for identifying tympanic membrane perforation based on deep learning, which comprises the following steps:

[0009] S1, obtain the tympanic membrane image and construct the tympanic membrane training dataset;

[0010] S2. Build a deep learning-based tympanic membrane perforation recognition model: The tympanic membrane perforation recognition model includes a lightweight backbone network, a multi-scale channel and spatial attention module, a feature fusion network, and a network output layer;

[0011] S21, constructing a lightweight backbone network for extracting perforation feature information of the tympanic membrane image, outputting a first perforation feature, a second perforation feature, and a third perforation feature, wherein the lightweight backbone network includes an initial convolution, a first layer network, a second layer network, a third layer network, a fourth layer network, a fifth layer network, and a spatial pyramid pooling module connected in series in sequence;

[0012] S22. Construct a multi-scale channel and spatial attention module to output spatial features. The multi-scale channel and spatial attention module includes a multi-scale channel attention module and a multi-scale depth separation convolution spatial attention module.

[0013] S23, constructing a feature fusion network to output the first eardrum fine-grained feature, the second eardrum fine-grained feature, and the third eardrum fine-grained feature. The feature fusion network is used to integrate the multi-scale feature information extracted by the network. The feature fusion network includes a feature pyramid network and a path aggregation network.

[0014] S24. Construct a network output layer for outputting the tympanic membrane image recognition result, where the tympanic membrane image recognition result includes the number of categories, the confidence level for each category, the predicted box height, the predicted box width, the predicted box center coordinates, and the mask;

[0015] S3. Train a deep learning-based tympanic membrane perforation recognition model to obtain a target tympanic membrane perforation recognition model;

[0016] S4. Based on the target tympanic membrane perforation recognition model, obtain a recognition result of the tympanic membrane image.

[0017] Preferably, S1 specifically includes the following sub-steps:

[0018] S11. Constructing a tympanic membrane labeled image database: Acquire several tympanic membrane images using an otoendoscopy, perform multi-category annotation to obtain corresponding label images, and construct a tympanic membrane labeled image database. The multi-category annotation includes the tympanic membrane area and perforation area.

[0019] S12. Data preprocessing: Based on the eardrum labeled image database, data enhancement technology is used to expand and normalize the data, unify the size and format, and obtain the eardrum training dataset;

[0020] S13. Divide the training data set and the test data set: Based on the tympanic membrane training data set, divide it according to the first ratio to obtain a training data set and a test data set.

[0021] Preferably, the initial convolution in S21 includes the first convolution, the first layer network includes the second convolution and the third convolution, the second layer network includes the fourth convolution and the fifth convolution, the third layer network includes the first universal inverted bottleneck module, the fourth layer network includes the second universal inverted bottleneck module, and the fifth layer network includes the sixth convolution and the seventh convolution; the first universal inverted bottleneck module includes the first additional depth convolution, the first inverted bottleneck module and the feature module connected in series in sequence, and the second universal inverted bottleneck module includes the second additional depth convolution, the third additional depth convolution and the second inverted bottleneck module connected in series in sequence; the first inverted bottleneck module includes the first depth convolution, the second inverted bottleneck module includes the second depth convolution, and the feature module includes the fourth additional depth convolution; the input of the initial convolution is the input of the lightweight backbone network, and the outputs of the third layer network, the fourth layer network and the spatial pyramid pooling module are the first perforated feature, the second perforated feature and the third perforated feature, respectively.

[0022] Preferably, in S22, the third perforated feature is input into the multi-scale channel attention module to obtain a channel attention map, and the channel attention map and the third perforated feature are element-wise multiplied to obtain the channel prior; the channel prior is input into the multi-scale depth separation convolution spatial attention module to obtain a spatial attention map, and the spatial attention map and the channel prior are element-wise multiplied to obtain the spatial feature; in S24, the input of the network output layer is the first tympanic membrane fine-grained feature, the second tympanic membrane fine-grained feature and the third tympanic membrane fine-grained feature, and the first segmentation feature map, the second segmentation feature map and the third segmentation feature map are obtained in turn, and the first segmentation feature map, the second segmentation feature map and the third segmentation feature map are gridded and predicted in turn, and the prediction boxes and masks with the highest confidence are screened and retained to obtain the tympanic membrane image recognition result.

[0023] Preferably, the feature pyramid network in S23 includes a third feature compression convolution, an eighth convolution and a fourth feature compression convolution, and the path aggregation network includes a ninth convolution, a first downsampling, a tenth convolution, a second downsampling and an eleventh convolution; the input of the feature pyramid network is the first perforation feature, the second perforation feature and the spatial feature, and the output is the first semantic feature, the second semantic feature and the third semantic feature. The first semantic feature, the second semantic feature and the third semantic feature serve as the input of the path aggregation network, and the output of the path aggregation network is the first tympanic membrane fine-grained feature, the second tympanic membrane fine-grained feature and the third tympanic membrane fine-grained feature.

[0024] Preferably, S3 specifically includes the following sub-steps:

[0025] S31. Set the loss function CIoU;

[0026] S32. Set model parameters and evaluation indicators, including precision, recall, accuracy, and F1-Score.

[0027] S33. Training based on the tympanic membrane training dataset: Based on the training dataset and the test dataset, the tympanic membrane images and corresponding label images in the training dataset are input into the tympanic membrane perforation recognition model based on deep learning for training.

[0028] Preferably, the spatial pyramid pooling module in S21 includes a first feature compression convolution, a second feature compression convolution, and a first maximum pooling unit, a second maximum pooling unit, and a third maximum pooling unit connected in series in sequence. The input of the first maximum pooling unit is the output of the first feature compression convolution, and the output of the first feature compression convolution and the outputs of the first maximum pooling unit, the second maximum pooling unit, and the third maximum pooling unit are input into the second feature compression convolution through a channel splicing operation. The input of the first feature compression convolution is the input of the spatial pyramid pooling module, and the output of the second feature compression convolution is the output of the spatial pyramid pooling module.

[0029] Preferably, CIoU in S31 is expressed as:

[0030] ;

[0031] in, Represents the Euclidean distance between the center points of the predicted box and the true box; Represents the diagonal distance of the minimum closure area that can contain both the predicted box and the true box; represents the intersection and union ratio; Represents a balance factor, which is used to weigh the loss caused by aspect ratio difference and area difference, and v represents the normalized value of the aspect ratio difference between the predicted box and the true box.

[0032] Preferably, in S31 and v are expressed as:

[0033] ;

[0034] ;

[0035] in, Indicates the width of the real box; Indicates the height of the real frame; Indicates the width of the prediction box; Indicates the height of the prediction box; Represents the tangent value.

[0036] Preferably, S33 specifically includes the following sub-steps:

[0037] The tympanic membrane images and corresponding labels in the training dataset are input into the tympanic membrane perforation recognition model based on deep learning, the tympanic membrane image recognition results are output, and the network weight parameters of the tympanic membrane perforation recognition model based on deep learning are obtained; the execution is repeated until all training generations are completed, and when the evaluation index of the test dataset in all training generations is optimal, the weight parameters of the tympanic membrane perforation recognition model based on deep learning are obtained, and the target tympanic membrane perforation recognition model is obtained.

[0038] Compared with the prior art, the technical effects of the present invention are:

[0039] 1. The deep learning-based tympanic membrane perforation recognition method designed by the present invention addresses the problems of difficulty in identifying tiny perforations and the diverse morphologies of perforations. It introduces multi-scale channels and multi-scale depthwise separable convolutional spatial attention, dynamically assigns weights in the channel and spatial dimensions to capture the spatial features of perforation location and size and the multi-scale features of perforation category, effectively supplementing the feature extraction capability of the lightweight backbone network, and can better capture the details and structures in the tympanic membrane image, thereby improving image segmentation accuracy.

[0040] 2. The deep learning-based tympanic membrane perforation recognition method designed by the present invention uses a feature pyramid network to transfer top-level semantic information to bottom-level semantic information step by step through context paths. The path aggregation network obtains feature maps of different scales by introducing lateral connections and bottom-up path information transmission, and realizes information fusion between feature maps of different resolutions through path aggregation. At the same time, a feature fusion network with a learnable downsampling module is adopted to reduce feature extraction loss through a learnable mechanism, achieve better feature integration, reduce the number of parameters, and improve image segmentation accuracy.

[0041] 3. The deep learning-based tympanic membrane perforation recognition method designed by the present invention realizes automatic and high-precision recognition of tympanic membrane perforation by constructing a multi-category labeled tympanic membrane image benchmark library and a deep learning-based tympanic membrane perforation recognition model. It uses objective medical image analysis to improve accuracy and efficiency, and also provides a reliable basis for legal identification, with significant application value. At the same time, based on the tympanic membrane perforation segmentation results, the pixel area occupied by the perforated part can be calculated, thereby better identifying tympanic membrane perforation. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Other features, objects and advantages of the present application will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings.

[0043] Figure 1 This is a flow chart of the method for identifying tympanic membrane perforation based on deep learning of the present invention;

[0044] Figure 2 Schematic diagram of the structure of the tympanic membrane perforation recognition model based on deep learning of the present invention;

[0045] Figure 3 It is a structural diagram of the lightweight backbone network of the present invention;

[0046] Figure 4 Schematic diagram of the structure of the multi-scale channel and spatial attention module of the present invention;

[0047] Figure 5 It is a structural diagram of the feature fusion network of the present invention;

[0048] Figure 6 is a normal tympanic membrane binary image in a specific embodiment of the present invention;

[0049] Figure 7 is a binary image of a healing tympanic membrane in a specific embodiment of the present invention;

[0050] Figure 8 is a diagram of a perforated tympanic membrane in accordance with a specific embodiment of the present invention;

[0051] Figure 9 is a binary image of a perforated tympanic membrane in a specific embodiment of the present invention;

[0052] Figure 10 It is a binary image of the perforated portion of the perforated eardrum in a specific embodiment of the present invention. DETAILED DESCRIPTION

[0053] The present application will be further described in detail below with reference to the accompanying drawings and examples. It will be understood that the specific embodiments described herein are intended only to explain the relevant inventions and are not intended to limit the inventions. It should also be noted that, for ease of description, only portions relevant to the relevant inventions are shown in the accompanying drawings. It should be noted that, unless there is a conflict, the embodiments and features in the embodiments of the present application may be combined with one another. The present application will be described in detail below with reference to the accompanying drawings and in conjunction with the examples.

[0054] Figure 1 The method for identifying tympanic membrane perforation based on deep learning of the present invention comprises the following steps:

[0055] S1. Obtain tympanic membrane images and construct a tympanic membrane training dataset: Obtain several tympanic membrane images and perform multi-category labeling to construct a tympanic membrane labeled image database. After data preprocessing, divide the dataset into training and test datasets.

[0056] S11. Construct a tympanic membrane label image database: Obtain several tympanic membrane images based on an otoendoscopy, perform multi-category annotation to obtain corresponding label images, and construct a tympanic membrane label image database. The multi-category annotation includes the tympanic membrane area and perforation area.

[0057] S12. Data preprocessing: Based on the eardrum labeled image database, data augmentation technology is used to expand and normalize the data, unify the size and format, and obtain the eardrum training dataset. Data augmentation techniques include rotation, flipping, occlusion, mosaic, copy-paste, etc. to improve the diversity of the dataset.

[0058] With the help of population standard deviation And normalized by mean m:

[0059] (1);

[0060] in, and Represents the pixel points in the tympanic membrane image before and after normalization The pixel value at .

[0061] The unified size is 640×640×3.

[0062] S13. Divide the training data set and the test data set: Based on the tympanic membrane training data set, divide it according to the first ratio to obtain a training data set and a test data set.

[0063] S2. Build a deep learning-based tympanic membrane perforation recognition model: Figure 2 As shown in the figure, the deep learning-based tympanic membrane perforation recognition model includes a lightweight backbone network, a multi-scale channel and spatial attention module CPMS, a feature fusion network and a network output layer. The tympanic membrane image is sequentially processed by the lightweight backbone network, the multi-scale channel and spatial attention module CPMS, and the feature fusion network to obtain the perforation feature image, the spatial feature image, the multi-scale feature image and the tympanic membrane image recognition result; the lightweight backbone network is used to extract the perforation feature information of the tympanic membrane image, the multi-scale channel and spatial attention module is used to improve the model's ability to extract spatial feature information, the feature fusion network is used to process and integrate the multi-scale feature information extracted by the network, and the network output layer is used to output the tympanic membrane image recognition result.

[0064] S21. Each network module in the lightweight backbone network adopts channel-separable convolution and SE channel attention mechanism and is connected through residual connection. It introduces the Universal Inverted Bottleneck (UIB) search block, merges the Inverted Bottleneck (IB), ConvNext, Feedforward Network (FFN) and a new additional depth convolution variant.

[0065] like Figure 3As shown, the lightweight backbone network includes an initial convolution, a first-layer network, a second-layer network, a third-layer network, a fourth-layer network, a fifth-layer network and a spatial pyramid pooling module SPPF. The initial convolution, the first-layer network, the second-layer network, the third-layer network, the fourth-layer network, the fifth-layer network and the spatial pyramid pooling module SPPF are connected in series in sequence, and the input of the initial convolution is the input of the lightweight backbone network; the initial convolution includes the first convolution, the first-layer network includes the second and third convolutions, the second-layer network includes the fourth and fifth convolutions, the third-layer network includes the first universal inverted bottleneck module, the fourth-layer network includes the second universal inverted bottleneck module, and the fifth-layer network includes the sixth and seventh convolutions; the first universal inverted bottleneck module includes the first additional depth convolution, the first inverted bottleneck module and the feature module, the second universal inverted bottleneck module includes the second additional depth convolution, the third additional depth convolution and the second inverted bottleneck module; the first inverted bottleneck module includes the first depth convolution, the second inverted bottleneck module includes the second depth convolution, and the feature module includes the fourth additional depth convolution. The first additional depthwise convolution, the first inverted bottleneck module, and the feature module are connected in series. The second additional depthwise convolution, the third additional depthwise convolution, and the second inverted bottleneck module are connected in series. The outputs of the third, fourth, and spatial pyramid pooling modules (SPPFs) are the first, second, and third perforated features, respectively, which serve as the output of the lightweight backbone network.

[0066] The convolution kernels of the first, second, and fourth convolutions are all 3×3 and have a stride of 2. The convolution kernels of the third, fifth, sixth, and seventh convolutions are all 1×1 and have a stride of 1. The convolution kernel of the first additional depth convolution is 5×5 and has a stride of 2. The convolution kernel of the second additional depth convolution is 3×3 and has a stride of 2. The convolution kernel of the third additional depth convolution is 5×5 and has a stride of 1. The convolution kernel of the fourth additional depth convolution is 3×3 and has a stride of 1. The convolution kernels of the first and second depth convolutions are 5×5 and have a stride of 2. The convolution kernels are all 3×3 and the stride is 1; the number of output channels of the first convolution, second convolution, third convolution, fourth convolution, fifth convolution, sixth convolution and seventh convolution are 32, 32, 32, 64, 64, 960, 1280 respectively, the number of output channels of the first extra depth convolution, second extra depth convolution, third extra depth convolution and fourth extra depth convolution are 96, 128, 128, 96 respectively, and the number of output channels of the first depth convolution and second depth convolution are 96 and 128 respectively.

[0067] In a preferred embodiment, four first bottleneck inverting modules and four second bottleneck inverting modules are provided and are sequentially connected in series.

[0068] The spatial pyramid pooling module SPPF includes a first feature compression convolution CBS, a first maximum pooling unit, a second maximum pooling unit, a third maximum pooling unit and a second feature compression convolution CBS. The first maximum pooling unit, the second maximum pooling unit and the third maximum pooling unit are connected in series in sequence. The input of the first maximum pooling unit is the output of the first feature compression convolution CBS. The output of the first feature compression convolution CBS and the outputs of the first maximum pooling unit, the second maximum pooling unit and the third maximum pooling unit are input into the second feature compression convolution CBS through a channel splicing operation. The input of the first feature compression convolution CBS is the input of the spatial pyramid pooling module SPPF, and the output of the second feature compression convolution CBS is the output of the spatial pyramid pooling module SPPF, that is, the third perforated feature. The convolution kernels of the first maximum pooling unit, the second maximum pooling unit, and the third maximum pooling unit are all 5×5 and the step size is 1. The convolution kernels of the first feature compression convolution and the second feature compression convolution CBS are both 1×1 and the step size is 1. The number of output channels of the first feature compression convolution and the second feature compression convolution CBS are 512 and 1024 respectively.

[0069] S22, the multi-scale channel and spatial attention module encodes contextual information and effectively extracts spatial features through parallel multi-scale feature extraction channels and depth-wise separable convolutional spatial attention, while retaining channel prior knowledge. It has the ability to focus on channel attention and spatial attention and dynamically allocates weights in the channel and spatial dimensions.

[0070] like Figure 4 As shown in Figure 1, the multi-scale channel and spatial attention module CPMS includes a multi-scale channel attention module and a multi-scale depth separation convolutional spatial attention module. The overall structure includes the sequential placement of channel attention and spatial attention. The input of the multi-scale channel and spatial attention module CPMS is the third perforated feature, and the channel attention map is obtained by the multi-scale channel attention module. The third perforated feature and the channel attention map are element-wise multiplied. Get the channel prior, and the channel prior is passed through the multi-scale deep separation convolution spatial attention module to obtain the spatial attention map. The channel prior and the spatial attention map are multiplied element by element. The spatial features are obtained as the output of the multi-scale channel and spatial attention module CPMS.

[0071] The multi-scale channel attention module is equipped with average pooling units, maximum pooling units, and shared multi-layer perceptron units, and the multi-scale deep separation convolutional spatial attention module is equipped with a deep convolution unit. The number of output channels of the multi-scale channel and spatial attention module CPMS is 1024.

[0072] S23, such as Figure 5As shown, the feature fusion network includes a feature pyramid network FPN and a path aggregation network PAN. The feature pyramid network FPN includes the third feature compression convolution CBS, the eighth convolution and the fourth feature compression convolution CBS. The path aggregation network PAN includes the ninth convolution, the first downsampling, the tenth convolution, the second downsampling and the eleventh convolution. The input of the feature pyramid network FPN is the first perforated feature, the second perforated feature and the spatial feature. The spatial feature is compressed by the third feature convolution CBS to obtain the third semantic feature. The third semantic feature is upsampled and channel-joined with the second perforated feature as the input of the eighth convolution. The output of the eighth convolution is compressed by the fourth feature convolution CBS to obtain the second semantic feature. The second semantic feature is upsampled and channel-joined with the first perforated feature. The first semantic feature is obtained by splicing; the input of the path aggregation network PAN is the first semantic feature, the second semantic feature and the third semantic feature. The first semantic feature is subjected to the ninth convolution to obtain the first eardrum fine-grained feature. The first eardrum fine-grained feature is channel-spliced ​​with the second semantic feature after the first downsampling as the input of the tenth convolution. The output of the tenth convolution is channel-spliced ​​with the third semantic feature after the second downsampling as the input of the eleventh convolution. The outputs of the tenth and eleventh convolutions are the second and third eardrum fine-grained features, respectively; the first eardrum fine-grained feature, the second eardrum fine-grained feature and the third eardrum fine-grained feature are the outputs of the feature fusion network to fully extract the features of the eardrum such as edge refinement and texture refinement, ensure the fine granularity of the final output result, and improve the recognition accuracy of the model.

[0073] The output channels of the third feature compressed convolution and the fourth feature compressed convolution CBS are 512 and 256 respectively. The output channels of the eighth convolution, ninth convolution, tenth convolution and eleventh convolution are 512, 256, 512 and 1024 respectively. The output channels of the first and second downsampling are 256 and 512 respectively. The parameters of the first downsampling and the second downsampling are learnable.

[0074] The feature fusion network consists of a feature pyramid network (FPN) combined with a path aggregation network (PAN). The feature pyramid network (FPN) transfers top-level semantic information to bottom-level semantic information step by step through context paths. The path aggregation network (PAN) obtains feature maps of different scales by introducing lateral connections and bottom-up path information transmission, and realizes information fusion between feature maps of different resolutions through path aggregation. The downsampling (Adown) with learning ability is introduced to reduce the number of parameters and better preserve image features.

[0075] S24. The network output layer includes the twelfth convolution, the thirteenth convolution and the fourteenth convolution. The input of the network output layer is the first eardrum fine-grained feature, the second eardrum fine-grained feature and the third eardrum fine-grained feature. The first eardrum fine-grained feature is subjected to the twelfth convolution to obtain the first segmentation feature map, the second eardrum fine-grained feature is subjected to the thirteenth convolution to obtain the second segmentation feature map, and the third eardrum fine-grained feature is subjected to the fourteenth convolution to obtain the third segmentation feature map; the number of output channels of the twelfth convolution, the thirteenth convolution and the fourteenth convolution are all 255.

[0076] The sizes of the first, second, and third segmentation feature maps are 255×80×80, 255×40×40, and 255×20×20, respectively. The first segmentation feature map is responsible for detecting small objects, the second segmentation feature map is responsible for detecting medium objects, and the third segmentation feature map is responsible for detecting large objects. Gridded predictions are performed on the first, second, and third segmentation feature maps in sequence, resulting in a feature map partitioning network and mask after detector convolution, which are then used to obtain the predicted boxes and masks. Based on the confidence scores and overlap of the predicted boxes and masks, the predicted boxes and masks are screened using the non-maximum suppression (NMS) algorithm. The prediction boxes and masks with the highest confidence are retained, resulting in the eardrum image recognition results. The eardrum image recognition results include the number of categories, the confidence score (nc) corresponding to each category, the predicted box height, the predicted box width, the coordinates of the predicted box center point, and the mask, for a total of six types of information.

[0077] S3. Train the deep learning-based tympanic membrane perforation recognition model to obtain the target tympanic membrane perforation recognition model: set the loss function, set the model parameters and evaluation indicators, train the deep learning-based tympanic membrane perforation recognition model based on training data sets and test data sets, obtain the network weight parameters of the deep learning-based tympanic membrane perforation recognition model corresponding to the best evaluation indicator of the test data set, and obtain the target tympanic membrane perforation recognition model.

[0078] S31. Set the loss function CIoU;

[0079] (2);

[0080] in Represents the Euclidean distance between the center points of the predicted box and the true box; Represents the diagonal distance of the minimum closure area that can contain both the predicted box and the true box; represents the intersection and union ratio; Represents a balance factor, which is used to weigh the loss caused by aspect ratio difference and area difference. v represents the normalized value of the aspect ratio difference between the predicted box and the true box, and has:

[0081] (3);

[0082] (4);

[0083] in, Indicates the width of the real box; Indicates the height of the real frame; Indicates the width of the prediction box; Indicates the height of the prediction box; Represents the tangent value.

[0084] S32. Set model parameters and evaluation indicators. The evaluation indicators include precision, recall, accuracy, and F1-Score, which are expressed as:

[0085] (5);

[0086] (6);

[0087] (7);

[0088] (8);

[0089] TP stands for true positive; FP stands for false positive; TN stands for true negative; and FN stands for false negative.

[0090] S33. Training based on the tympanic membrane training dataset: Based on the training dataset and the test dataset, the tympanic membrane images and corresponding label images in the training dataset are input into the tympanic membrane perforation recognition model based on deep learning for training.

[0091] S331. Use the tympanic membrane image and corresponding label in the training data set as input to the tympanic membrane perforation recognition model based on deep learning, and extract the first perforation feature, the second perforation feature and the third perforation feature of the tympanic membrane image through the lightweight backbone network.

[0092] S332. The third perforated feature is subjected to a multi-scale channel and spatial attention module CPMS to extract channel attention and spatial attention, encode context information, and obtain spatial features.

[0093] S333. The first perforation feature, the second perforation feature, and the spatial feature are input into a feature fusion network to obtain a first tympanic membrane fine-grained feature, a second tympanic membrane fine-grained feature, and a third tympanic membrane fine-grained feature.

[0094] S334. The fine-grained features of the first tympanic membrane, the second tympanic membrane, and the third tympanic membrane are input into the network output layer to obtain the first segmentation feature map, the second segmentation feature map, and the third segmentation feature map, obtain the tympanic membrane image recognition result, and obtain the network weight parameters of the tympanic membrane perforation recognition model based on deep learning.

[0095] S335. Repeat S331 to S334 until all training generations are completed. When the evaluation index of the test data set in all training generations is optimal, the weight parameters of the deep learning-based tympanic membrane perforation recognition model are obtained to obtain the target tympanic membrane perforation recognition model.

[0096] S4. Based on the target tympanic membrane perforation recognition model, obtain the recognition result of the tympanic membrane image: Based on the target tympanic membrane perforation recognition model, the tympanic membrane image is normalized and used as the input of the target tympanic membrane perforation recognition model to obtain the output of the target tympanic membrane perforation recognition model, that is, the recognition result of the tympanic membrane image.

[0097] Based on the segmentation mask of the recognition result of the eardrum image, the segmentation mask of the perforation is binarized, the mask pixels are traversed, the number of pixels of different categories of segmentation parts is counted, and the statistical results are updated to the output results to obtain the size of the perforation area.

[0098] like Figure 6-Figure 8 As shown in Table 1, examples of tympanic membrane image recognition results in different conditions are shown. For a normal tympanic membrane, the tympanic membrane outline and the recognition confidence level will be displayed; for a healing tympanic membrane, the tympanic membrane outline, the healed perforated part, and the recognition confidence level will be displayed; for a perforated tympanic membrane, the tympanic membrane outline, the perforated part, the area of ​​the tympanic membrane outline, the area of ​​the perforated part, and the recognition confidence level will be displayed. The tympanic membrane image recognition confidence level can reach up to 0.93, with good credibility, and the overall recognition accuracy rate reaches 95.7%, which has a high recognition accuracy rate. In a preferred embodiment, Figure 8 The perforated tympanic membrane diagram is a schematic diagram of the results obtained through network prediction and recognition, in which the box in the center represents the circumscribed rectangle of the perforated part, and the adjacent outer boxes represent the circumscribed rectangle of the tympanic membrane outline.

[0099] Figure 9 An example of a binary image showing the segmentation results of the perforated and non-perforated parts of a perforated tympanic membrane image is shown. The black area in the center is the binary representation of the perforated part, and the white area is the binary representation of the tympanic membrane outline, which shows the tympanic membrane part more intuitively.

[0100] Figure 10 The white area in the figure shows an example of a binary image of the perforated portion of the tympanic membrane perforation image.

[0101] In a specific embodiment, as shown in Table 1, the area of ​​the perforated part of the perforated eardrum is represented by 1425.5 pixels, and the area of ​​the eardrum image is represented by 9108 pixels. The proportion of the perforated part is calculated based on the ratio of the number of pixels, which represents the ratio of the image area of ​​the perforated part to the image area of ​​the eardrum, thereby providing image reference and assistance for identifying the condition of the eardrum.

[0102] Table 1

[0103]

[0104] The present invention proposes a method for identifying tympanic membrane perforations based on deep learning. In order to address the problems of small perforations being difficult to identify and the diverse perforation morphologies, multi-scale channels and multi-scale depth-separable convolutional spatial attention are introduced to dynamically allocate weights in the channel and spatial dimensions to capture the spatial features of perforation position and size and the multi-scale features of perforation categories, effectively supplementing the feature extraction capability of the lightweight backbone network, and being able to better capture the details and structures in the tympanic membrane image, thereby improving the image segmentation accuracy; the feature pyramid network transfers the top-level semantic information to the bottom-level semantic information step by step through the context path, and the path aggregation network obtains features of different scales by introducing lateral connections and bottom-up path information transmission. Figure, through path aggregation to achieve information fusion between feature maps of different resolutions; at the same time, a feature fusion network with a learnable downsampling module is adopted to reduce feature extraction loss through a learnable mechanism, achieve better feature integration, reduce the number of parameters, and improve image segmentation accuracy; by constructing a multi-category labeled tympanic membrane image benchmark library and a tympanic membrane perforation recognition model based on deep learning, automatic and high-precision recognition of tympanic membrane perforation is achieved, and objective medical image analysis is used to improve accuracy and efficiency, while providing a reliable basis for legal identification, which has significant clinical application value; at the same time, based on the tympanic membrane perforation segmentation results, the pixel area occupied by the perforated part can be calculated, providing an image reference for the identification of the tympanic membrane condition.

[0105] Finally, it should be noted that the above embodiments are only intended to illustrate rather than limit the technical solutions of the present invention. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that the present invention can still be modified or replaced by equivalents. Any modification or partial replacement that does not depart from the spirit and scope of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A method for identifying tympanic membrane perforation based on deep learning, characterized by: It includes the following steps: S1, obtain the tympanic membrane image and construct the tympanic membrane training dataset; S2. Build a deep learning-based tympanic membrane perforation recognition model: The tympanic membrane perforation recognition model includes a lightweight backbone network, a multi-scale channel and spatial attention module, a feature fusion network, and a network output layer; S21, constructing a lightweight backbone network for extracting perforation feature information of the tympanic membrane image, outputting a first perforation feature, a second perforation feature, and a third perforation feature, wherein the lightweight backbone network includes an initial convolution, a first layer network, a second layer network, a third layer network, a fourth layer network, a fifth layer network, and a spatial pyramid pooling module connected in series in sequence; S22. Construct a multi-scale channel and spatial attention module to output spatial features. The multi-scale channel and spatial attention module includes a multi-scale channel attention module and a multi-scale depth separation convolution spatial attention module. S23, constructing a feature fusion network to output the first eardrum fine-grained feature, the second eardrum fine-grained feature, and the third eardrum fine-grained feature. The feature fusion network is used to integrate the multi-scale feature information extracted by the network. The feature fusion network includes a feature pyramid network and a path aggregation network. S24. Construct a network output layer for outputting the tympanic membrane image recognition result, where the tympanic membrane image recognition result includes the number of categories, the confidence level for each category, the predicted box height, the predicted box width, the predicted box center coordinates, and the mask; S3. Train a deep learning-based tympanic membrane perforation recognition model to obtain a target tympanic membrane perforation recognition model; S4. Based on the target tympanic membrane perforation recognition model, obtain a recognition result of the tympanic membrane image.

2. The deep learning-based tympanic membrane perforation identification method according to claim 1, characterized in that: S1 specifically includes the following sub-steps: S11. Constructing a tympanic membrane labeled image database: Acquire several tympanic membrane images using an otoendoscopy, perform multi-category annotation to obtain corresponding label images, and construct a tympanic membrane labeled image database. The multi-category annotation includes the tympanic membrane area and perforation area. S12. Data preprocessing: Based on the eardrum labeled image database, data enhancement technology is used to expand and normalize the data, unify the size and format, and obtain the eardrum training dataset; S13. Divide the training data set and the test data set: Based on the tympanic membrane training data set, divide it according to the first ratio to obtain a training data set and a test data set.

3. The deep learning-based tympanic membrane perforation identification method according to claim 1, characterized in that: In S21, the initial convolution includes the first convolution, the first layer network includes the second and third convolutions, the second layer network includes the fourth and fifth convolutions, the third layer network includes the first universal inverted bottleneck module, the fourth layer network includes the second universal inverted bottleneck module, and the fifth layer network includes the sixth and seventh convolutions; the first universal inverted bottleneck module includes the first additional depth convolution, the first inverted bottleneck module and the feature module connected in series in sequence, and the second universal inverted bottleneck module includes the second additional depth convolution, the third additional depth convolution and the second inverted bottleneck module connected in series in sequence; the first inverted bottleneck module includes the first depth convolution, the second inverted bottleneck module includes the second depth convolution, and the feature module includes the fourth additional depth convolution; the input of the initial convolution is the input of the lightweight backbone network, and the outputs of the third layer network, the fourth layer network and the spatial pyramid pooling module are the first perforated feature, the second perforated feature and the third perforated feature, respectively.

4. The deep learning-based tympanic membrane perforation identification method according to claim 1, characterized in that: In S22, the third perforated feature is input into the multi-scale channel attention module to obtain the channel attention map, and the channel attention map and the third perforated feature are element-wise multiplied to obtain the channel prior; the channel prior is input into the multi-scale depth separation convolution spatial attention module to obtain the spatial attention map, and the spatial attention map and the channel prior are element-wise multiplied to obtain the spatial feature; in S24, the input of the network output layer is the first eardrum fine-grained feature, the second eardrum fine-grained feature and the third eardrum fine-grained feature, and the first segmentation feature map, the second segmentation feature map and the third segmentation feature map are obtained in turn. The first segmentation feature map, the second segmentation feature map and the third segmentation feature map are gridded and predicted in turn, and the prediction boxes and masks with the highest confidence are screened and retained to obtain the eardrum image recognition result.

5. The deep learning-based tympanic membrane perforation identification method according to claim 1, characterized in that: The feature pyramid network in S23 includes the third feature compression convolution, the eighth convolution and the fourth feature compression convolution, and the path aggregation network includes the ninth convolution, the first downsampling, the tenth convolution, the second downsampling and the eleventh convolution; the input of the feature pyramid network is the first perforation feature, the second perforation feature and the spatial feature, and the output is the first semantic feature, the second semantic feature and the third semantic feature. The first semantic feature, the second semantic feature and the third semantic feature serve as the input of the path aggregation network, and the output of the path aggregation network is the first tympanic membrane fine-grained feature, the second tympanic membrane fine-grained feature and the third tympanic membrane fine-grained feature.

6. The deep learning-based tympanic membrane perforation identification method according to claim 2, characterized in that: S3 specifically includes the following sub-steps: S31. Set the loss function CIoU; S32. Set model parameters and evaluation indicators, including precision, recall, accuracy, and F1-Score. S33. Training based on the tympanic membrane training dataset: Based on the training dataset and the test dataset, the tympanic membrane images and corresponding label images in the training dataset are input into the tympanic membrane perforation recognition model based on deep learning for training.

7. The deep learning-based tympanic membrane perforation identification method according to claim 1, characterized in that: The spatial pyramid pooling module in S21 includes a first feature compression convolution, a second feature compression convolution, and a first maximum pooling unit, a second maximum pooling unit, and a third maximum pooling unit connected in series in sequence. The input of the first maximum pooling unit is the output of the first feature compression convolution. The output of the first feature compression convolution and the outputs of the first maximum pooling unit, the second maximum pooling unit, and the third maximum pooling unit are input into the second feature compression convolution through a channel splicing operation. The input of the first feature compression convolution is the input of the spatial pyramid pooling module, and the output of the second feature compression convolution is the output of the spatial pyramid pooling module.

8. The deep learning-based tympanic membrane perforation identification method according to claim 6, characterized in that: CIoU in S31 is expressed as: ； in, Represents the Euclidean distance between the center points of the predicted box and the true box; Represents the diagonal distance of the minimum closure area that can contain both the predicted box and the true box; represents the intersection and union ratio; Represents a balance factor, which is used to weigh the loss caused by aspect ratio difference and area difference, and v represents the normalized value of the aspect ratio difference between the predicted box and the true box.

9. The deep learning-based tympanic membrane perforation identification method according to claim 8, characterized in that: S31 and v are expressed as: ； ； in, Indicates the width of the real box; Indicates the height of the real frame; Indicates the width of the prediction box; Indicates the height of the prediction box; Represents the tangent value.

10. The deep learning-based tympanic membrane perforation identification method according to claim 6, characterized in that: S33 specifically includes the following sub-steps: The tympanic membrane images and corresponding labels in the training dataset are input into the tympanic membrane perforation recognition model based on deep learning, the tympanic membrane image recognition results are output, and the network weight parameters of the tympanic membrane perforation recognition model based on deep learning are obtained; the execution is repeated until all training generations are completed, and when the evaluation index of the test dataset in all training generations is optimal, the weight parameters of the tympanic membrane perforation recognition model based on deep learning are obtained, and the target tympanic membrane perforation recognition model is obtained.

Images