Ear endoscope image classification method based on double-branch feature extraction and gating fusion

Abstract

This invention discloses a method for classifying otoscopic images based on bi-branch feature extraction and gated fusion. First, a feature extraction backbone network generates a deep feature map from the original otoscopic image. Then, a global branch and a local branch extract global and local feature vectors from the deep feature map, respectively. Next, a gated fusion mechanism is used to perform controlled fusion of the two types of features to obtain a fused feature vector. Finally, disease category classification is performed based on the fused feature vector. Compared to traditional methods that rely solely on global pooling features, this method can simultaneously consider global information and key local details, improving the accuracy, robustness, and interpretability of fine-grained otoscopic disease identification. It can be applied to auxiliary screening, intelligent diagnosis, and automated analysis in primary healthcare settings for ear diseases.

Description

Technical Field

[0001] This invention relates to the field of medical image processing and intelligent assisted diagnosis technology, specifically to a method for classifying endoscopic ear images based on bi-branch feature extraction and gating fusion. Background Technology

[0002] Otoscopic examination is one of the important methods for diagnosing ear diseases. After obtaining images of the tympanic membrane, external auditory canal, and lesion areas using an otoscope, doctors usually make a judgment on the type of ear disease based on visual features such as color, texture, shape, boundary, and lesion distribution in the images. Because otoscope images can directly reflect pathological changes in the ear canal and tympanic membrane area, they have high application value in clinical examination, follow-up visits, and primary care screening.

[0003] However, traditional endoscopic otoscopic image diagnosis relies heavily on physician experience, especially in non-specialist physicians, primary healthcare institutions, or large-scale initial screening scenarios, where it is easily affected by differences in experience, observation angles, and subjective judgment biases. With the development of artificial intelligence technology, deep learning algorithms have been gradually applied to the field of endoscopic otoscopic image processing, enabling computers to automatically learn and identify discriminative features in images, thereby improving diagnostic efficiency and consistency.

[0004] While existing methods for classifying endoscopic ear images have achieved high accuracy in some categories, they still face significant limitations in fine-grained classification scenarios. Ear diseases are diverse, and different categories often appear similar. For example, secretory otitis media and a normal tympanic membrane can look very similar at certain stages, and tympanic sclerosis and chronic otitis media can also be visually confused. If the model relies primarily on a single global feature for classification, it struggles to capture subtle differences within the lesion area, resulting in insufficient fine-grained discrimination capabilities.

[0005] To improve fine-grained classification capabilities, existing methods have attempted to incorporate local region features. However, otoscopy images are often accompanied by complex factors such as reflections, occlusions, blurring, incomplete fields of view, and background interference. Local regions may simultaneously contain crucial lesion information and irrelevant noise. If local and global features are fused using simple concatenation, direct superposition, or equal-weighted averaging, local noise can easily excessively interfere with the overall discrimination process, affecting the model's stability and generalization ability.

[0006] Furthermore, existing methods still lack an effective fusion mechanism that can balance overall semantic stability and local discriminative sensitivity in the fusion of global and local features. In other words, how to maintain the dominant role of global features while allowing local features to participate in the discrimination process only as controlled supplementary terms, so as to avoid local noise overshadowing global semantics, while improving the ability to distinguish easily confused categories, remains a pressing technical problem to be solved in the field of endoscopic ear image classification.

[0007] There are existing studies on intelligent classification of endoscopic images: [1]: Livingstone D, Talai AS, Chau J, et al. Building an Otoscopic screening prototype tool using deep learning[J]. Journal of Otolaryngology -Head & Neck Surgery, 2019, 48(1): 66.DOI: 10.1186 / s40463-019-0389-9. [2]: Zeng X, Jiang Z, Luo W, et al. Efficient and accurate identification of ear diseases using an ensemble deep learning model[J]. Scientific Reports, 2021, 11(1): 10839. DOI: 10.1038 / s41598-021-90345-w. The above-mentioned published literature provides a feasible path for the classification of endoscopic diseases, but there is still room for improvement in fine-grained category differentiation and global-local feature balance. Summary of the Invention

[0008] To address the aforementioned problems, this invention provides a method for classifying endoscopic ear images based on bi-branch feature extraction and gated fusion, comprising the following steps: S1. Acquire an endoscopic image of the ear and perform feature processing on the endoscopic image to generate a deep feature map; S2. Construct a dual-branch feature extraction structure to extract global and local feature vectors based on deep feature maps; S3. Generate a fused feature vector by combining the global and local feature vectors through a gating fusion mechanism; S4. Based on the fused feature vector, identify the disease category and output the category result corresponding to the otoscopy image; Specifically, the gating fusion mechanism is as follows: a scalar gating value is output based on the joint representation of the global feature vector and the local feature vector, and the scalar gating value is used to control the magnitude of the fusion of the local feature vector.

[0009] Furthermore, in step S1, the feature processing of the endoscopic image is specifically performed using a feature extraction backbone network. By extracting high-level semantic information and spatial discrimination information from the endoscopic image, a deep feature map is generated.

[0010] Furthermore, the deep feature map F is specifically represented as: B╳C╳H╳W; where B represents the number of images simultaneously fed into the network in one forward propagation; C represents the number of channels; H and W are both spatial dimensions, representing spatial height and spatial width, respectively.

[0011] Furthermore, the dual-branch feature extraction structure in step S2 specifically includes: a global branch and a local branch; wherein, the global branch specifically involves using adaptive average pooling to aggregate global information from the depth feature map, converging information in the spatial dimension into a one-dimensional vector representation, and obtaining a global feature vector that reflects the overall semantic information of the entire endoscopic ear image. The local branch specifically involves selecting key local regions from the depth feature map and aggregating the feature information at that location to obtain a local feature vector that reflects the differences in fine-grained lesions.

[0012] Furthermore, the local branch specifically includes: a local location selection module and a local feature aggregation module; Specifically, the local location selection module adaptively selects multiple key local regions with strong discriminative power based on the response of each spatial location in the deep feature map. The local feature aggregation module specifically performs mean aggregation on the feature vectors corresponding to local key regions, and then performs nonlinear projection mapping to obtain local feature vectors.

[0013] Furthermore, the local location selection module specifically maps the deep feature map through 1×1 convolutional scoring to obtain a single-channel spatial scoring map. The single-channel spatial scoring map is then flattened along the spatial dimension into a scoring vector of B×(H×W). Top-K selection is performed using the scoring vector to obtain the k indices with the highest scores. The deep feature map is then rearranged according to the spatial dimension into a feature sequence of B×(H×W)×C. Feature sampling is performed based on the k indices to obtain a local feature set of B×k×C.

[0014] Furthermore, the local feature vector also contains an auxiliary classification head, which outputs local classification results and calculates the corresponding auxiliary classification loss during the training phase; wherein, the auxiliary classification loss is incorporated into the total loss function according to a preset weight coefficient.

[0015] Furthermore, the total loss function L local The calculation formula is: ; In the formula, L main Primary classification loss; L aux λ is the auxiliary classification loss; λ is the auxiliary loss weight coefficient, set to 0.073937.

[0016] Furthermore, the fused feature vector is represented as: ; In the formula, f fused f represents the fused feature vector; global f represents the global feature vector; local Represents a local feature vector; Proj() represents the projection function; α represents the scalar gate value; The formula for calculating the scalar gate value α is as follows: ; In the formula, σ represents the Sigmoid function; MLP() represents the multilayer sensing mapping; [f global ;f local ] represents the concatenation of global and local feature vectors; b is the learnable bias term, initially set to 0.089795; α max This indicates the upper limit of the gate value, set to 0.15477.

[0017] Furthermore, the disease categories in step S4 specifically include: normal, secretory otitis media, acute otitis media, chronic otitis media, poor ventilation, bullous myringitis, tympanic membrane perforation, tympanostomy tube insertion, tympanosclerosis, ear canal cerumen, adhesive otitis media, cholesteatoma, fungal external otitis, tympanic cavity hemorrhage, and ear canal foreign body.

[0018] This invention provides a classification method for endoscopic ear images based on bi-branch feature extraction and gated fusion, which has the following beneficial effects: Compared to traditional methods that rely solely on global pooling features, this invention can simultaneously consider both global information and key local details, improving the accuracy, robustness, and interpretability of fine-grained otoscope disease identification. It can be applied to auxiliary screening, intelligent diagnosis, and automated analysis in primary healthcare settings for ear diseases. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram of the method flow provided by the present invention; Figure 2 A schematic diagram of the gating fusion mechanism structure provided by the present invention; Figure 3 This is a schematic diagram of the local branch calculation process provided by the present invention. Detailed Implementation

[0021] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0022] The following detailed description of the implementation method of the present invention is in conjunction with the accompanying drawings. The description is only a partial embodiment and not all embodiments. For clarity, representations and descriptions unrelated to the present invention are omitted in the drawings and description.

[0023] To provide a clearer understanding of the technical features, objectives, and beneficial effects of this invention, the following detailed description of the technical solution is provided. Obviously, the described embodiments are only a portion of the embodiments of this invention, not all of them, and should not be construed as limiting the scope of implementation of this invention. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without inventive effort are within the protection scope of this invention.

[0024] This invention addresses the technical problems in existing technologies, such as the inability of a single global feature to fully represent the fine-grained differences in endoscopic images, the susceptibility of direct fusion of local features to noise interference, and the difficulty in effectively balancing global and local features. It provides an endoscopic image classification method based on bi-branch feature extraction and gated fusion, mainly comprising six parts: data preparation, construction of a feature extraction backbone network, bi-branch feature extraction, gated fusion, classification output, and training optimization. The core of the method lies in: first, the feature extraction backbone network generates a deep feature map from the original endoscopic image; then, the global branch and local branch extract global and local feature vectors from the deep feature map, respectively; next, the two types of features are fused in a controlled manner through a gated fusion mechanism to obtain a fused feature vector; finally, disease category classification is performed based on the fused feature vector.

[0025] like Figure 1 As shown, this invention provides a method for classifying endoscopic ear images based on bi-branch feature extraction and gated fusion, comprising the following steps: S1. Acquire endoscopic images of the ear and perform feature processing on these images to generate a deep feature map. Specifically, feature extraction is performed using a feature extraction backbone network. This network extracts high-level semantic information and spatial discrimination information from the endoscopic images to generate a deep feature map F: B╳C╳H╳W; where B represents the number of images fed into the network in one forward propagation; C represents the number of channels; and H and W are spatial dimensions, representing spatial height and spatial width, respectively. EfficientNetV2-S can be used as the feature extraction backbone network to obtain a deep feature map that balances representational power and computational efficiency. This deep feature map retains both the high-level semantic expression of the image and sufficient spatial structural information to support the selection of key local locations, thus providing a foundation for subsequent collaborative extraction of global and local branches.

[0026] S2. Construct a dual-branch feature extraction structure to extract global and local feature vectors based on deep feature maps.

[0027] The dual-branch feature extraction structure specifically includes a global branch and a local branch. The global branch involves adaptive average pooling to aggregate global information from the deep feature map, transforming spatial information into a one-dimensional vector representation to obtain a global feature vector reflecting the overall semantic information of the entire endoscopic image. This global feature vector primarily carries the overall semantic context information of the image and serves as the dominant feature in the final classification process.

[0028] The local branch specifically selects key local regions from the depth feature map and aggregates the feature information at that location to obtain a local feature vector that reflects the differences in fine-grained lesions: local location selection module and local feature aggregation module.

[0029] The local location selection module adaptively selects multiple key local regions with strong discriminative power based on the response of each spatial location in the deep feature map. Specifically, it maps the deep feature map using a 1×1 convolutional scoring method to obtain a single-channel spatial scoring map. This single-channel spatial scoring map is then flattened along the spatial dimension into a B×(H×W) scoring vector. Top-K selection is performed using this scoring vector to obtain the k highest-scoring indices (k=min(K, H×W)). The deep feature map is then rearranged along the spatial dimension into a feature sequence of B×(H×W)×C, and feature sampling is performed based on the k indices to obtain a local feature set of B×k×C. The sampling process can be implemented using the gather operator to ensure a one-to-one correspondence between the sampled locations and the original spatial locations.

[0030] The B×k×C local feature set is aggregated along the k-dimensional mean to obtain an initial B×C local feature vector. This vector is then input into a projection module consisting of two layers of linear mapping, nonlinear activation, and random deactivation to output an enhanced local feature vector. A local branch auxiliary classification head is placed after the projection module. Based on the enhanced local feature vector, it outputs a local classification result and calculates the auxiliary classification loss during the training phase. The auxiliary classification loss is incorporated into the total loss function with preset weights to improve the ability of local branches to discriminate fine-grained lesion regions. The number of local regions K is a preset positive integer and can be set according to task complexity, image resolution, and computing resources. By adjusting the value of K, a balance can be struck between the ability to focus on key regions and the coverage of local information.

[0031] The local feature aggregation module specifically performs mean aggregation on the feature vectors corresponding to key local regions, followed by nonlinear projection mapping to obtain local feature vectors. This projection mapping process can employ a combination of linear transformation and nonlinear activation, and can be configured with intermediate dimension expansion and random deactivation mechanisms to achieve mapping from original local features to enhanced local features. The intermediate dimension is a preset positive integer, and the random deactivation ratio is a preset value within the interval [0,1). After linear transformation, nonlinear activation, and random deactivation, the local feature vectors are mapped back to the same dimension as the global feature vectors.

[0032] Through the aforementioned dual-branch design, this invention can simultaneously consider both the overall structural information and local discriminative information in endoscopic ear images. An auxiliary classification head is also present in the local feature vectors. This auxiliary classification head outputs local classification results and calculates the corresponding auxiliary classification loss during the training phase; wherein, the auxiliary classification loss is incorporated into the total loss function according to preset weight coefficients.

[0033] Total loss function L local The calculation formula is: ; In the formula, L main Primary classification loss; L aux The auxiliary classification loss is λ, which is the auxiliary loss weight coefficient, set to 0.073937. The main classification loss is calculated based on the output of the main classification head, and the auxiliary classification loss is calculated based on the output of the auxiliary classification head in the local branches. Both the main classification loss and the auxiliary classification loss can use class-weighted cross-entropy (CE) loss. ; In the formula, N represents the batch size; C=15 represents the number of disease categories; p i,c ω represents the probability that the model predicts the i-th sample as class c. C The class weights are calculated based on the inverse of the frequency of each class sample in the training set, and then cropped to [0.5, 10.0]. Label smoothing is also introduced into the loss calculation, with the label smoothing coefficient set to 0.004027.

[0034] The calculation formula is: In the formula, y i,c Let represent the label value of the i-th sample in class c. When label smoothing is used, a smoothing coefficient ε can be introduced, and ε is a preset parameter in the interval [0,1). The smoothed label can be obtained by linear weighting of the original label, thereby reducing model overconfidence and improving generalization ability.

[0035] S3. Generate a fused feature vector by combining the global feature vector and the local feature vector through a gating fusion mechanism; wherein, the gating fusion mechanism specifically involves: outputting a scalar gating value based on the joint representation of the global feature vector and the local feature vector, and using the scalar gating value to control the magnitude of the fusion of the local feature vector.

[0036] The fused feature vector is represented as follows: ; In the formula, f fused f represents the fused feature vector; global f represents the global feature vector; local Represents a local feature vector; Proj() represents the projection function; α represents the scalar gate value; The formula for calculating the scalar gate value α is as follows: ; In the formula, σ represents the Sigmoid function; MLP() represents the multilayer sensing mapping; [f global ;f local ] represents the concatenation of global and local feature vectors; b is the learnable bias term, initially set to 0.089795; α max This indicates the upper limit of the gate value, set to 0.15477.

[0037] S4. Based on the fused feature vector, identify disease categories (normal, secretory otitis media, acute otitis media, chronic otitis media, poor ventilation, bullous myringitis, tympanic membrane perforation, tympanostomy tube insertion, tympanosclerosis, earwax, adhesive otitis media, cholesteatoma, fungal external otitis, hemotympanic cavity, foreign body in the ear canal), and output the corresponding category results for the endoscopic image. Specifically, the classification head receives the fused feature vector and outputs the predicted score or predicted probability corresponding to each disease category through a fully connected mapping to complete the classification of the endoscopic image.

[0038] In one example training process, the optimizer used is AdamW with an initial learning rate of 9.7247 × 10⁻⁶. -5 The weight decay is 1.3022 × 10 -4 The learning rate scheduling uses a combination of 5-epoch linear warmup and cosine annealing; the training cycle can be set to 100 epochs, the batch size can be set to 32, the gradient accumulation step can be set to 1, and the exponential moving average decay coefficient can be set to 0.999.

[0039] To improve the executability of the training process under memory-constrained conditions, the training program can automatically reduce the batch size and increase the gradient accumulation steps accordingly when memory is insufficient, so as to ensure that the training process can continue to execute.

[0040] Example: The otoscope image classification dataset used in this example contains 4074 otoscope images across 15 categories. Specific categories include: normal, secretory otitis media, acute otitis media, chronic otitis media, poor ventilation, bullous myringitis, tympanic membrane perforation, tympanostomy tube insertion, tympanic sclerosis, earwax, adhesive otitis media, cholesteatoma, fungal external otitis, hemotympanic cavity, and foreign body in the ear canal. The dataset is divided into training, validation, and test sets in approximately an 8:1:1 ratio.

[0041] The training set contains 3252 samples, the validation set contains 407 samples, and the test set contains 415 samples, totaling 4074 samples. The distribution of samples across categories exhibits a clear long-tail characteristic; for example, the number of samples in the "normal" category is much higher than that of low-frequency categories such as "ear canal foreign body." Therefore, class-weighted cross-entropy loss can be used during training to alleviate the training bias caused by the imbalanced class distribution. The class weights are calculated by the inverse of the frequency of each category in the training set and cropped to the interval [0.5, 10.0].

[0042] Before inputting data into the network, the raw otoscope images can be preprocessed and augmented. In one example, this may include image scaling, random cropping, center cropping, random flipping, color perturbation, normalization, and other operations to enhance the model's adaptability to different shooting conditions, different field of view offsets, and local brightness variations.

[0043] The dual-branch feature extraction structure consists of a global branch and a local branch. The global branch extracts global feature vectors, while the local branch extracts local feature vectors. The core idea of ​​this dual-branch structure is to extract overall semantic information and key local discriminative information separately from the same deep feature map, thus balancing stability and sensitivity in classification tasks. The global branch can be implemented using the adaptive average pooling method described in the technical solution section to compress the deep feature map into a global feature vector.

[0044] like Figure 3 As shown, the Top-K position selection and feature sampling process of local branches can be executed according to the local position selection module described in the technical solution section. That is, spatial scoring is performed first, then key positions are selected and corresponding feature sampling is completed; then mean aggregation and projection mapping are performed on the sampled local features to obtain enhanced local feature vectors.

[0045] For the feature vectors corresponding to the K spatial locations, mean aggregation is first performed to obtain initial local feature vectors, and then mapping is performed through a nonlinear projection module to obtain enhanced local feature vectors. This design improves the stability of local representation through aggregation and enhances the discriminative power of local features through projection. In one example, selected local features can be aggregated and then mapped to form the final local feature vector. The number of local regions K is set to 3, the intermediate dimension in the local feature projection process is 768, and a random inactivation ratio of 0.291569 is used for local branches to improve the robustness of local feature representation.

[0046] like Figure 2 As shown, this embodiment employs a gated fusion mechanism to fuse global and local feature vectors. First, based on the joint semantics of the global and local feature vectors, a scalar gate value α is calculated using a gating network. Then, α is used to control the magnitude of the local feature vectors after the projection function transformation. Finally, the controlled correction term of the local features is superimposed onto the global feature vector to generate the fused feature vector. The gated fusion mechanism can employ a weakly gated fusion mechanism, that is, by constraining the upper limit of the scalar gate value, the local features will not exceed a reasonable range of influence during the fusion process. The upper limit of the gate value α is... max The learnable bias term can be initialized to an initial gate value of approximately 0.089795, which can be set to 0.15477. This allows for an optimization process where the global bias dominates during the initial training phase, while local information is gradually introduced during the later stages of training. To better adapt the local feature vectors to the feature space where the global feature vectors reside, a local feature projection unit can be included in the gated fusion mechanism. This local feature projection unit performs linear mapping, nonlinear activation, and random deactivation operations on the local feature vectors before feeding the mapping results into the fusion calculation.

[0047] The classification head maps the fused feature vectors to a predefined disease category space and outputs a predicted score or probability for each disease category to complete the classification of endoscopic ear images. In one example, the classification head is constructed using fully connected layers. In another example, both the main classification head and the local branch auxiliary classification heads can use fully connected classification structures. Figure 3 As shown, to enhance the ability of local branches to identify key lesion areas, this embodiment sets an auxiliary classification head after the local branch. During the training phase, the enhanced local feature vector is input into the auxiliary classification head, which outputs the local classification result and calculates the auxiliary classification loss L. aux On the other hand, the input control fusion module and global features jointly participate in subsequent fusion; the main classification head outputs the main classification result based on the fused feature vector and calculates the main classification loss, and the two jointly participate in training optimization.

[0048] Total loss function L local The calculation formula is: In the formula, L mainPrimary classification loss; L aux The auxiliary classification loss is λ, which is the auxiliary loss weight coefficient, set to 0.073937. The main classification loss is calculated based on the output of the main classification head, and the auxiliary classification loss is calculated based on the output of the auxiliary classification head in the local branches. Both the main classification loss and the auxiliary classification loss can use class-weighted cross-entropy (CE) loss. In the formula, N represents the batch size; C=15 represents the number of disease categories; p i,c ω represents the probability that the model predicts the i-th sample as class c. C The class weights are calculated based on the inverse of the frequency of each class sample in the training set, and then cropped to [0.5, 10.0]. Label smoothing is also introduced into the loss calculation, with the label smoothing coefficient set to 0.004027.

[0049] The calculation formula is: In the formula, y i,c Let represent the label value of the i-th sample in class c. When label smoothing is used, a smoothing coefficient ε can be introduced, and ε is a preset parameter in the interval [0,1). The smoothed label can be obtained by linearly weighting the original label, thereby reducing model overconfidence and improving generalization ability. In one example, the auxiliary loss weight coefficient λ can be set to 0.073937, and the label smoothing coefficient ε can be set to 0.004027.

[0050] The optimizer used is AdamW with a learning rate of 9.7247 × 10⁻⁶. -5 The weight decay is 1.3022 × 10 -4 The learning rate scheduling employs a combination of 5-epoch linear warmup and cosine annealing; the training cycle can be set to 100 epochs, the batch size to 32, and the gradient accumulation step count to 1; the exponential moving average decay coefficient can be set to 0.999. This configuration ensures stable convergence during training and contributes to more stable test performance. During the testing phase, a 5-crop testing enhancement strategy can be used to generate multiple cropping results from the same endoscopic image and fuse the classification outputs of each cropping result, thereby improving the stability of the test results.

[0051] In classification tasks, this invention achieves collaborative modeling of global and local features through a dual-branch feature extraction structure and a gating fusion mechanism. Global features provide a stable overall semantic context, while local features supplement fine-grained discriminative information for key lesion regions. The gating fusion mechanism dynamically adjusts the magnitude of local feature introduction, allowing local features to participate in discrimination as controlled correction terms for global features. This reduces the interference of local noise on the overall classification results in simple concatenation or equal-weighted fusion, as shown in Table 1 below.

[0052] Table 1. Overall performance comparison of the classification scheme and the baseline model on the test set. Where mAP is the mean average precision, Acc is the overall accuracy, CP is the class average precision, CR is the class average recall, and CF1 is the class average F1 score.

[0053] As shown in Table 1, the present invention outperforms the baseline model in terms of overall classification accuracy, average precision, and macro-average precision. Specifically, Acc increased from 94.44% to 94.94%, mAP increased from 89.34% to 91.60%, CP changed from 91.92% to 91.29%, CR changed from 89.04% to 87.11%, and CF1 changed from 89.22% to 87.83%, demonstrating an improvement in overall ranking ability and Top-1 discrimination ability under the precision-recall trade-off.

[0054] Regarding key category performance, this invention shows good improvement effects on some fine-grained inflammatory categories and easily confused categories, as shown in Table 2 below.

[0055] Table 2 shows the performance changes of the classification scheme in some key categories. Among them, the F1 rate for bullous myringitis increased from 89.66% to 100.00%, an increase of 10.34%; the F1 rate for poor ventilation increased from 88.46% to 90.57%, an increase of 2.10%; the F1 rate for acute otitis media increased from 94.55% to 96.43%, an increase of 1.88%; the F1 rate for normal otitis media increased from 97.86% to 98.19%, an increase of 0.33%; and the F1 rate for secretory otitis media decreased from 94.29% to 94.12%, a decrease of 0.17%.

[0056] The above description is merely a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and environments, and can be altered within the scope of the concept described herein through the above teachings or related technologies or knowledge. Modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims.

Claims

1. A classification method for endoscopic ear images based on bi-branch feature extraction and gated fusion, characterized in that, Includes the following steps: S1. Acquire an endoscopic image of the ear and perform feature processing on the endoscopic image to generate a deep feature map; S2. Construct a dual-branch feature extraction structure to extract global and local feature vectors based on deep feature maps; S3. Generate a fused feature vector by combining the global and local feature vectors through a gating fusion mechanism; S4. Based on the fused feature vector, identify the disease category and output the category result corresponding to the otoscopy image; Specifically, the gating fusion mechanism is as follows: a scalar gating value is output based on the joint representation of the global feature vector and the local feature vector, and the scalar gating value is used to control the magnitude of the fusion of the local feature vector.

2. The method for classifying endoscopic ear images based on bi-branch feature extraction and gated fusion according to claim 1, characterized in that, In step S1, the feature processing of the endoscopic image is specifically performed using a feature extraction backbone network. By extracting high-level semantic information and spatial discrimination information from the endoscopic image, a deep feature map is generated.

3. The method for classifying endoscopic ear images based on bi-branch feature extraction and gated fusion according to claim 2, characterized in that, The deep feature map F is specifically represented as: B╳C╳H╳W; where B represents the number of images simultaneously fed into the network in one forward propagation; C represents the number of channels; H and W are spatial dimensions, representing spatial height and spatial width, respectively.

4. The method for classifying endoscopic ear images based on bi-branch feature extraction and gated fusion according to claim 1, characterized in that, The dual-branch feature extraction structure in step S2 specifically includes: a global branch and a local branch; wherein, the global branch specifically involves using adaptive average pooling to aggregate global information from the depth feature map, converging information in the spatial dimension into a one-dimensional vector representation, and obtaining a global feature vector that reflects the overall semantic information of the entire endoscopic ear image. The local branch specifically involves selecting key local regions from the depth feature map and aggregating the feature information at that location to obtain a local feature vector that reflects the differences in fine-grained lesions.

5. The method for classifying endoscopic ear images based on bi-branch feature extraction and gated fusion according to claim 4, characterized in that, The local branch specifically includes: a local location selection module and a local feature aggregation module; Specifically, the local location selection module adaptively selects multiple key local regions with strong discriminative power based on the response of each spatial location in the deep feature map. The local feature aggregation module specifically performs mean aggregation on the feature vectors corresponding to local key regions, and then performs nonlinear projection mapping to obtain local feature vectors.

6. The method for classifying endoscopic ear images based on bi-branch feature extraction and gated fusion according to claim 3 or 5, characterized in that, The local location selection module specifically maps the deep feature map through 1×1 convolutional scoring to obtain a single-channel spatial scoring map. The single-channel spatial scoring map is then flattened along the spatial dimension into a scoring vector of B×(H×W). Top-K selection is performed using the scoring vector to obtain the k indices with the highest scores. The deep feature map is then rearranged according to the spatial dimension into a feature sequence of B×(H×W)×C. Feature sampling is performed based on the k indices to obtain a local feature set of B×k×C.

7. The method for classifying endoscopic ear images based on bi-branch feature extraction and gated fusion according to claim 5, characterized in that, The local feature vector also contains an auxiliary classification head, which outputs local classification results and calculates the corresponding auxiliary classification loss during the training phase; wherein, the auxiliary classification loss is incorporated into the total loss function according to a preset weight coefficient.

8. The method for classifying endoscopic ear images based on bi-branch feature extraction and gated fusion according to claim 7, characterized in that, The total loss function L local The calculation formula is: ； In the formula, L main Primary classification loss; L aux λ is the auxiliary classification loss; λ is the auxiliary loss weight coefficient, set to 0.073937.

9. The method for classifying endoscopic ear images based on bi-branch feature extraction and gated fusion according to claim 1, characterized in that, The fused feature vector is represented as follows: ； In the formula, f fused f represents the fused feature vector; global Represents the global feature vector; f local Represents a local feature vector; Proj() represents the projection function; α represents the scalar gate value; The formula for calculating the scalar gate value α is as follows: ； In the formula, σ represents the Sigmoid function; MLP() represents the multilayer sensing mapping; [f global ;f local ] represents the concatenation of global and local feature vectors; b is the learnable bias term, initially set to 0.089795; α max This indicates the upper limit of the gate value, set to 0.15477.

10. The method for classifying endoscopic ear images based on bi-branch feature extraction and gated fusion according to claim 1, characterized in that, The disease categories in step S4 specifically include: normal, secretory otitis media, acute otitis media, chronic otitis media, poor ventilation, bullous myringitis, tympanic membrane perforation, tympanostomy tube insertion, tympanosclerosis, ear canal cerumen, adhesive otitis media, cholesteatoma, fungal external otitis, tympanic cavity hemorrhage, and ear canal foreign body.

Images