Nasopharyngeal endoscope video lesion detection method and system based on prototype retrieval enhancement

Abstract

This application discloses a method and system for lesion detection in nasopharyngeal endoscopy videos based on prototype retrieval enhancement, belonging to the field of smart healthcare technology. The method includes: constructing a set of category prototypes; updating the set of category prototypes using a momentum update mechanism; extracting cues from the nasopharyngeal endoscopy video to be detected, and concatenating the cues with the target inference frame to form the original input; the detection head includes a regression branch and a classification branch, where the regression branch predicts the location of the lesion in the original input, and the classification branch predicts the category of the lesion in the original input. During the classification process, cosine similarity is calculated between the region features and the set of category prototypes, the most relevant category prototype is selected, and the most relevant category prototype is fused with the current features to obtain an enhanced classification feature representation, thereby completing the lesion category prediction. This application solves the problem of limited detection performance caused by the scarcity of tail lesion samples and blurred category boundaries in long-tailed data during nasopharyngeal endoscopy examinations.

Description

Technical Field

[0001] This application relates to the field of smart healthcare technology, and in particular to a method for detecting lesions in nasopharyngeal endoscopy videos using semantic prototype and retrieval enhancement techniques. Background Technology

[0002] Head and neck malignancies rank sixth in global cancer incidence, encompassing subtypes such as oral cancer, nasopharyngeal carcinoma, oropharyngeal carcinoma, laryngeal cancer, and hypopharyngeal cancer. Nasopharyngeal endoscopy, as an important non-invasive technique for detecting and diagnosing lesions and assessing their extent in clinical practice, can provide multi-dimensional analysis of the nasopharyngeal microstructure. It has significant clinical value in differentiating precancerous lesions such as intraepithelial neoplasia, detecting pathological features of carcinoma in situ, and quantitatively assessing the anatomical layers of tumor invasion.

[0003] However, nasopharyngeal endoscopy faces several practical challenges. First, medical imaging data typically exhibits a significant long-tail distribution: common lesions with high clinical incidence (such as cysts and neoplasms) have abundant samples, forming the "head" of the data distribution; while rare lesions and subtypes with low incidence (such as vocal cord nodules, vocal cord polyps, and postoperative manifestations) are scarce, forming the "tail." Although the sample size of each tail lesion is limited, they still have significant clinical diagnostic value; missed detections or misdiagnoses can directly lead to patients missing the optimal treatment window. Second, some tail lesions under nasopharyngeal endoscopy are highly similar in visual appearance to head lesions. For example, vocal cord nodules, vocal cord polyps, and vocal cord leukoplakia all occur in the vocal cord structure, and their pathological features all show tissue hyperplasia or mucosal coverage. Under NBI (narrow-band imaging) conditions, their color representation tends to converge, making them more prone to confusion in category discrimination.

[0004] Existing video object detection methods primarily target general object detection tasks in natural scenes, assuming a relatively balanced distribution of samples across categories, without specifically addressing the long-tail distribution characteristics of medical images. When directly applied to nasopharyngeal endoscopy video analysis, the model training process is easily dominated by the head category, while the tail categories, due to insufficient samples, struggle to form clear decision boundaries, causing detection results to "absorb" the head category, resulting in severe false negatives and negatives. Some studies have attempted to alleviate the class imbalance problem using strategies such as resampling and cost-sensitive learning, but these are prone to overfitting or disrupting the original structure of the feature space. While few-shot learning methods address scenarios with scarce samples, they are mostly designed for static images, failing to fully utilize the temporal continuity of video data and lacking explicit modeling of the class confusion problem in medical images. Summary of the Invention

[0005] This application provides a method and system for detecting lesions in nasopharyngeal endoscopy video based on prototype retrieval enhancement, in order to solve the problems in the prior art.

[0006] On the one hand, embodiments of this application provide a method for detecting lesions in nasopharyngeal endoscopic video based on prototype retrieval enhancement, including: Construct a category prototype set, in which the category prototypes are derived from ROI (region of interest) features generated during the training of the detection system that have a classification confidence higher than the threshold and are consistent with the real situation; As training progresses, the class prototype set is updated using a momentum update mechanism; Acquire the nasopharyngeal endoscopy video to be tested, extract the prompts from the nasopharyngeal endoscopy video to be tested, and concatenate the prompts with the target inference frames in the nasopharyngeal endoscopy video to be tested to form the original input; The original input is fed into a prototype memory-enhanced detection head, which includes a regression branch and a classification branch. The regression branch is used to predict the location of lesions in the original input, and the classification branch is used to predict the category of lesions in the original input. During the classification process, the classification branch calculates the cosine similarity between the regional features extracted from the original input and the updated category prototype set. Based on the cosine similarity calculation result, the most relevant category prototype in the category prototype set is selected. The most relevant category prototype is fused with the current features obtained based on the regional features to obtain the enhanced classification feature representation. The enhanced classification feature representation is then used to complete the lesion category prediction.

[0007] On the other hand, embodiments of this application also provide a nasopharyngeal endoscopic video lesion detection system based on prototype retrieval enhancement, including: The prototype memory module is used to construct a set of category prototypes. The category prototypes in the set are derived from ROI features that have a classification confidence higher than the threshold and are consistent with the real situation, generated during the training of the detection system. As training progresses, the set of category prototypes is updated using a momentum update mechanism. A prompt generation network is used to acquire the nasopharyngeal endoscopy video to be tested, extract prompts from the nasopharyngeal endoscopy video to be tested, and concatenate the prompts with the target inference frames in the nasopharyngeal endoscopy video to be tested to form the original input; The target detection network includes a detection head based on prototype memory enhancement. The detection head contains a regression branch and a classification branch. The regression branch is used to predict the location of lesions in the original input, and the classification branch is used to predict the category of lesions in the original input. During the classification process, the classification branch calculates the cosine similarity between the region features extracted from the original input and the updated category prototype set. Based on the cosine similarity calculation result, the most relevant category prototype in the category prototype set is selected. The most relevant category prototype is fused with the current features obtained based on the region features to obtain the enhanced classification feature representation. The enhanced classification feature representation is used to complete the lesion category prediction.

[0008] The prototype-based retrieval-enhanced nasopharyngeal endoscopic video lesion detection method and system in this application have the following advantages: This paper addresses the performance limitations of nasopharyngeal endoscopy in detecting tail lesions due to the scarcity of tail lesion samples and blurred category boundaries in long-tailed data. The proposed Retrieval-Augmented Spatio-temporal Prompting Network (RAGSTPN) introduces an explicitly maintained prototype memory module on a two-stage cascaded architecture. During end-to-end training, a stable category prototype semantic center is constructed through a momentum update mechanism. Simultaneously, the network employs a prototype memory-based retrieval-enhanced detection head structure. By decoupling the classification-regression branches, a prototype retrieval and feature fusion mechanism is introduced in the classification stage, providing reliable semantic prior support for tail lesions. The contrastive loss function used during training strengthens the alignment between sample features and category prototypes, improving intra-class compactness and inter-class separability. Compared with existing methods, the RAGSTPN proposed in this application significantly improves the detection accuracy and recall of tail minority class lesions on the self-built NVLDN (a video target detection dataset of nasopharyngeal and laryngeal lesions containing novel lesions) seven-class lesion detection dataset, with only a small loss in head class detection performance. It provides an effective technical solution for small sample target detection of long-tailed medical image data and has important clinical application value. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This is an architecture diagram of the nasopharyngeal endoscopic video lesion detection method based on prototype retrieval enhancement provided in the embodiments of this application.

[0011] Figure 2 The flowchart illustrates the execution of the nasopharyngeal endoscopic video lesion detection method based on prototype retrieval enhancement provided in this application embodiment.

[0012] Figure 3 This is a stacked mAP50 diagram of the network proposed in this application and eight prior art technologies in a few classes.

[0013] Figure 4 This is a stacked diagram of the Recall of the network proposed in this application and eight prior art technologies in a few classes.

[0014] Figure 5This study compares the detection results of the network proposed in this application with those of three existing technologies in terms of vocal cord nodules, vocal cord polyps, and postoperative manifestations. Detailed Implementation

[0015] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0016] This application provides a method for detecting lesions in nasopharyngeal endoscopic video based on prototype retrieval enhancement. The method includes the following steps: S100, construct a set of category prototypes. The category prototypes in the set are ROI features that have a classification confidence higher than the threshold and are consistent with the real situation when the detection system is trained.

[0017] For example, such as Figure 1 As shown, to alleviate the problem of unstable feature distribution caused by the scarcity of tail lesion samples, this application embodiment designs a Prototype Memory Block (PMB). This module explicitly maintains the semantic prototypes of various lesion categories in the feature space during end-to-end training, providing stable and continuously updated category semantic priors for subsequent detection heads.

[0018] Let the total number of categories be The prototype memory module maintains the following set of category prototypes. :

[0019] in, Indicate category c The semantic prototype, i.e., the category prototype. for The feature space is a dimensional embedding space. Each category prototype characterizes the global semantic center of the corresponding category in the feature space, which is used to eliminate the uncertainty caused by the fluctuation of features of individual samples.

[0020] S110, as training progresses, the set of class prototypes is updated using a momentum update mechanism.

[0021] For example, the momentum update process is represented as:

[0022] in, and The first t Second and thirdt -1 training iterations of class c Category prototype, This is the momentum coefficient, used to balance the relationship between historical semantic information and newly observed features. The number of iterations required to unfreeze the prototype memory (preset). For the first t In the next training iteration, the class c The sample embedding features are represented as:

[0023] in, For L2 norm normalization, For the first t Class during the next training iteration c The number of medium- and high-quality samples S For the first t The sample at the next iteration For the first t Samples in the next training iteration s ROI features extracted by the system For the first t Class at -1 iteration The sample embedding features. This application's embodiments utilize a momentum update mechanism to enable the category prototype to evolve synchronously with the feature distribution.

[0024] Furthermore, the loss function used during system training is:

[0025] in, For the total loss function, For classifying losses, For bounding box regression loss, For prototype comparison loss, and These are all weighting coefficients, used to balance the impact of different loss terms on the training process.

[0026] To standardize the optimization direction of the prototype modeling mechanism and promote the model to learn more discriminative feature representations, this application adopts Prototype Contrastive Loss (PCL) in its embodiments. Let the set of positive samples be... For the sample The cosine similarity between its embedded features and the class prototype is denoted as . The prototype contrastive loss is expressed as:

[0027] in, For the positive sample set, For the positive sample set The number of samples in for The samples in For the sample i The true category label, For the sample i Embedding features and Cosine similarity between them This is a temperature coefficient used to adjust the smoothness of the similarity distribution. C This represents the total number of categories. For the sample i Embedding features and categories c Cosine similarity between class prototypes.

[0028] The prototype contrastive loss calculates the similarity between the embedded features of each sample and the corresponding class prototype within the positive sample set. The log-likelihood form encourages the sample features to converge toward their true class prototype. The summation of the similarities of all class prototypes in the denominator introduces an implicit constraint on the out-of-class prototype, keeping the sample features at a large distance from the non-corresponding class prototype. This results in a distribution structure in the feature space where similar features cluster around their class prototype and out-of-class features are separated from each other.

[0029] Unlike inter-sample contrast loss, prototype contrast loss focuses on the alignment constraints between samples and prototypes. It uses the class prototype as a unified reference to avoid optimization bias caused by individual noise in inter-sample contrast, while enhancing the representational stability of prototype memory and providing a more reliable feature matching benchmark for the detection head.

[0030] In the embodiments of this application, the bounding box regression loss is calculated using the Smooth function, which is:

[0031] in, For Smooth L1 functions, It is a predicted value. Compared with the true value difference.

[0032] During training, ResNet101 and Swin-Tiny were used as the backbone networks, and AdamW was selected as the optimizer. The base learning rate was set to 2.5e-5, with a learning rate scaling factor of 10× for the trainable parameter g. The experimental environment was configured as follows: NVIDIA RTX 3090 GPU, Python 3.8, PyTorch 1.10.0, and CUDA 11.3 framework.

[0033] S120: Acquire the nasopharyngeal endoscopy video to be tested, extract the prompts from the nasopharyngeal endoscopy video to be tested, and concatenate the prompts with the target inference frame in the nasopharyngeal endoscopy video to be tested to form the original input.

[0034] S130: The original input is fed into a detection head enhanced with prototype memory. The detection head includes a regression branch and a classification branch. The regression branch is used to predict the location of the lesion in the original input. The prediction process is as follows: First, the multi-scale feature maps extracted by the backbone network from the original input are fused to generate a fused high-dimensional feature representation. Then, the fused features are reduced in dimensionality and spatial information enhanced through multi-layer convolution and pooling operations, and the initial coordinates (x1, y1, x2, y2) of the candidate anchor boxes are output, where (x1, y1) and (x2, y2) are the coordinates of the two diagonal vertices of the candidate anchor boxes, respectively. Next, the initial coordinates are optimized by offset regression (dx, dy, dw, dh), where dx and dy are the center coordinate offsets, and dw and dh are the width and height scaling factors. Finally, by removing redundant candidate boxes, the accurate bounding box coordinates of the lesion are output, thus completing the prediction of the lesion location. The classification branch is used to predict the category of lesions in the original input. During the classification process, the classification branch calculates the cosine similarity between the regional features extracted from the original input and the updated category prototype set. Based on the cosine similarity calculation result, the most relevant category prototype in the category prototype set is selected. The most relevant category prototype is fused with the current features obtained based on the regional features to obtain the enhanced classification feature representation. The enhanced classification feature representation is used to complete the lesion category prediction.

[0035] For example, with the support of the prototype memory module, the present invention designs a detection head structure based on prototype memory retrieval enhancement (RAG Head) to make full use of category prototype information at the detection head level and improve classification performance in tail lesion scenarios.

[0036] Unlike traditional ROI Heads that directly perform classification and regression based on shared ROI features, the detection head in this embodiment adopts a decoupled classification-regression branch design: the regression branch performs bounding box regression based on the original ROI features, unaffected by prototype memory; the classification branch introduces a prototype retrieval and feature fusion mechanism, and completes category prediction after semantically enhancing the ROI features through the category prototype.

[0037] Let the region features of the ROI-aligned output be represented as follows: ,in, This represents the number of candidate regions. For feature dimension, for B × DThe embedding space is dimensional. Since the dimension of ROI features is often inconsistent with that of the class prototypes, the classification branch first maps the region features to a low-dimensional embedding space to align with the feature dimensions of the class prototypes in the updated class prototype set before calculating the cosine similarity between the region features and the updated class prototype set.

[0038] in, The region features are mapped to the desired dimensions, i.e., the current features. Represents the ReLU function. This is a preset linear mapping parameter matrix, which is optimized along the gradient direction during model training. ReLU is a region feature. Its computational form is simple and its gradient propagation is stable. Its suppression of negative responses and linear preservation of positive responses help to form a sparse and discriminative feature distribution in the embedding space, thereby improving the reliability of prototype matching based on cosine similarity.

[0039] After dimensionality reduction, cosine similarity is calculated between the ROI features and the category prototypes in the prototype memory module:

[0040] in, For current features and categories c Category prototype Cosine similarity between them This represents the L2 norm. Cosine similarity normalizes feature vectors using L2, focusing only on directional consistency and eliminating interference from amplitude differences. This makes the similarity more focused on the semantic structure itself, making it particularly suitable for identifying tail lesions with unstable feature distributions in small sample scenarios.

[0041] Based on the similarity results, the system retrieved the most relevant category prototypes. The most relevant category prototype is then fused with the current features obtained based on region features. The fusion process is as follows:

[0042] in, The classification features are spliced ​​and merged. For linear parameter mapping matrix, For the most relevant category prototype, Indicates splicing; classification features after splicing and merging. By restoring the original feature dimensions through linear mapping, we obtain the enhanced classification feature representation. Ultimately, the enhanced classification features will be The input classification header is processed by a fully connected layer to calculate the score for each lesion category. Then, a Softmax function is used to convert these scores into a category probability distribution. The category with the highest probability is selected as the predicted category for the current lesion region, and the confidence score for that category is output.

[0043] This application also provides a prototype-based enhanced nasopharyngeal endoscopic video lesion detection system, which includes: The prototype memory module is used to construct a set of category prototypes. The category prototypes in the set are derived from ROI features that have a classification confidence higher than the threshold and are consistent with the real situation, generated during the training of the detection system. As training progresses, the set of category prototypes is updated using a momentum update mechanism. A prompt generation network is used to acquire the nasopharyngeal endoscopy video to be tested, extract prompts from the nasopharyngeal endoscopy video to be tested, and concatenate the prompts with the target inference frames in the nasopharyngeal endoscopy video to be tested to form the original input; The target detection network includes a detection head based on prototype memory enhancement. The detection head contains a regression branch and a classification branch. The regression branch is used to predict the location of lesions in the original input, and the classification branch is used to predict the category of lesions in the original input. During the classification process, the classification branch calculates the cosine similarity between the region features extracted from the original input and the updated category prototype set. Based on the cosine similarity calculation result, the most relevant category prototype in the category prototype set is selected. The most relevant category prototype is fused with the current features obtained based on the region features to obtain the enhanced classification feature representation. The enhanced classification feature representation is used to complete the lesion category prediction.

[0044] For example, in the early stages of system training, the prototype memory module is frozen, and the momentum update mechanism is executed after the spatial stability of the ROI features.

[0045] Furthermore, the system in this application adopts a two-stage cascaded architecture as its basic framework. The entire system consists of a prompt generation network, an object detection network, and a prototype memory module. The prompt generation network is responsible for spatiotemporal feature modeling and discriminative prompt generation; the object detection network performs feature enhancement, prototype retrieval, and task inference; and the prototype memory module explicitly maintains a set of category prototypes, providing stable semantic prior support for the detection head. All three components achieve collaborative training through end-to-end joint optimization.

[0046] In the prompt generation network, the target inference frame F in the nasopharyngeal endoscopy video to be detected is used. t Centered on F, select the location located at F t The K frames before and after constitute the reference frame F sup Reference frame F supThe input is fed into the backbone network to capture the temporal correlation and spatial feature information contained among K frames. A spatiotemporal attention predictor is used to compress the joint features (temporal correlation and spatial feature information) captured in the backbone network along both temporal and spatial dimensions. This spatiotemporal attention predictor is composed of temporal attention and spatial attention concatenated: first, temporal attention models inter-frame dependencies, highlighting key temporal information of the reference frame; then, spatial attention focuses on lesion-related regions and suppresses background noise. Through two-dimensional weighted fusion, the joint features are refined and compressed, ultimately outputting a highly discriminative cue P. In the target detection network, the target inference frame F is... t Divide the network into n non-overlapping patches, and construct the patch structure as the target inference frame F for backbone network computation. t The feature embedding will combine the cue P generated by the cue generation network with the target inference frame F. t The feature embeddings are pre-concatenated and used as the raw input for feature extraction in the inference frame backbone network. During the computation of the inference frame backbone network, cues P are added layer by layer. The self-learning scaling factor of each computational block in the backbone network is used to compare the cues P with the target inference frame F. t The features are fused to maintain stable spatiotemporal information of reference frames in the deep network. The detection network learns the most suitable fusion method of each layer during the training phase, thereby providing the detection head with high-quality spatiotemporal enhancement features of inference frames.

[0047] The prototype memory module explicitly maintains semantic prototype vectors for each category prototype, continuously optimizing the category prototype set through a momentum update mechanism during system training. In the actual detection application phase, prototype retrieval is achieved based on cosine similarity, fusing the retrieved category prototypes with features of the current region to provide a stable semantic reference for classification decisions. The training and detection processes are as follows: Figure 2 As shown.

[0048] Finally, the spatiotemporal enhancement features are applied to the prototype memory-enhanced detection head to generate the target inference frame F. t The test results include the location and type of lesions.

[0049] Experimental instructions To compare the performance of the nasopharyngeal endoscopy lesion detection model, this experiment collected original nasopharyngeal endoscopy videos of 851 patients from two tertiary hospitals across different regions, conducted between January 6, 2024 and June 3, 2024, and April 7, 2025 and May 26, 2025, according to the data collection and annotation specifications proposed in the "Expert Consensus on Quality Control System for Endoscopic Artificial Intelligence Data Acquisition and Annotation". Under the guidance of three professional physicians, 270 patient videos with clear visual characteristics of lesions were selected. To ensure the privacy and security of medical imaging data, all original video data underwent strict anonymization processing, removing sensitive content involving patient identity information. In the data preprocessing stage, the original videos were segmented and segments without obvious abnormalities were removed, ultimately forming 438 basic lesion video segments. The number of video samples for the four common lesion types were as follows: neoplasms (91 segments), vocal cord leukoplakia (47 segments), cysts (76 segments), and adenoid hyperplasia (43 segments). Based on this, videos of 41 patients with a minority of lesions were further selected from the original endoscopic examination data. Following the same data screening and processing principles, 75 video segments of a minority of lesions were obtained, including three types of lesions: vocal cord nodules, vocal cord polyps, and postoperative manifestations.

[0050] The final dataset comprises seven categories of nasopharyngeal lesions: neoplasms, vocal cord leukoplakia, cysts, adenoid hyperplasia, vocal cord nodules, vocal cord polyps, and postoperative manifestations. The number of video samples for each lesion category is as follows: neoplasms 85, vocal cord leukoplakia 42, cysts 70, adenoid hyperplasia 39, vocal cord nodules 7, vocal cord polyps 17, and postoperative manifestations 17, totaling 591 video clips from 245 different patients. Under the guidance of three nasopharyngeal specialists, and referring to the anatomical structure definitions and annotation specifications in "Clinical Application of Electronic Laryngoscopy—Narrow-Band Imaging Endoscopic Atlas of Nasopharyngeal and Laryngeal Tumors," all video clips were meticulously annotated frame by frame. Simultaneously, to enhance the system's robustness in real clinical settings, 131 lesion-free video clips from 64 healthy individuals were also included as negative samples. Through the above process, the final video dataset NVLDN for endoscopic lesion detection research in nasopharyngeal and laryngeal diseases was constructed.

[0051] Table 1 shows the comparative experimental results of the RAGSTPN model proposed in this application with other video detection models on the self-built dataset NVLDN. The models compared are SELSA (Sequence-level Semantic Aggregation), RDN (Relation Distillation Network), Mamba, DiffusionVID (Denoising Box Based on Spatiotemporal Conditions), STPN (Spatiotemporal Cueing Network), DynSTPN (Dynamic Spatiotemporal Cueing Network), TFA (Spatiotemporal Feature Aggregation), and DCFS (Dynamic Counting Fair Sampling).

[0052] Table 1. Comparison of RAGSTPN model and contrasting model on the NVLDN dataset.

[0053] In Table 1, bold indicates the best performance, and underline indicates the second best. From an overall detection performance perspective, the RAGSTPN proposed in this application achieved a high recall value of 85.8% across all task categories, consistent with the Mamba method using a ResNet101 backbone network, and significantly outperformed methods such as DiffusionVID, STPN, and DynSTPN. This demonstrates that even in the presence of complex interferences such as motion blur, occlusion, and lighting changes in endoscopic videos, RAGSTPN, through the combination of spatiotemporal cue information and a prototype retrieval mechanism, can more effectively capture lesion targets in video sequences, thereby significantly reducing the risk of missed detections.

[0054] In terms of mAP, RAGSTPN achieved an all-class mAP of 37.6%, slightly lower than methods such as SELSA, DiffusionVID, and DynSTPN. This is mainly because these methods have more stable classification boundaries when there are sufficient majority class samples, while RAGSTPN's design focuses on mitigating the long-tail distribution problem and enhancing features for the minority class, thus its performance improvement on the majority class is relatively limited. However, in the minority class detection task, RAGSTPN achieved an mAP of 26.8% and a recall of 74.8%, both significantly better than other comparative methods, validating the effectiveness of the proposed prototype retrieval enhancement mechanism in small-sample lesion detection.

[0055] Figure 3 and Figure 4 This is a stacked plot of mAP50 and Recall of the RAGSTPN model and eight contrast models on the self-built dataset NVLDN minority class in the embodiments of this application.

[0056] In the mAP50 stacking plot, RAGSTPN showed the highest overall height and a balanced contribution to the accuracy of the three types of rare lesions, demonstrating optimal overall detection accuracy. In the recall stacking plot, RAGSTPN also achieved the highest overall recall, particularly excelling in typical rare lesion cases such as postoperative conditions, proving its effectiveness in reducing the false negative rate of rare lesions. Overall, RAGSTPN outperformed the comparison methods in both accuracy and recall for detecting rare lesions in nasopharyngeal endoscopy video, validating its effectiveness and superiority in this scenario.

[0057] Table 2 shows the performance comparison results of the RAGSTPN model and three comparative models on the self-built dataset NVLDN in the embodiments of this application.

[0058] Table 2 shows the comparison of detection performance between the RAGSTPN model and the contrasting model on the NVLDN dataset.

[0059] In Table 2, bold indicates the best performance, and underline indicates the second best performance. AP (%) / Recall (%) was used as the evaluation criterion. The data in Table 2 shows that for most types of lesions with a large sample size and relatively stable appearance features, such as cysts and adenoid hyperplasia, the detection performance differences between methods are relatively small, with RAGSTPN maintaining a similar level to other models. This indicates that with sufficient samples, traditional video object detection models can already learn relatively stable discriminative features, and the marginal contribution of introducing additional prototype memory information to performance improvement is relatively limited.

[0060] In contrast, RAGSTPN demonstrated significant advantages in both AP and Recall metrics for typical minority lesions such as vocal cord nodules, vocal cord polyps, and postoperative manifestations. Specifically, the detection accuracy for vocal cord nodules and postoperative manifestations was significantly improved compared to other methods, indicating that the prototype memory module can provide stable semantic references for scarce sample categories, thereby alleviating the problem of insufficient feature representation under conditions of few samples. Simultaneously, the higher Recall value also indicates that this method can more fully capture minority lesion targets, improving the reliability of the model in clinical screening scenarios.

[0061] Figure 5 This document presents a visual comparison of the RAGSTPN model and three contrasting models on the self-built NVLDN dataset for detecting various lesions in this embodiment. Each stitched image consists of the original image above and a magnified local detection result below. All magnified original images are marked with red boxes. The text below the image indicates the model's best detection category and detection confidence for the current lesion region in the image; higher values ​​indicate higher detection accuracy, and red text indicates false positives. Figure 5 In this context, GT stands for Ground Truth, which is the truth value of the label.

[0062] from Figure 5 It can be seen that for vocal cord nodule samples, both SELSA and DynSTPN misclassified the lesions as vocal cord leukoplakia and gave high confidence scores to the misclassified categories, indicating insufficient separation between different lesion categories in their feature representation space. Although Mamba could correctly detect the target, its confidence score was low, reflecting its limited ability to discriminate this type of small sample lesion. In contrast, RAGSTPN not only accurately located the lesion region but also achieved a significant confidence advantage in the correct category, while effectively suppressing the misclassified response, demonstrating a clearer category decision boundary.

[0063] For vocal cord polyp samples, all four methods were able to detect the lesions, with the main differences lying in the confidence level allocation and bounding box control. SELSA and DynSTPN generated false positives for novel organisms with confidence levels of 0.84 and 0.99, respectively, in the lesion area. While Mamba correctly predicted vocal cord polyps with a confidence level of 0.32, it also included the lesion area and vocal cord secretions in the higher-confidence false positive result for novel organisms, exhibiting a clear class bias. The class judgments of the above models all tended to favor the category of novel organisms with sufficient training samples to varying degrees, reflecting the instability of the class decision boundary under long-tailed distribution conditions. RAGSTPN, through cross-sample constraints of prototype memory, gave a higher and more concentrated confidence output for the true category in this sample and effectively suppressed the wrong category response, demonstrating stronger feature discrimination and class separation capabilities.

[0064] Regarding postoperative manifestation categories, the comparative models generally showed a tendency to favor high-frequency categories (such as neoplasms and cysts). While SELSA and DynSTPN could perform relatively accurate location regression, their category determination was clearly biased towards high-frequency categories. Mamba, while exhibiting category bias, also suffered from localization errors. This phenomenon further reveals the category bias problem of models under long-tailed distribution conditions and the inadequacy of minority class feature modeling capabilities. In contrast, RAGSTPN, through its cross-sample prototype guidance mechanism, can effectively distinguish fine-grained differences between categories, forming a clear category distinction structure in the confidence distribution, thus maintaining stable recognition performance in complex backgrounds.

[0065] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0066] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. An enhanced nasopharyngolaryngoscope video lesion detection method based on prototype retrieval, characterized in that, include: Construct a category prototype set, wherein the category prototypes in the category prototype set are ROI features that have a classification confidence higher than a threshold and are consistent with the real situation when the detection system is trained; As training progresses, the set of class prototypes is updated using a momentum update mechanism; Acquire the nasopharyngeal endoscopy video to be tested, extract the prompts from the nasopharyngeal endoscopy video to be tested, and concatenate the prompts with the target inference frame in the nasopharyngeal endoscopy video to be tested to form the original input; The original input is fed into a prototype memory-enhanced detection head, which includes a regression branch and a classification branch. The regression branch is used to predict the location of lesions in the original input, and the classification branch is used to predict the category of lesions in the original input. During the classification process, the classification branch calculates the cosine similarity between the regional features extracted from the original input and the updated category prototype set. Based on the cosine similarity calculation result, the most relevant category prototype in the category prototype set is selected. The most relevant category prototype is fused with the current features obtained based on the regional features to obtain an enhanced classification feature representation. The enhanced classification feature representation is then used to complete the lesion category prediction.

2. The method of claim 1, wherein the method is based on a prototype retrieval augmented endoscopic video lesion detection method. The loss function used during the training of the system is: in, For the total loss function, For classifying losses, For bounding box regression loss, For prototype comparison loss, and All are weighting coefficients.

3. The method of claim 2, wherein the method is based on a prototype retrieval augmented endoscopic video lesion detection method. The prototype contrast loss is expressed as: in, For the positive sample set, For the positive sample set The number of samples in for The samples in For the sample i The true category label, For the sample i Embedding features and Cosine similarity between them For temperature coefficient, C This represents the total number of categories. For the sample i Embedding features and categories c Cosine similarity between class prototypes.

4. The method for detecting nasopharyngeal endoscopic video lesions based on prototype retrieval enhancement according to claim 2, characterized in that, The bounding box regression loss is calculated using the Smooth function, which is: in, For Smooth L1 functions, It is a predicted value. Compared with the true value difference.

5. The method for detecting nasopharyngeal endoscopic video lesions based on prototype retrieval enhancement according to claim 1, characterized in that, The momentum update process is represented as: in, and The first t Second and third t -1 training iterations of class c Category prototype, The momentum coefficient, For the first t In the next training iteration, the class c The sample embedding features are represented as: in, For L2 norm normalization, For the first t Class during the next training iteration c The number of medium- and high-quality samples S For the first t The sample at the next iteration For the first t Samples in the next training iteration s ROI features extracted by the system For the first t Class at -1 iteration The sample embedding features.

6. The method for detecting nasopharyngeal endoscopic video lesions based on prototype retrieval enhancement according to claim 1, characterized in that, Before calculating the cosine similarity between the region features and the updated set of category prototypes, the classification branch first maps the region features to a low-dimensional embedding space to align with the feature dimensions of the category prototypes in the set of category prototypes: in, The region features after dimension mapping, i.e., the current features, Represents the ReLU function. For the preset linear mapping parameter matrix, The region features are described above.

7. The method for detecting nasopharyngeal endoscopic video lesions based on prototype retrieval enhancement according to claim 6, characterized in that, The process of fusing the most relevant category prototype with the current feature obtained based on the region features is as follows: in, The classification features are spliced ​​and merged. For linear parameter mapping matrix, For the most relevant category prototype, Indicates splicing; Classification features after splicing and fusion By restoring the original feature dimensions through linear mapping, we obtain the enhanced classification feature representation. .

8. A system for detecting nasopharyngeal endoscopic video lesions based on prototype retrieval enhancement as described in any one of claims 1-7, characterized in that, include: The prototype memory module is used to construct a category prototype set, wherein the category prototypes in the category prototype set are ROI features that have a classification confidence higher than a threshold and are consistent with the real situation when the detection system is trained. As training progresses, the set of class prototypes is updated using a momentum update mechanism; A prompt generation network is used to acquire a nasopharyngeal endoscopy video to be tested, extract prompts from the nasopharyngeal endoscopy video to be tested, and concatenate the prompts with the target inference frames in the nasopharyngeal endoscopy video to be tested to form the original input; The target detection network includes a detection head based on prototype memory enhancement. The detection head includes a regression branch and a classification branch. The regression branch is used to predict the location of lesions in the original input, and the classification branch is used to predict the category of lesions in the original input. During the classification process, the classification branch calculates the cosine similarity between the regional features extracted from the original input and the updated category prototype set. Based on the cosine similarity calculation result, the most relevant category prototype in the category prototype set is selected. The most relevant category prototype is fused with the current features obtained based on the regional features to obtain an enhanced classification feature representation. The enhanced classification feature representation is used to complete the lesion category prediction.

9. The nasopharyngeal endoscopic video lesion detection system based on prototype retrieval enhancement according to claim 8, characterized in that, In the early stages of system training, the prototype memory module is frozen, and the momentum update mechanism is executed only after the spatial stability of the ROI features.

Images