Two-stage cascaded video focus detection method and system

Abstract

The invention discloses a two-stage cascaded video focus detection method and system, and relates to the technical field of image target recognition, and the method comprises the steps: obtaining an endoscope image; selecting a reference frame; carrying out iterative calculation on the feature embedding of the reference frame to obtain deep space-time representation; generating a time attention weight based on the deep spatio-temporal representation, and performing weighted fusion on the deep spatio-temporal representation to obtain an enhanced feature; performing dimension reduction on the enhanced features to obtain video prompt information; and extracting feature embedding of the target reasoning frame and reasoning frame features in the video prompt information, performing iterative calculation on the reasoning frame features to obtain deep reasoning features, and performing focus detection on the deep reasoning features to obtain a detection result. According to the method, a two-stage cascade Transform architecture is adopted, the quality degradation phenomena of dynamic blur, exposure imbalance, reflection artifacts and the like of inference frames are effectively relieved, and efficient joint modeling of spatial-temporal characteristics is achieved.

Description

Technical Field

[0001] This application relates to the field of image target recognition technology, and in particular to a method for identifying lesions in nasopharyngeal endoscopic images using a two-stage cascaded Transformer network. Background Technology

[0002] Head and neck malignancies rank sixth in global cancer incidence, encompassing subtypes such as oral cancer, nasopharyngeal carcinoma, oropharyngeal carcinoma, laryngeal cancer, and hypopharyngeal cancer. Nasopharyngeal endoscopy, as an important non-invasive technique for detecting and assessing the extent of lesions in clinical practice, can provide multi-dimensional analysis of the nasopharyngeal microstructure. It has significant clinical value in differentiating precancerous lesions such as intraepithelial neoplasia, detecting pathological features of carcinoma in situ, and quantitatively assessing the anatomical layers of tumor invasion.

[0003] However, nasopharyngeal endoscopy faces many practical challenges. On the one hand, the complex structure of the pharynx and larynx, covered by multiple layers of mucosa, results in low contrast and blurred boundaries of lesions. Furthermore, common lesions such as adenoid hyperplasia and cysts overlap with early pharyngeal cancer in morphological characteristics, leading to significant differences in interpretation among physicians of varying experience, resulting in frequent missed diagnoses and misdiagnoses. On the other hand, endoscopic imaging is limited by single-point illumination, and the endoscopic videos generally suffer from motion blur, exposure imbalance, and reflection artifacts, severely impairing the integrity of feature representation. Additionally, traditional detection methods employ a "video frame splitting - single-frame detection - result fusion" process, failing to effectively utilize the spatiotemporal continuity of video. The inference speed is also limited by the efficiency of frame splitting and merging, making it difficult to meet the real-time clinical requirements, i.e., a frame rate (FPS) ≥ 24.

[0004] Breakthroughs in the field of medical artificial intelligence have promoted the clinical application of computer-aided diagnostic systems. Deep learning frameworks have been successfully applied to scenarios such as multi-classification of nasopharyngeal lesions and detection of vocal cord cancer. However, the end-to-end detection architecture for nasopharyngeal endoscopy videos still needs improvement. Current research is still limited to discrete frame analysis and it is difficult to overcome technical bottlenecks.

[0005] Therefore, it is necessary to propose a new lesion detection method to address the limitations in detection performance caused by video quality degradation and insufficient utilization of spatiotemporal features in nasopharyngeal endoscopy. Summary of the Invention

[0006] This application provides a two-stage cascaded video lesion detection method and system to solve the problems in the prior art.

[0007] On the one hand, embodiments of this application provide a two-stage cascaded video lesion detection method, including: Acquire nasopharyngeal endoscopy videos and extract endoscopic images from the nasopharyngeal endoscopy videos; Using the endoscopic image at the current moment as the target inference frame, a reference frame is formed by selecting the target inference frame and multiple frames before and after the target inference frame. The reference frame is input into the prompt generation network, which consists of multiple sequentially connected Transformer encoders and a spatiotemporal attention predictor. The Transformer encoders include a multi-head attention mechanism and a feedforward network. The multiple Transformer encoders iteratively calculate the feature embeddings of the reference frame to obtain a deep spatiotemporal representation. In the temporal attention part, the spatiotemporal attention predictor generates temporal attention weights based on the deep spatiotemporal representation, and then weights and fuses the deep spatiotemporal representation according to the temporal attention weights to obtain enhanced features with temporal characteristics. In the spatial attention part, the spatiotemporal attention predictor uses a learnable query vector to reduce the dimensionality of the enhanced features to obtain video prompt information. The video prompt information and the target inference frame are input into the target detection network. The target detection network consists of multiple sequentially connected stages and a detection head. Each stage contains a dynamic prompt enhancement module and a Transformer computation block. The feature embedding of the target inference frame and the video prompt information are input into the dynamic prompt enhancement module. The dynamic prompt enhancement module extracts the features of the inference frame using gating parameters. The Transformer computation block iteratively calculates the features of the inference frame to obtain deep inference features. The detection head performs lesion detection on the deep inference features to obtain the detection result.

[0008] On the other hand, embodiments of this application also provide a two-stage cascaded video lesion detection system, including: The image acquisition module is used to acquire nasopharyngeal endoscopy videos and extract endoscopic images from the nasopharyngeal endoscopy videos. The reference frame assembly module is used to select the target inference frame and multiple frames before and after the target inference frame to form a reference frame, using the endoscopic image at the current moment as the target inference frame. The prompt generation network consists of multiple sequentially connected Transformer encoders and a spatiotemporal attention predictor. The Transformer encoders include a multi-head attention mechanism and a feedforward network. The multiple Transformer encoders iteratively compute the feature embeddings of the reference frames to obtain deep spatiotemporal representations. In the temporal attention part, the spatiotemporal attention predictor generates temporal attention weights based on the deep spatiotemporal representations and weights and fuses the deep spatiotemporal representations according to the temporal attention weights to obtain enhanced features with temporal characteristics. In the spatial attention part, the spatiotemporal attention predictor uses a learnable query vector to reduce the dimensionality of the enhanced features to obtain video prompt information. The target detection network consists of multiple sequentially connected stages and a detection head. Each stage includes a dynamic cue enhancement module and a Transformer computation block. The target inference frame feature embedding and video cue information are input into the dynamic cue enhancement module. The dynamic cue enhancement module extracts inference frame features using gating parameters. The Transformer computation block iteratively calculates the inference frame features to obtain deep inference features. The detection head performs lesion detection on the deep inference features to obtain the detection result.

[0009] This application also provides a computer storage medium storing a plurality of computer instructions for causing a computer to execute the above-described method.

[0010] The two-stage cascaded video lesion detection method and system of this application have the following advantages: This method employs a two-stage cascaded Transformer architecture. A dynamic cue generator is designed in the cue generation network, generating discriminative cue information based on the spatiotemporal feature representation of the reference frame, effectively mitigating quality degradation phenomena such as dynamic blur, exposure imbalance, and reflection artifacts in the inference frame. A dynamic cue enhancement module is designed in the target detection network, fusing video cue information based on an adaptive differentiable gating mechanism and dynamically adjusting its gain on the inference frame, achieving efficient joint modeling of spatiotemporal features. Compared with existing methods, the proposed method, DynSTPN (Dynamic Spatiotemporal Cue Network), achieves the best balance between detection accuracy and efficiency on the publicly available ImageNet VID (Video Detection) dataset. On the self-built NLLVOD (Nasopharyngeal Lesion Video Detection) dataset for detecting four categories of lesions in the nasopharynx and larynx, it achieves the best detection accuracy of 79.6% and detection efficiency of 29.4 FPS, meeting the real-time requirements of clinical practice. This method provides a new approach for the development of nasopharyngeal endoscopic video lesion detection technology and has greater clinical practical value. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a flowchart of a two-stage cascaded video lesion detection method provided in an embodiment of this application.

[0013] Figure 2 This is an architecture diagram of the prompt generation network and the target detection network provided in the embodiments of this application.

[0014] Figure 3 This is an architecture diagram of the spatiotemporal attention predictor provided in an embodiment of this application.

[0015] Figure 4 This is an architecture diagram of the dynamic prompt enhancement module provided in an embodiment of this application.

[0016] Figure 5 The visualization comparison results of the various methods provided in the embodiments of this application on a self-built dataset are shown, wherein (a) includes uneven lighting and local lens blur degradation, and (b) includes strong dynamic blur degradation.

[0017] Figure 6 Heatmaps of each stage of the target detection phase in the DynSTPN-SwinTiny dataset provided for embodiments of this application. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] Figure 1 A flowchart illustrating a two-stage cascaded video lesion detection method provided in this application embodiment. This application embodiment provides a two-stage cascaded video lesion detection method, including: S100: Acquire nasopharyngeal endoscopy video and extract endoscopic images from the nasopharyngeal endoscopy video.

[0020] For example, after the endoscope acquires nasopharyngeal endoscopy video, it is transmitted to the computer in real time. The computer temporarily stores the video and then splits it frame by frame to obtain multiple endoscopic images. The order of these endoscopic images is the same as the order of each frame in the nasopharyngeal endoscopy video.

[0021] S110, using the endoscopic image at the current moment as the target inference frame, select the target inference frame and multiple frames before and after the target inference frame to form a reference frame.

[0022] For example, at the current moment t Target inference frame Centered on, select the location located at Before and after frames K Each frame constitutes a reference frame. .

[0023] Specifically, given the current inference frame It can construct time series sets This time series set is the reference frame, in which This is the sampling step size, default. =1, The preset window size, with an even value.

[0024] S120, the reference frame is input into the prompt generation network. The prompt generation network includes multiple Transformer encoders connected in sequence and a spatiotemporal attention predictor. The Transformer encoders include a multi-head attention mechanism and a feedforward network. The multiple Transformer encoders iteratively calculate the feature embeddings of the reference frame in sequence to obtain a deep spatiotemporal representation. In the temporal attention part, the spatiotemporal attention predictor generates temporal attention weights based on the deep spatiotemporal representation. The deep spatiotemporal representation is weighted and fused according to the temporal attention weights to obtain enhanced features with temporal characteristics. In the spatial attention part, the spatiotemporal attention predictor uses a learnable query vector to reduce the dimensionality of the enhanced features to obtain video prompt information.

[0025] For example, in the prompt generation network, Input into the backbone network for capture K The temporal correlations and spatial features between reference frames are compressed along the spatiotemporal dimension using STAP (Spatiotemporal Attention Predictor) to extract the most discriminative video cue information from the current reference frame. .

[0026] Specifically, such as Figure 2 As shown, the prompt generation network also includes a patch partitioning layer and a linear embedding layer. The patch partitioning layer divides the reference frame into multiple non-overlapping patches, and the linear embedding layer constructs the patches into feature embeddings for computation by the first Transformer encoder.

[0027] The patch partitioning layer divides the input reference frame into... Non-overlapping facets ,in, The first division obtained k A piece of dough, It is a set of positive integers. For the spatial representation of the three-dimensional real tensor of the patch, and Representing the height and width of the patch, the linear embedding layer constructs the patch into feature embeddings for computation by the Transformer encoder. , It is a two-dimensional real tensor. N 0 represents the initial number of feature embeddings. d0 represents the feature embedding depth. Each Transformer encoder contains a multi-head attention mechanism (MSA) and a feedforward network (FFN), and its computation process can be represented as follows:

[0028] in, For the first i Feature embeddings output by the layer Transformer encoder LN (·) indicates layer normalization. MSA (·) indicates multi-head attention mechanism processing. FFN (·) indicates feedforward network processing. L This represents the total number of layers in the Transformer encoder.

[0029] In this embodiment, the multi-head attention mechanism is used to capture multi-dimensional correlation information of input features in parallel. Its structure and implementation steps are as follows: Input Transformation Unit: Receives the feature tensor to be processed. X The query matrix is ​​generated through three independent linear transformation layers. Q Key matrix K Value matrix V ,in Q , K , V The dimensions of each are the same as those of the input feature tensor. X Consistent; Multi-head splitting unit: This unit splits the query matrix... Q Key matrix K Value matrix V Each is divided into h’ Submatrices of equal dimensions Q i’ , K i’ , V i’ , i’ ∈[1, h’ ], h’ The preset number of heads; Single-head attention computation unit: for each submatrix Q i’ , K i’ , V i’ Perform scaling dot product attention operation: calculate Q i’ , K i’ The dot product of the transpose of the matrix, divided by ,in forK i’ The dimension is adjusted to mitigate the gradient vanishing problem caused by excessively large dot product results; a Softmax activation operation is performed on the scaled dot product results to obtain an attention weight matrix, which characterizes the correlation strength between feature locations; the attention weight matrix is ​​then compared with... V i’ Perform a weighted summation to obtain the single-head attention output; Multi-head integration unit: h’ The single-head attention outputs are concatenated and then mapped to the model dimension through a set of linear transformation layers. This yields the final output of the multi-head attention mechanism.

[0030] The function of the multi-head attention mechanism is to improve the global representation ability of features by simultaneously capturing the correlation information of input features under different semantic dimensions through parallel attention calculation of multiple subspaces.

[0031] In this embodiment, the feedforward network is used to nonlinearly enhance the output features of the multi-head attention mechanism, and it consists of an up-dimensional transformation unit, a nonlinear activation unit, and an down-dimensional transformation unit. Its main purpose is to receive the output features of the multi-head attention mechanism (dimension 1). The features are mapped to a higher dimension through a linear transformation layer, and then a non-linear activation operation is performed on the upgraded features to introduce non-linear transformation capabilities. Finally, a second linear transformation layer maps the activated features back to the previous dimension. This yields the final output of the feedforward network.

[0032] In this embodiment, the Transformer computation block is used to capture global and local dependencies between features and enhance the nonlinear expressive power of features. The patch fusion layer aggregates local spatial information by changing the number of patches and feature dimensions, thereby reducing the high computational complexity caused by Transformer attention computation.

[0033] through L After iterative computation by the layer Transformer encoder, the deep spatiotemporal representation is finally obtained. ,in, It is a two-dimensional real tensor. N L The number of feature embeddings in layer L. d L Let L be the feature embedding depth. Since the number of reference frames is equal to the number of target inference frames... K The data is several times larger, and there is significant data redundancy within a short time window, therefore it is necessary to... Feature filtering is performed to generate discriminative video prompts for the inference frames. .

[0034] Furthermore, methods for generating temporal attention weights based on deep spatiotemporal representations in spatiotemporal attention predictors include: Max pooling and average pooling operations are performed on the deep spatiotemporal representation to obtain global context features and local detail features, respectively. The aggregated features are obtained by weighting and fusing global context features and local detail features using learnable weight parameters. The aggregated features are averaged along the channel dimension, and then the temporal attention weights are generated using the Sigmoid activation function.

[0035] Specifically, such as Figure 3 As shown, given a deep spatiotemporal representation In the time attention part, parallel max pooling and average pooling operations are first performed, and then learnable weight parameters are set. α and β During training, optimization is performed along the gradient descent direction. Max pooling and average pooling are used to dynamically balance global contextual information and local detail features. The mathematical expression for this is:

[0036] in, As an aggregation feature, and These are max pooling and average pooling, respectively.

[0037] Based on this, firstly Average pooling is performed along the channel dimension, followed by the application of the Sigmoid activation function to generate temporal attention weights.

[0038] Furthermore, the spatiotemporal attention predictor utilizes learnable query vectors to reduce the dimensionality of enhanced features, and methods for obtaining video prompt information include: Two different linear layers are used to generate the K matrix and V matrix respectively for the enhanced features; Multiply the query vector by the K matrix to obtain the spatial attention weights; The spatial attention weights are normalized, then weighted and calculated with the V matrix, and finally passed through a linear layer to generate video prompt information.

[0039] Specifically, the obtained temporal attention weights are weighted and fused with the original deep spatiotemporal representation, and then integrated through a residual connection mechanism to finally output enhanced features with temporal characteristics. The calculation process can be expressed as follows:

[0040] in, As time weight, To take the average, To activate using the Sigmoid function.

[0041] In the spatial attention part, a multi-head attention mechanism is used as the core component, which is trained by query vectors. Through forward computation, loss feedback, gradient backpropagation, and parameter updates, a mapping relationship adapted to the task is learned. This allows the input features, after a linear transformation, to effectively retrieve relevant information from the sequence. Dimensionality reduction and extraction of spatial semantic features, among which It is a two-dimensional real tensor. N p The reference frame indicates the number of feature embeddings. d The reference frame is used to indicate the feature embedding depth. Generated using two different linear layers K Matrix and V matrix. Q and K Matrix multiplication produces attention weights:

[0042] in, For dimension Two-dimensional tensor, THW Characteristic matrix T , H , W The product of three dimensions, For spatial attention weights.

[0043] After normalizing the spatial attention weights using the Softmax function, a weighted average is calculated with the V matrix and then fed into the linear layer to generate video cues. :

[0044] in, For the SoftMax function, This is for linearization.

[0045] S130, the video prompt information and the target inference frame are input into the target detection network. The target detection network includes multiple sequentially connected stages and a detection head. Each stage contains a dynamic prompt enhancement module and a Transformer computation block. The feature embedding of the target inference frame and the video prompt information are input into the dynamic prompt enhancement module. The dynamic prompt enhancement module extracts the features of the inference frame using gating parameters. The Transformer computation block iteratively calculates the features of the inference frame to obtain deep inference features. The detection head performs lesion detection on the deep inference features to obtain the detection result.

[0046] For example, in an object detection network, the current inference frame is obtained. The feature embedding will prompt the generator network to produce and The feature embeddings are pre-concatenated and used as the raw input data for the feature extraction part of the backbone network. Before executing each Transformer computation block, an independent dynamic cueing enhancement module is used to dynamically and adaptively fuse the input sequence to prevent the loss of reference frame feature information during deep computation. Finally, the deep inference features generated by the entire network architecture are applied to the detection head to produce the target inference frame. The test results.

[0047] Specifically, to address the performance degradation caused by the fixed-ratio reference frame video cue information fusion method in deep networks, this application cascades a dynamic cue enhancement module before each Transformer computation block. This module first performs feature reorganization on the preset video cues and dynamically adjusts the information fusion ratio through a differentiable gating mechanism. Then, it uses gating parameters... The gain of the reference frame relative to the target inference frame is adaptively adjusted along the gradient descent direction during training. Simultaneously, for the multi-stage pyramid architecture of the Swin Transformer, cross-stage feature expansion is performed to achieve dimensional alignment. Finally, the detection results are output through a detection head composed of a region proposal network head and a bounding box prediction head. The region proposal network head predicts the binary classification score of each anchor box (whether it is an object) using cross-entropy loss, and then... L1 The loss predicts the position offset of the anchor boxes, thus outputting anchor boxes at different scales, corresponding to the initial candidate regions of the target to be detected. The bounding box prediction head is responsible for classifying and refining the box positions of the candidate boxes generated by the region proposal network head, outputting the final detection category and accurate bounding boxes.

[0048] Furthermore, such as Figure 4 As shown, the method by which the dynamic cue enhancement module extracts inference frame features using gating parameters includes: The feature embedding of the target inference frame or the deep inference features output from the previous stage are split into inference frame feature embedding and corresponding reference frame hints; The gating parameters are activated using the Sigmoid activation function, multiplied by the video prompt information, added to the reference frame prompt, and then concatenated with the inference frame feature embedding to obtain the inference frame features of the current stage.

[0049] Specifically, given the target detection network stage, the Transformer backbone network computation block number... i -1 layer output inference frame features Based on the prior condition of fixed position encoding, the reference frame cue corresponding to this layer can be accurately extracted. The dynamic propagation of prompts between levels is achieved through a gated adaptive fusion mechanism. The calculation process of the Dynamic Prompt Enhancement (DPE) module is as follows:

[0050] in, This represents the embedding of inference frame features after removing video prompt information. This indicates a concatenation operation along the sequence dimension. For the first i Features of the inference frame output by the layer.

[0051] The processing procedure for the Transformer computation block in this step is the same as that for the Transformer computation block in the prompt generation network. The dynamic prompt enhancement module and the Transformer computation block are combined to form the feature fusion Transformer computation block. In each stage, a patch fusion layer is also connected after the feature fusion Transformer computation block, and its processing procedure is the same as that for the patch fusion layer in the prompt generation network.

[0052] Furthermore, the object detection network includes a patch partitioning layer and a linear embedding layer. The patch partitioning layer divides the target inference frame into multiple non-overlapping patches, and the linear embedding layer constructs basic feature embeddings from the patches. Then, the basic feature embeddings and video cue information are concatenated to form the feature embedding of the target inference frame used for computation in the first stage.

[0053] Specifically, target inference frame Basic feature embeddings are obtained after the first layer of network encoding. Then, with the video prompt information Perform pre-assembly operations to form the initialization input of the DPE module. :

[0054] in, This represents the number of feature tokens in the target inference frame. The number of tokens for the video prompt message. For feature dimensions.

[0055] Furthermore, after the prompt generation network outputs video prompt information, the video prompt information is subjected to feature copying and concatenation processing to generate dimension-aligned prompts that match each stage, which are then spliced ​​with the inference frame features of each stage to jointly participate in the computation of the Transformer block.

[0056] Specifically, given that the multi-stage pyramid architecture of the Swin Transformer leads to feature dimension downsampling, this application performs a cross-stage feature expansion operation at the input of each Transformer computation block: for video cue information... Perform feature copying and cascading to generate a model adapted to the current stage. Dimensional alignment hints :

[0057] in, This indicates a feature copying operation along a spatial dimension.

[0058] As can be seen, the two Transformer networks in this application adopt a two-stage cascaded architecture. The prompt generation network is responsible for spatiotemporal feature modeling, while the object detection network performs feature enhancement and task inference. The two achieve end-to-end optimization through a dynamic feature interaction mechanism to perform reference frame feature extraction and inference frame object detection tasks.

[0059] This application uses The function is used to optimize the loss between the noise predicted by the model during the backward process and the Gaussian noise added during the forward process. The formula for calculating the function is:

[0060] in, It is a predicted value. Compared with the true value difference.

[0061] During training, ResNet101 and Swin-Tiny were used as the backbone networks, and AdamW was selected as the optimizer. The base learning rate was set to 2.5e-5, and the gating parameters were... g The learning rate scaling factor was 10×. The experimental environment was configured as follows: NVIDIA RTX 3090 GPU, Python 3.8, PyTorch 1.10.0 and CUDA 11.3 framework.

[0062] This application also provides a two-stage cascaded video lesion detection system, including: The image acquisition module is used to acquire nasopharyngeal endoscopy videos and extract endoscopic images from the nasopharyngeal endoscopy videos. The reference frame assembly module is used to select the target inference frame and multiple frames before and after the target inference frame to form a reference frame, using the endoscopic image at the current moment as the target inference frame. The prompt generation network consists of multiple sequentially connected Transformer encoders and a spatiotemporal attention predictor. The Transformer encoders include a multi-head attention mechanism and a feedforward network. The multiple Transformer encoders iteratively compute the feature embeddings of the reference frames to obtain deep spatiotemporal representations. In the temporal attention part, the spatiotemporal attention predictor generates temporal attention weights based on the deep spatiotemporal representations and weights and fuses the deep spatiotemporal representations according to the temporal attention weights to obtain enhanced features with temporal characteristics. In the spatial attention part, the spatiotemporal attention predictor uses a learnable query vector to reduce the dimensionality of the enhanced features to obtain video prompt information. The target detection network consists of multiple sequentially connected stages and a detection head. Each stage includes a dynamic cue enhancement module and a Transformer computation block. The target inference frame feature embedding and video cue information are input into the dynamic cue enhancement module. The dynamic cue enhancement module extracts inference frame features using gating parameters. The Transformer computation block iteratively calculates the inference frame features to obtain deep inference features. The detection head performs lesion detection on the deep inference features to obtain the detection result.

[0063] This application also provides a computer storage medium storing a plurality of computer instructions for causing a computer to execute the above-described method.

[0064] Table 1 compares the performance metrics of various existing video object detection methods trained on the public ImageNet VID dataset using the default configuration of the original codebase. In the table, bold text indicates the best results, underlined text indicates the second-best results, and Ours represents the method proposed in this application.

[0065] Table 1 shows the comparison results of each method on the public dataset ImageNet VID.

[0066] In Table 1, SELSA is a sequence-level semantic aggregation video object detection algorithm, RDN is a relation distillation network video object detection algorithm, MEGA is a memory-enhanced global-local aggregation video object detection algorithm, and Faster R-CNN is a faster region-based convolutional neural network.

[0067] To compare the model's effectiveness in detecting lesions under nasopharyngoscopy, the experiment collected original nasopharyngeal endoscopy videos of 851 patients examined at two tertiary hospitals across different regions (January 6, 2024 to June 3, 2024, and April 7, 2025 to May 26, 2025) according to the data collection and annotation standards recognized in the expert consensus on the quality control system for endoscopic artificial intelligence data collection and annotation. Under the guidance of three professional physicians, 270 patient videos with clearly defined visual lesion features were selected. To ensure the privacy and security of medical imaging data, all original video data underwent anonymization processing, removing all sensitive information involving patient privacy. This resulted in 438 lesion video segments, encompassing four different types of lesions: neoplasms, vocal cord leukoplakia, cysts, and adenoid hyperplasia. The video sequence length ranged from a minimum of 33 frames to a maximum of 779 frames, with an average of 185.37 frames per video and a median of 156.00 frames per video. The number of video lesion samples for the four types of lesions were as follows: 91 segments of neoplasms, 47 segments of vocal cord leukoplakia, 76 segments of cysts, and 43 segments of adenoid hyperplasia, totaling 262 lesion samples from different patients. Under the guidance of three nasopharyngeal specialists, and referring to the specific anatomical structure definitions and standards in "Clinical Application of Electronic Laryngoscopy - Atlas of Narrow-Band Imaging Endoscopy for Nasopharyngeal Tumors," 438 lesion video segments were annotated frame by frame, and 131 lesion-free negative samples from 64 normal examinees were added. This resulted in the nasopharyngeal endoscopic video lesion detection dataset NLLVOD (Nasopharyngeal-Laryngeal Lesion Video Object Detection).

[0068] Table 2 shows the performance metrics of various methods on the self-built nasopharyngeal endoscopy video lesion detection dataset NLLVOD. Bold text indicates the best results, underlined text indicates the second-best results, and Ours represents the method proposed in this application. In Table 2, DEIM-S is a smaller version of the improved matching DETR fast convergence model, where DETR stands for Detection Transformer.

[0069] Table 2 shows the comparison results of various methods on the self-built nasopharyngeal endoscopy video lesion detection dataset.

[0070] Table 3 shows the comparison of the detection capabilities of each method on the self-built nasopharyngeal endoscopy video lesion detection dataset for each lesion. In the table, bold text indicates the best results, underlined text indicates the second best results, and Ours refers to the method proposed in this application.

[0071] Table 3 shows the comparison results of various methods on the self-built nasopharyngeal endoscopy video lesion detection dataset.

[0072] Table 1 shows the performance comparison results of existing video object detection methods trained based on the default configuration of the original codebase on the publicly available large-scale ImageNet VID dataset. The DynSTPN method shows a significant performance advantage. Specifically, the SwinBase version of DynSTPN achieves an mAP (mean accuracy) of 83.9%, which is an improvement of 3.6%, 7.6%, and 5.5% compared to baseline Faster R-CNN models using the ResNet-101 architecture (e.g., SELSA 80.3%, RDN 76.3%, and MEGA 78.4%). Compared to Diffusion VID, the mAP is only 3.0% lower, but the computational efficiency is higher. In terms of processing speed, the SwinBase version reaches 17.8 FPS, showing better inference efficiency compared to SELSA (11 FPS) and Diffusion VID (15.7 FPS). Similarly, the SwinTiny version of DynSTPN also performs well. When using SwinTiny as the backbone, DynSTPN achieves 82.9% mAP while maintaining comparable detection efficiency, surpassing the baseline method STPN (80.6%). This demonstrates that the method can effectively balance accuracy and speed in a lightweight architecture.

[0073] As shown in Table 2, in the self-built nasopharyngeal lesion detection video dataset, the proposed DynSTPN achieved the best detection result with an mAP of 79.6, which is 4.6 percentage points higher than the benchmark method STPN. Furthermore, since the proposed two-module design does not add any additional parameters, DynSTPN maintains the original detection efficiency while achieving performance improvement. Compared with advanced static image detectors YOLOv12 and DEIM, existing video object detection methods, while improving mAP, are usually accompanied by a significant increase in computational cost. The proposed DynSTPN-SwinTiny outperforms YOLOv12-M by 11.3 percentage points and DEIM-S by 15.1 percentage points in mAP, while maintaining a computational efficiency of 29.4 FPS. This result demonstrates that in the application scenario of nasopharyngeal endoscopy lesion detection, DynSTPN not only significantly surpasses static image detection methods in detection accuracy, but its inference speed of 29.4 frames per second also fully meets the real-time requirements of nasopharyngeal endoscopy clinical image inference (≥25 FPS, local storage frame rate of 25 FPS).

[0074] As can be seen, the DynSTPN proposed in this application achieves high detection accuracy while maintaining fast inference speed in video target detection tasks, realizing an effective balance between accuracy and efficiency, and providing a technical solution with practical application value for this field.

[0075] exist Figure 5 In the image, each sub-image consists of the complete image above and a magnified local detection result below, with the magnified area marked with a red box in the complete image. The numbers in the sub-image represent the optimal detection confidence score of the method for the current lesion region in the image; a larger number indicates higher detection accuracy. Figure 6 The heatmaps in the image show the model's attention levels at different locations, with red indicating higher attention levels and blue indicating lower attention levels. The first row of each subplot represents instances where the prompt information extracted from the reference frame is not used, and the second row represents instances where the prompt information extracted from the reference frame is used.

[0076] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0077] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A two-stage cascaded video lesion detection method, characterized in that, include: Acquire nasopharyngeal endoscopy video and extract endoscopic images from the nasopharyngeal endoscopy video; Using the endoscopic image at the current moment as the target inference frame, a reference frame is formed by selecting the target inference frame and multiple frames before and after the target inference frame. The reference frame is input into the prompt generation network, which includes multiple Transformer encoders connected in sequence and a spatiotemporal attention predictor. The Transformer encoders include a multi-head attention mechanism and a feedforward network. The multiple Transformer encoders iteratively calculate the feature embeddings of the reference frame in sequence to obtain a deep spatiotemporal representation. The spatiotemporal attention predictor generates temporal attention weights based on the deep spatiotemporal representation in the temporal attention part, and then weights and fuses the deep spatiotemporal representation according to the temporal attention weights to obtain an enhanced feature with temporal characteristics. In the spatial attention section, the spatiotemporal attention predictor uses a learnable query vector to reduce the dimensionality of the enhanced features to obtain video prompt information; The video prompt information and the target inference frame are input into the target detection network, which includes multiple sequentially connected stages and a detection head. Each stage contains a dynamic prompt enhancement module and a Transformer computation block. The feature embedding of the target inference frame and the video prompt information are input into the dynamic prompt enhancement module. The dynamic prompt enhancement module extracts inference frame features using gating parameters. The Transformer computation block iteratively calculates the inference frame features to obtain deep inference features. The detection head performs lesion detection on the deep inference features to obtain the detection result.

2. The dual-stage cascaded video lesion detection method according to claim 1, characterized in that, The prompt generation network further includes a patch partitioning layer and a linear embedding layer. The patch partitioning layer divides the reference frame into multiple non-overlapping patches, and the linear embedding layer constructs the patches into feature embeddings for computation by the first Transformer encoder.

3. The dual-stage cascaded video lesion detection method according to claim 1, characterized in that, The method for generating the temporal attention weights based on the deep spatiotemporal representation by the spatiotemporal attention predictor includes: Max pooling and average pooling operations are performed on the deep spatiotemporal representation to obtain global context features and local detail features, respectively. The global context features and the local detail features are weighted and fused using learnable weight parameters to obtain aggregated features; The aggregated features are subjected to average pooling along the channel dimension, and then the temporal attention weights are generated using the Sigmoid activation function.

4. The dual-stage cascaded video lesion detection method according to claim 1, characterized in that, The spatiotemporal attention predictor uses a learnable query vector to reduce the dimensionality of the enhanced features to obtain video prompt information. The method includes: The enhanced features are generated using two different linear layers to produce the K matrix and V matrix, respectively; Multiply the query vector by the K matrix to obtain the spatial attention weights; The spatial attention weights are normalized, then weighted and calculated with the V matrix, and finally passed through a linear layer to generate the video prompt information.

5. The dual-stage cascaded video lesion detection method according to claim 1, characterized in that, The method by which the dynamic prompt enhancement module extracts inference frame features using gating parameters includes: The feature embedding of the target inference frame or the deep inference features output from the previous stage are split into inference frame feature embedding and corresponding reference frame hints; The gating parameters are activated using the Sigmoid activation function, multiplied by the video prompt information, added to the reference frame prompt, and then concatenated with the inference frame feature embedding to obtain the inference frame feature of the current stage.

6. The dual-stage cascaded video lesion detection method according to claim 5, characterized in that, The target detection network includes a patch partitioning layer and a linear embedding layer. The patch partitioning layer divides the target inference frame into multiple non-overlapping patches. The linear embedding layer constructs basic feature embeddings from the patches. Then, the basic feature embeddings and the video prompt information are concatenated to form the feature embedding of the target inference frame for the first stage of computation.

7. The dual-stage cascaded video lesion detection method according to claim 1, characterized in that, After the prompt generation network outputs the video prompt information, the video prompt information is subjected to feature copying and cascading processing to generate dimension-aligned prompts that match each stage.

8. A system applying the dual-stage cascaded video lesion detection method according to any one of claims 1-7, characterized in that, include: The image acquisition module is used to acquire nasopharyngeal endoscopy videos and extract endoscopic images from the nasopharyngeal endoscopy videos. The reference frame assembly module is used to select the target inference frame and multiple frames before and after the target inference frame as the target inference frame, using the endoscopic image at the current moment as the target inference frame. The prompt generation network includes multiple Transformer encoders connected in sequence and a spatiotemporal attention predictor. The Transformer encoders include a multi-head attention mechanism and a feedforward network. The multiple Transformer encoders iteratively calculate the feature embeddings of the reference frame in sequence to obtain a deep spatiotemporal representation. The spatiotemporal attention predictor generates temporal attention weights based on the deep spatiotemporal representation in the temporal attention part, and weights and fuses the deep spatiotemporal representation according to the temporal attention weights to obtain enhanced features with temporal characteristics. In the spatial attention section, the spatiotemporal attention predictor uses a learnable query vector to reduce the dimensionality of the enhanced features to obtain video prompt information; The target detection network comprises multiple sequentially connected stages and a detection head. Each stage includes a dynamic cue enhancement module and a Transformer computation block. The feature embedding of the target inference frame and the video cue information are input to the dynamic cue enhancement module. The dynamic cue enhancement module extracts inference frame features using gating parameters. The Transformer computation block iteratively calculates the inference frame features to obtain deep inference features. The detection head performs lesion detection on the deep inference features to obtain the detection result.

9. A computer storage medium, characterized in that, The computer storage medium stores a plurality of computer instructions, which are used to cause the computer to perform the method described in any one of claims 1-7.

Images