A two-stage cascaded video lesion detection method and system

Abstract

The application discloses a two-stage cascaded video lesion detection method and system, relates to the technical field of image target recognition, and comprises the following steps: acquiring an endoscope image; selecting a reference frame; iteratively calculating feature embedding of the reference frame to obtain deep space-time representation; generating time attention weight based on the deep space-time representation, and weighting and fusing the deep space-time representation to obtain enhanced features; dimensionally reducing the enhanced features to obtain video prompt information; extracting feature embedding of a target inference frame and inference frame features in the video prompt information, iteratively calculating the inference frame features to obtain deep inference features, and performing lesion detection on the deep inference features to obtain a detection result. The method adopts a two-stage cascaded Transformer architecture, effectively alleviates quality degradation phenomena such as dynamic blur, exposure imbalance and reflection artifacts of the inference frame, and realizes efficient joint modeling of space-time features.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image target recognition, and particularly relates to a method for recognizing lesions in nasopharyngolaryngoscope video by using a two-stage cascaded Transformer network. BACKGROUND

[0002] Head and neck cancer ranks sixth in the global incidence of malignant tumors, covering oral cancer, nasopharyngeal cancer, oropharyngeal cancer, laryngeal cancer and hypopharyngeal cancer subtypes. Nasopharyngolaryngoscopy, as an important non-invasive technique in clinical practice for finding lesions and evaluating lesion extent, can analyze the microstructure of the nasopharynx in multiple dimensions, and has important clinical value in differentiating precancerous lesions such as intraepithelial neoplasia, detecting pathological features of carcinoma in situ, and quantitatively evaluating tumor infiltration and anatomical levels.

[0003] However, nasopharyngolaryngoscopy faces many real-world challenges. On the one hand, the structure of the throat is complex and covered by multiple layers of mucosa, resulting in low contrast of the lesion tissue and blurred boundary shape. In addition, there is an overlap in morphological features between common lesions such as adenoid hyperplasia and cysts and early throat cancer, and there are large differences in interpretation among doctors with different years of experience. Misdiagnosis and misdiagnosis are common. On the other hand, endoscopic imaging is limited by single-point light source illumination, and the quality of the endoscopic video is generally degraded by motion blur, exposure imbalance, reflection artifacts and other problems, which seriously damages the integrity of feature expression. At the same time, the traditional detection method uses the process of "video frame splitting-single frame detection-result fusion", which does not effectively utilize the spatiotemporal continuity features of the video, and the inference speed is limited by the frame splitting and frame combining efficiency, making it difficult to meet the real-time needs of the clinic, i.e. frame rate FPS≥24.

[0004] Breakthroughs in the field of medical artificial intelligence have driven the clinical popularization of computer-aided diagnosis systems, and deep learning frameworks have been successfully applied to nasopharyngeal lesion multi-classification, vocal cord cancer detection and other scenarios, but the end-to-end detection architecture for nasopharyngolaryngoscope video still needs to be improved, and current research is still limited to discrete frame analysis, making it difficult to break through the technical bottleneck.

[0005] Therefore, it is necessary to propose a new lesion detection method to solve the problem of limited detection performance caused by video quality degradation and insufficient utilization of spatiotemporal features in nasopharyngolaryngoscopy. SUMMARY

[0006] The embodiments of the present application provide a two-stage cascaded video lesion detection method and system to solve the problems in the prior art.

[0007] In one aspect, the embodiments of the present application provide a two-stage cascaded video lesion detection method, comprising:

[0008] Obtaining a nasopharyngolaryngoscope video and extracting endoscopic images in the nasopharyngolaryngoscope video;

[0009] taking the endoscopic image at the current moment as a target reasoning frame, selecting the target reasoning frame and multiple frames before and after the target reasoning frame to form a reference frame;

[0010] inputting the reference frame into a prompt generation network, the prompt generation network comprising multiple sequentially connected Transformer encoders and a spatio-temporal attention predictor, the Transformer encoders comprising multi-head attention mechanisms and feedforward networks, the multiple Transformer encoders sequentially performing iterative calculation on feature embeddings of the reference frame to obtain deep spatio-temporal representations; the spatio-temporal attention predictor generating time attention weights based on the deep spatio-temporal representations in a time attention part, weighting and fusing the deep spatio-temporal representations according to the time attention weights to obtain enhanced features with time sequence characteristics; and the spatio-temporal attention predictor using a learnable query vector to reduce the dimension of the enhanced features in a spatial attention part to obtain video prompt information;

[0011] inputting the video prompt information and the target reasoning frame into a target detection network, the target detection network comprising multiple sequentially connected stages and a detection head, each stage comprising a dynamic prompt enhancement module and a Transformer calculation block, the feature embeddings of the target reasoning frame and the video prompt information being input into the dynamic prompt enhancement module, the dynamic prompt enhancement module extracting reasoning frame features using gating parameters, the Transformer calculation block performing iterative calculation on the reasoning frame features to obtain deep reasoning features, and the detection head performing lesion detection on the deep reasoning features to obtain a detection result.

[0012] On the other hand, the embodiments of the present application also provide a two-stage cascaded video lesion detection system, comprising:

[0013] an image acquisition module configured to acquire a nasopharyngolaryngoscope video and extract endoscopic images in the nasopharyngolaryngoscope video;

[0014] a reference frame construction module configured to take an endoscopic image at a current moment as a target reasoning frame, select the target reasoning frame and multiple frames before and after the target reasoning frame to form a reference frame;

[0015] a prompt generation network, the prompt generation network comprising multiple sequentially connected Transformer encoders and a spatio-temporal attention predictor, the Transformer encoders comprising multi-head attention mechanisms and feedforward networks, the multiple Transformer encoders sequentially performing iterative calculation on feature embeddings of the reference frame to obtain deep spatio-temporal representations; the spatio-temporal attention predictor generating time attention weights based on the deep spatio-temporal representations in a time attention part, weighting and fusing the deep spatio-temporal representations according to the time attention weights to obtain enhanced features with time sequence characteristics; and the spatio-temporal attention predictor using a learnable query vector to reduce the dimension of the enhanced features in a spatial attention part to obtain video prompt information;

[0016] The target detection network comprises a plurality of stages connected in sequence and a detection head, each stage comprises a dynamic prompt enhancement module and a Transformer calculation block, feature embedding of a target inference frame and video prompt information are input into the dynamic prompt enhancement module, the dynamic prompt enhancement module extracts inference frame features by using a gating parameter, the Transformer calculation block iteratively calculates the inference frame features to obtain deep inference features, and the detection head performs lesion detection on the deep inference features to obtain a detection result.

[0017] The embodiment of the application further provides a computer storage medium, which stores a plurality of computer instructions for causing a computer to execute the method.

[0018] The two-stage cascaded video lesion detection method and system provided in the application have the following advantages:

[0019] The method adopts a two-stage cascaded Transformer architecture, designs a dynamic prompt generator in a prompt generation network, generates discriminative prompt information based on the spatiotemporal feature representation of a reference frame, effectively alleviates the quality degradation phenomena such as dynamic blur, exposure imbalance and reflection artifacts of an inference frame, designs a dynamic prompt enhancement module in a target detection network, fuses video prompt information based on an adaptive differentiable gating mechanism and dynamically adjusts the gain degree of the inference frame, and realizes efficient joint modeling of spatiotemporal features. Compared with existing methods, the method DynSTPN (dynamic spatiotemporal prompt network) of the application achieves the best balance between detection accuracy and efficiency on the public dataset ImageNet VID (video object detection), and achieves the optimal detection accuracy 79.6% and detection efficiency 29.4FPS on the self-built four-class lesion detection video dataset NLLVOD (nasopharyngeal laryngeal lesion video object detection), which meets the real-time requirement of the clinic. The method provides a new idea for the development of nasopharyngeal laryngeal endoscopy video lesion detection technology and has higher clinical practical value. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. Obviously, the drawings in the following description are only some embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0021] Figure 1 A flowchart of a two-stage cascaded video lesion detection method provided by the embodiment of the application.

[0022] Figure 2An architectural diagram of the hint generation network and the target detection network provided by the embodiments of the present application.

[0023] Figure 3 An architectural diagram of the spatio-temporal attention predictor provided by the embodiments of the present application.

[0024] Figure 4 An architectural diagram of the dynamic hint enhancement module provided by the embodiments of the present application.

[0025] Figure 5 The visual comparison results of various methods provided by the embodiments of the present application on a self-built data set, wherein (a) contains uneven illumination and local lens blur degradation phenomena, and (b) contains strong dynamic blur degradation phenomena.

[0026] Figure 6 The heat map of each stage of the target detection stage of DynSTPN-SwinTiny provided by the embodiments of the present application under the self-built nasopharyngeal laryngoscope data set. DETAILED DESCRIPTION

[0027] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0028] Figure 1 A flowchart of a two-stage cascaded video lesion detection method provided by the embodiments of the present application. The embodiments of the present application provide a two-stage cascaded video lesion detection method, which comprises:

[0029] S100, acquiring a nasopharyngeal laryngoscope video and extracting endoscope images in the nasopharyngeal laryngoscope video.

[0030] Exemplarily, after the endoscope collects the nasopharyngeal laryngoscope video, the nasopharyngeal laryngoscope video is transmitted to a computer in real time, temporarily stored by the computer, and then split frame by frame to obtain multiple endoscope images, and the order of the endoscope images is the same as the order of each frame of image in the nasopharyngeal laryngoscope video.

[0031] S110, taking the endoscope image at the current time as a target inference frame, selecting the target inference frame and multiple frames before and after the target inference frame to form a reference frame.

[0032] Exemplarily, taking the target inference frame at the current time t as the center, selecting frames before and after the target inference frame to form a reference frame . K .​

[0033] Specifically, given the current inference frame , a temporal set can be constructed , the temporal set is the reference frame, wherein is the sampling step, by default =1, is the preset window size, which is an even number.

[0034] S120, input the reference frame into the prompt generation network, the prompt generation network includes a plurality of sequentially connected Transformer encoders and a space-time attention predictor, the Transformer encoder includes a multi-head attention mechanism and a feedforward network, and the plurality of Transformer encoders perform iterative calculation on feature embeddings of the reference frame to obtain deep space-time representations; the space-time attention predictor generates time attention weights based on the deep space-time representations in the time attention part, and fuses the deep space-time representations according to the time attention weights to obtain enhanced features with time sequence characteristics; in the spatial attention part, the space-time attention predictor uses a learnable query vector to reduce the dimension of the enhanced features to obtain video prompt information.

[0035] Exemplarily, in the prompt generation network, is input into the backbone network to capture K the temporal correlation and spatial feature information contained between the reference frames, and the STAP (space-time attention predictor) compresses the joint representation of the reference frames along the space-time dimension to extract the most discriminative video prompt information in the current reference frame .

[0036] Specifically, as shown in Figure 2 , the prompt generation network further includes a patch division layer and a linear embedding layer, the patch division layer divides the reference frame into a plurality of non-overlapping patches, and the linear embedding layer constructs the patches into feature embeddings for calculation by the first Transformer encoder.

[0037] The patch division layer divides the input reference frame into non-overlapping patches , wherein is the k th patch obtained by division, is a set of positive integers, is a spatial representation of the patch three-dimensional real tensor, and represent the height and width of the patch, and the linear embedding layer constructs the patch into a feature embedding for calculation by the Transformer encoder , N0 is the initial feature embedding number, d 0 is the feature embedding depth. Each Transformer encoder contains a multi-head attention mechanism (MSA) and a feed-forward network (FFN), and the calculation process can be represented as:

[0038]

[0039] wherein, is the feature embedding output by the i-th i layer Transformer encoder, LN (·) represents layer normalization, MSA (·) represents multi-head attention processing, FFN (·) represents feed-forward network processing, L is the total number of layers of the Transformer encoder.

[0040] In the embodiments of the present application, the multi-head attention mechanism is used to capture multi-dimensional associated information of the input features in parallel, and the structure and implementation steps are as follows:

[0041] Input transformation unit: receives the feature tensor to be processed X , and generates query matrix Q , key matrix K , and value matrix V through three independent linear transformation layers, wherein Q , K , V The dimensions of X ,

[0042] Multi-head splitting unit: divides the query matrix Q , key matrix K , and value matrix V into h’ sub-matrices Q i’ , K i’ , V i’ , i’ ∈[1, h’ ], h’ is the preset number of heads;

[0043] Single-head attention calculation unit: for each group of sub-matrices Q i’ , K i’ , V i’ , perform scaled dot-product attention operation: calculate Q i’ ,K i’ The dot product of the transpose of the matrix, divided by ,in for K i’ The dimension is adjusted to mitigate the gradient vanishing problem caused by excessively large dot product results; a Softmax activation operation is performed on the scaled dot product results to obtain an attention weight matrix, which characterizes the correlation strength between feature locations; the attention weight matrix is ​​then compared with... V i’ Perform a weighted summation to obtain the single-head attention output;

[0044] Multi-head integration unit: h’ The single-head attention outputs are concatenated and then mapped to the model dimension through a set of linear transformation layers. This yields the final output of the multi-head attention mechanism.

[0045] The function of the multi-head attention mechanism is to improve the global representation ability of features by simultaneously capturing the correlation information of input features under different semantic dimensions through parallel attention calculation of multiple subspaces.

[0046] In this embodiment, the feedforward network is used to nonlinearly enhance the output features of the multi-head attention mechanism, and it consists of an up-dimensional transformation unit, a nonlinear activation unit, and an down-dimensional transformation unit. Its main purpose is to receive the output features of the multi-head attention mechanism (dimension 1). The features are mapped to a higher dimension through a linear transformation layer, and then a non-linear activation operation is performed on the upgraded features to introduce non-linear transformation capabilities. Finally, a second linear transformation layer maps the activated features back to the previous dimension. This yields the final output of the feedforward network.

[0047] In this embodiment, the Transformer computation block is used to capture global and local dependencies between features and enhance the nonlinear expressive power of features. The patch fusion layer aggregates local spatial information by changing the number of patches and feature dimensions, thereby reducing the high computational complexity caused by Transformer attention computation.

[0048] through L After iterative computation by the layer Transformer encoder, the deep spatiotemporal representation is finally obtained. ,in, It is a two-dimensional real tensor. N L The number of feature embeddings in layer L. d L Let L be the feature embedding depth. Since the number of reference frames is equal to the number of target inference frames... KThe data redundancy is significant in a short time window, and thus needs to be filtered out to generate discriminative video cues for the inference frame .

[0049] Further, the method for generating time attention weights based on the deep spatio-temporal representation comprises:

[0050] performing maximum pooling and average pooling operations on the deep spatio-temporal representation respectively to obtain global context features and local detail features respectively;

[0051] performing weighted fusion on the global context features and the local detail features through learnable weight parameters to obtain aggregated features;

[0052] performing average pooling processing on the aggregated features along the channel dimension, and then generating time attention weights using a Sigmoid activation function.

[0053] Specifically, as shown in Figure 3 , given the deep spatio-temporal representation , the time attention part first performs parallel maximum pooling and average pooling operations, and then sets learnable weight parameters α β which are optimized along the gradient descent direction in the training process, dynamically balancing global context information and local detail features through maximum pooling and average pooling, and the mathematical expression is:

[0054]

[0055] wherein, is the aggregated feature, are maximum pooling and average pooling respectively.

[0056] Based on this, first, the performs average pooling operation along the channel dimension, and then applies a Sigmoid activation function to generate time attention weights.

[0057] Further, the method for generating video cues by using a learnable query vector to reduce the dimension of the enhanced features by the spatio-temporal attention predictor comprises:

[0058] using two different linear layers on the enhanced features to generate K matrix and V matrix respectively;

[0059] multiplying the query vector and the K matrix to obtain spatial attention weights;

[0060] performing normalization processing on the spatial attention weights, then performing weighted calculation with the V matrix, and finally generating video cues through a linear layer. ​​

[0061] Specifically, the obtained temporal attention weight is fused with the original deep spatio-temporal representation, and is integrated through a residual connection mechanism, and finally an enhanced feature with temporal characteristics is output . The calculation process can be represented as:

[0062]

[0063] wherein, is the time weight, is the mean value, is activated using the Sigmoid function.

[0064] In the spatial attention part, a multi-head attention mechanism is used as the core component. Through training of the query vector , forward calculation, loss feedback, gradient backpropagation and parameter update, the mapping relationship suitable for the task is learned. After linear transformation of the input feature, the relevant information in the sequence can be effectively queried. In , the spatial semantic features are reduced and extracted, wherein is a two-dimensional real tensor, N p is the number of reference frame hint feature embeddings, d is the reference frame hint feature embedding depth. For , two different linear layers are used to generate K matrix and V matrix. Q and K matrix are multiplied to generate attention weight:

[0065]

[0066] wherein, is a two-dimensional tensor with a dimension of , and THW is a three-dimensional product of feature matrices T , H , W , and is the spatial attention weight.

[0067] After the spatial attention weight is normalized by the Softmax function, it is weighted with the V matrix, and then sent to the linear layer to generate the video hint information :

[0068]

[0069] wherein, is the SoftMax function, is linearized.

[0070] S130, input the video prompt information and the target reasoning frame into a target detection network, the target detection network includes a plurality of stages connected in turn and a detection head, each stage includes a dynamic prompt enhancement module and a Transformer calculation block, the feature embedding of the target reasoning frame and the video prompt information are input into the dynamic prompt enhancement module, the dynamic prompt enhancement module extracts the reasoning frame feature by using a gating parameter, the Transformer calculation block iteratively calculates the reasoning frame feature to obtain deep reasoning features, and the detection head performs lesion detection on the deep reasoning features to obtain a detection result.

[0071] Exemplarily, in the target detection network, the feature embedding of the current reasoning frame is obtained, the feature embedding of the video prompt information generated by the prompt generation network is pre-spliced with and , so as to serve as the original input data of the backbone network feature extraction part. Before executing each Transformer calculation block, an independent dynamic prompt enhancement module is used to dynamically and adaptively fuse the input sequence, so as to prevent the loss of reference frame feature information in the deep calculation process. Finally, the deep reasoning features generated by the entire network architecture calculation are applied to the detection head to generate the detection result of the target reasoning frame .

[0072] Specifically, in order to solve the performance decline problem caused by the fixed proportion reference frame video prompt information fusion mode in the deep network, the application cascades a dynamic prompt enhancement module before each Transformer calculation block. The module first reorganizes the features of the preset video prompt information, dynamically adjusts the information fusion proportion through a differentiable gating mechanism. Then, the gating parameter adaptively adjusts the gain degree of the reference frame to the target reasoning frame in the direction of gradient descent in the training process. Meanwhile, for the multi-stage pyramid architecture of Swin Transformer, a cross-stage feature expansion operation is performed to realize dimension alignment, and finally the detection result is output by the detection head composed of a region proposal network head and a bounding box prediction head. The region proposal network head predicts the binary classification score of whether each anchor box is a target through cross-entropy loss, and then predicts the position offset of the anchor box through Smooth L1 loss, so as to output anchor boxes of different scales corresponding to the preliminary candidate regions of the target to be detected. The bounding box prediction head is responsible for classifying and refining the position of the candidate box generated by the region proposal network head, and outputs the final detection category and accurate bounding box.

[0073] Further, as shown in Figure 4 , the method for the dynamic prompt enhancement module to extract the reasoning frame feature by using the gating parameter includes:

[0074] The feature embedding of the target reasoning frame or the deep reasoning feature output of the previous stage is split into a reasoning frame feature embedding and a corresponding reference frame hint;

[0075] After the gating parameter is activated by a Sigmoid activation function and multiplied by the video hint information, the reference frame hint is added and then spliced with the reasoning frame feature embedding to obtain the reasoning frame feature of the current stage.

[0076] Specifically, given the reasoning frame feature i output by the Transformer backbone network calculation block of the target detection network stage , the reference frame hint corresponding to this layer can be accurately extracted based on the fixed prior condition of the position encoding . The dynamic propagation of the hierarchical hint information is realized through the gating adaptive fusion mechanism, and the calculation process of the dynamic hint enhancement module (DPE) is as follows:

[0077]

[0078] wherein, represents the reasoning frame feature embedding after stripping the video hint information, represents the splicing operation along the sequence dimension, is the reasoning frame feature output by the i layer.

[0079] The processing process of the Transformer calculation block in this step and the Transformer calculation block in the hint generation network is the same, and the dynamic hint enhancement module and the Transformer calculation block form a feature fusion Transformer calculation block. In each stage, the feature fusion Transformer calculation block is further connected with a patch fusion layer, and the processing process of the patch fusion layer is the same as that of the patch fusion layer in the hint generation network.

[0080] Further, the target detection network includes a patch division layer and a linear embedding layer. The patch division layer divides the target reasoning frame into a plurality of non-overlapping patches, and the linear embedding layer constructs the patches into a basic feature embedding. Then, the basic feature embedding and the video hint information are spliced to form the feature embedding of the target reasoning frame for the first stage calculation.

[0081] Specifically, the target reasoning frame is encoded by the first layer of the network to obtain a basic feature embedding , and then a pre-splicing operation is performed with the video hint information to constitute the initialization input of the DPE module :

[0082]

[0083] wherein, denotes the number of tokens of the target reasoning frame, denotes the number of tokens of the video prompt information, denotes the feature dimension.

[0084] Further, after the video prompt information is output by the prompt generation network, the video prompt information is subjected to feature replication and concatenation processing to generate a dimension-aligned prompt matched with each stage, so as to be spliced with the reasoning frame features of each stage to participate in the calculation of the Transformer block together.

[0085] Specifically, in view of the multi-stage pyramid architecture of the Swin Transformer leading to feature dimension down-sampling, the present application performs a cross-stage feature expansion operation at the input end of each Transformer calculation block: the video prompt information is subjected to feature replication and concatenation to generate a dimension-aligned prompt adapted to the current stage :

[0086]

[0087] wherein, denotes a feature replication operation along the spatial dimension.

[0088] It can be seen that the two Transformer networks connected in series in the present application adopt a two-stage cascading architecture, the prompt generation network is responsible for spatio-temporal feature modeling, and the target detection network performs feature enhancement and task reasoning, both of which realize end-to-end optimization through a dynamic feature interaction mechanism to perform reference frame feature extraction and reasoning frame target detection tasks.

[0089] The present application uses function to optimize the loss between the noise predicted by the model in the reverse process and the Gaussian noise added in the forward process. The calculation formula of the function is:

[0090]

[0091] wherein, is the difference between the predicted value and the true value .

[0092] In the training process, ResNet101 and Swin-Tiny are used as the backbone network of the network, AdamW is selected as the optimizer of the network, the basic learning rate is set to 2.5e-5, and the gating parameter gThe learning rate scaling factor of the learning rate scaling factor is 10x. The experimental environment is configured as: NVIDIA RTX 3090 GPU, Python 3.8, PyTorch 1.10.0 and CUDA 11.3 framework.

[0093] The embodiment of the application further provides a two-stage cascaded video lesion detection system, comprising:

[0094] An image acquisition module is configured to acquire a nasopharyngeal laryngoscope video and extract an endoscope image in the nasopharyngeal laryngoscope video.

[0095] A reference frame assembly module is configured to use the endoscope image at the current time as a target inference frame, select the target inference frame and a plurality of frames before and after the target inference frame to form a reference frame.

[0096] A prompt generation network comprises a plurality of sequentially connected Transformer encoders and a spatio-temporal attention predictor. The Transformer encoders comprise a multi-head attention mechanism and a feedforward network. The plurality of Transformer encoders iteratively calculate the feature embedding of the reference frame to obtain deep spatio-temporal representations. The spatio-temporal attention predictor generates a time attention weight based on the deep spatio-temporal representations in the time attention part, and fuses the deep spatio-temporal representations according to the time attention weight to obtain enhanced features with time sequence characteristics. In the spatial attention part, the spatio-temporal attention predictor uses a learnable query vector to reduce the dimension of the enhanced features to obtain video prompt information.

[0097] A target detection network comprises a plurality of sequentially connected stages and a detection head. Each stage comprises a dynamic prompt enhancement module and a Transformer calculation block. The feature embedding of the target inference frame and the video prompt information are input into the dynamic prompt enhancement module. The dynamic prompt enhancement module extracts inference frame features using a gating parameter. The Transformer calculation block iteratively calculates the inference frame features to obtain deep inference features. The detection head performs lesion detection on the deep inference features to obtain a detection result.

[0098] The embodiment of the application further provides a computer storage medium, which stores a plurality of computer instructions for causing a computer to execute the above method.

[0099] Table 1 shows the performance comparison results of the existing video target detection methods trained using the original code library default configuration on the public dataset ImageNet VID. The bold font indicates the best result, the underlined text indicates the suboptimal result, and Ours is the method proposed in the application.

[0100] Table 1 Comparative results of various methods on the public dataset ImageNet VID

[0101]

[0102] In Table 1, SELSA is a sequence-level semantic aggregation video object detection algorithm, RDN is a relationship distillation network video object detection algorithm, MEGA is a memory-enhanced global-local aggregation video object detection algorithm, and FasterRCNN is a faster region-based convolutional neural network.

[0103] In order to compare the model effect under the lesion detection of nasopharyngeal laryngoscope, 851 patients who underwent nasopharyngeal laryngoscope examination in two cross-regional tertiary hospitals (from January 6, 2024 to June 3, 2024, and from April 7, 2025 to May 26, 2025) were collected according to the data collection and labeling specifications identified in the expert consensus of the endoscope artificial intelligence data collection and labeling quality control system. Under the guidance of three professional doctors, 270 patient videos with clear lesion visual features were selected. In order to protect the privacy and security of medical image data, all original video data were desensitized, and all sensitive information related to patient privacy was removed. 438 segments of lesion video clips were formed, including neoplasm, vocal cord leukoplakia, cyst, and adenoid hyperplasia. The video sequence length ranged from a minimum of 33 frames to a maximum of 779 frames, with an average of 185.37 frames per video and a median of 156.00 frames per video. The number of video lesion samples for the four types of lesions was as follows: 91 segments of neoplasm, 47 segments of vocal cord leukoplakia, 76 segments of cyst, and 43 segments of adenoid hyperplasia, totaling 262 lesion samples from different patients. Under the guidance of three professional doctors of nasopharyngeal laryngology, referring to the specific anatomical structure definitions and specifications in the "Electronic Laryngoscope Clinical Application - Nasopharyngeal Laryngeal Tumor Narrowband Imaging Endoscopy Atlas", 438 segments of lesion video clips were annotated frame by frame, and 131 segments of negative samples without lesions from 64 normal examinees were added. The nasopharyngeal laryngeal lesion video object detection dataset NLLVOD (Nasopharyngeal - Laryngeal Lesion Video Object Detection) was formed.

[0104] Table 2 is a comparison of the performance indicators of various methods on the self-built nasopharyngeal laryngeal video lesion detection dataset NLLVOD, where bold font indicates the best result, underlined text indicates the suboptimal result, and Ours is the method proposed in this application. DEIM-S in Table 2 is an improved matching DETR fast convergence model-small version, and DETR is a detection Transformer.

[0105] Table 2 Comparative results of various methods on the self-built nasopharyngeal laryngeal video lesion detection dataset

[0106]

[0107] Table 3 is the detection capability comparison results of each method on the self-built nasopharyngeal laryngoscope video lesion detection data set for each lesion, wherein the bold font indicates the best result, the underlined text indicates the suboptimal result, and Ours is the method proposed in the present application.

[0108] Table 3 is the detection capability comparison results of each method on the self-built nasopharyngeal laryngoscope video lesion detection data set for each lesion, wherein the bold font indicates the best result, the underlined text indicates the suboptimal result, and Ours is the method proposed in the present application.

[0109]

[0110] As shown in Table 1, the performance comparison results of the existing video target detection method based on the default configuration of the original code library on the public large-scale ImageNet VID data set. The DynSTPN method shows significant advantages in performance. Specifically, the SwinBase version of DynSTPN achieves an mAP (mean average precision) value of 83.9%, which is 3.6%, 7.6% and 5.5% higher than the baseline FasterRCNN model using ResNet-101 architecture (e.g. 80.3% for SELSA, 76.3% for RDN and 78.4% for MEGA). Compared with Diffusion VID, the mAP is only 3.0% lower, but the computational efficiency is higher. In terms of processing speed, the SwinBase version reaches 17.8 FPS, which is more efficient in inference than SELSA (11 FPS) and Diffusion VID (15.7 FPS). Similarly, the SwinTiny version of DynSTPN also performs well in performance. When using SwinTiny as Backbone, DynSTPN achieves an mAP of 82.9% while maintaining comparable detection efficiency, surpassing the baseline method STPN (80.6%), which indicates that the method can effectively balance accuracy and speed under a lightweight architecture.

[0111] As shown in Table 2, in terms of self-built nasopharyngeal and laryngeal lesion detection video data set, the DynSTPN proposed in the application achieves the best detection result with mAP of 79.6. Compared with the benchmark method STPN, the mAP of DynSTPN is improved by 4.6 percentage points. At the same time, since the two modules designed in the application do not increase additional parameters, DynSTPN maintains the original detection efficiency while achieving performance improvement. Compared with the advanced static image detector Yolov12 and DEIM, although the existing video target detection method improves the mAP, it usually accompanies significant increase in computational overhead. The DynSTPN-SwinTiny proposed in the application exceeds Yolov12-M by 11.3 percentage points and DEIM-S by 15.1 percentage points in mAP index, and maintains the calculation efficiency of 29.4 FPS. This result shows that in the application scenario of nasopharyngeal and laryngeal endoscope lesion detection, DynSTPN not only significantly surpasses the static image detection method in detection accuracy, but also fully meets the real-time requirement of nasopharyngeal and laryngeal endoscope clinical image inference (≥25FPS, the localized storage frame rate is 25FPS).

[0112] As can be seen, the DynSTPN proposed in the application achieves higher detection accuracy while maintaining faster inference speed in the video target detection task, effectively balances the accuracy and efficiency, and provides a technically feasible solution with practical application value in this field.

[0113] In Figure 5 each subgraph is composed of the complete image on the top and the detection result of local amplification on the bottom, and the amplification area is marked with a red box in the complete image. The numbers in the subgraph represent the best detection confidence score of the method for the current lesion area of the image; the larger the number, the higher the detection accuracy. Figure 6 The heat map in shows the attention level of the model to different positions, with red indicating a higher attention level and blue indicating a lower attention level. The first row of each subgraph represents not using the prompt information extracted from the reference frame, and the second row represents using the prompt information extracted from the reference frame.

[0114] Although the preferred embodiments of the application have been described, those skilled in the art can make further changes and modifications to these embodiments once they know the basic inventive concept. Therefore, the appended claims are intended to be interpreted as including all changes and modifications falling within the scope of the application.

[0115] Obviously, those skilled in the art can make various modifications and variations to the application without departing from the spirit and scope of the application. Thus, if these modifications and variations of the application fall within the scope of the claims of the application and their equivalents, the application also intends to include these modifications and variations.

Claims

1. A two-stage cascaded video lesion detection method, characterized in that, The method comprises the following steps: acquiring a nasopharyngolaryngoscope video, and extracting an endoscope image in the nasopharyngolaryngoscope video; taking the endoscope image at the current time as a target inference frame, and selecting the target inference frame and a plurality of frames before and after the target inference frame to form a reference frame; inputting the reference frame into a prompt generation network, the prompt generation network comprising a plurality of sequentially connected Transformer encoders and a spatio-temporal attention predictor, the Transformer encoders comprising a multi-head attention mechanism and a feedforward network, and the plurality of Transformer encoders performing iterative calculation on feature embeddings of the reference frame to obtain deep spatio-temporal representations; the spatio-temporal attention predictor generates a time attention weight based on the deep spatio-temporal representations in a time attention part, and performs weighted fusion on the deep spatio-temporal representations according to the time attention weight to obtain enhanced features with time sequence characteristics; in a spatial attention part, the spatio-temporal attention predictor uses a learnable query vector to reduce the dimension of the enhanced features to obtain video prompt information; inputting the video prompt information and the target inference frame into a target detection network, the target detection network comprising a plurality of sequentially connected stages and a detection head, each stage comprising a dynamic prompt enhancement module and a Transformer calculation block, the feature embeddings of the target inference frame and the video prompt information being input into the dynamic prompt enhancement module, the dynamic prompt enhancement module extracting inference frame features using a gating parameter, the Transformer calculation block performing iterative calculation on the inference frame features to obtain deep inference features, and the detection head performing lesion detection on the deep inference features to obtain a detection result.

2. The two-stage cascaded video lesion detection method of claim 1, wherein, The prompt generation network further comprises a patch division layer and a linear embedding layer, the patch division layer divides the reference frame into a plurality of non-overlapping patches, and the linear embedding layer constructs the patches into feature embeddings for calculation by the first Transformer encoder.

3. The two-stage cascaded video lesion detection method of claim 1, wherein, The method for the spatio-temporal attention predictor to generate the time attention weight based on the deep spatio-temporal representations comprises: performing maximum pooling and average pooling operations on the deep spatio-temporal representations respectively to obtain global context features and local detail features respectively; performing weighted fusion on the global context features and the local detail features using learnable weight parameters to obtain aggregated features; performing average pooling processing on the aggregated features along the channel dimension, and then using a Sigmoid activation function to generate the time attention weight.

4. The two-stage cascaded video lesion detection method of claim 1, wherein, The method for the spatio-temporal attention predictor to use a learnable query vector to reduce the dimension of the enhanced features to obtain video prompt information comprises: using two different linear layers to respectively generate a K matrix and a V matrix from the enhanced features; multiplying the query vector and the K matrix to obtain a spatial attention weight; performing normalization processing on the spatial attention weight, then performing weighted calculation with the V matrix, and finally passing through a linear layer to generate the video prompt information.

5. The two-stage cascaded video lesion detection method of claim 1, wherein, The method for the dynamic prompt enhancement module to use a gating parameter to extract inference frame features comprises: split the feature embedding of the target reasoning frame or the deep reasoning feature output by the previous stage into a reasoning frame feature embedding and a corresponding reference frame hint; after the gating parameter is activated by a Sigmoid activation function, the gating parameter is multiplied by the video hint information, and the video hint information is added to the reference frame hint and then spliced with the reasoning frame feature embedding to obtain the reasoning frame feature of the current stage.

6. The two-stage cascaded video lesion detection method of claim 5, wherein, The target detection network includes a patch division layer and a linear embedding layer. The patch division layer divides the target reasoning frame into a plurality of non-overlapping patches. The linear embedding layer constructs the patches into a base feature embedding. Then, the base feature embedding is spliced with the video hint information to form a feature embedding of the target reasoning frame for first stage calculation.

7. The two-stage cascaded video lesion detection method of claim 1, wherein, After the hint generation network outputs the video hint information, the video hint information is subjected to feature replication and concatenation processing to generate a dimensionally aligned hint matched with each stage.

8. A system for applying a two-stage cascaded video lesion detection method according to any one of claims 1 to 7, characterized in that, comprise: an image acquisition module configured to acquire a nasopharyngolaryngoscope video and extract an endoscope image in the nasopharyngolaryngoscope video; a reference frame building module configured to take the endoscope image at a current time as a target reasoning frame, and select the target reasoning frame and a plurality of frames before and after the target reasoning frame to form a reference frame; a hint generation network, the hint generation network comprising a plurality of sequentially connected Transformer encoders and a spatiotemporal attention predictor, the Transformer encoders comprising a multi-head attention mechanism and a feedforward network, and the plurality of Transformer encoders being configured to iteratively calculate feature embeddings of the reference frames to obtain deep spatiotemporal representations; the spatiotemporal attention predictor being configured to generate temporal attention weights based on the deep spatiotemporal representations in a temporal attention part, and to weight and fuse the deep spatiotemporal representations according to the temporal attention weights to obtain enhanced features with temporal characteristics; in a spatial attention part, the spatiotemporal attention predictor is configured to use a learnable query vector to reduce the dimension of the enhanced features to obtain video hint information; a target detection network, the target detection network comprising a plurality of sequentially connected stages and a detection head, each stage comprising a dynamic hint enhancement module and a Transformer calculation block, the feature embedding of the target reasoning frame and the video hint information being input into the dynamic hint enhancement module, the dynamic hint enhancement module being configured to extract a reasoning frame feature using a gating parameter, the Transformer calculation block being configured to iteratively calculate the reasoning frame feature to obtain a deep reasoning feature, and the detection head being configured to perform lesion detection on the deep reasoning feature to obtain a detection result.

9. A computer storage medium, characterized in that The computer storage medium stores a plurality of computer instructions for causing a computer to execute the method of any one of claims 1-7. The computer storage medium stores a plurality of computer instructions for causing a computer to execute the method of any one of claims 1-7.

Images