Visual Transform-based dynamic screening medical image target tracking method and device

Abstract

The invention provides a dynamic screening medical image target tracking method and device based on visual Transform, and relates to the technical field of computer vision, and the method comprises the steps: standardizing near-infrared or visible light fundus video frames into uniform resolution, constructing a template-search frame pair, and then jointly mapping the two frames of images into a Token sequence; a dynamic local interaction module is embedded in front of each pruning layer of the whole network, a local context is captured by using depth separable convolution and point convolution, a dynamic convolution kernel generator is driven, and a neighborhood Token is adaptively weighted and aggregated. Next, the Token screening and the compression mechanism TSC are operated in the same pruning layer, only the Top-K key Token is reserved, the redundant Token is cut off, and the original index is recorded; the objective of the invention is to improve the positioning stability and reasoning efficiency of a focus area (such as an optic disc) in a complex operation video.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method and device for dynamically screening medical image target tracking based on a visual Transformer. Background Art

[0002] In the field of medical image-assisted diagnosis and surgical navigation, real-time, robust tracking of the optic disc is a fundamental task in clinical scenarios such as glaucoma screening, retinal surgery, and intraoperative laser positioning. Because fundus imaging is often accompanied by complex factors such as uneven illumination, specular reflections, vascular occlusion, micro-ocular movements, and instrument interference, the appearance of the optic disc region can vary significantly across consecutive video frames. Traditional tracking methods based on correlation filtering, Siamese networks, or lightweight convolutional neural networks struggle to achieve a balanced balance of positioning accuracy, cross-frame stability, and computational efficiency. In recent years, the Visual Transformer (ViT) has achieved breakthrough progress in general object tracking by leveraging its global self-attention mechanism. However, directly migrating standard ViT to medical videos has exposed three prominent contradictions: First, ViT performs dense token interactions on the entire image or the entire search area, resulting in a quadratic increase in the number of tokens. Under the real-time requirements of intraoperative surgery (usually requiring above 30 fps), redundant computation cannot be ignored; second, the optic disc occupies only a very small part of the entire fundus image, and the background retinal texture and vascular structure are complex. Global self-attention is easily overwhelmed by high-response background areas, resulting in target drift; third, ViT lacks local structural priors and is unable to adequately depict the low-contrast, weak-edge optic disc boundary. When the image degrades due to hemorrhage or sudden changes in illumination, the tracking box is prone to offset or jitter.

[0003] To alleviate these issues, the industry has attempted to compress the number of tokens using sparse attention, local window attention, or progressive downsampling strategies. However, most of these methods use manual thresholds or fixed ratios for pruning, which cannot adapt to the current frame content and can easily lead to the inadvertent deletion of key tokens when the target is blurred or occluded. Furthermore, some methods employ late fusion between the template branch and the search branch, which results in target semantic information interacting only at deeper layers, failing to fully utilize shallow local details and limiting the model's robustness to subtle deformations. Other studies have introduced deformable convolutions or dynamic filters to enhance local modeling, but the convolution kernel parameters are typically independent of the global context and are disconnected from the Transformer's self-attention mechanism, failing to form a unified end-to-end optimization framework. Simply put, while existing mainstream visual tracking frameworks offer superior performance, they suffer from issues such as high feature computation redundancy in the search area, a lack of local modeling capabilities, and weak target perception.

[0004] In view of this, this application is filed. Summary of the Invention

[0005] The present invention provides a method and device for dynamically screening medical image target tracking based on visual Transformer, which can at least partially improve the above problems.

[0006] To achieve the above object, the present invention adopts the following technical solutions: A method for dynamically screening medical image target tracking based on a visual Transformer, comprising: Obtaining an original fundus video image sequence acquired by a preset acquisition device, and performing standardized preprocessing on the original fundus video image sequence to generate an input image pair; The input image pairs are divided into blocks, converted into a joint input Token sequence, and the joint input Token sequence is input into the Transformer network; A dynamic local interaction module is introduced before each pruning layer of the Transformer network to perform local context enhancement on the search region token in the joint input token sequence. An attention-guided learnable token filtering and compression mechanism is used to filter the joint input token sequence after local context enhancement. After filtering, the values ​​of the retained joint input Token sequence are subjected to fusion modeling processing to generate the position prediction result of the optic disc area of ​​the current frame.

[0007] The present invention also provides a device for dynamically screening medical image target tracking based on a visual Transformer, which comprises: A preprocessing unit is used to obtain an original fundus video image sequence acquired by a preset acquisition device, and perform standardized preprocessing on the original fundus video image sequence to generate an input image pair; The conversion unit is used to process the input image pairs into blocks, convert them into a joint input Token sequence, and input the joint input Token sequence into the Transformer network; The DLIM unit is used to introduce a dynamic local interaction module before each pruning layer of the Transformer network, and use the dynamic local interaction module to perform local context enhancement processing on the search area token in the joint input token sequence; The TSC unit is used for a learnable token filtering and compression mechanism based on attention guidance, filtering the joint input token sequence after local context enhancement; The prediction unit is used to perform fusion modeling on the values ​​of the joint input Token sequence retained after screening to generate a position prediction result of the optic disc area of ​​the current frame.

[0008] In summary, the proposed dynamic screening medical image object tracking method based on a visual Transformer aims to improve the localization stability and inference efficiency of lesion regions (such as the optic disc) in complex surgical videos. This method constructs a unified Transformer backbone network with a total of 12 encoder layers, of which the 3rd, 6th, and 9th layers are token pruning layers (Token Selection and Compression Mechanism (TSC)), and the remaining layers are standard encoding layers. A dynamic local interaction module (DLIM) is introduced before each TSC layer to prioritize the local contextual semantic enhancement of tokens in the search region. The DLIM module extracts contextual structural information through depthwise and pointwise convolutions, and generates a dynamic convolution kernel based on query features. It then adaptively weights and aggregates neighborhood features to improve the representation of tokens within the local region. Subsequently, the TSC mechanism ranks the search region tokens by combining attention scores with a learnable importance scorer to assign weights. It dynamically retains the top-K most discriminative tokens and removes redundant tokens, significantly reducing inefficient computation. At the same time, to maintain structural consistency and sequence integrity, the pruning process records the original token index to ensure that subsequent layers can smoothly restore and aggregate features.

[0009] Furthermore, to enhance goal-oriented feature modeling capabilities, this method employs a template-search frame bidirectional interaction strategy: the template frame and search frame tokens are directly concatenated into the Transformer's initial input, enabling target semantic transmission from shallow to deep layers, improving the model's perception of the optic disc region and its ability to suppress background interference. This holistic approach, combining the contextual enhancement capabilities of DLIM, the structural sparsity control of TSC, and the information fusion advantages of dual-stream interaction, significantly improves optic disc localization accuracy and cross-frame stability while maintaining model compactness. It is particularly suitable for intelligent medical image processing applications such as intraoperative optic disc tracking and assisted navigation in ophthalmic surgery. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 1 is a flowchart of a method for dynamic screening of medical image target tracking based on visual Transformer provided by the first embodiment of the present invention; Figure 2 This is an overall structural diagram of a method for dynamic screening of medical image target tracking based on visual Transformer provided by an embodiment of the present invention; Figure 3is a structural diagram of a DLIM module provided by an embodiment of the present invention; Figure 4 is a structural diagram of a TSC module provided by an embodiment of the present invention; Figure 5 This is a module diagram of a dynamic screening medical image target tracking device based on visual Transformer provided in the second embodiment of the present invention. DETAILED DESCRIPTION

[0011] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0012] refer to Figure 1 、 Figure 2 As shown, the first embodiment of the present invention discloses a method for tracking a target in a medical image for dynamic screening based on a visual transformer. The method can be performed by a device for tracking a target in a medical image for dynamic screening based on a visual transformer (hereinafter referred to as a tracking device), and in particular, by one or more processors in the tracking device to implement the following method: S1, obtaining an original fundus video image sequence acquired by a preset acquisition device, and performing standardized preprocessing on the original fundus video image sequence to generate an input image pair; Specifically, step S1 further includes: obtaining a medical video captured by a preset near-infrared fundus camera or a visible light fundus camera, and obtaining an original fundus video image sequence containing the optic disc area from the medical video frame by frame; The original fundus video image sequence containing the optic disc area is standardized and preprocessed to generate input image pairs. The specific steps are as follows: Perform pixel normalization on the original fundus video image sequence containing the optic disc area, and uniformly linearly map the grayscale range or RGB channel value of the image to the [0,1] interval; The original fundus video image sequence containing the optic disc area is resized uniformly, and the sizes of all image frames in the original fundus video image sequence are converted to a standard input size by bilinear interpolation or center cropping; Based on the processed original fundus video image sequence, a sliding window strategy is used to generate an input image pair for tracking modeling. The input image pair includes a template frame and a search frame. The search frame is the current processing frame, indicating the position of the optic disc that needs to be located. The template frame is a frame with high image quality and clear optic disc edge selected from multiple current historical frames, which serves as a stable representation of the reference target structure.

[0013] In this embodiment, to achieve stable tracking of the optic disc region in fundus images, medical image frames must be acquired and standardized. Acquiring raw image data frame by frame from medical videos is often used in ophthalmic surgical navigation or diagnostic recording scenarios to construct a temporally continuous image input source.

[0014] Due to the wide variety of acquisition hardware, and considering the differences in resolution, lighting, noise, brightness, and image scale within each frame during actual acquisition, image frames undergo further standardization preprocessing to enhance the robustness of the subsequent modeling of the visual Transformer network. First, the image's grayscale range or RGB channel values ​​are uniformly scaled to the [0, 1] interval. This reduces the impact of varying acquisition devices or lighting conditions on the model input distribution, effectively minimizing the effect of device sensitivity differences on image contrast and improving the perception of weak edges (such as the optic disc boundary). Second, all image frames are converted to a standard input size (e.g., 224×224 pixels) through bilinear interpolation or center cropping. This ensures a consistent number of tokens generated during subsequent patching and prevents model instability with inputs of varying resolutions. This also ensures a consistent number of tokens generated after subsequent block operations, preventing irregular inputs from causing feature confusion or decreased model learning efficiency. Finally, to adapt to the inter-frame dependency characteristics of the tracking task, a sliding window strategy is used to generate image pairs for modeling, namely the input image pairs. The currently processed frame is used as the search frame, and a frame with good image quality or no obvious occlusion in the optic disc area is usually selected as the template frame to provide structural guidance and attention guidance information, that is, to provide target reference information.

[0015] In this embodiment, the template frame and search frame serve as the input basis for this method. Their synergistic relationship determines the accuracy of target structure guidance and the stability of subsequent tracking. This preprocessing step plays a crucial role in the method, directly affecting the quality of subsequent feature modeling and the accuracy of prediction results. Through this standardized image acquisition process, a clean, unified, and well-structured image input pair is established, laying a solid foundation for subsequent model processing.

[0016] S2, divides the input image pair into blocks, converts it into a joint input Token sequence, and inputs the joint input Token sequence into the Transformer network; Specifically, step S2 further includes: using a Patch Embedding module to divide the template frame and the search frame to obtain a plurality of sub-image blocks of fixed size, which are mathematically represented as patches; A preset linear transformation module or a small convolution module is used to transform all sub-image blocks into a C-dimensional embedded vector representation, which is mathematically represented as a Token. The embedded vector representation is used as the basic calculation unit for modeling to obtain the Token sequence of the template frame. , and the Token sequence of the search frame , where B is the batch size, is the number of Tokens in the template frame. is the number of tokens in the search frame, and C is the feature dimension of each token; The spatial position information represented by each embedding vector is explicitly encoded and added to the embedding vector representation; The Token sequence of the encoded template frame and the Token sequence of the search frame are concatenated along the sequence dimension to obtain a joint input Token sequence. The joint input Token sequence is used as the input for global modeling and transmitted to the Transformer network backbone structure.

[0017] In this embodiment, step S2 specifically describes the process of constructing a structured token representation of an image embedding. Specifically, after image processing is complete, to adapt the input format of a Transformer network, the first stage of visual Transformer modeling is required. This involves converting the structured image into a sequence of tokens that can be input into the neural network. This is a sequence of tokens.

[0018] First, the Patch Embedding module divides the template frame and search frame into multiple fixed-size sub-image patches. For example, a 224×224 image can be divided into 196 non-overlapping patches of 14×14 in units of 16×16. Each patch is expanded into a vector and then converted into a vector representation (via a convolutional projection layer or a fully connected layer (also known as a linear transformation module or a small convolution module)). This vector representation becomes the basic computational unit used for modeling in the network, namely a token.

[0019] This operation is then performed on each image, ultimately yielding a sequence of tokens for the template frame and a sequence of tokens for the search frame. Furthermore, to preserve spatial structure information—that is, the spatial order of regions within the image structure—a two-dimensional positional embedding is added to each token to reflect its original position. This ensures that the model can perceive the spatial arrangement of tokens within the attention mechanism, enabling it to learn the spatial dependencies between regions.

[0020] After obtaining the token representations of the template and search, the token sequences of the template frame and the search frame are concatenated in the sequence dimension to form a joint input sequence, which is then fed into the Transformer backbone for subsequent modeling. This step implements the mapping from image space to sequence space, providing a structured representation foundation for the Transformer to perform cross-image region contrast modeling. It is the core pre-process for constructing the Transformer model input format and provides a semantically unified representation space for subsequent context enhancement, pruning and compression, and feature modeling.

[0021] See also Figure 3 ,S3, introduces a dynamic local interaction module before each pruning layer of the Transformer network, and uses the dynamic local interaction module to perform local context enhancement processing on the search area token in the joint input token sequence; Specifically, step S3 further includes: the Transformer network includes 12 coding layers, wherein the 3rd, 6th and 9th layers are pruned layers, and the remaining layers are standard coding layers; A query-guided dynamic local interaction module is introduced before each pruning layer. The dynamic local interaction module consists of three parts: a one-dimensional depth-wise separable convolution layer, a point-by-point convolution layer, and a dynamic convolution kernel generator. A one-dimensional depth-wise separable convolutional layer is used to perform local structure modeling on each channel token sequence in the joint input token sequence. A point-by-point convolutional layer is used to integrate cross-channel semantics to form contextual key features. The original embedding vector representation is used as the interaction condition between the original query vector and the key feature. The dynamic convolution kernel generator generates the dynamic convolution kernel weight for each position, performs weighted aggregation according to the neighborhood window, and outputs the context-enhanced token representation, specifically: Convert the Token sequence of the search frame in the joint input Token sequence into Structure, input to a one-dimensional depth-separable convolutional layer; The point-by-point convolution layer is used to fuse the output results of the one-dimensional depth-separable convolution layer to generate the fused context key features; The original query vector is interacted with the context key features point by point. A lightweight MLP structure is used to generate dynamic convolution kernel weights unique to each token position. Based on the dynamic convolution kernel weights, adjacent tokens are weighted and aggregated within the local token window to output a context-enhanced token representation. The formula is: , , ,in, , Q, and V are all Token values ​​of the search area. The processed Token value of the search area. is a one-dimensional depth-wise separable convolutional layer, is the point-by-point convolution layer, is a local window, is the activation function, is the calculation function of the attention score, is the weight matrix, is the multiplication operator, is a dynamic local interaction module, Token serial number.

[0022] In this embodiment, the encoder of the entire Transformer network consists of 12 blocks (encoding layers). The unpruned layers function as standard Transformer Encoders to implement context accumulation and long-range dependency modeling. DLIM enhancement is performed before the pruned layers, followed by the TSC selection mechanism. Layers 3, 6, and 9 integrate the TSC module for token screening, while the remaining layers function as standard Encoder layers for sequence modeling and context fusion. A dynamic local interaction module (DLIM) is introduced before each pruned layer to perform local context enhancement on the search area tokens. Contextual semantic information is obtained through a combination of deep convolution and point convolution, and dynamic kernel weights are generated based on the query to achieve adaptive modeling.

[0023] Specifically, considering that target regions in medical images (such as the optic disc) typically exhibit strong local spatial consistency, a local context enhancement module, the Dynamic Local Interaction Module (DLIM), is introduced before the Transformer backbone encoding to enhance the local feature representation of the search token, in order to improve the search token's ability to perceive local spatial variations. This module utilizes a dynamic convolution mechanism to model the structural dependencies between the token and its neighborhood, thereby enhancing fine-grained features. This module differs from traditional attention mechanisms or sparse enhancement strategies in that it establishes a local modeling path guided by structural priors through a "token-adaptive dynamic convolution" mechanism.

[0024] Among them, the dynamic local interaction module is used to enhance the local modeling capability of the token in the search area. Its structure consists of three parts: one-dimensional depthwise separable convolution (Depthwise Conv), pointwise convolution (Pointwise Conv) and dynamic convolution kernel generator. In actual use, first, the search token sequence is converted into After the structure, the network is fed into a depthwise one-dimensional convolution, which operates independently on a channel-by-channel basis, efficiently extracting structural relationships between local adjacent tokens. The output is then fed into a pointwise convolution layer to fuse information across channels, generating fused contextual key features. At this point, the original query vector is point-by-point interacted (dot product) with the generated key features to produce a fused representation. To achieve adaptive token modeling based on this interaction, a lightweight MLP architecture is used to generate dynamic convolution kernel weights specific to each token position. Next, within a local token window, adjacent tokens are weighted and aggregated based on this dynamic kernel, outputting a context-enhanced token sequence to achieve local context enhancement. This module simulates a local self-attention mechanism, offering lower computational cost and greater adaptability. This process not only mimics the local dependency modeling capabilities of the self-attention mechanism, but also offers lower computational cost and stronger positional awareness. This makes it particularly suitable for maintaining the clarity and stability of token features when the optic disc edge is blurred or background noise is strong.

[0025] The dynamic local interaction module enables structure-adaptive local enhancement, preserving local discrimination capabilities even when the target region has blurred edges or low contrast. Compared to traditional sparse spatial attention mechanisms, this method incorporates query-based modeling and dynamically adjusts convolution kernels, enhancing the model's expressiveness and robustness in structurally uncertain regions.

[0026] See also Figure 4 ,S4, based on the attention-guided learnable Token screening and compression mechanism, the joint input Token sequence after local context enhancement processing is screened; Specifically, step S4 further includes: dynamically assigning weights based on the semantic similarity between the token sequence of the search frame and the token sequence of the template frame at each pruning layer through an importance scorer composed of two layers of perception units, thereby obtaining a weight ratio, wherein the token sequence of the search frame that is closer to the token sequence of the template frame and has a higher similarity will be given a higher weight; Extract the attention score of the template area from the token value of each search area from the multi-head attention module of the Transformer network, and perform weighted fusion processing with the weight ratio to generate a joint score for ranking; Perform Top-K filtering based on the joint score, retaining only the most discriminative Token values. The remaining Token values ​​are discarded as redundant information, and their original position indexes are retained to facilitate subsequent feature restoration. Its mathematical formula is: , , , , , , ,in, is the score vector, is the Sigmoid activation function, and are scorer weights, is the activation function, is the first layer bias value, is the second layer bias value, is the rating vector, T is the transpose operation, is the dimension of s, is the attention response sub-matrix of template to search token, Select the interaction between the template area and the search area in the attention matrix, for Dimensions, is the mean attention value of the j-th search token, is the number of attention heads in the multi-head attention mechanism, is the number of Tokens in the template area, is the number of tokens in the search area, is the ith template region mask, is the batch size, is the dimension of the number of attention heads, To search for token number, is the attention weight of the i-th template Token and the j-th search area Token, is the weight ratio, is the global position index of the search area, is the comprehensive score of the jth Token, Score the importance of the jth Token, It is the comprehensive score of Token.

[0027] In this embodiment, to reduce the redundant computational burden incurred by the Transformer model when processing search region tokens and to effectively control the computational complexity of the Transformer when processing large-scale search region tokens, a learnable attention-guided token selection and compression (TSC) mechanism is employed to dynamically select and structurally compress the search region token sequence at each pruning layer. This mechanism dynamically selects key tokens based on the importance weights of the search region tokens learned from the template tokens and the attention scores, accurately removing redundant or background region tokens.

[0028] This mechanism introduces a learnable importance scorer that dynamically assigns weights based on the contextual semantic information of the search region tokens. The scorer uses a two-layer fully connected structure to generate a normalized scalar to measure the token's contribution to the current task. Based on the template token, the scorer dynamically generates a weighted ratio for each search region token and combines it with the corresponding token's attention score to form a comprehensive ranking metric. A top-K retention strategy is used to filter out the most important tokens, eliminating redundant tokens. By maintaining an index mapping table, the original order and structural consistency of tokens are restored in subsequent layers, significantly improving computational efficiency while enhancing model tracking accuracy.

[0029] Specifically, first, an importance scorer consisting of two layers of perception units is used to dynamically assign weights based on the semantic similarity between the search token and the template token at each pruning layer. The closer the search token is to the template token and the higher the similarity, the higher the weight will be. This will then generate a normalized importance score to quantify the semantic effectiveness of each token in the current task.

[0030] At the same time, the Transformer's multi-head attention module extracts the intensity of each search token's attention on the template region. Each search token's attention score is then weighted and fused with the weighted ratio of the importance scorer's output to form a final joint score (joint ranking metric) for ranking. The system then performs a top-K filter based on this fused score, retaining only the most discriminative tokens. The remaining tokens are discarded as redundant information, retaining their original position indexes for subsequent feature restoration. The original indexes are also recorded for consistent reconstruction of token order and structure in subsequent layers. Simply put, the Transformer architecture employs an integrated template-search bidirectional interaction strategy. Template and search tokens are input in parallel into each Transformer encoder layer, and a multi-head self-attention mechanism establishes direct information flow. This ensures that the structural semantics of the optic disc in the template frame are transmitted layer by layer to the search frame, strengthening the search region's ability to focus on the target and improving the robustness and consistency of optic disc region positioning.

[0031] Traditional static pruning methods, such as fixed-ratio cropping or filtering with a preset energy threshold, often employ uniform, unchanging filtering rules that are unable to adapt to the dynamic changes in input data and the specificity of task contexts. These methods rely on a single attention sparsity, fixed attention thresholds, or non-learnable rigid pruning rules. These strategies lack the ability to dynamically adapt to task scenarios. When target boundaries are fuzzy or context changes drastically, they are prone to inadvertently deleting key tokens or retaining redundant tokens, significantly reducing filtering stability.

[0032] In comparison, the TSC mechanism in this method exhibits fundamental differences. It is learnable, adaptive, and structurally resilient, dynamically adjusting its retention strategy based on the real-time importance of tokens in a specific context. This dynamic nature makes it particularly effective in medical image scenarios with ambiguous target region features and complex background interference—it accurately retains key information while effectively eliminating redundant data. Combined with the local context enhancement features provided by the pre-processed DLIM module, the TSC mechanism not only significantly improves the accuracy and stability of token screening, but also reduces computational complexity while maintaining model expressiveness, achieving a dual optimization of efficiency and tracking robustness. This method explicitly introduces a trainable scoring network and deeply integrates it with an attention-guided mechanism to construct a dual-track collaborative selection strategy: "semantically driven + attention-aware." On the one hand, the learnable scorer learns task-specific knowledge from the data and dynamically quantifies the semantic importance of tokens. On the other hand, the contextual information provided by the attention mechanism further enhances the contextual awareness of the screening decision. This design not only makes the screening process more robust but also allows for flexible adaptation to inter-frame differences and changes in target morphology, making it particularly effective in medical image tracking scenarios with complex backgrounds and frequent occlusions. Combined with the local structure enhancement features provided by the pre-processed DLIM module, the TSC pruning mechanism significantly reduces computational overhead (effectively eliminating redundant tokens) while accurately preserving key semantic representations. This significantly improves the task adaptability and inference efficiency of the Transformer model, achieving a synergistic optimization of performance and efficiency.

[0033] S5, performing fusion modeling processing on the values ​​of the combined input Token sequence retained after screening to generate a position prediction result of the optic disc area of ​​the current frame.

[0034] Specifically, step S5 further includes: calling the standard coding layer of the Transformer network to perform multi-level feature fusion and target positioning modeling processing on the retained Token sequence of the template frame and the Token sequence of the search frame to generate a position prediction result of the optic disc area of ​​the current frame.

[0035] In this embodiment, global feature modeling and fusion are performed in the remaining Transformer encoding layers, and the predicted position of the optic disc region in the current frame is output. This method utilizes a unified, integrated Transformer architecture to perform multi-layer feature encoding and cross-region fusion modeling. Each encoding module incorporates a bidirectional interaction mechanism between template and search tokens. This multi-head self-attention architecture allows for direct information fusion between the two in the representation space, allowing the target structure in the template to guide the search region's attention focus layer by layer.

[0036] In each layer, template and search tokens are input together, and a multi-head attention mechanism facilitates bidirectional information transfer between them. Compared to traditional approaches that employ branch encoding followed by modeling, this approach establishes a joint template-search architecture from the outset, enabling target structural information to continuously influence feature extraction and localization strategies within the search region, thereby enhancing consistent target perception during tracking. At the network's end, feature embeddings for position determination are extracted from the search token sequence and fed into the localization prediction head module. This module can be implemented in various forms, such as fully connected layers, convolutional layers, or an attention-fusion regression architecture, and its output can be in the form of center point coordinates, bounding box parameters, and more. The final prediction results are fed back to the tracking system for use in template updates, inter-frame navigation, or integration with fundus surgery systems, enabling high-precision, low-latency, real-time optic disc tracking. This encoding and output process fully integrates contextual awareness and spatial alignment capabilities, and is the final key step in achieving efficient medical image modeling in this system.

[0037] This architecture forms an alternating interlayer strategy of "enhancement-compression-modeling" along the computational path, significantly improving operational efficiency while maintaining the model's expressive power. After the Transformer backbone completes processing, target representation features are extracted from the search token and fed into a localization prediction head (such as a linear regressor or MLP) to output the optic disc position for the current frame. Predictions can be expressed in the form of center coordinates, bounding box parameters, or heatmaps. The final output can be used for subsequent template updates, tracking visualization, or integration with medical analysis systems, enabling real-time, high-precision optic disc tracking and intelligent assisted diagnosis.

[0038] Simply put, the DLIM module is fixedly used in the context modeling phase before the 3rd, 6th, and 9th pruning layers to enhance local token representation; while the TSC module performs feature screening and structural compression within the corresponding pruning layers. Different pruning layers can be set to different token retention ratios to meet representation requirements at different depths. This modular combination of DLIM and TSC not only improves token feature representation but also significantly reduces redundant computation, constructing an efficient Transformer encoding framework that balances content awareness and structural compactness.

[0039] It should be noted that the dynamic screening medical image target tracking method based on visual Transformer can be applied to the continuous detection task of the optic disc area in infrared or visible light fundus videos. It has the ability to enhance cross-frame stability and is highly robust to background interference such as occlusion, blur, and lighting changes. It is suitable for medical scenarios such as fundus examination, intraoperative navigation, and lesion tracking.

[0040] In summary, this method converts fundus video frames into token sequences after unified preprocessing, and directly splices the template frame and search frame tokens into a 12-layer integrated Transformer. Before the 3rd, 6th, and 9th pruning layers, the dynamic local interaction module DLIM is used to generate dynamic convolution kernels guided by queries, and the neighborhood tokens are adaptively weighted enhanced. Then, a learnable candidate importance scorer and attention score are combined for Top-K screening, and redundant tokens are dynamically removed and the indexes are recorded to maintain structural consistency, so that the target semantics continuously guide the search area from the shallow to the deep layer. Finally, after multi-layer bidirectional interaction and global modeling, the optic disc position is output, achieving low-latency, high-precision, anti-occlusion and illumination change real-time optic disc tracking, which is suitable for medical scenarios such as fundus screening and intraoperative navigation.

[0041] See also Figure 5 The second embodiment of the present invention provides a device for dynamically screening medical image target tracking based on a visual Transformer, which includes: The preprocessing unit 101 is used to obtain the original fundus video image sequence collected by the preset collection device, and perform standardization preprocessing on the original fundus video image sequence to generate an input image pair; The conversion unit 102 is used to perform block processing on the input image pair, convert it into a joint input Token sequence, and input the joint input Token sequence into the Transformer network; The DLIM unit 103 is used to introduce a dynamic local interaction module before each pruning layer of the Transformer network, and use the dynamic local interaction module to perform local context enhancement processing on the search area token in the joint input token sequence; The TSC unit 104 is used to filter the joint input token sequence after local context enhancement based on the attention-guided learnable token filtering and compression mechanism; The prediction unit 105 is configured to perform fusion modeling processing on the values ​​of the combined input Token sequence retained after screening to generate a position prediction result of the optic disc area of ​​the current frame.

[0042] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for dynamic screening of medical image target tracking based on visual Transformer, characterized in that: include: Obtaining an original fundus video image sequence acquired by a preset acquisition device, and performing standardized preprocessing on the original fundus video image sequence to generate an input image pair; The input image pairs are divided into blocks, converted into a joint input Token sequence, and the joint input Token sequence is input into the Transformer network; A dynamic local interaction module is introduced before each pruning layer of the Transformer network. The dynamic local interaction module is used to perform local context enhancement on the search region token in the joint input token sequence. An attention-guided learnable token filtering and compression mechanism is used to filter the joint input token sequence after local context enhancement. After filtering, the values ​​of the retained joint input Token sequence are subjected to fusion modeling processing to generate the position prediction result of the optic disc area of ​​the current frame.

2. The method for dynamic screening of medical image target tracking based on visual Transformer according to claim 1 is characterized in that: The original fundus video image sequence collected by the preset acquisition device is obtained, and the original fundus video image sequence is standardized and preprocessed to generate an input image pair, specifically: Obtaining a medical video captured by a preset near-infrared fundus camera or a visible light fundus camera, and acquiring an original fundus video image sequence containing the optic disc area from the medical video frame by frame; The original fundus video image sequence containing the optic disc area is standardized and preprocessed to generate input image pairs. The specific steps are as follows: Perform pixel normalization on the original fundus video image sequence containing the optic disc area, and uniformly linearly map the grayscale range or RGB channel value of the image to the [0,1] interval; The original fundus video image sequence containing the optic disc area is resized uniformly, and the sizes of all image frames in the original fundus video image sequence are converted to a standard input size by bilinear interpolation or center cropping; Based on the processed original fundus video image sequence, a sliding window strategy is used to generate an input image pair for tracking modeling. The input image pair includes a template frame and a search frame. The search frame is the current processing frame, indicating the position of the optic disc that needs to be located. The template frame is a frame with high image quality and clear optic disc edge selected from multiple current historical frames, which serves as a stable representation of the reference target structure.

3. The method for dynamic screening of medical image target tracking based on visual Transformer according to claim 2, characterized in that: The input image pairs are divided into blocks and converted into a joint input Token sequence, which is then input into the Transformer network as follows: The Patch Embedding module is used to divide the template frame and the search frame into multiple fixed-size sub-image blocks, which are mathematically represented as patches. A preset linear transformation module or a small convolution module is used to transform all sub-image blocks into a C-dimensional embedded vector representation, which is mathematically represented as a Token. The embedded vector representation is used as the basic calculation unit for modeling to obtain the Token sequence of the template frame. , and the Token sequence of the search frame , where R is a set of real numbers, B is the batch size, is the number of Tokens in the template frame. is the number of tokens in the search frame, and C is the feature dimension of each token; The spatial position information represented by each embedding vector is explicitly encoded and added to the embedding vector representation; The Token sequence of the encoded template frame and the Token sequence of the search frame are concatenated along the sequence dimension to obtain a joint input Token sequence. The joint input Token sequence is used as the input for global modeling and transmitted to the Transformer network backbone structure.

4. The method for dynamic screening of medical image target tracking based on visual Transformer according to claim 3 is characterized in that: A dynamic local interaction module is introduced before each pruning layer of the Transformer network. The dynamic local interaction module is used to perform local context enhancement on the search area token in the joint input token sequence. Specifically: The Transformer network contains 12 coding layers, of which the 3rd, 6th and 9th layers are pruned layers, and the remaining layers are standard coding layers; A query-guided dynamic local interaction module is introduced before each pruning layer. The dynamic local interaction module consists of three parts: a one-dimensional depth-wise separable convolution layer, a point-by-point convolution layer, and a dynamic convolution kernel generator. A one-dimensional depthwise separable convolutional layer is used to perform local structure modeling on each channel token sequence in the joint input token sequence. A point-by-point convolutional layer is used to integrate cross-channel semantics to form contextual key features. The original embedding vector representation is used as the interaction condition between the original query vector and the key feature. The dynamic convolution kernel weight for each position is generated by the dynamic convolution kernel generator, and weighted aggregation is performed according to the neighborhood window to output the context-enhanced token representation.

5. The method for dynamic screening of medical image target tracking based on visual Transformer according to claim 4 is characterized in that: A one-dimensional depth-wise separable convolutional layer is used to perform local structure modeling on each channel token sequence in the joint input token sequence. A point-by-point convolutional layer is used to integrate cross-channel semantics to form contextual key features. The original embedding vector representation is used as the interaction condition between the original query vector and the key feature. The dynamic convolution kernel generator generates the dynamic convolution kernel weight for each position, performs weighted aggregation according to the neighborhood window, and outputs the context-enhanced token representation, specifically: Convert the Token sequence of the search frame in the joint input Token sequence into Structure, input to a one-dimensional depth-separable convolutional layer; The point-by-point convolution layer is used to fuse the output results of the one-dimensional depth-separable convolution layer to generate the fused context key features; The original query vector is interacted with the context key features point by point. A lightweight MLP structure is used to generate dynamic convolution kernel weights unique to each token position. Based on the dynamic convolution kernel weights, adjacent tokens are weighted and aggregated within the local token window to output a context-enhanced token representation. The formula is: , , ,in, , Q, and V are all Token values ​​of the search area. The processed Token value of the search area. is a one-dimensional depth-wise separable convolutional layer, is the point-by-point convolution layer, is a local window, is the activation function, is the calculation function of the attention score, is the weight matrix, is the multiplication operator, is a dynamic local interaction module, Token serial number.

6. The method for dynamic screening of medical image target tracking based on visual Transformer according to claim 5, characterized in that: Based on the attention-guided learnable token filtering and compression mechanism, the joint input token sequence after local context enhancement is filtered and processed. Specifically: The importance scorer, composed of two layers of perception units, dynamically assigns weights to the search frame's token sequence based on the semantic similarity between the search frame's token sequence and the template frame's token sequence at each pruning layer. A weighted ratio is obtained, where the token sequence of the search frame that is closer to the template frame's token sequence and has a higher similarity will receive a higher weight. Extract the attention score of the template area from the token value of each search area from the multi-head attention module of the Transformer network, and perform weighted fusion processing with the weight ratio to generate a joint score for ranking; Perform Top-K filtering based on the joint score, retaining only the most discriminative Token values. The remaining Token values ​​are discarded as redundant information, and their original position indexes are retained to facilitate subsequent feature restoration. Its mathematical formula is: , , , , , , ,in, is the score vector, is the Sigmoid activation function, and are scorer weights, is the activation function, is the first layer bias value, is the second layer bias value, is the rating vector, T is the transpose operation, is the dimension of s, is the attention response sub-matrix of template to search token, Select the interaction between the template area and the search area in the attention matrix, for Dimensions, is the mean attention value of the j-th search token, is the number of attention heads in the multi-head attention mechanism, is the number of Tokens in the template area, is the number of tokens in the search area, is the i-th template region mask, is the batch size, is the dimension of the number of attention heads, To search for token number, is the attention weight of the i-th template Token and the j-th search area Token, is the weight ratio, is the global position index of the search area, is the comprehensive score of the jth Token, Score the importance of the jth Token, It is the comprehensive score of Token.

7. The method for dynamic screening of medical image target tracking based on visual Transformer according to claim 6, characterized in that: After filtering, the values ​​of the retained joint input Token sequence are fused and modeled to generate the position prediction result of the optic disc area of ​​the current frame. Specifically, the standard coding layer of the Transformer network is called to perform multi-level feature fusion and target positioning modeling on the retained Token sequence of the template frame and the Token sequence of the search frame to generate the position prediction result of the optic disc area of ​​the current frame.

8. A dynamic screening medical image target tracking device based on visual Transformer, characterized in that: include: A preprocessing unit is used to obtain an original fundus video image sequence acquired by a preset acquisition device, and perform standardized preprocessing on the original fundus video image sequence to generate an input image pair; The conversion unit is used to process the input image pairs into blocks, convert them into a joint input Token sequence, and input the joint input Token sequence into the Transformer network; The DLIM unit is used to introduce a dynamic local interaction module before each pruning layer of the Transformer network, and use the dynamic local interaction module to perform local context enhancement processing on the search area token in the joint input token sequence; The TSC unit is used for a learnable token filtering and compression mechanism based on attention guidance, filtering the joint input token sequence after local context enhancement; The prediction unit is used to perform fusion modeling on the values ​​of the joint input Token sequence retained after screening to generate a position prediction result of the optic disc area of ​​the current frame.

Images