A dynamic screening medical image target tracking method and device based on visual Transformer

Abstract

The application provides a dynamic screening medical image target tracking method and device based on visual Transformer, and relates to the technical field of computer vision, which standardizes near-infrared or visible light fundus video frames into a unified resolution and constructs a template-search frame pair, then jointly maps the two frames of images into a Token sequence. The overall network embeds a dynamic local interaction module before each pruning layer, captures local context using depth separable convolution and point convolution, and drives a dynamic convolution kernel generator to adaptively weight and aggregate neighboring Tokens. Next, the Token screening and compression mechanism TSC operates within the same pruning layer, retaining only the Top-K key Tokens and pruning redundant Tokens while recording the original index. The purpose is to improve the positioning stability and inference efficiency of the lesion area (such as the optic disc) in complex surgical videos.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular to a dynamic screening medical image target tracking method and device based on visual Transformer. BACKGROUND

[0002] In the field of medical image assisted diagnosis and surgical navigation, real-time and robust tracking of the optic disc of the fundus is a basic task in clinical scenarios such as glaucoma screening, retinal surgery, and intraoperative laser positioning. Due to the complexity of uneven lighting, mirror reflection, blood vessel occlusion, eyeball micro-motion, and instrument interference in fundus imaging, the appearance of the optic disc region in consecutive video frames can change significantly. Traditional tracking methods based on correlation filtering, Siamese networks, or lightweight convolutional neural networks cannot simultaneously consider positioning accuracy, cross-frame stability, and computational efficiency. In recent years, visual Transformer (ViT) has made breakthrough progress in general object tracking with its global self-attention mechanism. However, directly migrating standard ViT to medical videos exposes three major contradictions: first, ViT performs dense token interaction on the entire image or search region, leading to a quadratic increase in computational load with the number of tokens. In the intraoperative real-time requirement (usually above 30 fps), redundant computation cannot be ignored; second, the optic disc only occupies a small part of the entire fundus image, and the background retinal texture and blood vessel structure are complex. Global self-attention is easily overwhelmed by high-response background areas, leading to target drift; third, ViT lacks local structure prior, and its ability to describe low-contrast and weak-edge optic disc boundaries is insufficient. When the image is degraded due to bleeding or sudden changes in lighting, the tracking box is prone to shift or jitter.

[0003] To alleviate the above problems, the industry has tried to use sparse attention, local window attention, or progressive downsampling strategies to compress the number of tokens. However, most of these methods use manual thresholds or fixed ratios for pruning, which cannot adaptively adjust according to the current frame content and may mistakenly delete key tokens when the target is blurred or occluded. In addition, some methods use late fusion between the template branch and the search branch, which limits the model's robustness to subtle deformations. Some research has introduced deformable convolution or dynamic filters to enhance local modeling, but the convolution kernel parameters are usually independent of the global context and are disconnected from the self-attention mechanism of the Transformer, failing to form a unified end-to-end optimization framework. In short, although the existing mainstream visual tracking framework performs well in terms of performance, it still has problems such as large search region feature computation redundancy, lack of local modeling ability, and weak target perception.

[0004] Therefore, the present application is proposed. SUMMARY

[0005] The application provides a visual Transformer-based dynamic screening medical image target tracking method and device, which can at least partially improve the above problems.

[0006] To achieve the above object, the application adopts the following technical solutions:

[0007] A visual Transformer-based dynamic screening medical image target tracking method, comprising:

[0008] An original fundus video image sequence collected by a preset collection device is acquired, and the original fundus video image sequence is standardized and preprocessed to generate an input image pair;

[0009] The input image pair is processed in blocks, a joint input Token sequence is converted, and the joint input Token sequence is input into a Transformer network;

[0010] A dynamic local interaction module is introduced before each pruning layer of the Transformer network, and the search region Token in the joint input Token sequence is processed by the dynamic local interaction module for local context enhancement;

[0011] A learnable Token screening and compression mechanism based on attention guidance is used to screen the joint input Token sequence after the local context enhancement;

[0012] The values of the joint input Token sequence remaining after the screening are fused and modeled to generate a position prediction result of the current frame disc region.

[0013] The application also provides a visual Transformer-based dynamic screening medical image target tracking device, comprising:

[0014] A preprocessing unit is configured to acquire an original fundus video image sequence collected by a preset collection device, and to standardize and preprocess the original fundus video image sequence to generate an input image pair;

[0015] A conversion unit is configured to process the input image pair in blocks, convert a joint input Token sequence, and input the joint input Token sequence into a Transformer network;

[0016] a DLIM unit configured to introduce a dynamic local interaction module before each pruning layer of the Transformer network, and perform local context enhancement processing on search region Tokens in the joint input Token sequence using the dynamic local interaction module;

[0017] a TSC unit configured to perform filtering processing on the joint input Token sequence after the local context enhancement processing based on an attention-guided learnable Token selection and compression mechanism;

[0018] a prediction unit configured to perform fusion modeling processing on values of the joint input Token sequence remaining after the filtering, and generate a position prediction result of the optic disc region of the current frame.

[0019] In summary, the dynamic filtering medical image target tracking method based on the visual Transformer aims to improve the positioning stability and inference efficiency of the lesion region (such as the optic disc) in a complex surgical video. The method constructs a unified Transformer backbone network with a total of 12 encoder layers, of which the 3rd, 6th, and 9th layers are Token pruning layers (Token selection and compression mechanism, TSC), and the remaining layers are standard encoding layers. A dynamic local interaction module (DLIM) is introduced before each TSC layer to preferentially perform local context semantic enhancement on the search region Tokens. The DLIM module extracts context structure information through deep convolution and point convolution, and generates a dynamic convolution kernel by combining the query feature to adaptively weight and aggregate the neighborhood features, thereby improving the representation ability of the Tokens in the local region. Subsequently, the TSC mechanism assigns weight ratio results by fusing attention scores and a learnable importance scorer, sorts the search region Tokens, and dynamically retains the Top-K Tokens with the highest discriminability, while eliminating the remaining redundant Tokens, thereby significantly reducing invalid calculations. At the same time, to maintain structural consistency and sequence integrity, the pruning process records the original Token index to ensure that the subsequent layers can successfully restore and aggregate the features.

[0020] In addition, to enhance the target-oriented feature modeling capability, the method also adopts a template-search frame bidirectional interaction strategy: the Tokens of the template frame and the search frame are directly concatenated and input in the initial stage of the Transformer to realize target semantic conduction from the shallow to the deep layers, thereby improving the model's perception ability of the optic disc region and the background interference suppression ability. The overall method combines the context enhancement capability of the DLIM, the structure sparsity control of the TSC, and the information fusion advantages of the double-flow interaction, while maintaining the compactness of the model, significantly improves the accuracy and cross-frame stability of the optic disc positioning, and is particularly suitable for medical image intelligent processing application scenarios such as intraoperative optic disc tracking and ophthalmic surgery assisted navigation. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 is a flowchart of a dynamic screening medical image target tracking method based on visual Transformer provided by the first embodiment of the present application;

[0022] Figure 2 is a whole structure diagram of the dynamic screening medical image target tracking method based on visual Transformer provided by the embodiment of the present application;

[0023] Figure 3 is a structure diagram of the DLIM module provided by the embodiment of the present application;

[0024] Figure 4 is a structure diagram of the TSC module provided by the embodiment of the present application;

[0025] Figure 5 is a module schematic diagram of the dynamic screening medical image target tracking device based on visual Transformer provided by the second embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to make the objectives, technical solutions and advantages of the present application clearer, further detailed description will be made to the present application with reference to embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0027] Reference Figure 1 , Figure 2 The first embodiment of the present application discloses a dynamic screening medical image target tracking method based on visual Transformer, which can be executed by a dynamic screening medical image target tracking device based on visual Transformer (hereinafter referred to as tracking device), and in particular, executed by one or more processors in the tracking device to realize the following method:

[0028] S1, obtaining an original fundus video image sequence collected by a preset collection device, and performing standardization preprocessing on the original fundus video image sequence to generate an input image pair;

[0029] Specifically, step S1 further comprises: obtaining a medical video collected by a preset near-infrared fundus camera or a visible light fundus camera, and obtaining an original fundus video image sequence containing an optic disc region from the medical video frame by frame;

[0030] The original fundus video image sequence containing the optic disc region is standardized and preprocessed to generate an input image pair, and the specific steps are as follows:

[0031] The original fundus video image sequence containing the optic disc region is subjected to pixel normalization processing, and the gray scale range or RGB channel value of the image is uniformly linearly mapped to the interval [0, 1];

[0032] The original fundus video image sequence containing the optic disc region is subjected to size unification processing, and the size of all image frames in the original fundus video image sequence is converted to a standard input size by means of bilinear interpolation or center cropping;

[0033] Based on the processed original fundus video image sequence, an input image pair for tracking modeling is generated by using a sliding window strategy, the input image pair comprising a template frame and a search frame, wherein the search frame is a current processing frame, representing the position of the optic disc that needs to be located at present, and the template frame is an image with good image quality and clear optic disc edge selected from the current multiple historical frames, serving as a stable representation of the reference target structure.

[0034] In the embodiment, in order to realize stable tracking of the optic disc region in the fundus image, the medical image frames need to be acquired and standardized pretreated. The original image data is acquired frame by frame from the medical video, which is commonly used in the scene of ophthalmic surgery navigation or diagnosis recording, to construct an image input source with time sequence continuity.

[0035] Due to the variety of acquisition hardware, considering the differences in resolution, illumination, noise, brightness, image proportion and other factors of each image in the actual acquisition process, in order to improve the modeling robustness of the subsequent visual Transformer network, the image frames are further standardized pretreated. First, the gray scale range or RGB channel value of the image is uniformly scaled to the interval [0, 1], thereby reducing the influence of different acquisition devices or illumination conditions on the input distribution of the model, effectively reducing the influence of device photosensitivity difference on image contrast, and improving the perception ability of weak edges (such as the optic disc boundary). Secondly, the size of all image frames is converted to a standard input size (such as 224x224 pixels) by means of bilinear interpolation or center cropping, so as to ensure that a consistent number of Tokens are generated when patch division is performed subsequently, and to avoid unstable performance of the model under different resolution inputs; to ensure that the number of Tokens obtained after subsequent blocking operation is consistent, and to avoid feature confusion or model learning efficiency reduction caused by irregular input. Finally, in order to adapt to the inter-frame dependence characteristics of the tracking task, a sliding window strategy is used to generate an image pair for modeling, i.e. the input image pair; wherein the current processing frame is taken as the search frame (Search Frame), and a frame with good image quality or no obvious occlusion in the optic disc region is usually selected as the template frame (Template Frame), to provide structural guidance and attention guidance information, i.e. to provide target reference information.

[0036] In the embodiment, the template frame and the search frame are jointly used as the input basis of the method, and the cooperative relationship between the two determines the accuracy of the target structure guidance and the stability in the subsequent tracking process. The preprocessing step plays a crucial role in the method and directly affects the quality of subsequent feature modeling and the accuracy of prediction results. Through this standardized image acquisition process, a clean, unified, and structured image input pair is established, laying a solid foundation for subsequent model processing.

[0037] S2, the input image pair is processed by block, and the joint input Token sequence is obtained, and the joint input Token sequence is input into the Transformer network;

[0038] Specifically, step S2 further includes: using a Patch Embedding module to divide and process the template frame and the search frame to obtain a plurality of fixed-size sub-image blocks, which are mathematically represented as patch.

[0039] All sub-image blocks are converted into C-dimensional embedding vector representations using a preset linear transformation module or a small convolution module, which are mathematically represented as Token. The embedding vector representation is used as a basic calculation unit for modeling, and the Token sequence of the template frame , and the Token sequence of the search frame , wherein B is the batch size, is the Token number of the template frame, is the Token number of the search frame, and C is the feature dimension of each Token.

[0040] The spatial position information of each embedding vector representation is explicitly coded and added to the embedding vector representation.

[0041] The encoded Token sequence of the template frame and the Token sequence of the search frame are spliced along the sequence dimension to obtain a joint input Token sequence , and the joint input Token sequence is used as the input of global modeling and transmitted to the Transformer network backbone structure.

[0042] In the embodiment, step S2 specifically describes the image embedding process of constructing structured Token representation. Specifically, after image processing, in order to adapt to the input form of the Transformer network, the first stage of visual Transformer modeling is needed, that is, to convert the structured image into a Token sequence that can be input into the neural network, that is, a sequence of Token data.

[0043] Firstly, the template frame and the search frame are respectively divided into a plurality of fixed-size sub-image blocks (patches) by using a Patch Embedding module, for example, a 224x224 image is divided into 14x14=196 non-overlapping patches with 16x16 as a unit. After each patch is expanded into a vector, it is converted into a vector representation by a convolutional projection layer or a fully connected layer (which can also be understood as a linear transformation module or a small convolutional module), becoming the basic calculation unit for modeling in the network, i.e., Token.

[0044] Further, for each image, the operation is performed respectively, and finally the Token sequence of the template frame and the Token sequence of the search frame are obtained. In addition, in order to preserve the spatial structure information, i.e., the spatial order of each region in the image structure, a two-dimensional positional encoding (Positional Embedding) is added to each Token to reflect its original position, ensuring that the model can perceive the spatial arrangement of the Token in the attention mechanism, so that the model can learn the spatial dependency between regions.

[0045] After obtaining the Token representation of the template and the search, the Token sequence of the template frame and the Token sequence of the search frame are concatenated in the sequence dimension to form a joint input sequence, which is then input into the Transformer backbone structure for subsequent modeling. This step realizes the mapping conversion from the image space to the sequence space, provides a structured representation basis for the contrast modeling of the Transformer across the image regions, and is the core pre-process for building the input format of the Transformer model, which provides a semantic unified representation space for subsequent context enhancement, pruning compression and feature modeling.

[0046] Please refer to Figure 3 , S3, a dynamic local interaction module is introduced before each pruning layer of the Transformer network, and the search region Token in the joint input Token sequence is processed by the dynamic local interaction module for local context enhancement;

[0047] Specifically, step S3 further includes that the Transformer network includes 12 encoding layers, of which the 3rd layer, the 6th layer and the 9th layer are pruning layers, and the remaining layers are standard encoding layers.

[0048] A Query-guided dynamic local interaction module is introduced before each pruning layer, wherein the dynamic local interaction module includes a one-dimensional depth separable convolution layer, a pointwise convolution layer and a dynamic convolution kernel generator.

[0049] Each channel token sequence in the joint input token sequence is processed by a one-dimensional deep separable convolution layer for local structure modeling, a point-wise convolution layer is used to integrate cross-channel semantics to form context key features, and the original embedding vector representation is used as the interaction condition of the original query vector and the key features, a dynamic convolution kernel generator is used to generate dynamic convolution kernel weights for each position, and a neighborhood window is used for weighted aggregation to output context-enhanced token representation, specifically:

[0050] The token sequence of the search frame in the joint input token sequence is converted into structure, input into a one-dimensional deep separable convolution layer;

[0051] The output of the one-dimensional deep separable convolution layer is processed by a point-wise convolution layer for information fusion to generate fused context key features;

[0052] The original query vector and the context key features are interacted point by point, a lightweight MLP structure is used to generate dynamic convolution kernel weights for each token position, and based on the dynamic convolution kernel weights, adjacent tokens are weighted and aggregated within the local window of the token to output context-enhanced token representation;

[0053] The formula is: , , wherein, , Q, V are token values of the search area, is the processed token value of the search area, is a one-dimensional deep separable convolution layer, is a point-wise convolution layer, is a local window, is an activation function, is a calculation function of attention score, is a weight matrix, is a multiplication operator, is a dynamic local interaction module, is a token number.

[0054] In this embodiment, the entire Transformer network encoder contains 12 layers of Block (encoding layer), among which the non-pruned layers serve as the context accumulation and long-distance dependence modeling of the standard Transformer Encoder implementation, and the pruned layers are enhanced by DLIM before applying the TSC selection mechanism. Among them, the 3rd, 6th and 9th layers integrate the TSC module to perform Token selection, and the remaining layers serve as standard Encoder layers for sequence modeling and context fusion. Among them, a dynamic local interaction module DLIM is introduced before each pruned layer to enhance the local context of the search area Token. The context semantic information is obtained by combining deep convolution and point convolution, and the dynamic kernel weight is generated based on the query to realize adaptive modeling.

[0055] Specifically, considering that the target region (such as the optic disc) in the medical image usually has strong local consistency in space, in order to improve the search area Token's ability to perceive spatial local changes, a local context enhancement module, namely a dynamic local interaction module (DLIM), is introduced before the Transformer backbone encoding to strengthen the local feature expression ability of the search Token. This module uses a dynamic convolution mechanism to model the structural dependence between Tokens and their neighborhoods, thereby improving the fine-grained features. This module is different from the traditional attention mechanism or sparse enhancement strategy, and it establishes a local modeling path guided by structural prior through the "Token adaptive dynamic convolution" mechanism.

[0056] Among them, the dynamic local interaction module is used to enhance the local modeling ability of the Token in the search area, and its structure is composed of one-dimensional depth separable convolution (Depthwise Conv), pointwise convolution (Pointwise Conv) and dynamic convolution kernel generator. In actual use, first, the search Token sequence is converted to The structure is input into a depthwise one-dimensional convolution, which is independently processed in units of channels and can efficiently extract the structural relationship between locally adjacent tokens. Then, the output result is input into a pointwise convolution layer to realize information fusion across channels, thereby generating a fused context key feature. At this time, a fusion representation is obtained by point-by-point interaction (point multiplication) between the original query vector and the generated key feature. In order to realize adaptive modeling of tokens according to the interaction feature, a lightweight MLP structure is used to generate dynamic convolution kernel weights specific to each token position. Subsequently, within the local window of the token, the adjacent tokens are weighted and aggregated according to the dynamic kernel, and a context-enhanced token sequence is output, realizing local context enhancement. This module simulates the self-attention mechanism in a local range, with lower computational cost and stronger adaptability. This process not only simulates the dependency modeling capability of the self-attention mechanism in a local range, but also has lower computational overhead and stronger position awareness, especially suitable for maintaining the clarity and stability of token features when the edge of the optic disc is blurred or the background interference is strong.

[0057] The dynamic local interaction module can realize structure-adaptive local enhancement, and can still retain local discrimination ability when the target region edge is blurred or the contrast is weak. Compared with the traditional sparse spatial attention mechanism, the method introduces query participation in modeling and dynamically adjusts the convolution kernel, enhancing the expressiveness and robustness of the model in structure-uncertain regions.

[0058] Please refer to Figure 4 , S4, a learnable Token filtering and compression mechanism based on attention-guided, filters the joint input Token sequence after local context enhancement processing;

[0059] Specifically, step S4 further comprises: through an importance scorer composed of two layers of perception units, dynamically allocating weights based on the semantic similarity between the Token sequence of the search frame and the Token sequence of the template frame at each pruning layer to obtain a weight ratio, wherein the Token sequence of the search frame that is closer to the Token sequence of the template frame and has a higher similarity will obtain a higher weight;

[0060] The attention score of the Token value of each search region to the template region is extracted from the multi-head attention module of the Transformer network, and is weighted and fused with the weight ratio to generate a joint score for sorting;

[0061] According to the joint score, Top-K filtering processing is performed, only the most discriminative Token value is retained, the rest of the Token values are discarded as redundant information, and the original position index is retained to facilitate subsequent feature restoration;

[0062] The mathematical formula is: , , , , , , wherein, is a score vector, is a Sigmoid activation function, and are both scorer weights, is an activation function, is a first layer bias value, is a second layer bias value, is a score vector, and T is a transpose operation, is a dimension of s, is a sub-matrix of attention response of the template to the search token, is an interaction part between the selected template area and the search area in the attention matrix, is a dimension of is an attention mean value of the jth search token, is a number of attention heads in the multi-head attention mechanism, is a number of tokens in the template area, is a number of tokens in the search area, is an i-th template token mask, is a batch size, is an attention head dimension, is a search token number, is an attention weight of the i-th template token and the j-th search area token, is a weight ratio, is a global position index of the search area, is a comprehensive score of the j-th token, is an importance score of the j-th token, is a token comprehensive score.

[0063] In this embodiment, in order to reduce the redundant calculation burden generated by the Transformer model in processing the search area Token and effectively control the calculation complexity brought by the Transformer in processing the large-scale search area Token. With the help of a kind of attention guided learnable Token screening and compression mechanism (Token Selection and Compression, TSC), the search area Token sequence is dynamically screened and structured at each pruning layer, that is, the search area Token importance weight learned based on the template Token is combined with the attention score to dynamically screen the key Token and accurately remove the redundant or background area Token.

[0064] The mechanism dynamically allocates weights by introducing a learnable importance scorer in combination with the context semantic information of the search area Token. The scorer uses a two-layer fully connected structure to generate a normalized scalar to measure the contribution of the Token to the current task. The scorer dynamically generates the weight ratio of each search area Token based on the template Token, and combines it with the attention score of the corresponding Token to form a comprehensive ranking index. The Top-K reservation strategy is used to select the most important Token, remove the redundant Token, and maintain the index mapping table to restore the original order and structural consistency of the Token in the subsequent level, thereby improving the model tracking accuracy while significantly improving the calculation efficiency.

[0065] Specifically, first, the importance scorer composed of two layers of perception units dynamically allocates weights based on the semantic similarity between the search Token and the template Token at each pruning layer — the search Token that is closer to the template Token and has higher similarity will get higher weight, and then generates a normalized importance score to quantify the semantic effectiveness of each Token in the current task.

[0066] Meanwhile, the attention intensity of each search Token to the template region is extracted from the multi-head attention module of the Transformer, i.e. the attention score of each search Token, which is weighted and fused with the output weight of the importance scorer to form a final joint score (joint ranking indicator) for ranking. The system performs Top-K screening according to the fusion score, only retains the most discriminative Token, and the remaining Token is discarded as redundant information, and its original position index is retained for subsequent feature restoration, while its original index is recorded for subsequent hierarchical Token sequence and structure reconstruction. In short, the Transformer structure adopts an integrated template-search bidirectional interaction strategy, which parallelly inputs template Token and search Token in each Transformer encoder, and establishes direct information flow through multi-head self-attention mechanism, ensuring that the structural semantics of the optic disc in the template frame can be transmitted layer by layer to the search frame, strengthening the search region's attention to the target and improving the robustness and consistency of optic disc region positioning.

[0067] Traditional static pruning methods, such as fixed ratio clipping or preset energy threshold filtering. These methods often use uniform and invariant screening rules, which cannot adapt to the dynamic changes of input data and the specificity of task context; they rely on single attention sparsity, fixed attention threshold or non-learnable rigid pruning rules, which lack the ability to dynamically adapt to task scenarios. In the case of ambiguous target boundaries or dramatic context changes, key Token deletion or redundant Token retention is likely to occur, resulting in a significant decline in screening stability.

[0068] In comparison, the TSC mechanism of the present method exhibits essential differences. It has learnability, adaptability and structural reduction capability, and can dynamically adjust the retention strategy according to the real-time importance of the Token in a specific context. This dynamic characteristic makes it particularly outstanding in medical image scenes where the target area features are ambiguous and the background interference is complex - it can not only accurately retain key information, but also effectively eliminate redundant data. Combined with the local context-enhanced features provided by the preposed DLIM module, the TSC mechanism not only significantly improves the accuracy and stability of Token screening, but also maintains the model's expression ability while reducing computational complexity, achieving dual optimization of efficiency and tracking robustness. The present method explicitly introduces a trainable scoring network and deeply integrates an attention-guided mechanism to build a "semantic-driven + attention-aware" dual-track collaborative selection strategy: on the one hand, the learnable scorer can learn task-specific knowledge from data and dynamically quantify the semantic importance of Tokens; on the other hand, the contextual awareness of the screening decision is further enhanced by the associated information provided by the attention mechanism. This design makes the screening process not only more robust, but also more flexible in adapting to different inter-frame differences and target shape changes, making it particularly outstanding in medical image tracking scenarios with complex backgrounds and frequent occlusions. With the local structure-enhanced features provided by the preposed DLIM module, the TSC pruning mechanism not only significantly reduces computational overhead (efficiently eliminates redundant Tokens), but also accurately retains key semantic representations, thereby significantly improving the task adaptability and inference efficiency of the Transformer model, achieving a synergistic optimization of performance and efficiency.

[0069] S5, after screening, the values of the retained joint input Token sequence are fused and modeled to generate the position prediction result of the current frame's optic disc region.

[0070] Specifically, step S5 further includes: calling the standard encoding layer of the Transformer network to perform multi-level feature fusion and target positioning modeling on the Token sequence of the template frame and the Token sequence of the search frame, and generate the position prediction result of the current frame's optic disc region.

[0071] In the present embodiment, after global feature modeling and fusion via the remaining Transformer encoding layers, the position prediction result of the current frame's optic disc region is output. The present method uses a unified Transformer structure to perform multi-level feature encoding and cross-region fusion modeling, and includes a bidirectional interaction mechanism for template Tokens and search Tokens in each encoding module, i.e., through a multi-head self-attention (Multi-Head Attention) structure, the two can directly fuse information in the representation space, allowing the target structure in the template to guide the attention focusing direction of the search region layer by layer.

[0072] In each layer, the template Token is input together with the search Token, and the information of both is bidirectionally transmitted through the multi-head attention mechanism. Compared with the traditional method of branch coding and then modeling, this method constructs a joint structure of template-search from the beginning, so that the target structure information can continuously affect the feature extraction and positioning strategy of the search area, improving the consistent perception of the target in the tracking process. At the end of the network, the feature embedding for position determination is extracted from the search Token sequence and input into the positioning prediction head module. This module can be implemented in various forms, such as fully connected layers, convolutional layers, or attention fusion regression structures, and the output can be center point coordinates, bounding box parameters, etc. The final prediction result is fed back to the tracking system and can be used for template updating, inter-frame navigation, or integrated with the fundus surgery system to achieve real-time, high-precision, and low-latency disc tracking capability. This encoding and output process fully integrates context awareness and spatial alignment capability, which is the last key step for the system to achieve efficient medical image modeling.

[0073] This structure forms an alternating interlayer strategy of "enhancement-compression-modeling" in the calculation path, greatly improving the running efficiency while maintaining the model's expression ability. After the Transformer backbone processing is completed, the target representation features in the search Token are extracted and input into the positioning prediction head (such as a linear regressor or MLP) to output the position information of the current frame disc. The prediction can be expressed in the form of center coordinates, bounding box parameters, or heat maps. The final output result can be used for template updating, tracking visualization, or integrated medical analysis system to achieve real-time, high-precision disc tracking and intelligent auxiliary diagnosis functions.

[0074] In simple terms, the DLIM module is fixed in the context modeling link before the 3rd, 6th, and 9th pruning layers to enhance the local representation of Tokens; while the TSC module completes feature selection and structure compression within the corresponding pruning layer. Different pruning layers can set different Token retention ratios to adapt to the representation needs at different depths. This modular combination of DLIM + TSC not only improves the Token feature expression capability, but also significantly reduces redundant calculations, building an efficient Transformer encoding framework that balances content awareness and structural compactness.

[0075] It should be noted that the dynamic screening medical image target tracking method based on visual Transformer can be applied to continuous detection tasks of the disc area in infrared or visible light fundus video, has enhanced cross-frame stability, is robust to background interference such as occlusion, blur, and light changes, and is suitable for medical scenarios such as fundus examination, intraoperative navigation, and lesion tracking.

[0076] In summary, the method converts the fundus video frames into Token sequences after unified preprocessing, splices the template frames and search frame Tokens directly into the 12-layer integrated Transformer, uses the dynamic local interaction module (DLIM) to generate a dynamic convolution kernel guided by the query before the 3rd, 6th, and 9th pruning layers, performs adaptive weighted enhancement on the neighborhood Tokens, combines the learnable candidate importance scorer and attention score for Top-K screening, dynamically removes redundant Tokens, records the index to maintain structural consistency, and enables the target semantics to continuously guide the search area from the shallow layer to the deep layer. Finally, the optic disc position is output after multi-layer bidirectional interaction and global modeling, realizing real-time optic disc tracking with low latency, high precision, anti-occlusion, and illumination changes, and being suitable for medical scenarios such as fundus screening and intraoperative navigation.

[0077] Please refer to Figure 5 The second embodiment of the present application provides a visual Transformer-based dynamic screening medical image target tracking device, which comprises:

[0078] A preprocessing unit 101 is configured to acquire an original fundus video image sequence collected by a preset acquisition device, perform standardization preprocessing on the original fundus video image sequence, and generate an input image pair.

[0079] A conversion unit 102 is configured to perform block processing on the input image pair, convert a joint input Token sequence, and input the joint input Token sequence into a Transformer network.

[0080] A DLIM unit 103 is configured to introduce a dynamic local interaction module before each pruning layer of the Transformer network, and perform local context enhancement processing on the search region Tokens in the joint input Token sequence using the dynamic local interaction module.

[0081] A TSC unit 104 is configured to perform screening processing on the joint input Token sequence after the local context enhancement processing based on an attention-guided learnable Token screening and compression mechanism.

[0082] A prediction unit 105 is configured to perform fusion modeling processing on the values of the joint input Token sequence retained after the screening, and generate a position prediction result of the current frame optic disc region.

[0083] The above describes the preferred embodiments of the present application. It should be noted that those skilled in the art can make several improvements and refinements without departing from the principles of the present application, and these improvements and refinements are also considered within the protection scope of the present application.

Claims

1. A visual Transformer-based dynamic screening medical image target tracking method, characterized in that, The application relates to a method for predicting the position of an optic disc region in an eye fundus video, and belongs to the technical field of computer vision. The method comprises the following steps: acquiring an original eye fundus video image sequence collected by a preset collection device, and performing standardization preprocessing on the original eye fundus video image sequence to generate an input image pair; performing block processing on the input image pair to convert a joint input Token sequence, and inputting the joint input Token sequence into a Transformer network; introducing a dynamic local interaction module before each pruning layer of the Transformer network, and performing local context enhancement processing on search area Tokens in the joint input Token sequence by using the dynamic local interaction module; performing screening processing on the joint input Token sequence after the local context enhancement processing based on an attention-guided learnable Token screening and compression mechanism; performing fusion modeling processing on the values of the joint input Token sequence remaining after the screening to generate a position prediction result of a current frame optic disc region; introducing a dynamic local interaction module before each pruning layer of the Transformer network, and performing local context enhancement processing on search area Tokens in the joint input Token sequence by using the dynamic local interaction module, specifically as follows: the Transformer network comprises 12 encoding layers, wherein the third layer, the sixth layer and the ninth layer are pruning layers, and the remaining layers are standard encoding layers; a Query-guided dynamic local interaction module is introduced before each pruning layer, wherein the dynamic local interaction module comprises a one-dimensional depth separable convolution layer, a pointwise convolution layer and a dynamic convolution kernel generator; 2. The method of claim 1, wherein the method is a visual Transformer-based dynamic screening medical image object tracking method. the one-dimensional depth separable convolution layer is used for performing local structure modeling processing on each channel Token sequence in the joint input Token sequence, the pointwise convolution layer is used for integrating cross-channel semantics to form context Key features, the original embedding vector representation is used as an interaction condition of an original query vector and the Key features, a dynamic convolution kernel weight of each position is generated by using the dynamic convolution kernel generator, weighted aggregation is performed in a neighborhood window, and a context-enhanced Token representation is output. acquiring an original eye fundus video image sequence collected by a preset collection device, and performing standardization preprocessing on the original eye fundus video image sequence to generate an input image pair, specifically as follows: acquiring a medical video collected by a preset near-infrared eye fundus camera or a visible light eye fundus camera, and acquiring an original eye fundus video image sequence containing an optic disc region from the medical video frame by frame; performing standardization preprocessing on the original eye fundus video image sequence containing the optic disc region to generate an input image pair, and the specific steps are as follows: performing pixel normalization processing on the original eye fundus video image sequence containing the optic disc region, and uniformly linearly mapping the gray scale range or the RGB channel value of the image to the [0, 1] interval; performing size unification processing on the original eye fundus video image sequence containing the optic disc region, and converting the size of all image frames in the original eye fundus video image sequence into a standard input size by using a bilinear interpolation method or a center cropping method; Based on the processed original fundus video image sequence, a sliding window strategy is used to generate an input image pair for tracking modeling, which includes a template frame and a search frame, wherein the search frame is the current processing frame, representing the current position of the optic disc that needs to be located, and the template frame is an image with good image quality and clear optic disc edge selected from the current multiple historical frames as a stable representation of the reference target structure.

3. The method of claim 2, wherein the method is a visual Transformer-based dynamic screening medical image target tracking method. The input image pair is subjected to block processing to obtain a joint input Token sequence, which is input into the Transformer network, specifically: The template frame and the search frame are divided using the Patch Embedding module to obtain multiple fixed-size sub-image blocks, which are mathematically represented as patch. All sub-image blocks are converted into embedding vector representations coded as C dimensions using a pre-set linear transformation module or small convolution module, which is mathematically expressed as Token, the embedding vector representation is used as a basic calculation unit for modeling, and a Token sequence of the template frame is obtained , and a Token sequence of the search frame , wherein R is a real set, B is a batch size, is the number of Tokens of the template frame, is the number of Tokens of the search frame, and C is the feature dimension of each Token; The spatial position information represented by each embedding vector is explicitly coded and added to the embedding vector representation. The Token sequence of the coded template frame and the Token sequence of the search frame are spliced along the sequence dimension to obtain a joint input Token sequence The joint input Token sequence is transmitted to a Transformer network backbone structure as an input of global modeling.

4. The method of claim 3, wherein the method is a visual Transformer-based dynamic screening medical image object tracking method. A one-dimensional depth separable convolution layer is used to model the local structure of each channel Token sequence in the joint input Token sequence, a point-by-point convolution layer is used to integrate cross-channel semantics to form context Key features, and the original embedding vector representation is used as the interaction condition between the original query vector and the Key features. A dynamic convolution kernel generator is used to generate dynamic convolution kernel weights for each position, which are weighted and aggregated in a neighborhood window to output context-enhanced Token representations, specifically: transforming the token sequence of the search frame in the joint input token sequence into a structure, into a one-dimensional depth separable convolutional layer; A point-by-point convolution layer is used to fuse the output of the one-dimensional depth separable convolution layer to generate fused context Key features. The original query vector and the context Key features are interacted point by point, and a lightweight MLP structure is used to generate dynamic convolution kernel weights specific to each Token position. Based on the dynamic convolution kernel weights, adjacent Tokens are weighted and aggregated within the Token local window to output context-enhanced Token representations. The formula is: , , wherein, , Q, V are Token values of the search area, is a processed Token value of the search area, is a one-dimensional depth separable convolution layer, is a point-wise convolution layer, is a local window, is an activation function, is a calculation function of the attention score, is a weight matrix, is a multiplication operator, is a dynamic local interaction module, is a Token serial number.

5. The visual Transformer-based dynamic screening medical image target tracking method according to claim 4, characterized in that, Based on the attention-guided learnable Token filtering and compression mechanism, the joint input Token sequence after local context enhancement processing is filtered, specifically: An importance scorer composed of two perception units performs dynamic weight allocation based on the semantic similarity between the Token sequence of the search frame and the Token sequence of the template frame at each pruning layer to obtain a weight ratio, wherein the Token sequence of the search frame that is closer to the Token sequence of the template frame and has a higher similarity will obtain a higher weight. The attention score of the Token value of each search region to the template region is extracted from the multi-head attention module of the Transformer network, and it is weighted and fused with the weight ratio to generate a joint score for sorting. Top-K filtering is performed according to the joint score, only the most discriminative Token value is retained, the remaining Token values are discarded as redundant information, and their original position indexes are retained for subsequent feature restoration. The mathematical formula is: , , , , , , wherein, is a score vector, is a Sigmoid activation function, and are scorer weights, is an activation function, is a first layer bias value, is a second layer bias value, is a score vector, and T is a transpose operation, is the dimension of s, is a sub-matrix of the attention response of the template to the search token, is the interaction part between the selected template area and the search area in the attention matrix, is the dimension of is the attention mean of the jth search token, is the number of attention heads in the multi-head attention mechanism, is the number of tokens in the template area, is the number of tokens in the search area, is the i-th template token mask, is the batch size, is the attention head dimension, is the search token number, is the attention weight of the i-th template token and the j-th search area token, is the weight ratio, is the global position index of the search area, is the comprehensive score of the jth token, is the importance score of the jth token, is the token comprehensive score.

6. The method of claim 5, wherein the method further comprises: After screening, the values of the joint input Token sequence remaining are fused and modeled to generate a position prediction result of the optic disc area of the current frame, specifically: calling the standard encoding layer of the Transformer network to perform multi-level feature fusion and target positioning modeling on the Token sequence of the template frame and the Token sequence of the search frame, and generate a position prediction result of the optic disc area of the current frame.

7. A dynamic screening medical image target tracking device based on visual Transformer, characterized in that, Comprise: A preprocessing unit is configured to acquire an original fundus video image sequence collected by a preset acquisition device, and perform standardization preprocessing on the original fundus video image sequence to generate an input image pair; A conversion unit is configured to perform block processing on the input image pair to convert a joint input Token sequence, and input the joint input Token sequence into a Transformer network; A DLIM unit is configured to introduce a dynamic local interaction module before each pruning layer of the Transformer network, and use the dynamic local interaction module to perform local context enhancement processing on the search area Token in the joint input Token sequence; A TSC unit is configured to perform screening processing on the joint input Token sequence after local context enhancement processing based on an attention-guided learnable Token screening and compression mechanism; A prediction unit is configured to perform fusion modeling processing on the values of the joint input Token sequence remaining after screening to generate a position prediction result of the optic disc area of the current frame; A dynamic local interaction module is introduced before each pruning layer of the Transformer network, and the dynamic local interaction module is used to perform local context enhancement processing on the search area Token in the joint input Token sequence, specifically: The Transformer network comprises 12 encoding layers, of which the 3rd layer, the 6th layer and the 9th layer are pruning layers, and the remaining layers are standard encoding layers; A Query-guided dynamic local interaction module is introduced before each pruning layer, wherein the dynamic local interaction module comprises a one-dimensional depth separable convolution layer, a pointwise convolution layer and a dynamic convolution kernel generator; The one-dimensional depth separable convolution layer is used to perform local structure modeling processing on each channel Token sequence in the joint input Token sequence, the pointwise convolution layer is used to integrate cross-channel semantics to form context Key features, and the original embedding vector representation is used as the interaction condition of the original query vector and the Key features, the dynamic convolution kernel generator is used to generate the dynamic convolution kernel weight of each position, the weighted aggregation is performed in the neighborhood window, and the context enhanced Token representation is output.

Images