Scene text tracking method and system based on semantic guidance and structure correction

Abstract

The invention discloses a scene text tracking method and system based on semantic guidance and structure correction, and belongs to the technical field of image or video recognition. The method comprises the following steps: carrying out feature interaction on a template token sequence and a search area token sequence, and outputting an original search area visual feature map; performing predictive token correction to obtain a corrected feature map; performing cross-expert calibration on the template frame and the search frame to generate a text semantic calibration mask; multiplying the corrected feature map by a text semantic calibration mask element by element to obtain a final fusion feature, inputting the final fusion feature into a prediction head for position prediction, and outputting a response map containing a target center coordinate and a size; and performing adaptive adjustment on the search scale and outputting a prediction result, and then fusing a Kalman filtering result of the constant velocity motion model to update a final target position. According to the method, the problems of structural imbalance and deformation are solved, and stable motion prediction is realized, so that efficient and high-precision specific text instance tracking is realized.

Description

Technical Field

[0001] This invention relates to the field of image or video recognition technology, and in particular to a scene text tracking method and system based on semantic guidance and structural correction. Background Technology

[0002] Text instances in videos serve as crucial semantic media for scene understanding, human-computer interaction, and assistive technologies. Currently, the mainstream paradigm for processing video text is video text localization, which aims to simultaneously detect, track, and identify all text instances. While this end-to-end approach provides comprehensive semantic understanding, its core process typically employs a detection-then-tracking model. This involves locating text using a detector in each frame and then connecting the detection boxes from different frames into a trajectory using an association algorithm. Video text localization methods are computationally intensive, requiring frame-by-frame detection. When text instances are occluded, blurred, or undergo drastic deformation, detectors are prone to failure and missed detections, leading to fragmented trajectories that cannot meet the requirement for continuous and stable tracking of specific text instances. Another related background technology is general visual object tracking. These methods are typically based on Siamese networks or Transformer architectures, using joint feature extraction and relationship modeling to find targets in search frames that match template frames.

[0003] General visual object tracking methods are poorly adapted to text deformation. Scene text, as a special type of planar object, undergoes drastic perspective deformation when the camera moves. General trackers typically employ a strong backbone and weak predictor head structure, resulting in an information bottleneck between high-level semantic features and shallow predictor heads, making it difficult to adapt to drastic changes in the geometric structure of text. Furthermore, text instances often have similar character structures, such as different words containing similar letter combinations, and texture features are weak. General visual object tracking models lack specific feature modeling for text, making them prone to drifting into similar-looking background text during tracking. Simultaneously, text tracking requires pixel-level localization accuracy; even slight localization deviations can alter the semantic content of the text. Existing methods mainly rely on general motion priors and lack specific optimization for the fine-grained structure of text. Summary of the Invention

[0004] The technical problem to be solved by this invention is to provide a scene text tracking method and system based on semantic guidance and structural correction, which solves the problems of structural imbalance and deformation, resolves visual ambiguity, and achieves stable motion prediction, thereby realizing efficient and high-precision tracking of specific text instances.

[0005] This invention is achieved through the following technical solution: A scene text tracking method based on semantic guidance and structural correction includes the following steps: S1: Use the target image block of the first frame in the video sequence as the template frame to generate a template token sequence, use the image block of the search region in the current tracking frame as the search frame to generate a search region token sequence, and then concatenate the template token sequence and the search region token sequence and input them into the visual transformer backbone network for feature interaction, and output the original search region visual feature map. S2: Extract the global semantic query vector from the template feature token sequence, and map the global semantic query vector to the channel dimension modulation field vector. Use the modulation field vector to perform depth correlation operation on the original search region visual feature map, calculate the spatial response map, and perform dimensionality reduction operation on the spatial response map to generate a spatial gated mask. Then multiply the spatial gated mask element-wise with the original search region visual feature map to obtain the corrected feature map. S3: Extract high-dimensional text features of the template frame and the search frame based on the template frame and the search frame respectively. Then, map the high-dimensional text features of the template frame and the search frame to the same dimensional space as the features of the visual transformer backbone network to obtain low-dimensional text features of the template frame and the search frame. Use the low-dimensional text features of the search frame as the query and the low-dimensional text features of the template frame as the key and value to calculate the enhanced text features. Perform residual connection and convolution operation on the enhanced text features and the low-dimensional text features of the search frame to generate a text semantic calibration mask. S4: Multiply the corrected feature map element-wise with the text semantic calibration mask to obtain the final fused feature. Input the final fused feature into the prediction head and output a response map containing the target center coordinates and size. S5: Determine whether the response map containing the target center coordinates and size needs to be adjusted for search scale and output the prediction results, then fuse the Kalman filter results of the constant velocity motion model to update the final target position.

[0006] Furthermore, the methods for generating the template token sequence in step S1 and the search region token sequence in the search frame are as follows: S111: Divide the template frame and the search frame into fixed-size tiles respectively; S112: Flatten the tiles and map them into corresponding feature vectors through linear projection; S113: Add learnable positional encoding to the corresponding feature vectors to generate template token sequences and search region token sequences.

[0007] In the optimized step S1, the visual transformer backbone network performs feature interaction between the template token sequence and the search region token sequence through a multi-layer self-attention mechanism, and outputs the original search region visual feature map.

[0008] Furthermore, in step S2, the global semantic query vector is extracted according to equation (1) by performing global average pooling aggregation on the template feature token sequence, and the global semantic query vector is mapped to the channel-dimensional modulation field vector according to equation (2): (1); (2); in: Represents the global semantic query vector. Indicates the first character in the template feature token sequence. A template feature token, This indicates the number of template feature tokens in the template feature token sequence. Represents the modulation field vector. This represents a mapping network.

[0009] Furthermore, in step S2, a spatial gated mask is generated according to equation (3), and the corrected feature map is obtained according to equation (4): (3); (4); in: Indicates a spatial gating mask. This represents the Sigmoid activation function. express Convolutional layer This represents the visual feature map of the original search region. This indicates depth-related operations. Represents the modulation field vector. This represents the corrected feature map. This indicates element-wise multiplication.

[0010] In the optimized step S3, a pre-trained and parameter-frozen scene text recognition model is used as a text expert encoder to extract high-dimensional text features of the template frame and the search frame.

[0011] In the optimized step S3, the enhanced text features are calculated according to equation (5): (5); in: This represents the enhanced text features. This indicates a multi-head cross-attention mechanism. This represents the low-dimensional text features of the search frame. This represents the low-dimensional text features of the template frame.

[0012] Furthermore, the method for generating the text semantic calibration mask in step S3 is as follows: S311: The enhanced text features and the low-dimensional text features of the search frame are residually connected according to Equation (6), and after layer normalization, the initial fused features are obtained: (6); in: Indicates the initial fusion features. Representation layer normalization, This represents the low-dimensional text features of the search frame. Indicates the enhanced text features; S312: According to equation (7), perform residual connection and convolution operations on the enhanced text features and the low-dimensional text features of the search frame to generate a text semantic calibration mask: (7); in: This represents a text semantic calibration mask. This represents the Sigmoid activation function. This indicates a lightweight convolution head. This indicates a dimensional reshaping.

[0013] Furthermore, step S5 updates the final target position using the following method: S511: Apply a Hanning window penalty to the response map output by the prediction head, which includes the target center coordinates and dimensions, to obtain the processed response map; S512: Take the peak value of the processed response map as the tracking confidence of the current frame, and compare the tracking confidence of the current frame with the preset confidence threshold. If the tracking confidence of the current frame is greater than or equal to the preset confidence threshold, the current prediction result is directly output. If the tracking confidence of the current frame is less than the preset confidence threshold, the resampling mechanism is triggered, the size of the image patch in the search area is adjusted using the preset scaling factor set, and steps S1 to S4 are repeated for feature extraction and prediction. Then the result with the highest confidence is selected as the prediction result. S513: Based on the constant velocity motion model, Kalman filtering is used to predict the motion position of the current tracking frame of the constant velocity motion model according to the state of the previous tracking frame. S514: The prediction result of step S512 is weighted and fused with the predicted motion position of the current tracking frame of the constant velocity motion model to obtain the updated final target position.

[0014] A scene text tracking system based on semantic guidance and structural correction includes a template token sequence and search region token sequence splicing module, a visual transformer backbone network, a corrected feature map generation module, a text semantic calibration mask generation module, a feature fusion module, a prediction head and a final target position update module. The template token sequence and search region token sequence splicing module is used to generate a template token sequence by taking the target image block of the first frame in the video sequence as the template frame, and to generate a search region token sequence by taking the search region image block of the current tracking frame as the search frame, and then splicing the template token sequence and the search region token sequence. The visual transformer backbone network uses a multi-layer self-attention mechanism to interact with the template token sequence and the search region token sequence to output the original search region visual feature map. The corrected feature map generation module is used to extract the global semantic query vector from the template feature token sequence, and map the global semantic query vector to the channel dimension modulation field vector. The modulation field vector is used to perform a depth correlation operation on the original search region visual feature map to calculate the spatial response map. The spatial response map is then reduced in dimensionality to generate a spatial gate mask. The spatial gate mask is then multiplied element-wise with the original search region visual feature map to obtain the corrected feature map. The text semantic calibration mask generation module extracts high-dimensional text features of the template frame and the search frame based on the template frame and the search frame, respectively. Then, it maps the high-dimensional text features of the template frame and the search frame to the same dimensional space as the features of the visual transformer backbone network to obtain low-dimensional text features of the template frame and the search frame. Using the low-dimensional text features of the search frame as the query and the low-dimensional text features of the template frame as the key and value, the enhanced text features are calculated. The enhanced text features are then connected with the low-dimensional text features of the search frame through residual connection and convolution operation to generate the text semantic calibration mask. The feature fusion module is used to multiply the corrected feature map element-wise with the text semantic calibration mask to obtain the final fused features; The prediction head outputs a response map containing the target center coordinates and size based on the final fused features; The final target position update module is used to determine whether the response map containing the target center coordinates and size needs to be adjusted in terms of search scale and output the prediction result, and then fuse the Kalman filter result of the constant velocity motion model to update the final target position.

[0015] Beneficial effects of the invention: The present invention provides a scene text tracking method and system based on semantic guidance and structural correction, which has the following advantages: 1. It abandons the redundant frame-by-frame detection process and adopts a tracking-based paradigm, predicting subsequent positions based only on the initial state of the first frame. This not only avoids trajectory breakage caused by detection failure, but also significantly improves the continuity of tracking.

[0016] 2. By utilizing the advanced semantics of templates to pre-optimize the feature space of the search region, a corrected feature map is obtained, which solves the text deformation problem and effectively alleviates the imbalance of feature structure caused by changes in perspective.

[0017] 3. High-fidelity text semantic priors are injected into visual tracking features to generate a text semantic calibration mask, which can distinguish interfering text with similar appearance, prevent drift, and solve the problem of visual ambiguity.

[0018] 4. By dynamically adjusting the search space and then fusing the Kalman filter results of the constant velocity motion model, time consistency regularization was achieved. This significantly improved the adaptability to rapid motion and scale changes without increasing training costs, thus enhancing inference robustness. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the process of this invention.

[0020] Figure 2 This is a visual comparison of the results of this invention and existing methods. Detailed Implementation

[0021] A scene text tracking method based on semantic guidance and structural correction is illustrated in the flowchart below. Figure 1 As shown, it includes the following steps: S1: Use the target image block of the first frame in the video sequence as the template frame to generate a template token sequence, use the image block of the search region in the current tracking frame as the search frame to generate a search region token sequence, and then concatenate the template token sequence and the search region token sequence and input them into the visual transformer backbone network for feature interaction, and output the original search region visual feature map. Specifically, the template frame can be used to generate a template token sequence, and the search frame can be used to generate a search region token sequence, as follows: S111: Divide the template frame and the search frame into fixed-size tiles, the size of which can preferably be... Pixel.

[0022] S112: Flatten the tiles and map them into corresponding feature vectors through linear projection; S113: Add learnable positional encoding to the corresponding feature vectors to generate template token sequences and search region token sequences.

[0023] The visual transformer backbone network is preferably a visual transformer backbone network with shareable weights.

[0024] The visual transformer backbone network can perform feature interaction between the template token sequence and the search region token sequence through a multi-layer self-attention mechanism, and output the original search region visual feature map. The output of the original search region visual feature map corresponds to the feature output of the search region part.

[0025] This step constructs a symbiotic spatiotemporal propagation feature extraction network, namely a visual transformer backbone network with shared weights, designed to extract visual features from images and perform preliminary spatiotemporal information interaction, outputting a visual feature map of the original search region. It abandons the redundant frame-by-frame detection process and adopts a tracking-based paradigm, predicting subsequent positions based solely on the initial state of the first frame in the video sequence. The efficiency and continuity brought by this detection-free architecture not only avoid trajectory breaks caused by detection failures but also significantly improve the coherence of tracking.

[0026] S2: Extract the global semantic query vector from the template feature token sequence, and map the global semantic query vector to the channel dimension modulation field vector. Use the modulation field vector to perform depth correlation operation on the original search region visual feature map, calculate the spatial response map, and perform dimensionality reduction operation on the spatial response map to generate a spatial gated mask. Then multiply the spatial gated mask element-wise with the original search region visual feature map to obtain the corrected feature map. Specifically, a global semantic query vector can be extracted according to equation (1) by performing global average pooling aggregation on the template feature token sequence. The extracted global semantic query vector can be input into a mapping network and mapped to a channel-dimensional modulation field vector according to equation (2). (1); (2); in: Represents the global semantic query vector. Indicates the first character in the template feature token sequence. A template feature token, This indicates the number of template feature tokens in the template feature token sequence. Represents the modulation field vector. , The number of feature channels, This represents a mapping network, which can be constructed from multilayer perceptrons (MLPs).

[0027] Furthermore, the spatial response map can be transmitted through a... The convolutional layer performs dimensionality reduction, generates a spatial gated mask according to equation (3), and obtains the corrected feature map according to equation (4): (3); (4); in: Indicates a spatial gating mask. This represents the Sigmoid activation function. express Convolutional layer This represents the visual feature map of the original search region. This indicates depth-related operations. Represents the modulation field vector. This represents the corrected feature map. This indicates element-wise multiplication.

[0028] The generated spatial gating mask has values ​​ranging from between.

[0029] This step utilizes the high-level semantic information of the template to generate a spatially gated mask, which can correct the structural feature imbalance caused by perspective distortion in the search area. The spatially gated mask is multiplied element-wise with the original visual feature map of the search area to obtain the corrected feature map. The corrected feature map suppresses background noise and highlights the deformed text area, thus achieving predictive token correction.

[0030] S3: Extract high-dimensional text features of the template frame and the search frame based on the template frame and the search frame respectively. Then, map the high-dimensional text features of the template frame and the search frame to the same dimensional space as the features of the visual transformer backbone network to obtain low-dimensional text features of the template frame and the search frame. Use the low-dimensional text features of the search frame as the query and the low-dimensional text features of the template frame as the key and value to calculate the enhanced text features. Perform residual connection and convolution operation on the enhanced text features and the low-dimensional text features of the search frame to generate a text semantic calibration mask. The optimized method can use a pre-trained and parameter-frozen scene text recognition model as a text expert encoder to extract high-dimensional text features of template frames and search frames. The extracted high-dimensional text features of template frames and search frames are text-specific features with high recognizability, thus realizing the extraction of text expert features.

[0031] In an optimized manner, a shared linear projection layer can be used to map the high-dimensional text features of the template frame and the high-dimensional text features of the search frame to the same dimensional space as the features of the visual transformer backbone network, thereby obtaining the low-dimensional text features of the template frame and the low-dimensional text features of the search frame, which can achieve feature projection alignment.

[0032] The specific calculation formula is as follows: ,in: This represents the low-dimensional text features of the search frame. This represents the low-dimensional text features of the template frame. This represents a shared linear projection layer. This represents the high-dimensional text features of the template frame. This represents the high-dimensional text features of the search frame.

[0033] The optimized approach can employ a multi-head cross-attention mechanism, using the low-dimensional text features of the search frame as the query and the low-dimensional text features of the template frame as the key and value. The enhanced text features are calculated according to equation (5). The enhanced text features achieve cross-attention enhancement, strengthening the part of the search region that is consistent with the template semantics.

[0034] (5); in: This represents the enhanced text features. This represents a multi-head cross-attention mechanism.

[0035] Furthermore, the following method can be used to generate a text semantic calibration mask: S311: The enhanced text features and the low-dimensional text features of the search frame are residually connected according to Equation (6), and after layer normalization, the initial fused features are obtained: (6); in: Indicates the initial fusion features. Representation layer normalization, This represents the low-dimensional text features of the search frame. Indicates the enhanced text features; S312: According to equation (7), perform residual connection and convolution operations on the enhanced text features and the low-dimensional text features of the search frame to generate a text semantic calibration mask: (7); in: This represents a text semantic calibration mask. This represents the Sigmoid activation function. This indicates a lightweight convolution head. This indicates a dimensional reshaping.

[0036] This step injects high-fidelity text semantic priors into the visual tracking features, achieving initial fusion of enhanced text features and low-dimensional text features of the search frame. It also enables cross-expert calibration, and the generated text semantic calibration mask can distinguish interfering texts with similar appearances, prevent drift, and solve the problem of visual ambiguity.

[0037] S4: Multiply the corrected feature map element-wise with the text semantic calibration mask to obtain the final fused feature. Input the final fused feature into the prediction head and output a response map containing the target center coordinates and size. The corrected feature map is multiplied element-wise with the text semantic calibration mask to obtain the final fused feature calculation formula. ,in: Indicates the final fusion characteristics, express, This indicates element-wise multiplication.

[0038] The optimized prediction head consists of a fully convolutional network (FCN).

[0039] The final fusion feature combines features from both predictive token correction and cross-expert calibration, inputs them into the prediction head, and outputs a response map containing the target center coordinates and dimensions, making the location prediction more accurate.

[0040] S5: Determine whether the response map containing the target center coordinates and size needs to be adjusted for search scale and output the prediction results, then fuse the Kalman filter results of the constant velocity motion model to update the final target position.

[0041] Furthermore, step S5 updates the final target position using the following method: S511: Apply a Hanning window penalty to the response map output by the prediction head, which includes the target center coordinates and size, to obtain a processed response map. The processed response map can suppress edge effects. S512: Take the peak value of the processed response map as the tracking confidence of the current frame, and compare the tracking confidence of the current frame with the preset confidence threshold. If the tracking confidence of the current frame is greater than or equal to the preset confidence threshold, the current prediction result is directly output. If the tracking confidence of the current frame is less than the preset confidence threshold, it indicates that the target may undergo drastic scale changes or be in an uncertain state. Then, the resampling mechanism is triggered, the size of the image patch in the search area is adjusted using the preset scaling factor set, and steps S1 to S4 are repeated for feature extraction and prediction. Then, the result with the highest confidence is selected as the prediction result. Employing a multi-scale resampling mechanism can improve the confidence level of prediction results.

[0042] S513: Based on the constant velocity motion model, Kalman filtering is used to predict the motion position of the current tracking frame of the constant velocity motion model according to the state of the previous tracking frame. Specifically, the constructed constant-velocity motion model will determine the center point coordinates of the text target. and its velocity in the corresponding direction Jointly defined as the system state vector , Using Kalman filtering based on the previous frame (the first frame) The optimal motion state vector of a frame ,according to Predict the prior state vector of the current frame ,in Indicates matrix transpose. The state transition matrix is ​​represented in the constant velocity motion model constructed in this invention. ,Will and The physical meaning of multiplication expansion is: the prior center coordinates of the current frame are derived from the displacement caused by the historical velocity superimposed on the coordinates of the previous frame, i.e. , Indicates the prior center coordinates of the current frame. This indicates the center coordinates of the target text in the previous frame. This indicates the speed at which the target text moved in the previous frame. This indicates the time interval between frames.

[0043] S514: The prediction result of step S512 is weighted and fused with the predicted motion position of the current tracking frame of the constant velocity motion model to obtain the updated final target position.

[0044] The specific fusion formula is as follows: ;in: Indicates the final target location. Indicates the fusion weight coefficient. This represents the center point corresponding to the prediction result in step S512.

[0045] This step achieves spatial and temporal consistency regularization by adaptively and dynamically adjusting the search space and then fusing the Kalman filter results of the constant velocity motion model. It smooths out the jitter in visual prediction by utilizing the laws of physical motion, achieving pixel-level fine localization. Without increasing training costs, it significantly improves the adaptability to rapid motion and scale changes, enhances inference robustness, and solves the problem of target loss caused by rapid motion and occlusion.

[0046] A scene text tracking system based on semantic guidance and structural correction includes a template token sequence and search region token sequence splicing module, a visual transformer backbone network, a corrected feature map generation module, a text semantic calibration mask generation module, a feature fusion module, a prediction head and a final target position update module. The template token sequence and search region token sequence splicing module is used to generate a template token sequence by taking the target image block of the first frame in the video sequence as the template frame, and to generate a search region token sequence by taking the search region image block of the current tracking frame as the search frame, and then splicing the template token sequence and the search region token sequence. The visual transformer backbone network uses a multi-layer self-attention mechanism to interact with the template token sequence and the search region token sequence to output the original search region visual feature map. The corrected feature map generation module is used to extract the global semantic query vector from the template feature token sequence, and map the global semantic query vector to the channel dimension modulation field vector. The modulation field vector is used to perform a depth correlation operation on the original search region visual feature map to calculate the spatial response map. The spatial response map is then reduced in dimensionality to generate a spatial gate mask. The spatial gate mask is then multiplied element-wise with the original search region visual feature map to obtain the corrected feature map. The text semantic calibration mask generation module extracts high-dimensional text features of the template frame and the search frame based on the template frame and the search frame, respectively. Then, it maps the high-dimensional text features of the template frame and the search frame to the same dimensional space as the features of the visual transformer backbone network to obtain low-dimensional text features of the template frame and the search frame. Using the low-dimensional text features of the search frame as the query and the low-dimensional text features of the template frame as the key and value, the enhanced text features are calculated. The enhanced text features are then connected with the low-dimensional text features of the search frame through residual connection and convolution operation to generate the text semantic calibration mask. The feature fusion module is used to multiply the corrected feature map element-wise with the text semantic calibration mask to obtain the final fused features; The prediction head outputs a response map containing the target center coordinates and size based on the final fused features; The final target position update module is used to determine whether the response map containing the target center coordinates and size needs to be adjusted in terms of search scale and output the prediction result, and then fuse the Kalman filter result of the constant velocity motion model to update the final target position.

[0047] In three scene text tracking benchmark datasets adapted from video text localization (ArTVideot) SOT DSText SOT BOVText SOT Extensive experiments were conducted on it.

[0048] Table 1 shows the various modules of the present invention in ArTVideot. SOT Table 2 shows the ablation experiment data completed on the dataset, comparing the present invention (SymTrack) with the current state-of-the-art trackers. Figure 2 The results of a visual comparison between the present invention and existing methods are shown. Figure 2 In the diagram, the green line Ground Truth represents the correct location box of the target text, the red line Ours represents the target text location box predicted by this invention, the yellow line represents the target text location box predicted by the OSTrack model, the blue line represents the target text location box predicted by ODTrack, and the orange line represents the target text location box predicted by ARTrack.

[0049] Table 1

[0050] Table 2

[0051] As can be seen from Tables 1 and 2, the present invention significantly outperforms the invention in all indicators, fully demonstrating the effectiveness of the system-specific architecture of the present invention. Figure 2 As can be seen, in complex scenarios with severe perspective distortion, motion blur, and interference from similar text, existing methods are prone to losing the target or drifting, while the present invention can always closely track the target text, demonstrating extremely high robustness.

[0052] In summary, the present invention provides a scene text tracking method and system based on semantic guidance and structural correction, which solves the problems of structural imbalance and deformation, resolves visual ambiguity, and achieves stable motion prediction, thereby realizing efficient and high-precision tracking of specific text instances.

[0053] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A scene text tracking method based on semantic guidance and structural correction, characterized in that: Includes the following steps: S1: Use the target image block of the first frame in the video sequence as the template frame to generate a template token sequence, use the image block of the search region in the current tracking frame as the search frame to generate a search region token sequence, and then concatenate the template token sequence and the search region token sequence and input them into the visual transformer backbone network for feature interaction, and output the original search region visual feature map. S2: Extract the global semantic query vector from the template feature token sequence, and map the global semantic query vector to the channel dimension modulation field vector. Use the modulation field vector to perform depth correlation operation on the original search region visual feature map, calculate the spatial response map, and perform dimensionality reduction operation on the spatial response map to generate a spatial gated mask. Then multiply the spatial gated mask element-wise with the original search region visual feature map to obtain the corrected feature map. S3: Extract high-dimensional text features of the template frame and the search frame based on the template frame and the search frame respectively. Then, map the high-dimensional text features of the template frame and the search frame to the same dimensional space as the features of the visual transformer backbone network to obtain low-dimensional text features of the template frame and the search frame. Use the low-dimensional text features of the search frame as the query and the low-dimensional text features of the template frame as the key and value to calculate the enhanced text features. Perform residual connection and convolution operation on the enhanced text features and the low-dimensional text features of the search frame to generate a text semantic calibration mask. S4: Multiply the corrected feature map element-wise with the text semantic calibration mask to obtain the final fused feature. Input the final fused feature into the prediction head and output a response map containing the target center coordinates and size. S5: Determine whether the response map containing the target center coordinates and size needs to be adjusted for search scale and output the prediction results, then fuse the Kalman filter results of the constant velocity motion model to update the final target position.

2. The scene text tracking method based on semantic guidance and structural correction according to claim 1, characterized in that: The methods for generating the template token sequence in step S1 and the search region token sequence in the search frame are as follows: S111: Divide the template frame and the search frame into fixed-size tiles respectively; S112: Flatten the tiles and map them into corresponding feature vectors through linear projection; S113: Add learnable positional encoding to the corresponding feature vectors to generate template token sequences and search region token sequences.

3. The scene text tracking method based on semantic guidance and structural correction according to claim 1, characterized in that: In step S1, the visual transformer backbone network performs feature interaction between the template token sequence and the search region token sequence through a multi-layer self-attention mechanism, and outputs the original search region visual feature map.

4. The scene text tracking method based on semantic guidance and structural correction according to claim 1, characterized in that: In step S2, the global semantic query vector is extracted according to equation (1) by performing global average pooling aggregation on the template feature token sequence, and then mapped to the channel-dimensional modulation field vector according to equation (2): （1）； （2）； in: Represents the global semantic query vector. Indicates the first character in the template feature token sequence. A template feature token, This indicates the number of template feature tokens in the template feature token sequence. Represents the modulation field vector. This represents a mapping network.

5. The scene text tracking method based on semantic guidance and structural correction according to claim 1, characterized in that: In step S2, a spatial gated mask is generated according to equation (3), and the corrected feature map is obtained according to equation (4): （3）； （4）； in: Indicates a spatial gating mask. This represents the Sigmoid activation function. express Convolutional layer This represents the visual feature map of the original search region. This indicates depth-related operations. Represents the modulation field vector. This represents the corrected feature map. This indicates element-wise multiplication.

6. The scene text tracking method based on semantic guidance and structural correction according to claim 1, characterized in that: In step S3, a pre-trained scene text recognition model with frozen parameters is used as a text expert encoder to extract high-dimensional text features of template frames and search frames.

7. The scene text tracking method based on semantic guidance and structural correction according to claim 1, characterized in that: In step S3, the enhanced text features are calculated according to equation (5): （5）； in: This represents the enhanced text features. This indicates a multi-head cross-attention mechanism. This represents the low-dimensional text features of the search frame. This represents the low-dimensional text features of the template frame.

8. The scene text tracking method based on semantic guidance and structural correction according to claim 1, characterized in that: The method for generating the text semantic calibration mask in step S3 is as follows: S311: The enhanced text features and the low-dimensional text features of the search frame are residually connected according to Equation (6), and after layer normalization, the initial fused features are obtained: （6）； in: Indicates the initial fusion features. Representation layer normalization, This represents the low-dimensional text features of the search frame. Indicates the enhanced text features; S312: According to equation (7), perform residual connection and convolution operations on the enhanced text features and the low-dimensional text features of the search frame to generate a text semantic calibration mask: （7）； in: This represents a text semantic calibration mask. This represents the Sigmoid activation function. This indicates a lightweight convolution head. This indicates a dimensional reshaping.

9. A scene text tracking method based on semantic guidance and structural correction according to claim 1, characterized in that: Step S5 updates the final target position using the following method: S511: Apply a Hanning window penalty to the response map output by the prediction head, which includes the target center coordinates and dimensions, to obtain the processed response map; S512: Take the peak value of the processed response map as the tracking confidence of the current frame, and compare the tracking confidence of the current frame with the preset confidence threshold. If the tracking confidence of the current frame is greater than or equal to the preset confidence threshold, the current prediction result is directly output. If the tracking confidence of the current frame is less than the preset confidence threshold, the resampling mechanism is triggered, the size of the image patch in the search area is adjusted using the preset scaling factor set, and steps S1 to S4 are repeated for feature extraction and prediction. Then the result with the highest confidence is selected as the prediction result. S513: Based on the constant velocity motion model, Kalman filtering is used to predict the motion position of the current tracking frame of the constant velocity motion model according to the state of the previous tracking frame. S514: The prediction result of step S512 is weighted and fused with the predicted motion position of the current tracking frame of the constant velocity motion model to obtain the updated final target position.

10. A scene text tracking system based on semantic guidance and structural correction, used to execute a scene text tracking method based on semantic guidance and structural correction as described in any one of claims 1 to 9, characterized in that, It includes a template token sequence and search region token sequence concatenation module, a visual transformer backbone network, a corrected feature map generation module, a text semantic calibration mask generation module, a feature fusion module, a prediction head and final target position update module; The template token sequence and search region token sequence splicing module is used to generate a template token sequence by taking the target image block of the first frame in the video sequence as the template frame, and to generate a search region token sequence by taking the search region image block of the current tracking frame as the search frame, and then splicing the template token sequence and the search region token sequence. The visual transformer backbone network uses a multi-layer self-attention mechanism to interact with the template token sequence and the search region token sequence to output the original search region visual feature map. The corrected feature map generation module is used to extract the global semantic query vector from the template feature token sequence, and map the global semantic query vector to the channel dimension modulation field vector. The modulation field vector is used to perform a depth correlation operation on the original search region visual feature map to calculate the spatial response map. The spatial response map is then reduced in dimensionality to generate a spatial gate mask. The spatial gate mask is then multiplied element-wise with the original search region visual feature map to obtain the corrected feature map. The text semantic calibration mask generation module extracts high-dimensional text features of the template frame and the search frame based on the template frame and the search frame, respectively. Then, it maps the high-dimensional text features of the template frame and the search frame to the same dimensional space as the features of the visual transformer backbone network to obtain low-dimensional text features of the template frame and the search frame. Using the low-dimensional text features of the search frame as the query and the low-dimensional text features of the template frame as the key and value, the enhanced text features are calculated. The enhanced text features are then connected with the low-dimensional text features of the search frame through residual connection and convolution operation to generate the text semantic calibration mask. The feature fusion module is used to multiply the corrected feature map element-wise with the text semantic calibration mask to obtain the final fused features; The prediction head outputs a response map containing the target center coordinates and size based on the final fused features; The final target position update module is used to determine whether the response map containing the target center coordinates and size needs to be adjusted in terms of search scale and output the prediction result, and then fuse the Kalman filter result of the constant velocity motion model to update the final target position.

Images