High-precision video text tracking method and device based on topological structure feature association

Abstract

The application discloses a video text tracking method based on topological structure feature association, which comprises the following steps: first, a text detector is used to generate high-recall text detection frames for video frames; second, a three-stage matching association strategy is used to perform feature matching between text instances of front and rear frames; and finally, all trajectories containing position information and identity information of the same text target in the video are generated. The text detection results are divided into high-score frames and low-score frames, and the first and second stage data association matching is performed with unpaired trajectories in sequence, then the local search tracker is used to refer to historical trajectory text features for the second stage interrupted trajectory to perform local search on the missing text frame at the breakpoint, and the third stage performs data association matching between the local search text and unpaired trajectories. The method is accurate and efficient, and the best tracking accuracy and efficiency are achieved on multiple mainstream video text tracking benchmarks. The application also provides a corresponding video text tracking device based on topological structure feature association.

Description

Technical Field

[0001] This invention belongs to the fields of deep learning and computer vision technology, and more specifically, relates to a high-precision video text tracking method and apparatus based on topological structure feature association. Background Technology

[0002] Multi-object tracking is a classic research topic in computer vision, and video text tracking is a crucial branch of multi-object tracking. Video text tracking requires predicting the trajectories of all text objects in a video, including IDs representing the text object's identity and quadrilateral bounding boxes representing its location. Current mainstream multi-stage paradigms divide video text tracking into two tasks: first, detecting all text instances in the current frame; and second, performing data association between historical trajectories and text instances in the current frame. Video text tracking is widely used in comprehensive tasks such as road sign understanding, video retrieval, video content commentary, and smart cities.

[0003] The main challenges in video text tracking tasks are text blurring, deformation, and occlusion caused by various factors. These issues not only cause trajectory confusion at the data association level but also lead to missed text detection and interruptions in corresponding trajectories at the detection level. The association of data between consecutive frames is primarily based on the appearance and location information of the text. However, in scenarios with densely distributed text, such as supermarkets and warehouses, there may be text that is difficult to distinguish in terms of both location and appearance, severely interfering with data association in video text tracking.

[0004] One of the key challenges in video text tracking lies in how to fully represent the structural features of text images. Confusion arising from appearance during text association is primarily due to inaccurate representation of text images. At the appearance level, the most obvious feature of a text image is the left-to-right arrangement of characters. Specific character orders represent specific semantics; therefore, current video text tracking methods mainly utilize RNN structures to represent the semantics of text images to achieve better text association results. However, the feature differences of the same character in different text images limit the methods for representing semantics. Similarly, regarding the internal structural features of text, treating the internal character order as a unidirectional topological structure and introducing a graph convolutional neural network to model and represent this structure avoids learning high-dimensional semantic information, enabling a more accurate representation of the structural features of text images.

[0005] Furthermore, in specific scenarios, video text tracking faces the challenge of identifying misplaced text beyond its physical appearance. In environments like supermarkets and warehouses where identical items are densely distributed, text with similar appearances and locations exists, requiring differentiation based on factors beyond just its physical location. Currently, no video text tracking methods address this type of misplaced text. Background dilation can be used to differentiate between texts with similar appearances and locations by utilizing differences in the surrounding background image.

[0006] In conclusion, many unresolved issues remain in the field of video text tracking, such as how to more effectively fuse features from consecutive frames and how to design more robust and efficient text representation branches in conjunction with detectors. With continued in-depth research across various fields, these issues are expected to be resolved, ultimately driving the development of video text tracking. Summary of the Invention

[0007] To address the shortcomings of current technologies and the aforementioned improvement needs, this invention aims to design a robust video text tracking method based on topological feature association. This method utilizes a text detector and a local search tracker to generate high-recall text candidate boxes during the current frame tracking process. Then, based on a multi-stage, hierarchical association strategy, feature matching is performed between historical trajectories and candidate texts, continuously updating the text trajectory. This method uses high-scoring text boxes, low-scoring text boxes output by the text detector, and text boxes recalled by the local search tracker for three stages of data association matching. Its data association strategy effectively distinguishes between background and low-confidence foreground using text matching mechanisms, improving overall tracking performance at the video text detection level. Simultaneously, this invention proposes a text topological structure representation network, treating the unique sorting structure of text images as a topological structure of unidirectional connections between characters. A graph convolutional neural network architecture is used to model the topological structure of text images, effectively distinguishing visually confusing text based on its unique structural features. Furthermore, for scenarios with densely distributed similar text, this invention performs background expansion on text image blocks and designs a cross-convolutional matching network for text similarity learning, effectively distinguishing texts with similar appearances from the perspective of the surrounding background. Therefore, this invention represents text instances in ways beyond conventional graphics and motion laws, achieving accurate data association performance and efficient overall tracking performance.

[0008] To achieve the above objectives, according to one aspect of the present invention, a high-precision video text tracking method based on topological feature association is provided, comprising the following steps:

[0009] (1) Training the text topology representation network and the cross-convolution matching network using the video text tracking training set, including the following sub-steps:

[0010] (1.1) The backbone network of the text detector is used to extract feature maps corresponding to the training images of the preceding and following frames at random intervals (maximum 10). The feature maps of the text annotation box regions are processed using the candidate region alignment operator to generate all text features of the preceding and following frames.

[0011] (1.2) Train the text topology representation network using the text features generated in step (1.1). This includes the following sub-steps:

[0012] (1.2.1) Use the text topology representation network to extract the appearance feature embedding corresponding to the text features of the previous and next frames.

[0013] (1.2.2) For each text trajectory in the preceding and following frames, determine the text instances a and p within the trajectory, and the instance n outside the trajectory. The trajectory identities of the three and their corresponding appearance feature embeddings in step (1.3.1) are id, id, and p, respectively. a id p id n and e a e p e n .

[0014] (1.2.3) Using triplet loss L tri The parameters of the text topology representation network are optimized to learn text similarity. The boundary value M is set to 1, and |*| represents the L2 Euclidean distance. The specific formula is shown below:

[0015]

[0016] (1.3) For the text annotation boxes of all training images, randomly extract the text of the adjacent frames to generate text annotation pairs.

[0017] (1.4) Train the cross-convolutional matching network using the text annotations generated in step (1.2). This includes the following sub-steps:

[0018] (1.4.1) Increase the area of ​​the text box in the annotation pair by 5 and 2.5 times respectively to generate the background-expanded trajectory text and the detected text image.

[0019] (1.4.2) Predicting trajectory text and detecting text similarity using the binary cross-entropy loss function L cls Optimize the parameters of the text topology representation network. N represents the number of input text image pairs, x i With y i Let be the predicted value and the ground truth value for the i-th image pair, respectively. The specific formula is shown below:

[0020]

[0021] (2) Use a text detection model to generate high-recall text detection boxes for the current video frame, then divide the text detection results into confidence levels and perform data association matching between the high-scoring detection boxes and historical trajectories in the first stage, including the following sub-steps:

[0022] (2.1) A large number of text detection boxes for the current frame are generated using an advanced text detection model. Two thresholds, λ0 and λ1, are set to extract all detection boxes and classify them into high-scoring and low-scoring detection boxes. Detection boxes with a confidence score greater than λ0 are high-scoring detection boxes, while detection boxes with a confidence score between λ0 and λ1 are low-scoring detection boxes.

[0023] (2.2) The position of the historical text trajectory in the current frame is predicted using the exponential moving average algorithm, and then the position distance cost matrix C between it and the high-resolution detection box in step (2.1) is calculated based on the intersection-union ratio. p .

[0024] (2.3) The text topology representation network trained in step (1.2) is used to generate the appearance feature embedding of the high-resolution detection box in step (2.1), thereby calculating the appearance distance cost matrix C between it and the historical text trajectory. a Then through C a With C p The weighted average yields the overall distance cost matrix C. s .

[0025] (2.4) Set the distance threshold ω1 and the confusion parameter ω2. For each historical text trajectory, first obtain the minimum distance C between the trajectory text and all high-scoring detection texts. min Determine that all total distances are less than ω1 and less than (C). min +ω2) and not unique text pairs are confused text pairs. The cross-convolution matching network trained in step (1.4) is used to perform background expansion similarity learning for confused texts with the same trajectory and directly pair the text with the highest similarity.

[0026] (2.5) In matrix C s In step (2.4), the paired texts are invalidated, and then an overall distance threshold h0 is set to eliminate text pairs with excessively large distances. The Hungarian algorithm is used to filter the overall distance matrix C after text filtering. s By conducting analysis, the best match between all historical trajectories and high-scoring detected texts can be established.

[0027] (2.6) Update the historical trajectory using the paired text from step (2.5) and open up new trajectories using the unpaired high-scoring text.

[0028] (3) Perform data association matching between the low-scoring detection boxes in the second stage and the unpaired historical trajectories in step (2.4), including the following sub-steps:

[0029] (3.1) Referring to the algorithms in steps (2.2) and (2.3), calculate the overall distance cost matrix C between the unpaired historical trajectory and the low-scoring detection box in step (2.4). ls .

[0030] (3.2) Set the overall distance threshold h1 to eliminate C. ls For text pairs with excessively large distances, the Hungarian algorithm is used to filter the overall distance matrix C after text filtering. ls The analysis was performed to determine the best match between the unpaired historical trajectory and the low-scoring detected text in step (2.4).

[0031] (3.3) Use the paired text from step (3.2) to update the historical trajectory, while discarding the unpaired low-scoring detection text.

[0032] (4) Perform data association matching between the low-scoring detection boxes in the third stage and the unpaired historical trajectories in step (3.2), including the following sub-steps:

[0033] (4.1) Use the exponential moving average algorithm to predict the position of the unpaired historical text trajectory in the current frame in step (3.2), then increase the area of ​​the historical text box by 3 times and generate the missing text search area with the predicted position as the center.

[0034] (4.2) A local search tracker is used to perform local search text box regression within the missing text search area generated in step (4.1) with reference to the corresponding historical trajectory text image block.

[0035] (4.3) Referring to the algorithms in steps (3.1) and (3.2), determine the best match between the unpaired historical trajectory and the local search text in step (3.2).

[0036] (4.4) Update the historical trajectory using the paired text from step (4.3), discard unpaired local search text, and set a maximum time interval n. Move the historical trajectory that has not been matched in the inference of consecutive n frames into the deletion pool so that it can no longer participate in text matching. The trajectory that has not entered the deletion pool is used as the historical trajectory in the next frame to return to step (3.1) and continue to participate in the text tracking process of the video stream.

[0037] According to another aspect of the present invention, a video text tracking device based on topological feature association is also provided, including at least one processor and a memory, wherein the at least one processor and the memory are connected via a data bus, and the memory stores instructions that can be executed by the at least one processor, wherein the instructions, after being executed by the processor, are used to complete the video text tracking method based on topological feature association.

[0038] In summary, the technical solutions conceived by this invention have the following beneficial effects compared with the prior art:

[0039] (1) High accuracy: The text topology representation network proposed in this invention is based on graph convolutional neural networks to model and represent the topology of text images. While avoiding the learning of high-dimensional semantic information, it expresses the internal structural features of text classes. Compared with RNN architecture networks, it can generate text appearance feature embeddings more accurately, and achieve more advanced data association performance and video text tracking accuracy.

[0040] (2) High efficiency: The three-stage matching strategy proposed in this invention only performs local search of missing text for interrupted trajectories, avoiding feature association processing of non-interrupted trajectories. At the same time, it proposes judgment conditions for confused text pairs, reducing the number of inferences of the cross-convolution matching network. The overall speed is faster than similar paradigms.

[0041] (3) Strong compatibility: All key components of this invention adopt a pluggable design, which can be applied to all multi-stage video text tracking methods. Attached Figure Description

[0042] Figure 1 This is a flowchart of a video text tracking method based on topological feature association according to the present invention;

[0043] Figure 2 This invention presents a text topology representation of a network structure diagram.

[0044] Figure 3 This is a diagram of the cross-convolution matching network structure proposed in this invention. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0046] like Figure 1As shown, the video text tracking method based on topological structure feature association of the present invention includes the following sub-steps:

[0047] (1) Training the text topology representation network and the cross-convolution matching network using the video text tracking training set, including the following sub-steps:

[0048] (1.1) The backbone network of the text detector is used to extract feature maps corresponding to the training images of the preceding and following frames at random intervals (maximum 10). The feature maps of the text annotation box regions are processed using the candidate region alignment operator to generate all text features of the preceding and following frames.

[0049] (1.2) The text features generated in step (1.1) are used to train the text topology representation network, such as... Figure 2 As shown. Specifically, it includes the following sub-steps:

[0050] (1.2.1) Use the text topology representation network to extract the appearance feature embedding corresponding to the text features of the previous and next frames.

[0051] (1.2.2) For each text trajectory in the preceding and following frames, determine the text instances a and p within the trajectory, and the instance n outside the trajectory. The trajectory identities of the three and their corresponding appearance feature embeddings in step (1.3.1) are id, id, and p, respectively. a id p id n and e a e p e n .

[0052] (1.2.3) Using triplet loss L tri The parameters of the text topology representation network are optimized to learn text similarity. The boundary value M is set to 1, and |*| represents the L2 Euclidean distance. The specific formula is shown below:

[0053]

[0054] (1.3) For the text annotation boxes of all training images, randomly extract the text of the adjacent frames to generate text annotation pairs.

[0055] (1.4) Train the cross-convolutional matching network using the text annotations generated in step (1.2), such as... Figure 3 As shown. Specifically, it includes the following sub-steps:

[0056] (1.4.1) Increase the area of ​​the text box in the annotation pair by 5 and 2.5 times respectively to generate the background-expanded trajectory text and the detected text image.

[0057] (1.4.2) Predicting trajectory text and detecting text similarity using the binary cross-entropy loss function L cls Optimize the parameters of the text topology representation network. N represents the number of input text image pairs, x i With y i Let be the predicted value and the ground truth value for the i-th image pair, respectively. The specific formula is shown below:

[0058]

[0059] (2) Use a text detection model to generate high-recall text detection boxes for the current video frame, then divide the text detection results into confidence levels and perform data association matching between the high-scoring detection boxes and historical trajectories in the first stage, including the following sub-steps:

[0060] (2.1) A large number of text detection boxes for the current frame are generated using an advanced text detection model. Two thresholds, λ0 and λ1, are set to extract all detection boxes and classify them into high-scoring and low-scoring detection boxes. Detection boxes with a confidence score greater than λ0 are high-scoring detection boxes, while detection boxes with a confidence score between λ0 and λ1 are low-scoring detection boxes.

[0061] (2.2) The position of the historical text trajectory in the current frame is predicted using the exponential moving average algorithm, and then the position distance cost matrix C between it and the high-resolution detection box in step (2.1) is calculated based on the intersection-union ratio. p .

[0062] (2.3) The text topology representation network trained in step (1.2) is used to generate the appearance feature embedding of the high-resolution detection box in step (2.1), thereby calculating the appearance distance cost matrix C between it and the historical text trajectory. a Then through C a With C p The weighted average yields the overall distance cost matrix C. s .

[0063] (2.4) Set the distance threshold ω1 and the confusion parameter ω2. For each historical text trajectory, first obtain the minimum distance C between the trajectory text and all high-scoring detection texts. min Determine that all total distances are less than ω1 and less than (C). min +ω2) and not unique text pairs are confused text pairs. The cross-convolution matching network trained in step (1.4) is used to perform background expansion similarity learning for confused texts with the same trajectory and directly pair the text with the highest similarity.

[0064] (2.5) In matrix C sIn step (2.4), the paired texts are invalidated, and then an overall distance threshold h0 is set to eliminate text pairs with excessively large distances. The Hungarian algorithm is used to filter the overall distance matrix C after text filtering. s By conducting analysis, the best match between all historical trajectories and high-scoring detected texts can be established.

[0065] (2.6) Update the historical trajectory using the paired text from step (2.5) and open up new trajectories using the unpaired high-scoring text.

[0066] (3) Perform data association matching between the low-scoring detection boxes in the second stage and the unpaired historical trajectories in step (2.4), including the following sub-steps:

[0067] (3.1) Referring to the algorithms in steps (2.2) and (2.3), calculate the overall distance cost matrix C between the unpaired historical trajectory and the low-scoring detection box in step (2.4). ls .

[0068] (3.2) Set the overall distance threshold h1 to eliminate C. ls For text pairs with excessively large distances, the Hungarian algorithm is used to filter the overall distance matrix C after text filtering. ls The analysis was performed to determine the best match between the unpaired historical trajectory and the low-scoring detected text in step (2.4).

[0069] (3.3) Use the paired text from step (3.2) to update the historical trajectory, while discarding the unpaired low-scoring detection text.

[0070] (4) Perform data association matching between the low-scoring detection boxes in the third stage and the unpaired historical trajectories in step (3.2), including the following sub-steps:

[0071] (4.1) Use the exponential moving average algorithm to predict the position of the unpaired historical text trajectory in the current frame in step (3.2), then increase the area of ​​the historical text box by 3 times and generate the missing text search area with the predicted position as the center.

[0072] (4.2) A local search tracker is used to perform local search text box regression within the missing text search area generated in step (4.1) with reference to the corresponding historical trajectory text image block.

[0073] (4.3) Referring to the algorithms in steps (3.1) and (3.2), determine the best match between the unpaired historical trajectory and the local search text in step (3.2).

[0074] (4.4) Update the historical trajectory using the paired text from step (4.3), discard unpaired local search text, and set a maximum time interval n. Move the historical trajectory that has not been matched in the inference of consecutive n frames into the deletion pool so that it can no longer participate in text matching. The trajectory that has not entered the deletion pool is used as the historical trajectory in the next frame to return to step (3.1) and continue to participate in the text tracking process of the video stream.

[0075] Furthermore, the present invention also provides a video text tracking device based on topological feature association, including at least one processor and a memory, wherein the at least one processor and the memory are connected via a data bus, and the memory stores instructions that can be executed by the at least one processor, wherein the instructions, after being executed by the processor, are used to complete the video text tracking method based on topological feature association.

[0076] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A video text tracking method based on topological feature association, characterized in that, include: (1) Training the text topology representation network and the cross-convolution matching network using the video text tracking training set, including the following sub-steps: (1.1) Use the backbone network of the text detector and the candidate region alignment operator to extract all text image block features corresponding to the training images in the previous and next frames; (1.2) Train the text topology representation network using the text image patch features generated in step (1.1); specifically including the following sub-steps: (1.2.1) Use the text topology representation network to extract the appearance feature embedding corresponding to the text image block features of the previous and next frames; (1.2.2) For each text trajectory in the preceding and following frames, determine the text instances in the trajectory. , and text instances outside of that trajectory The trajectory identities of the three and the corresponding appearance feature embeddings in step (1.2.1) are respectively , , as well as , , ; (1.2.3) Using triplet loss The parameters of the text topology representation network are optimized to learn text similarity; among them, boundary values... Set to 1, This represents the L2 Euclidean distance; the specific formula is shown below: ； (1.3) For the text annotation boxes of all training images, randomly select the text of adjacent frames of the training image to generate text annotation pairs; (1.4) Train the cross-convolutional matching network using the text annotations generated in step (1.3); specifically including the following sub-steps: (1.4.1) Increase the area of ​​the text box in the annotation pair by 5 and 2.5 times respectively to generate the trajectory text with background expansion and the detected text image; (1.4.2) Predicting trajectory text and detecting the similarity between text and images, and using the binary cross-entropy loss function. Optimize the parameters of the cross-convolution matching network; This represents the logarithm of the input text image. and The first The predicted and ground truth values ​​for the image; the specific formulas are shown below: ； (2) The high-recall text detection boxes generated by the text detection model are divided into confidence levels. Based on the text topology representation network and the cross-convolution matching network in (1), the first-stage high-scoring detection boxes and historical trajectories are matched for data association. The feature is that it includes: (2.1) Generate a large number of text detection boxes for the current frame using a text detection model, and set two thresholds, one high and one low. and Let the confidence level be greater than The detection bounding boxes are high-resolution detection bounding boxes, and the confidence level is between [value missing]. and The detection bounding box is a low-resolution detection bounding box; (2.2) Predict the position of the historical trajectory in the current frame using the exponential moving average algorithm, and then calculate the position distance cost matrix between it and the high-resolution detection box in step (2.1) based on the intersection-union ratio; (2.3) Use the text topology representation network trained in step (1.2) to generate the appearance distance cost matrix of the high-resolution detection box and historical trajectory in step (2.1), and then combine it with the position distance cost matrix in step (2.2) to generate the overall distance cost matrix; (2.4) Using the cross-convolutional matching network trained in step (1.4), perform background expansion similarity learning on the confused text of historical trajectories and directly match the text with the highest similarity. (2.5) Analyze the overall distance cost matrix in step (2.3) after filtering using the Hungarian algorithm to establish the best match between all historical trajectories and high-scoring detection boxes; (2.6) Update the historical trajectory using the paired text from step (2.5), and open up new trajectories using the unpaired high-scoring detection boxes; (3) Perform the second-stage low-scoring detection box data association matching with historical trajectories that were not paired in step (2.4), including the following sub-steps: (3.1) Referring to the algorithms in steps (2.2) and (2.3), calculate the overall distance cost matrix between the unpaired historical trajectory and the low-scoring detection box in step (2.4); (3.2) Analyze the overall distance cost matrix in step (2.3) after filtering using the Hungarian algorithm to determine the best match between unpaired historical trajectories and low-scoring detection texts in step (2.4); (3.3) Update the historical trajectory using the paired text from step (3.2) and discard the unpaired low-scoring detection text; (4) Perform data association matching between the low-scoring detection boxes in the third stage and the unpaired historical trajectories in step (3.2), including the following sub-steps: (4.1) Use the exponential moving average algorithm to predict the position of the unpaired historical trajectory in the current frame in step (3.2) and generate the missing text search area; (4.2) A local search tracker is used to perform regression of the local search text box within the missing text search area generated in step (4.1); (4.3) Referring to the algorithms in steps (3.1) and (3.2), determine the best match between the unpaired historical trajectory and the local search text in step (3.2); (4.4) Use the matching results of step (4.3) to update and delete the historical trajectory, discard the unmatched local search text, and return to step (2.1) to continue the reasoning process of the next frame.

2. The video text tracking method based on topological feature association as described in claim 1, characterized in that, In step (1.3), the maximum value of the number of random interval frames randomly selected from adjacent frames of the training image is 10.

3. A video text tracking device based on topological feature association, characterized in that, The method includes at least one processor and a memory, which are connected via a data bus. The memory stores instructions that can be executed by the at least one processor. After being executed by the processor, the instructions are used to perform the video text tracking method based on topological feature association as described in any one of claims 1-2.

Images