A Traffic Video Text Tracking Method Based on SwinTransformer

Abstract

The application discloses a traffic video text tracking method based on a Swin Transformer, which is characterized in that the correlation of the text in the images of the front and rear frames is utilized, the occluded text is recovered through a DropMAE module, then a deformable text detection module is used to provide a flexible receptive field for the model, and then a spatial attention mechanism is combined to fuse second detection feature images of different scales; then, in the text matching, the visual features, semantic features and geometric features of the text instances are combined to obtain a text similarity adjacency matrix; finally, the text instance images are associated to a target text track formed by the text instance images with the minimum distance of text association in the video text tracking process; the method has the advantages that the text missing detection problem is solved, the boundary of the text instance can be accurately captured, the detection accuracy of the fuzzy and deformed text instance is improved, and the text detection and tracking performance of the target traffic video is better.

Description

Technical Field

[0001] This invention relates to a traffic video text tracking method, and more particularly to a traffic video text tracking method based on Swing Transformer. Background Technology

[0002] In recent years, deep learning technology has achieved tremendous success in computer vision tasks, especially with the introduction of Transformer models, which has greatly improved natural language processing tasks such as machine translation, text generation, and text classification. Recently, the Swin Transformer network, as an emerging Transformer network architecture, has achieved remarkable results in the field of computer vision. By introducing local attention mechanisms and window-based image representations, the Swin Transformer network effectively handles long-range dependencies in images and has achieved excellent performance in multiple vision tasks. However, despite its success in the image domain, the application of the Swin Transformer network in traffic video text detection and tracking tasks is relatively limited.

[0003] Technology that improves vehicle systems to assist drivers in safe driving is called intelligent driver assistance systems (ADAS). Installed inside the vehicle, ADAS provides essential information about traffic, road closures, and road congestion ahead, and suggests possible routes to avoid congestion. A robust and reliable ADAS is a prerequisite for safe driving in complex, uncontrolled driving environments. For example, if a road is under emergency construction due to a traffic accident, and a "Construction Ahead" sign is placed on the roadside, traffic video text detection and tracking methods can provide fast and accurate text detection and tracking for ADAS, making it crucial for drivers to understand surrounding text signs during driving. Current traffic video text detection and tracking methods are mainly based on traditional object detection algorithms, such as those based on convolutional neural networks (CNNs) and recurrent neural networks (RNNs). These methods perform well in processing static images but have limitations in processing temporal video data and multi-scale text. CNNs are primarily used for image processing and have weak modeling capabilities for temporal data; while RNNs are susceptible to problems such as vanishing or exploding gradients. These methods solve sequence prediction tasks in a single-domain manner, obtaining image detection and recognition results by classifying features of text regions. Obviously, these methods cannot handle challenging scenarios such as blurriness and occlusion. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a traffic video text tracking method based on Swing Transformer that effectively solves the problem of missed text detection and has better text detection and tracking performance for target traffic videos. It can accurately capture the boundaries of text instances and improve the detection accuracy of blurred and deformed text instances.

[0005] The technical solution adopted by this invention to solve the above-mentioned technical problems is as follows: a traffic video text tracking method based on Swing Transformer, comprising the following steps:

[0006] Step 1): Obtain the first and second frame images of the target traffic video from the existing traffic video database, arranged in chronological order from front to back;

[0007] Step 2): The DropMAE module is constructed using a video-based occlusion autoencoder method. Based on the temporal correlation between the first frame image and the second frame image, the DropMAE module recovers the image information of the respective occlusion areas from the visible areas of the first frame image and the visible areas of the second frame image. The first reconstructed image is obtained by recovering the image information of the occlusion areas from the first frame image, and the second reconstructed image is obtained by recovering the image information of the occlusion areas from the second frame image.

[0008] Step 3-1): Construct a Siamese Swing Transformer extended network, including a ResNet50 backbone network, a Swing Transformer extended network, and a deep cross-correlation operation layer. The structure of the Swing Transformer extended network is to add an extended convolutional block to the end of each layer of the existing Swing Transformer network. The extended convolutional block consists of an extended convolution, a BN layer, and a ReLU activation layer. The size of the convolutional kernel of the extended convolution is 3×3, the horizontal dilation rate is 2, and the vertical dilation rate is 3.

[0009] Step 3-2): Extract the image features of the first reconstructed image through the ResNet50 backbone network to obtain the corresponding first feature image; extract the image features of the second reconstructed image through the ResNet50 backbone network to obtain the corresponding second feature image.

[0010] Step 3-3): The first feature image is thinned using the Swing Transformer expansion network to obtain the first thinned feature map, and the second feature image is thinned using the Swing Transformer expansion network to obtain the second thinned feature map;

[0011] Steps 3-4): Perform deep cross-correlation operation on the first and second thinned feature maps through a deep cross-correlation operation layer to obtain feature similarity encoding that contains information about the overlapping regions of the first and second frame images;

[0012] Step 4) Construct a mask optimization module. The mask optimization module concatenates the first feature image passed from bottom to top and the feature similarity code passed from the upper layer, upsamples it and restores it to the same size as the second frame image in the target traffic video. After restoration, the correlation detection mask between the previous and next frames is obtained.

[0013] Step 5): Construct a deformable text detection module. The deformable text detection module extracts the image features of the second frame image using the ResNet50 backbone network to obtain the corresponding second detection feature image. Then, deformable convolutions are added to the tail of the third, fourth and fifth layers of the ResNet50 backbone network. The second detection feature images of different scales are then fused together with the spatial attention mechanism to obtain the current frame detection mask of the target traffic video.

[0014] Step 6): Perform feature enhancement on the detection mask of the previous and next frames and the detection mask of the current frame through differential binarization to obtain the final detection mask of the current frame. Construct a detection box generation module. The detection box generation module performs connected component analysis on the final detection mask of the current frame to obtain the connected components of the text region. Calculate the minimum bounding rectangle for each connected component of the text region to obtain the minimum bounding rectangle of the text region. Obtain the position and size of the text bounding box based on the coordinate information of the minimum bounding rectangle of the text region. Based on the obtained position and size of the text bounding box, perform a cropping operation on the second frame image to obtain the corresponding text instance image. Output the detected text instance image.

[0015] Step 7): Obtain the text similarity adjacency matrix through the text matching module. The specific process is as follows:

[0016] Step 7-1): Construct a text matching module. The text matching module includes a semantic feature branch, a visual feature branch, and a geometric feature branch. The semantic feature branch consists of ROIAlign layer, convolutional layer, and Bi-LSTM layer in sequence. The visual feature branch consists of ROIAlign layer, convolutional layer, flattening layer, and fully connected layer in sequence. The geometric feature branch consists of fully connected layer.

[0017] Step 7-2): Extract the semantic features of the text instance image through the semantic feature branch to obtain the corresponding semantic feature embedding; extract the visual features of the text instance image through the visual feature branch to obtain the corresponding visual feature embedding; and extract the geometric features of the text instance image through the geometric feature branch to obtain the corresponding geometric feature embedding.

[0018] Step 7-3): The text matching module calculates the semantic feature embedding distance Ds by performing L2 Euclidean distance calculations on the semantic feature embeddings of the text instance images in the first frame and the second frame. The text matching module also calculates the visual feature embedding distance Dv by performing L2 Euclidean distance calculations on the visual feature embeddings of the text instance images in the first and second frames. Finally, the text matching module calculates the geometric feature embedding distance Dg by performing L2 Euclidean distance calculations on the geometric feature embeddings of the text instance images in the first and second frames. Then, the semantic feature embedding distance, visual feature embedding distance, and geometric feature embedding distance are used for text similarity matching to obtain the text similarity D between any text instance image detected in the first frame and any text instance image detected in the second frame. ij D ij = aDs + bDv + cDg, where a = 0.6, b = 0.2, c = 0.2, 1 ≤ i ≤ p, 1 ≤ j ≤ q, p is the number of text instance images detected in the first frame, and q is the number of text instance images detected in the second frame. Finally, all text similarities are stored in the text similarity adjacency matrix A. Text similarity adjacency matrix;

[0019] Step 8): Construct a text association module. The text association module performs the Hungarian algorithm on the text similarity adjacency matrix to obtain the minimum distance of text association. If the minimum distance of text association is greater than the preset association threshold, the text instance image is associated with the target text trajectory formed by the text instance image with the minimum distance of text association during the video text tracking process.

[0020] Compared with existing technologies, the advantages of this invention lie in utilizing the correlation between text in consecutive frames of a target traffic video. It uses the existing DropMAE module, specifically designed for recovering occluded text, to recover the occluded text in consecutive frames, effectively solving the problem of missed text detection and accurately capturing the boundaries of text instances. Next, a deformable text detection module provides the model with a flexible receptive field, improving the inaccuracy of long text instance detection. Furthermore, a spatial attention mechanism is used to fuse second detection feature images of different scales, enhancing the detection accuracy of blurred and deformed text instances. Finally, during text matching, visual, semantic, and geometric features of text instances are combined, making the text matching module robust to changes in text appearance. The traffic video text tracking method based on Swing Transformer proposed in this invention can effectively alleviate the poor performance of traffic video text detection and tracking caused by motion blur, illumination occlusion, and similar appearance.

[0021] The specific process of steps 3-4) is as follows:

[0022] Let the first frame image be denoted as x. t-1 Let the second frame image be denoted as x. t In the deep cross-correlation operation layer, the first refined feature map is denoted as φ(x). t-1 ), φ(x t-1 Using φ(x) as the convolution kernel, the second refined feature map φ(x) is processed through a deep cross-correlation layer. t Perform convolution operations to obtain a cross-correlation feature map o(x) containing information about the overlapping regions of the first and second frame images. t-1 ,x t ), will o(x t-1 ,x t It is defined as feature similarity coding.

[0023] The specific process of step 4) is as follows:

[0024] Step 4-1): The mask optimization module takes the first feature image F passed from bottom to top and passes it through a convolutional layer with a 3×3 convolutional kernel to obtain an intermediate feature S;

[0025] Step 4-2): Concatenate the intermediate feature S and the feature similarity encoding M passed down from the upper layer according to the channel dimension to generate k channels. s +k m The cascaded feature map, where k s k represents the number of channels in S. m This represents the number of channels in M;

[0026] Step 4-3): Pass the concatenated feature map through a convolutional layer with a 3×3 kernel to obtain intermediate concatenated features, and then pass a bilinear upsampling layer to perform bilinear upsampling on the intermediate concatenated features to obtain the front and back frame association detection mask.

[0027] The specific process of step 5) is as follows:

[0028] Step 5-1): Define the second detection feature image as consisting of N feature maps X, where N = 4;

[0029] Step 5-2): Connect the four feature maps X together, and then input them into a convolutional layer with a 3×3 kernel to obtain the intermediate feature Y;

[0030] Step 5-3): Obtain the attention weight S by applying a spatial attention mechanism to the intermediate feature Y;

[0031] Step 5-4): Decompose the attention weight S into 4 components along the channel dimension, and multiply the 4 components with the corresponding feature map X by weight to obtain the final fused feature Z, where Z represents the final feature image representation after processing by the spatial attention weighted fusion mechanism.

[0032] Step 5-5): Obtain the detection mask of the current frame by performing convolution and deconvolution operations on the fused feature Z.

[0033] The specific process of step 7-2) is as follows:

[0034] Step 7-2-1): The ROI Align layer crops each text instance image to obtain a cropped feature of fixed size 5×16×256. Then, a convolutional layer with a 5×3 kernel is used to convert the cropped feature into a feature map of size 1×16×256. This feature map is a sequence of length 16 composed of 1×256 feature vectors. Then, a Bi-LSTM layer is used to capture the correlation between the feature vectors in the feature map of size 1×16×256 to obtain semantic features. Finally, a global average pooling layer reduces the semantic features to a sequence of length 16, which is composed of 1×16 feature vectors. Then, a fully connected layer concatenates the sequence of length 16 into a 256-dimensional vector semantic feature embedding.

[0035] Step 7-2-2): The ROI Align layer is used to crop each text instance image to obtain a cropped feature of fixed size 5×16×256. Then, it is flattened into a one-dimensional vector of size 1×1×20480 by two convolutional layers with 3×3 kernels. Finally, the fully connected layer performs a linear transformation operation on the one-dimensional vector of size 1×1×20480 to generate a 1×256 feature vector. The generated 1×256 feature vector is embedded as a visual feature representing the overall appearance of the text instance.

[0036] Step 7-2-3): Extract the position and size of the text bounding box of the text instance image through the fully connected layer and embed it as a geometric feature. Detailed Implementation

[0037] The present invention will be further described in detail below with reference to specific embodiments.

[0038] A traffic video text tracking method based on Swing Transformer includes the following steps:

[0039] Step 1): Obtain the first and second frame images of the target traffic video from the existing traffic video database, arranged in chronological order from front to back;

[0040] Step 2): The DropMAE module is constructed using a video-based occlusion autoencoder method. Based on the temporal correlation between the first frame image and the second frame image, the DropMAE module recovers the image information of the respective occlusion areas from the visible areas of the first frame image and the visible areas of the second frame image. The first reconstructed image is obtained by recovering the image information of the occlusion areas from the first frame image, and the second reconstructed image is obtained by recovering the image information of the occlusion areas from the second frame image.

[0041] Step 3-1): Construct a Siamese Swing Transformer extended network, including a ResNet50 backbone network, a Swing Transformer extended network, and a deep cross-correlation operation layer. The structure of the Swing Transformer extended network is to add an extended convolutional block to the end of each layer of the existing Swing Transformer network. The extended convolutional block consists of an extended convolution, a BN layer, and a ReLU activation layer. The size of the convolutional kernel of the extended convolution is 3×3, the horizontal dilation rate is 2, and the vertical dilation rate is 3. The extended convolution can provide a larger receptive field and enhance the contextual information over a wide range to solve the problem of mutual occlusion of small-scale objects.

[0042] Step 3-2): Extract the image features of the first reconstructed image through the ResNet50 backbone network to obtain the corresponding first feature image; extract the image features of the second reconstructed image through the ResNet50 backbone network to obtain the corresponding second feature image.

[0043] Step 3-3): The first feature image is thinned using the Swing Transformer expansion network to obtain the first thinned feature map, and the second feature image is thinned using the Swing Transformer expansion network to obtain the second thinned feature map;

[0044] Steps 3-4): Perform deep cross-correlation operations on the first and second thinned feature maps using a deep cross-correlation layer to obtain feature similarity encodings that contain information about the overlapping regions of the first and second frame images. The specific process is as follows:

[0045] Let the first frame image be denoted as x. t-1 Let the second frame image be denoted as x. t In the deep cross-correlation operation layer, the first refined feature map is denoted as φ(x). t-1 ), φ(x t-1 Using φ(x) as the convolution kernel, the second refined feature map φ(x) is processed through a deep cross-correlation layer. t Perform convolution operations to obtain a cross-correlation feature map o(x) containing information about the overlapping regions of the first and second frame images. t-1 ,x t), will o(x t-1 ,x t It is defined as feature similarity coding.

[0046] Step 4) Construct a mask optimization module. This module concatenates the first feature image passed from bottom to top and the feature similarity encoding passed from the upper layer, upsamples it, and restores it to the same size as the second frame image in the target traffic video. After restoration, the correlation detection mask between the preceding and following frames is obtained. The specific process is as follows:

[0047] Step 4-1): The mask optimization module takes the first feature image F passed from bottom to top and passes it through a convolutional layer with a 3×3 convolutional kernel to obtain an intermediate feature S;

[0048] Step 4-2): Concatenate the intermediate feature S and the feature similarity encoding M passed down from the upper layer according to the channel dimension to generate k channels. s +k m The cascaded feature map, where k s k represents the number of channels in S. m This represents the number of channels in M;

[0049] Step 4-3): Pass the concatenated feature map through a convolutional layer with a 3×3 kernel to obtain intermediate concatenated features, and then pass a bilinear upsampling layer to perform bilinear upsampling on the intermediate concatenated features to obtain the front and back frame association detection mask.

[0050] Step 5): Construct a deformable text detection module. This module extracts image features from the second frame image using a ResNet50 backbone network to obtain the corresponding second detection feature image. Then, deformable convolutions are added to the tails of the third, fourth, and fifth layers of the ResNet50 backbone network. These deformable convolutions provide the model with a flexible receptive field. Finally, combined with a spatial attention mechanism, the second detection feature images at different scales are fused to obtain the current frame detection mask of the target traffic video. The specific process is as follows:

[0051] Step 5-1): Define the second detection feature image as consisting of N feature maps X, where N = 4;

[0052] Step 5-2): Connect the four feature maps X together, and then input them into a convolutional layer with a 3×3 kernel to obtain the intermediate feature Y;

[0053] Step 5-3): Obtain the attention weight S by applying a spatial attention mechanism to the intermediate feature Y;

[0054] Step 5-4): Decompose the attention weight S into 4 components along the channel dimension, and multiply the 4 components with the corresponding feature map X by weight to obtain the final fused feature Z, where Z represents the final feature image representation after processing by the spatial attention weighted fusion mechanism.

[0055] Step 5-5): Obtain the detection mask of the current frame by performing convolution and deconvolution operations on the fused feature Z.

[0056] Step 6): Perform feature enhancement on the detection mask of the previous and next frames and the detection mask of the current frame through differential binarization to obtain the final detection mask of the current frame. Construct a detection box generation module. The detection box generation module performs connected component analysis on the final detection mask of the current frame to obtain the connected components of the text region. Calculate the minimum bounding rectangle for each connected component of the text region to obtain the minimum bounding rectangle of the text region. Obtain the position and size of the text bounding box based on the coordinate information of the minimum bounding rectangle of the text region. Based on the obtained position and size of the text bounding box, perform a cropping operation on the second frame image to obtain the corresponding text instance image. Output the detected text instance image.

[0057] Step 7): Obtain the text similarity adjacency matrix through the text matching module. The specific process is as follows:

[0058] Step 7-1): Construct a text matching module. The text matching module includes a semantic feature branch, a visual feature branch, and a geometric feature branch. The semantic feature branch consists of a ROIAlign layer, a convolutional layer, and a Bi-LSTM layer in sequence. Since text instances have specific text meanings represented by character order, traditional convolution cannot derive the semantic meaning of these sequence-like objects. Therefore, the text instance features are converted into a sequence, and semantic features are obtained through a Bi-LSTM layer. The visual feature branch consists of a ROIAlign layer, a convolutional layer, a flattening layer, and a fully connected layer in sequence. Visual features represent the overall appearance representation of text instances. The geometric feature branch consists of a fully connected layer. Geometric features represent the detection box information of text instances.

[0059] Step 7-2): Extract semantic features from the text instance image through the semantic feature branch to obtain the corresponding semantic feature embedding; extract visual features from the text instance image through the visual feature branch to obtain the corresponding visual feature embedding; and extract geometric features from the text instance image through the geometric feature branch to obtain the corresponding geometric feature embedding. The specific process is as follows:

[0060] Step 7-2-1): The ROI Align layer crops each text instance image to obtain a cropped feature of fixed size 5×16×256. Then, a convolutional layer with a 5×3 kernel is used to convert the cropped feature into a feature map of size 1×16×256. This feature map is a sequence of length 16 composed of 1×256 feature vectors. Then, a Bi-LSTM layer is used to capture the correlation between the feature vectors in the feature map of size 1×16×256 to obtain semantic features. Finally, a global average pooling layer reduces the semantic features to a sequence of length 16, which is composed of 1×16 feature vectors. Then, a fully connected layer concatenates the sequence of length 16 into a 256-dimensional vector semantic feature embedding.

[0061] Step 7-2-2): The ROI Align layer is used to crop each text instance image to obtain a cropped feature of fixed size 5×16×256. Then, it is flattened into a one-dimensional vector of size 1×1×20480 by two convolutional layers with 3×3 kernels. Finally, the fully connected layer performs a linear transformation operation on the one-dimensional vector of size 1×1×20480 to generate a 1×256 feature vector. The generated 1×256 feature vector is embedded as a visual feature representing the overall appearance of the text instance.

[0062] Step 7-2-3): Extract the position and size of the text bounding box of the text instance image through the fully connected layer and embed it as a geometric feature.

[0063] Step 7-3): The text matching module calculates the semantic feature embedding distance Ds by performing L2 Euclidean distance calculations on the semantic feature embeddings of the text instance images in the first frame and the second frame. The text matching module also calculates the visual feature embedding distance Dv by performing L2 Euclidean distance calculations on the visual feature embeddings of the text instance images in the first and second frames. Finally, the text matching module calculates the geometric feature embedding distance Dg by performing L2 Euclidean distance calculations on the geometric feature embeddings of the text instance images in the first and second frames. Then, the semantic feature embedding distance, visual feature embedding distance, and geometric feature embedding distance are used for text similarity matching to obtain the text similarity D between any text instance image detected in the first frame and any text instance image detected in the second frame. ij D ij = aDs + bDv + cDg, where a = 0.6, b = 0.2, c = 0.2, 1 ≤ i ≤ p, 1 ≤ j ≤ q, p is the number of text instance images detected in the first frame, and q is the number of text instance images detected in the second frame. Finally, all text similarities are stored in the text similarity adjacency matrix A. Text similarity adjacency matrix;

[0064] Step 8): Construct a text association module. The text association module performs the Hungarian algorithm on the text similarity adjacency matrix to obtain the minimum distance of text association. If the minimum distance of text association is greater than the preset association threshold, the text instance image is associated with the target text trajectory formed by the text instance image with the minimum distance of text association during the video text tracking process.

[0065] In the above embodiments, the correlation between text in consecutive frames of the target traffic video is utilized to recover the occluded text in the consecutive frames using the existing DropMAE module specifically designed for recovering occluded text. This effectively solves the problem of missed text detection and can accurately capture the boundaries of text instances. Next, the deformable text detection module provides the model with a flexible receptive field, improving the inaccuracy of long text instance detection. Furthermore, the spatial attention mechanism is combined to fuse second detection feature images of different scales, improving the detection accuracy of blurred and deformable text instances. Finally, during text matching, the visual features, semantic features, and geometric features of the text instance are combined, making the text matching module robust to changes in the appearance of the text.

[0066] The following comparative examples will be used to compare the tracking performance of the traffic video text tracking method of this embodiment, which will be referred to as "this method".

[0067] Table 1:

[0068]

[0069] Table 1 shows the detection and tracking results of the traffic video text tracking method of this embodiment and other traditional video text tracking methods on the ICDAR2015 dataset.

[0070] As can be seen from Table 1, compared with the existing Free video text tracking method, the traffic video text tracking method of this embodiment improves the MOTA evaluation index by 1.54%, the IDF1 evaluation index by 3.47%, the maximum number of matches (MM) by 32.0%, the number of partial matches (PM) by 15.9%, and the maximum number of missing matches (ML) by 33.2%. It has the advantages of high detection and tracking accuracy and strong generalization to detection and tracking of similar appearance text instances under illumination occlusion, motion blur, and similar appearance.

[0071] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A traffic video text tracking method based on Swing Transformer, characterized in that... Includes the following steps: Step 1): Obtain the first and second frame images of the target traffic video from the existing traffic video database, arranged in chronological order from front to back; Step 2): The DropMAE module is constructed using a video-based occlusion autoencoder method. Based on the temporal correlation between the first frame image and the second frame image, the DropMAE module recovers the image information of the respective occlusion areas from the visible areas of the first frame image and the visible areas of the second frame image. The first reconstructed image is obtained by recovering the image information of the occlusion areas from the first frame image, and the second reconstructed image is obtained by recovering the image information of the occlusion areas from the second frame image. Step 3-1): Construct a Siamese Swing Transformer extended network, including a ResNet50 backbone network, a Swing Transformer extended network, and a deep cross-correlation operation layer. The structure of the Swing Transformer extended network is to add an extended convolutional block to the end of each layer of the existing Swing Transformer network. The extended convolutional block consists of an extended convolution, a BN layer, and a ReLU activation layer. The size of the convolutional kernel of the extended convolution is 3×3, the horizontal dilation rate is 2, and the vertical dilation rate is 3. Step 3-2): Extract the image features of the first reconstructed image through the ResNet50 backbone network to obtain the corresponding first feature image; extract the image features of the second reconstructed image through the ResNet50 backbone network to obtain the corresponding second feature image. Step 3-3): The first feature image is thinned using the Swin Transformer expansion network to obtain the first thinned feature map, and the second feature image is thinned using the Swin Transformer expansion network to obtain the second thinned feature map; Steps 3-4): Perform deep cross-correlation operation on the first and second thinned feature maps through a deep cross-correlation operation layer to obtain feature similarity encoding that contains information about the overlapping regions of the first and second frame images; Step 4) Construct a mask optimization module. The mask optimization module concatenates the first feature image passed from bottom to top and the feature similarity code passed from the upper layer, upsamples it and restores it to the same size as the second frame image in the target traffic video. After restoration, the correlation detection mask between the previous and next frames is obtained. Step 5): Construct a deformable text detection module. The deformable text detection module extracts the image features of the second frame image using the ResNet50 backbone network to obtain the corresponding second detection feature image. Then, deformable convolutions are added to the tail of the third, fourth and fifth layers of the ResNet50 backbone network. The second detection feature images of different scales are then fused together with the spatial attention mechanism to obtain the current frame detection mask of the target traffic video. Step 6): Perform feature enhancement on the detection mask of the previous and next frames and the detection mask of the current frame through differential binarization to obtain the final detection mask of the current frame. Construct a detection box generation module. The detection box generation module performs connected component analysis on the final detection mask of the current frame to obtain the connected components of the text region. Calculate the minimum bounding rectangle for each connected component of the text region to obtain the minimum bounding rectangle of the text region. Obtain the position and size of the text bounding box based on the coordinate information of the minimum bounding rectangle of the text region. Based on the obtained position and size of the text bounding box, perform a cropping operation on the second frame image to obtain the corresponding text instance image. Output the detected text instance image. Step 7): Obtain the text similarity adjacency matrix through the text matching module. The specific process is as follows: Step 7-1): Construct a text matching module. The text matching module includes a semantic feature branch, a visual feature branch, and a geometric feature branch. The semantic feature branch consists of ROIAlign layer, convolutional layer, and Bi-LSTM layer in sequence. The visual feature branch consists of ROIAlign layer, convolutional layer, flattening layer, and fully connected layer in sequence. The geometric feature branch consists of fully connected layer. Step 7-2): Extract the semantic features of the text instance image through the semantic feature branch to obtain the corresponding semantic feature embedding; extract the visual features of the text instance image through the visual feature branch to obtain the corresponding visual feature embedding; and extract the geometric features of the text instance image through the geometric feature branch to obtain the corresponding geometric feature embedding. Step 7-3): The text matching module calculates the semantic feature embedding distance Ds by performing L2 Euclidean distance calculations on the semantic feature embeddings of the text instance images in the first frame and the second frame. The text matching module also calculates the visual feature embedding distance Dv by performing L2 Euclidean distance calculations on the visual feature embeddings of the text instance images in the first and second frames. Finally, the text matching module calculates the geometric feature embedding distance Dg by performing L2 Euclidean distance calculations on the geometric feature embeddings of the text instance images in the first and second frames. Then, the semantic feature embedding distance, visual feature embedding distance, and geometric feature embedding distance are used for text similarity matching to obtain the text similarity D between any text instance image detected in the first frame and any text instance image detected in the second frame. ij D ij = aDs + bDv + cDg, where a = 0.6, b = 0.2, c = 0.2, 1 ≤ i ≤ p, 1 ≤ j ≤ q, p is the number of text instance images detected in the first frame, and q is the number of text instance images detected in the second frame. Finally, all text similarities are stored in the text similarity adjacency matrix A. Text similarity adjacency matrix; Step 8): Construct a text association module. The text association module performs the Hungarian algorithm on the text similarity adjacency matrix to obtain the minimum distance of text association. If the minimum distance of text association is greater than the preset association threshold, the text instance image is associated with the target text trajectory formed by the text instance image with the minimum distance of text association during the video text tracking process.

2. The traffic video text tracking method based on Swing Transformer according to claim 1, characterized in that... The specific process of steps 3-4) is as follows: Let the first frame image be denoted as x. t-1 Let the second frame image be denoted as x. t In the deep cross-correlation operation layer, the first refined feature map is denoted as φ(x). t-1 ), φ(x t-1 Using φ(x) as the convolution kernel, the second refined feature map φ(x) is processed through a deep cross-correlation layer. t Perform convolution operations to obtain a cross-correlation feature map o(x) containing information about the overlapping regions of the first and second frame images. t-1 x t ), will o(x t-1 x t It is defined as feature similarity encoding.

3. The traffic video text tracking method based on Swing Transformer according to claim 2, characterized in that... The specific process of step 4) is as follows: Step 4-1): The mask optimization module takes the first feature image F passed from bottom to top and passes it through a convolutional layer with a 3×3 convolutional kernel to obtain an intermediate feature S; Step 4-2): Concatenate the intermediate feature S and the feature similarity encoding M passed down from the upper layer according to the channel dimension to generate a channel number k. s +k m The cascaded feature map, where k s k represents the number of channels in S. m This represents the number of channels in M; Step 4-3): Pass the concatenated feature map through a convolutional layer with a 3×3 kernel to obtain intermediate concatenated features, and then pass a bilinear upsampling layer to perform bilinear upsampling on the intermediate concatenated features to obtain the front and back frame association detection mask.

4. The traffic video text tracking method based on Swing Transformer according to claim 3, characterized in that... The specific process of step 5) is as follows: Step 5-1): Define the second detection feature image as consisting of N feature maps X, where N = 4; Step 5-2): Connect the four feature maps X together, and then input them into a convolutional layer with a 3×3 kernel to obtain the intermediate feature U; Step 5-3): Obtain the attention weight S by applying a spatial attention mechanism to the intermediate feature Y; Step 5-4): Decompose the attention weight S into 4 components along the channel dimension, and multiply the 4 components with the corresponding feature map X by weight to obtain the final fused feature Z, where Z represents the final feature image representation after processing by the spatial attention weighted fusion mechanism. Step 5-5): Obtain the detection mask of the current frame by performing convolution and deconvolution operations on the fused feature Z.

5. A traffic video text tracking method based on Swing Transformer according to claim 4, characterized in that... The specific process of step 7-2) is as follows: Step 7-2-1): The ROI Align layer crops each text instance image to obtain a cropped feature of fixed size 5×16×256. Then, a convolutional layer with a 5×3 kernel is used to convert the cropped feature into a feature map of size 1×16×256. This feature map is a sequence of length 16 composed of 1×256 feature vectors. Then, a Bi-LSTM layer is used to capture the correlation between the feature vectors in the feature map of size 1×16×256 to obtain semantic features. Finally, a global average pooling layer reduces the semantic features to a sequence of length 16, which is composed of 1×16 feature vectors. Then, a fully connected layer concatenates the sequence of length 16 into a 256-dimensional vector semantic feature embedding. Step 7-2-2): The ROI Align layer is used to crop each text instance image to obtain a cropped feature of fixed size 5×16×256. Then, it is flattened into a one-dimensional vector of size 1×1×20480 by two convolutional layers with 3×3 kernels. Finally, the fully connected layer performs a linear transformation operation on the one-dimensional vector of size 1×1×20480 to generate a 1×256 feature vector. The generated 1×256 feature vector is embedded as a visual feature representing the overall appearance of the text instance. Step 7-2-3): Extract the position and size of the text bounding box of the text instance image through the fully connected layer and embed it as a geometric feature.

Images