Traffic target tracking method and system based on language updating and memory modeling

Abstract

The application discloses a traffic target tracking method and system based on language updating and memory modeling, and the method comprises the following steps: performing pretreatment and text extraction processing on an obtained traffic monitoring video sequence to obtain original language features and visual features of the traffic target after pretreatment, wherein the visual features comprise template visual features corresponding to template frames and search region visual features corresponding to search frames; constructing a target tracking network model based on language updating and memory modeling and training the target tracking network model to obtain a trained target tracking network model; and inputting the pretreated language features and visual features into the trained target tracking network model to obtain target tracking results corresponding to each search frame in the traffic monitoring video sequence. The application significantly improves the tracking robustness in complex road conditions, similar vehicle interference and long-time monitoring scenes, and is suitable for the fields of intelligent traffic management and video investigation.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and pattern recognition technology, specifically relating to a traffic target tracking method and system based on language update and memory modeling. Background Technology

[0002] With the rapid development of intelligent transportation systems, accurate detection and continuous tracking of specific targets in complex traffic scenarios has become a core requirement for applications such as autonomous driving, traffic flow monitoring, and abnormal behavior analysis. Traditional traffic target tracking methods mainly rely on visual features, but they are prone to target loss or identity switching when faced with frequent occlusion in congested traffic, interference from vehicles with similar appearances, and complex lighting changes. Visual language tracking aims to continuously and accurately locate specific targets in video sequences based on given natural language descriptions. Introducing language modalities as high-level semantic guidance into traffic scenarios can significantly enhance the model's contextual understanding and semantic discrimination capabilities in such complex scenarios, compensating for the shortcomings of a single visual modality.

[0003] However, existing visual language tracking methods face two main challenges when applied to long-term, highly dynamic traffic scenarios: First, there is an inconsistency between static language descriptions and dynamically evolving visual targets. In actual tracking, natural language descriptions are typically only provided in the initial frame, while the target inevitably undergoes occlusion, deformation, scale changes, and illumination variations in subsequent video frames. Most existing methods use static language representations and fail to dynamically update language features according to changes in visual context, leading to semantic alignment deviations. Second, there is a lack of effective cross-modal continuous state modeling. Traffic monitoring videos are typically long-lasting, and targets may reappear after prolonged occlusion. Long-term tracking requires robust temporal memory modeling capabilities, but existing methods primarily focus on visual dependencies, often treating language descriptions as fixed inputs, lacking cross-modal temporal collaborative mechanisms, and struggling to maintain robustness in long video sequences. Existing Transformer-based methods have high computational complexity when processing long sequences and often ignore the co-evolution of language and vision over time. Summary of the Invention

[0004] This invention provides a traffic target tracking method and system based on language updating and memory modeling. By introducing a visually guided dynamic language adaptation module and a collaborative memory interaction module, it solves the problems of static language representation and insufficient cross-modal temporal modeling in existing technologies. The technical problem to be solved by this invention is achieved through the following technical solution:

[0005] One aspect of the present invention provides a traffic target tracking method based on language update and memory modeling, comprising:

[0006] S1: The acquired traffic monitoring video sequence is preprocessed and text extraction is performed to obtain the original language features and visual features of the traffic target after preprocessing. The visual features include the template visual features corresponding to the template frame and the search area visual features corresponding to the search frame.

[0007] S2: Construct a target tracking network model based on language update and memory modeling, and train the target tracking network model to obtain a trained target tracking network model. The target tracking network model includes a visually guided dynamic language adaptation module, a collaborative memory interaction module, and an encoding tracking module. The visually guided dynamic language adaptation module is used to dynamically update language features according to the real-time visual context in the traffic scene. The collaborative memory interaction module is used to perform cross-modal long-term modeling and further update the language features and the visual features. The encoding tracking module is used to obtain the location information of the traffic target using the updated language features and visual features.

[0008] S3: Input the preprocessed language features and visual features into the trained target tracking network model to obtain the target tracking result corresponding to each search frame in the traffic monitoring video sequence.

[0009] Another aspect of the present invention provides a traffic target tracking system based on language update and memory modeling, the system comprising:

[0010] The preprocessing module is used to preprocess and extract text from traffic monitoring video sequences to obtain preprocessed linguistic and visual features.

[0011] The trained target tracking network model is used to obtain the target tracking result corresponding to each search frame in the traffic monitoring video sequence based on the preprocessed language features and visual features.

[0012] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0013] This invention proposes a traffic target tracking method based on language update and memory modeling. This method introduces a visually guided dynamic language adaptation module, which achieves real-time updates of language features according to visual changes through a multi-branch visual perturbation mechanism. It can dynamically adjust language representations based on visual context, solving the problem that static language descriptions cannot adapt to changes in the appearance of the tracked traffic target, achieving fine-grained semantic alignment, and effectively overcoming the limitations of static language descriptions. Simultaneously, a collaborative memory interaction module is introduced to construct collaborative memory and state vectors, integrating multimodal information into the state space sequence modeling process. The collaborative memory state space model is used to collaboratively encode and perform long-term temporal interaction on multimodal historical information (including text memory tokens, template memory tokens, and temporal state vectors), achieving efficient and expressive cross-modal long-term temporal interaction, significantly improving the consistency and robustness of long-video tracking. Compared to traditional methods, this invention significantly enhances the depth and breadth of cross-modal temporal modeling, thus exhibiting superior accuracy and stability in complex scenes and long-term tracking tasks.

[0014] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0015] Figure 1 This is a flowchart of a traffic target tracking method based on language update and memory modeling provided in an embodiment of the present invention;

[0016] Figure 2 This is a schematic diagram of the structure of a traffic target tracking network model based on language update and memory modeling provided in an embodiment of the present invention;

[0017] Figure 3 This is a schematic diagram illustrating the processing procedure of a traffic target tracking method based on language update and memory modeling provided in an embodiment of the present invention;

[0018] Figure 4 This is a schematic diagram of the structure of a visually guided dynamic language adaptation module provided in an embodiment of the present invention;

[0019] Figure 5 This is a schematic diagram of the structure of a collaborative memory interaction module provided in an embodiment of the present invention. Detailed Implementation

[0020] To further illustrate the technical means and effects adopted by the present invention to achieve the intended purpose, the following describes in detail a traffic target tracking method and system based on language update and memory modeling proposed according to the present invention, in conjunction with the accompanying drawings and specific embodiments.

[0021] The foregoing and other technical contents, features, and effects of the present invention will be clearly presented in the following detailed description of specific embodiments in conjunction with the accompanying drawings. Through the description of the specific embodiments, a more in-depth and concrete understanding can be gained of the technical means and effects adopted by the present invention to achieve its intended purpose. However, the accompanying drawings are for reference and illustration only and are not intended to limit the technical solutions of the present invention.

[0022] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations are intended to cover non-exclusive inclusion, such that an article or apparatus comprising a list of elements includes not only those elements but also other elements not expressly listed. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the article or apparatus that includes said element.

[0023] Example 1

[0024] First, it's important to note that Visual-Language Object Tracking (VLT) offers a novel perspective for overcoming challenges in traditional visual tracking, such as target ambiguity and similar interfering objects, by introducing deep semantic cues inherent in natural language descriptions. Looking at existing VLT technologies, researchers have explored feature fusion and interaction extensively. For example, JointNLT uses semantically guided temporal modules for joint localization and tracking; UVLTrack designed a multimodal contrastive loss function and a modality-adaptive detection head; and SIEVLTrack implemented a multi-stage tracking architecture utilizing multi-layered semantic and temporal information. Furthermore, methods like QueryNLT attempt to optimize semantic ambiguity using cue word modulation mechanisms. While these methods enhance the model's understanding of target semantics to some extent, they still face several significant technical bottlenecks when dealing with complex and varied real-world scenarios.

[0025] Therefore, this embodiment proposes a traffic target tracking method based on language update and memory modeling. Please refer to [link to relevant documentation]. Figure 1 and Figure 3 The method includes the following steps:

[0026] S1: Preprocess and extract text from the acquired traffic monitoring video sequence to obtain the original linguistic and visual features of traffic targets such as vehicles or pedestrians after preprocessing. The visual features include the template visual features corresponding to the template frame and the search area visual features corresponding to the search frame.

[0027] This step aims to convert the original traffic monitoring video sequences and corresponding text (i.e., natural language descriptions) into standardized digital feature representations suitable for neural network processing, and to establish initial visual and linguistic benchmarks for subsequent tracking tasks. Specifically, it includes the following steps:

[0028] S1.1: Obtain the traffic monitoring video sequence of the traffic target to be tracked, and select the template frame and search frame according to the time sequence. The template frame is the first frame image of the traffic monitoring video sequence.

[0029] The system acquires a sequence of traffic surveillance videos of the target traffic object. The video source for this sequence can be road checkpoint surveillance, drone aerial photography, or vehicle-mounted cameras. At the beginning of the tracking task, the first frame of the traffic surveillance video sequence is selected as the template frame to provide a baseline for the appearance of the target traffic object. Each subsequent frame is used as a search frame to locate the target traffic object within it.

[0030] S1.2: Crop and scale the template frame and search frame to obtain the template image and search area image, respectively.

[0031] To accommodate the input requirements of the subsequent target tracking network model, the template frame and search frame need to be cropped and scaled. Specifically, the template frame is cropped and scaled according to the required size and the location of the traffic target; the search frame is cropped based on the predicted location of the traffic target from the previous search frame to ensure that the traffic target to be tracked is located within the search area.

[0032] S1.3: A pre-trained language encoder is used to encode the natural language descriptions in the traffic monitoring video sequence to obtain the corresponding initial language features.

[0033] In this embodiment, a natural language description of the traffic target to be tracked is obtained based on the traffic monitoring video sequence. The natural language description can be a query command input by traffic management personnel, such as "track the black SUV with a roof rack".

[0034] In this step, a pre-trained BERT-base model is used as a language encoder to perform text normalization processing on the natural language description, segmenting the natural language description into word sequences, and adding special classification labels [CLS] and separator labels [SEP] to convert the word sequences into digitized sequences and output high-dimensional initial language features.

[0035] S1.4: A trained hierarchical visual transformer is used to visually encode the template image and the search region image to obtain the corresponding template visual features and search region visual features.

[0036] Specifically, a trained hierarchical visual transformer is used as the backbone network. The template image and search region image obtained in step S1.2 are input into this backbone network to extract the visual features of the template and the visual features of the search region, respectively. These initial features will serve as the basic input for subsequent dynamic adaptation and memory modeling. It should be noted that in actual use, this hierarchical visual transformer first needs to be trained. Its training process is carried out simultaneously with the subsequent target tracking network model based on language updates and memory modeling. The training process will be described in detail in step S2.

[0037] S2: Construct and train a target tracking network model based on language update and memory modeling to obtain a trained target tracking network model.

[0038] This step, by constructing a target tracking network model based on language updating and memory modeling, enables dynamic updating of language features in response to visual changes and long-term temporal collaborative memory of cross-modal information. For details, please refer to... Figure 2 , Figure 2 This is a schematic diagram of a target tracking network model based on language updating and memory modeling, provided by an embodiment of the present invention. The target tracking network model includes a vision-guided dynamic language adaptation module, a collaborative memory interaction module, and an encoding tracking module. The vision-guided dynamic language adaptation module dynamically updates language features based on the real-time visual context of the traffic scene. The collaborative memory interaction module performs cross-modal long-term time-series modeling and further updates the language and visual features. The encoding tracking module uses the updated language and visual features to obtain the location information of the traffic target. Throughout the data processing, visual and language features are not processed independently, but rather deeply interact and merge through the vision-guided dynamic language adaptation module and the collaborative memory interaction module.

[0039] The visually guided dynamic language adaptation module includes a token-level visual attention unit and a multi-expert feedforward network. The token-level visual attention unit is used to process the language features corresponding to the previous search frame. As a query vector, the visual features corresponding to the current search frame As key and value vectors, they are retrieved and obtained using a token-level visual attention mechanism, along with language features. The relevant visual context is then processed and layer-normalized to obtain the layer-normalized visual context; subsequently, this layer-normalized visual context is compared with language features. By stitching together the data, intermediate features containing visual guidance information are obtained. A multi-expert feedforward network contains multiple expert branches for computing intermediate features that incorporate visual guidance information. The similarity between the learnable route vectors of each expert branch and the dynamic route weights is used to generate dynamic route weights. The visual perturbation vector components output by multiple expert branches are then weighted and aggregated based on these dynamic route weights to obtain the aggregated visual perturbation vector. Subsequently, a local context modeling of the aggregated visual perturbation vector is performed using depthwise separable convolution, and then combined with language features. Perform residual connections to obtain the updated language features corresponding to the current search frame. .

[0040] It should be noted that the processing of template frames and search frames corresponding to the traffic monitoring video sequence is performed frame by frame, meaning that the features corresponding to one search frame are processed at a time. In the current frame (i.e., the first...) t During frame processing, the language features corresponding to the previous search frame are input to the token-level visual attention unit. This refers to the language features updated by the visually guided dynamic language adaptation module in the previous frame processing. The visual features corresponding to the current search frame. The template visual features corresponding to the current search frame and visual features of the search area The features formed after splicing.

[0041] The specific implementation process of this dynamic language adaptation module is as follows:

[0042] like Figure 4 As shown, firstly, the language features corresponding to the previous search frame are... As a query vector, the visual features corresponding to the current search frame The key and value vectors are input to the first multi-head attention unit. This first multi-head attention unit uses a token-level visual attention mechanism to calculate attention weights and aggregate visual information, thereby retrieving the most relevant visual context for each language word. Then, it undergoes layer normalization processing through the first-layer normalization unit to obtain the layer-normalized visual context. Subsequently, the layer-normalized visual context is combined with language features. The data is stitched together along the channel dimension to form an intermediate feature that contains visual guidance information. .

[0043] Subsequently, a multi-expert feedforward network is introduced to generate the visual perturbation vector. In this embodiment, the multi-expert feedforward network includes, in sequence, a multi-expert branch feedforward unit, a depthwise separable convolutional unit, a second-layer normalization unit, and a first residual connection unit. The multi-expert branch feedforward unit includes... K Multiple expert branches, with the multi-expert-branch feedforward unit using linear projection to integrate intermediate features containing visual guidance information. Mapped to context vector Calculate the context vector Learnable route vector corresponding to each expert branch The cosine similarity between the two branches is calculated, and the similarity score is normalized using the softmax function to obtain the dynamic routing weight corresponding to each expert branch. Then, the visual perturbation vector components output by each expert branch are weighted and aggregated according to the dynamic routing weight to obtain the weighted aggregated visual perturbation vector. The expression of the weighted aggregated visual perturbation vector is as follows:

[0044] ,

[0045] in, This represents the weighted aggregated visual perturbation vector. Indicates the first i The linear projection matrix corresponding to each expert branch Indicates the first j The linear projection matrix corresponding to each expert branch Indicates the first i The learnable routing vectors corresponding to each expert branch Indicates the first j The learnable routing vectors corresponding to each expert branch This represents an intermediate feature that contains visual guidance information. Indicates after the first i The visual perturbation vector components output after processing by the feedforward layer of each expert branch. It should be noted that this learnable routing vector... It is initialized with a Gaussian random distribution with zero mean and is learned and updated during the model training process.

[0046] Furthermore, depthwise separable convolutional units and second-layer normalization units are used to process the weighted aggregated visual perturbation vector. Local depthwise separable convolution and layer normalization are performed to obtain the processed visual perturbation vector. The first residual connection unit is used to combine the processed visual perturbation vector with the language features corresponding to the previous search frame. Perform element-wise addition (residual addition) to obtain the updated language features of the current search frame:

[0047] ,

[0048] in, This indicates the updated language features of the current search frame. This indicates the language features corresponding to the previous search frame. This represents depthwise separable convolution. Presentation layer standardization.

[0049] Through this process, language features are no longer static, but are adaptively adjusted according to the visual content of the current frame.

[0050] The collaborative memory interaction module in this embodiment is used for long-term memory modeling. Please refer to [link / reference]. Figure 5 , Figure 5 This is a schematic diagram of the structure of a collaborative memory interaction module provided in an embodiment of the present invention. The collaborative memory interaction module includes a second multi-head attention unit, a third multi-head attention unit, a first residual normalization unit, a second residual normalization unit, a first linear layer, and a second linear layer. The second multi-head attention unit is used to input the text memory token corresponding to the previous search frame. As a query vector, the visual features of the language template corresponding to the current search frame are used. As key and value vectors, the text memory token corresponding to the previous search frame is used through a cross-attention mechanism. The update is performed, and after residual connection and standardization processing by the first residual normalization unit and the first linear layer, the updated text memory token of the current search frame is obtained. Among them, the visual features of the language template corresponding to the current search frame. Language features corresponding to the current search frame Visual features of the template The concatenated features; the third multi-head attention unit is used as input for the template memory token corresponding to the previous search frame. As a query vector, the visual features of the language template corresponding to the current search frame are used. As key and value vectors, the template memory token corresponding to the previous search frame is used through a cross-attention mechanism. The template memory token for the current search frame is updated and then processed through the second residual normalization unit and the residual connection and normalization of the second linear layer. It should be noted that, at the initial moment, the text memory token and the template memory token are each generated into a 512-dimensional vector through random initialization. The elements of the vector are sampled from a zero-mean Gaussian distribution and used as the initial memory representation for the input of the first search frame, and are continuously updated during the subsequent frame-by-frame data processing.

[0051] Furthermore, the collaborative memory interaction module in this embodiment also includes a collaborative memory state space model, a third linear layer, a convolutional layer, and a first SiLU activation function layer. The collaborative memory state space model is used to combine the updated text memory token with an exponential moving average strategy. and template memory token For the temporal state vector corresponding to the previous search frame Update the data to obtain the fusion state vector corresponding to the current search frame. The fusion expression is:

[0052] ,

[0053] in, This represents the fusion state vector corresponding to the current search frame. This represents the temporal state vector corresponding to the previous search frame. This represents the update coefficient of the exponential moving average. Represents the linear projection matrix of the tokens. This represents the Sigmoid activation function.

[0054] The third linear layer is used to input the visual language features corresponding to the current search frame. The features are obtained after processing through a third linear layer, a convolutional layer, and a first SiLU activation function layer. Among them, the visual language features corresponding to the current search frame Language features corresponding to the current search frame Template visual features and visual features of the search area Features after splicing.

[0055] This collaborative memory state-space model is also used to determine features. and fusion state vector Obtain the temporal state vector corresponding to the current search frame. and intermediate layer visual language fusion features :

[0056] ,

[0057] ,

[0058] in, This represents the temporal state vector corresponding to the current search frame. Represents the state transition matrix. Indicates adaptive step size, Indicates the input modulation coefficients. This represents the discretized state transition matrix. This represents the output intermediate-layer visual-language fusion features (the hidden state corresponding to the current search frame). This represents the output mapping matrix corresponding to the current search frame. This indicates the input residual modulation parameters.

[0059] It should be noted that the temporal state vector is set to zero at the initial moment to represent the initial state before historical information has been accumulated.

[0060] See also Figure 5The collaborative memory interaction module also includes a fourth linear layer, a second SiLU activation function layer, a gating unit, a fifth linear layer, and a second residual connection unit, wherein the input visual language features corresponding to the current search frame are... After being processed sequentially through the fourth linear layer and the second SiLU activation function layer, the visual language fusion features of the gated unit and the intermediate layer are then processed. Perform linear combination (element-by-element multiplication); then pass through the fifth linear layer and combine with visual language features. The residuals are summed in the second residual connection unit to obtain the final multimodal features enhanced by temporal memory.

[0061] It should be noted that the collaborative memory interaction module in this embodiment is configured to only access the current frame image and the state information cached in the previous frame when processing each frame, without having to repeatedly calculate the features of historical frames. It achieves long-term memory reasoning with linear complexity through the collaborative memory state space model.

[0062] Further, please see Figure 2 and Figure 3 The encoding and tracking module of this embodiment includes a unified multimodal encoder and a tracking and positioning head, which are sequentially connected to the output of the collaborative memory interaction module. The unified multimodal encoder performs deep fusion of the multimodal features output by the collaborative memory interaction module and inputs the fused multimodal features to the tracking and positioning head. The tracking and positioning head outputs the target tracking result of the current search frame based on the fused multimodal features. The target tracking result includes the predicted center position coordinates and bounding box size of the target in the current search frame. The unified multimodal encoder of this embodiment includes multiple sequentially connected encoding blocks.

[0063] This encoding and tracking module aims to utilize features processed by deep networks, containing rich dynamic semantics and long temporal memory, to accurately calculate the position and size of traffic targets in the current search frame, thus completing the tracking task. The tracking and localization head in this embodiment contains two parallel fully convolutional branches: a classification branch and a regression branch. The classification branch predicts the confidence level of each pixel in the search area belonging to the center of the traffic target; the regression branch predicts the offset of each pixel relative to the true bounding box of the traffic target, as well as the width and height of the traffic target. The output of the tracking and localization head is post-processed to obtain the final result. In the score map output by the classification branch, the center point coordinates of the traffic target are determined by finding the location of the maximum value. Then, the output of the regression branch at the corresponding center point position is read to obtain the width and height of the traffic target. The center point coordinates and dimensions are combined to generate the bounding box of the traffic target in the current search frame, which is output as the final target tracking result.

[0064] It should be noted that before using this target tracking network model based on language update and memory modeling for traffic target tracking, the network needs to be trained to improve the accuracy of target tracking. During training, the parameters of the language encoder are kept frozen, and the hierarchical visual transformer is trained and updated together with the modules in the target tracking network model. The training dataset consists of several publicly available tracking datasets, including portions of the TNL2K, LaSOT, OTB99, and WebUAV-3M tracking datasets. This training dataset includes multiple training samples, each consisting of a language description, three template images, and three search images. The template images include traffic target location labels.

[0065] Losses during training include focus loss. , Loss and generalized IoU loss The total loss function is:

[0066] ,

[0067] in, Indicates the total loss. and These represent the generalized IoU loss. and The weighting coefficients of the losses are used to balance the contributions of each loss term to the overall optimization process. The AdamW optimizer is used for training, with an initial learning rate of... The weight decay coefficient is The total number of training rounds is 300, and each round contains 30K training samples.

[0068] Specifically, the hierarchical visual transformer and the target tracking network model are trained using the above training parameters. During the training process, the parameters in each module of the hierarchical visual transformer and the target tracking network model are updated using the above loss function. When the total number of training rounds is reached, the training ends, and the trained hierarchical visual transformer and the trained target tracking network model are obtained.

[0069] S3: Input the preprocessed language features and visual features into the trained target tracking network model to obtain the target tracking results corresponding to each search frame of the traffic monitoring video sequence.

[0070] Specifically, as in step S1, the acquired traffic monitoring video sequence undergoes preprocessing and text extraction. Preprocessed original language features and visual features are obtained through a pre-trained language encoder and a trained hierarchical visual transformer, respectively. These preprocessed original language features and visual features are then input into a trained target tracking network model for frame-by-frame processing and interaction. For the first search frame, based on the original language features (which can be considered the language features corresponding to frame 0), the template visual features corresponding to the template frame, and the visual features of the search region corresponding to the first search frame, the updated language features of the first search frame are obtained in the visually guided dynamic language adaptation module. Subsequently, the collaborative memory interaction module uses the updated language features of the first search frame, along with the set original text memory token and template memory token (which can be considered the text memory token and template memory token corresponding to frame 0), to obtain the updated text memory token and template memory token of the first search frame. Combined with the set original temporal state vector (which can be considered the temporal state vector corresponding to frame 0), the updated fusion state vector, temporal state vector, and hidden state of the first search frame are obtained. The hidden state is then used to process the original visual language features. The system is updated to obtain the final output multimodal features enhanced by temporal memory. Then, the target tracking result of the first search frame is identified through the encoding tracking module. At the same time, the updated language features, text memory tokens, template memory tokens and temporal state vectors obtained from the processing of the first search frame are saved and used in the processing of the second search frame.

[0071] The second frame image is then processed. Based on the updated language features of the first search frame, the template visual features corresponding to the template frame, and the visual features of the search region corresponding to the second search frame, the updated language features of the second search frame are obtained in the visually guided dynamic language adaptation module. Subsequently, the collaborative memory interaction module uses the updated language features of the second search frame, along with the text memory token and template memory token corresponding to the first search frame, to obtain the updated text memory token and template memory token of the second search frame. Combined with the temporal state vector corresponding to the first search frame, the module obtains the updated fusion state vector, temporal state vector, and hidden state of the second search frame. The hidden state is then used to update the input features, resulting in the final output multimodal features enhanced by temporal memory. The target tracking result of the second search frame is then identified by the encoding tracking module. Simultaneously, the updated language features, text memory token, template memory token, and temporal state vector obtained from the processing of the second search frame are saved for use in the processing of the third search frame. This process continues until the target tracking result for each search frame is obtained.

[0072] Example 2

[0073] Based on Embodiment 1, this embodiment provides a traffic target tracking system based on language update and memory modeling. The traffic target tracking system includes a preprocessing module and a trained target tracking network model. The preprocessing module is used to preprocess and extract text from the traffic monitoring video sequence to obtain preprocessed language features and visual features. For the specific execution process, please refer to S1 in the embodiment, which will not be repeated here. The trained target tracking network model is used to obtain the target tracking result corresponding to each search frame in the traffic monitoring video sequence based on the preprocessed language features and visual features. For the specific execution process, please refer to S2 and S3 in the embodiment, which will not be repeated here.

[0074] This embodiment proposes a traffic target tracking method based on language update and memory modeling. This method introduces a vision-guided dynamic language adaptation module, which uses a multi-branch visual perturbation mechanism to achieve real-time updates of language features as visual changes occur. It dynamically adjusts language features according to the visual context, solving the problem that static language descriptions cannot adapt to changes in the appearance of the tracked traffic target, achieving fine-grained semantic alignment and effectively overcoming the limitations of static language descriptions. Simultaneously, a collaborative memory interaction module is introduced to construct collaborative memory and state vectors, integrating multimodal information into the state space sequence modeling process. The state space model is used to collaboratively encode and perform long-term temporal interaction on multimodal historical information, achieving efficient and expressive cross-modal long-term temporal interaction, significantly improving the consistency and robustness of long-video tracking. Compared to traditional methods, the method in this embodiment significantly enhances the depth and breadth of cross-modal temporal modeling, thus exhibiting superior accuracy and stability in complex scenes and long-term tracking tasks.

[0075] In the several embodiments provided by this invention, it should be understood that the apparatus and methods disclosed in this invention can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of modules is merely a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0076] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated module can be implemented in hardware or in the form of hardware plus software functional modules.

[0077] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A traffic target tracking method based on language update and memory modeling, characterized in that, include: S1: The acquired traffic monitoring video sequence is preprocessed and text extraction is performed to obtain the original language features and visual features of the traffic target after preprocessing. The visual features include the template visual features corresponding to the template frame and the search area visual features corresponding to the search frame. S2: Construct a target tracking network model based on language update and memory modeling, and train the target tracking network model to obtain a trained target tracking network model. The target tracking network model includes a visually guided dynamic language adaptation module, a collaborative memory interaction module, and an encoding tracking module. The visually guided dynamic language adaptation module is used to dynamically update language features according to the real-time visual context in the traffic scene. The collaborative memory interaction module is used to perform cross-modal long-term modeling and further update the language features and the visual features. The encoding tracking module is used to obtain the location information of the traffic target using the updated language features and visual features. S3: Input the preprocessed language features and visual features into the trained target tracking network model to obtain the target tracking result corresponding to each search frame in the traffic monitoring video sequence; The visually guided dynamic language adaptation module includes a token-level visual attention unit and a multi-expert feedforward network, wherein... The token-level visual attention unit is used to process the language features corresponding to the previous search frame. As a query vector, the visual features corresponding to the current search frame As key and value vectors, the language features are retrieved and obtained through a token-level visual attention mechanism. The relevant visual context is then subjected to layer normalization to obtain the layer-normalized visual context; subsequently, the layer-normalized visual context is compared with the language features. By stitching together the data, intermediate features containing visual guidance information are obtained. Among them, the visual features corresponding to the current search frame The template visual features corresponding to the current search frame and visual features of the search area The features formed after splicing; The multi-expert feedforward network comprises multiple expert branches for computing the intermediate features containing visual guidance information. The similarity between the learnable route vectors of each expert branch and the actual route vectors is used to generate dynamic route weights. These dynamic route weights are then used to weight and aggregate the visual perturbation vector components output by the multiple expert branches to obtain an aggregated visual perturbation vector. Subsequently, a local context model is performed on the aggregated visual perturbation vector using depthwise separable convolution, and then combined with the language features. Perform residual connections to obtain the updated language features corresponding to the current search frame. .

2. The traffic target tracking method based on language update and memory modeling according to claim 1, characterized in that, S1 includes: S1.1: Obtain the traffic monitoring video sequence of the traffic target, and select the template frame and search frame according to the time sequence. The template frame is the first frame image of the traffic monitoring video sequence. S1.2: The template frame and the search frame are cropped and scaled to obtain the template image and the search area image, respectively; S1.3: A pre-trained language encoder is used to encode the natural language descriptions in the traffic monitoring video sequence to obtain the corresponding initial language features; S1.4: The template image and the search region image are encoded using a trained hierarchical visual transformer to obtain the corresponding template visual features and search region visual features.

3. The traffic target tracking method based on language update and memory modeling according to claim 1, characterized in that, The multi-expert feedforward network comprises, in sequence, multi-expert branch feedforward units, depthwise separable convolutional units, second-layer normalization units, and first residual connection units, wherein... The multi-expert branch feedforward unit includes parallel... K Each expert branch, the multi-expert branch feedforward unit, uses linear projection to transform the intermediate features containing visual guidance information... Mapped to context vector Calculate the context vector Learnable route vector corresponding to each expert branch The cosine similarity between the branches is used, and Softmax normalization is used to obtain the dynamic routing weights corresponding to each expert branch. Then, the routing is performed according to these dynamic routing weights. K The visual perturbation vector components output by each expert branch are weighted and aggregated to obtain a weighted aggregated visual perturbation vector, wherein the expression of the weighted aggregated visual perturbation vector is: ]， in, This represents the weighted aggregated visual perturbation vector. Indicates the first i The linear projection matrix corresponding to each expert branch Indicates the first j The linear projection matrix corresponding to each expert branch Indicates the first i The learnable routing vectors corresponding to each expert branch Indicates the first j The learnable routing vectors corresponding to each expert branch This represents an intermediate feature that contains visual guidance information. Indicates after the first i The visual perturbation vector components output after processing by the feedforward layer of each expert branch; The depthwise separable convolutional unit and the second-layer normalization unit are used to process the weighted aggregated visual perturbation vector. Local depthwise separable convolution and layer normalization are performed to obtain the processed visual perturbation vector; The first residual connection unit is used to connect the processed visual perturbation vector with the language features corresponding to the previous search frame. By summing the residuals, we can obtain the updated language features for the current search frame: ， in, This indicates the updated language features of the current search frame. This represents the language features corresponding to the previous element frame. This represents depthwise separable convolution. Presentation layer standardization.

4. The traffic target tracking method based on language update and memory modeling according to claim 1, characterized in that, The collaborative memory interaction module includes a second multi-head attention unit, a third multi-head attention unit, a first residual normalization unit, a second residual normalization unit, a first linear layer, and a second linear layer, wherein... The second multi-head attention unit is used to input the text memory token corresponding to the previous search frame. As a query vector, the visual features of the language template corresponding to the current search frame are used. As key and value vectors, the text memory token corresponding to the previous search frame is used through a cross-attention mechanism. The update is performed, and after passing through the first residual normalization unit and the residual connection and normalization processing of the first linear layer, the updated text memory token of the current search frame is obtained. The visual features of the language template corresponding to the current search frame. Language features corresponding to the current search frame Visual features of the template Features after splicing; The third multi-head attention unit is used to input the template memory token corresponding to the previous search frame. As a query vector, the visual features of the language template corresponding to the current search frame are used. As key and value vectors, template memory tokens from the previous search frame are used through a cross-attention mechanism. The update is performed, and after passing through the second residual normalization unit and the residual connection and normalization processing of the second linear layer, the updated template memory token for the current frame is obtained. .

5. The traffic target tracking method based on language update and memory modeling according to claim 4, characterized in that, The collaborative memory interaction module further includes a collaborative memory state space model, a third linear layer, a convolutional layer, and a first SiLU activation function layer, wherein... The collaborative memory state-space model is used based on an exponential moving average strategy, combined with the updated text memory token. and template memory token For the temporal state vector corresponding to the previous search frame Update the data to obtain the fusion state vector corresponding to the current search frame. : ， in, This represents the fusion state vector corresponding to the current search frame. This represents the temporal state vector corresponding to the previous search frame. This represents the update coefficient of the exponential moving average. Represents the linear projection matrix of the tokens. This represents the Sigmoid activation function; The third linear layer is used to input the visual language features corresponding to the current search frame. The features are obtained after processing by the third linear layer, the convolutional layer, and the first SiLU activation function layer. The visual language features corresponding to the current search frame. Language features corresponding to the current search frame Template visual features and visual features of the search area Features after splicing; The collaborative memory state-space model is also used to determine the features. and the fusion state vector Obtain the temporal state vector corresponding to the current search frame. And intermediate layer visual language fusion features: ， ， in, This represents the temporal state vector corresponding to the current search frame. Indicates the visual language features corresponding to the current search frame. Features processed by linear layers, convolutional layers, and the SiLU activation function Represents the state transition matrix. Indicates adaptive step size, Indicates the input modulation coefficients. This represents the discretized state transition matrix. This represents the intermediate-layer visual-language fusion features of the output. This represents the output mapping matrix corresponding to the current search frame. This indicates the input residual modulation parameters.

6. The traffic target tracking method based on language update and memory modeling according to claim 5, characterized in that, The collaborative memory interaction module further includes a fourth linear layer, a second SiLU activation function layer, a gating unit, a fifth linear layer, and a second residual connection unit, wherein the visual language features corresponding to the current search frame... After being processed sequentially by the fourth linear layer and the second SiLU activation function layer, the visual language fusion features of the gating unit and the intermediate layer are then processed. Perform linear combination; then, after passing through the fifth linear layer, combine with the visual language features. The residuals are summed in the second residual connection unit to obtain the final multimodal features enhanced by temporal memory.

7. The traffic target tracking method based on language update and memory modeling according to claim 1, characterized in that, The encoding and tracking module includes a unified multimodal encoder and a tracking and positioning head, which are sequentially connected to the output of the collaborative memory interaction module. The unified multimodal encoder is used to deeply fuse the multimodal features output by the collaborative memory interaction module, and input the fused multimodal features into the tracking and positioning head; The tracking and positioning head is used to output the target tracking result of the current search frame based on the fused multimodal features. The target tracking result includes the predicted center position coordinates and bounding box size of the target in the current search frame.

8. The traffic target tracking method based on language update and memory modeling according to claim 2, characterized in that, Training the target tracking network model includes: Keeping the parameters of the language encoder frozen, the hierarchical visual transformer and each module in the target tracking network model are trained and their parameters are updated together. The training dataset includes multiple training samples, each training sample including a language description, three template images and three search images, wherein the template images include traffic target location labels. Losses during training include focus loss. , Loss and generalized IoU loss The total loss function is: ， in, Indicates the total loss. and These represent the generalized IoU loss. and The weighting coefficients of the loss.

9. A traffic target tracking system based on language updating and memory modeling, characterized in that, The system for performing the traffic target tracking method according to any one of claims 1 to 8, the system comprising: The preprocessing module is used to preprocess and extract text from traffic monitoring video sequences to obtain preprocessed linguistic and visual features. The trained target tracking network model is used to obtain the target tracking result corresponding to each search frame in the traffic monitoring video sequence based on the preprocessed language features and visual features.

Images