A multi-modal monitoring video target tracking method based on a double-flow attention mechanism

Abstract

The present application belongs to the technical field of target tracking, and particularly relates to a multi-modal monitoring video target tracking method based on a double-flow attention mechanism. The present application takes the target tracking of multi-modal monitoring video data as the background, and in view of the weak global feature modeling capability in the existing multi-modal target tracking, a target tracking method based on a double-flow attention mechanism is proposed. The method first uses a double-flow attention network to extract and fuse the features of the target, applies the attention mechanism to model the relationship between positive and negative samples, and focuses on the global feature connection. In addition, in order to enhance the feature expression, the features of different scales are interacted. Finally, the position of the target is predicted by using a regression classification sub-network. In this way, both local details and global features are ensured, more discriminative features are obtained, and the prediction result of the regression classification sub-network is more accurate.

Description

Technical Field

[0001] This invention belongs to the field of target tracking technology, specifically relating to a multimodal surveillance video target tracking method based on a dual-stream attention mechanism. Background Technology

[0002] Target tracking refers to tracking the position of a target in subsequent frames given the target in the first frame. Unlike traditional target detection, target tracking does not require identifying the type of target; it only needs to identify the target's position within each frame. Previous target tracking methods have largely focused on visible light surveillance video data. However, in reality, visible light images from surveillance videos often have low resolution. Furthermore, visible light video, due to its imaging mechanism of light reflection, is highly sensitive to lighting conditions; excessive light can cause exposure problems, while insufficient light makes the target unclear. Therefore, infrared images are introduced for feature complementation. Infrared images utilize the temperature of different objects, making them less sensitive to lighting conditions; however, they cannot reflect target details such as color and texture. Traditional multimodal target tracking often uses convolutional neural networks to extract target features. These networks process and learn each frame's samples individually, neglecting the relationship between positive and negative samples. This method obtains the target position based on the target classification confidence score, which may result in the highest classification score not being the target's actual location. Additionally, different feature depths focus on different characteristics. Summary of the Invention

[0003] Based on the above problems, this invention introduces multimodal fusion and complementarity of visible light and infrared images for target tracking in surveillance video data. It uses a dual-stream attention mechanism to focus on global features and fuses features at different scales, striving to focus on the relationship between the target and the background while preserving local details, thereby obtaining more discriminative features.

[0004] This invention focuses on target tracking using multimodal surveillance video data. Addressing the weakness of existing multimodal target tracking methods in modeling global features, it proposes a target tracking method based on a dual-stream attention mechanism. This method first employs a dual-stream attention network to extract and fuse target features, then applies an attention mechanism to model the relationship between positive and negative samples, focusing on global feature connections. Furthermore, to enhance feature representation, features at different scales are interacted. Finally, a regression classification sub-network is used to predict the target's position. This approach preserves local details while also considering global features, resulting in more discriminative features and making the prediction results of the regression classification sub-network more accurate.

[0005] The technical solution of this invention is:

[0006] A multimodal surveillance video target tracking method based on a dual-stream attention mechanism, characterized by the following steps:

[0007] S1. Select matching visible light-infrared video pairs from the surveillance video data, and save the video pairs as image pairs at the same frame rate to obtain input samples; select the image representing the first frame in the image pair as the matching image, and use a rectangle to select the target to be tracked in the matching image; use images of subsequent frames as search images; denoted as the input sample. i = 1, 2, ..., N j N j This represents the number of samples in the j-th class. Represents visible light data. Infrared data is represented; sample labels are denoted as... i = 1, 2, ..., N j , Represents visible light data tags, This represents infrared data tags; in the actual training process, in order to utilize as much data as possible, the matching branch randomly selects video sequence M. train A certain frame in the data is denoted as (Z). v Z t The search branch selects an image pair following the matching branch image, denoted as (X). v ,X t );

[0008] S2. Construct a neural network, including a two-stream feature extraction network, a multimodal fusion network, and a tracking network;

[0009] The dual-stream feature extraction network includes two branches with the same structure: a matching branch and a search branch. Both the matching branch and the search branch extract features from visible light images and infrared images, respectively. Specifically, feature extraction is performed using three cascaded transformer blocks.

[0010] The multimodal fusion network is used to fuse the features extracted by the dual-stream feature extraction network. Specifically, it defines the outputs of the three transformer blocks of the matching branch in the dual-stream feature extraction network as (X... 1V ,X 1T ),(X 2V ,X 2T ),(X 3V ,X 3T The outputs of the three transformer blocks in the search branch are defined as (Z...). 1V Z 1T ),(Z 2V Z 2T),(Z 3V Z 3T The multimodal fusion network obtains the output of each transformer block separately and fuses them to obtain:

[0011] X1 = cat(X v1 X t1 )

[0012] X2 = cat(X) v2 X t2 )

[0013] X3 = cat(X v3 X t3 )

[0014] Z1 = cat(Z v1 Z t1 )

[0015] Z2 = cat(Z) v2 Z t2 )

[0016] Z3 = cat(Z v3 Z t3 )

[0017] Here, cat indicates channel fusion of the image;

[0018] Then, relevance calculation is performed to obtain the similarity matrix S. i :

[0019] S i =corr(Z) i ,X i ), i = 1, 2, 3

[0020] Here, corr represents the similarity calculation, which is the smaller matching image Z. i As a convolution kernel pair X i The process of performing convolution operations;

[0021] By concatenating the similarity matrices, we obtain a similarity matrix S containing multi-scale information:

[0022] S = cat(S) i )

[0023] S = downsample(S)

[0024] Here, downsample means downsampling, which projects S onto a dimension space suitable for tracking.

[0025] The tracking network comprises three branches:

[0026] The classification loss branch predicts the center location of the target:

[0027]

[0028] Among them, y t Let y represent the true label of the anchor box for the t-th sample. t =1 indicates that the anchor frame represents the foreground; m t This indicates the probability that the anchor box is foreground;

[0029] The regression loss branch predicts the distances from the four edges of the bounding box to the center point:

[0030]

[0031]

[0032] Where bbox represents the size of the predicted bounding box, which is represented by the distance between the four borders and the center, and (i,j) represents a point on S;

[0033] Central loss branch, suppressing center point drift:

[0034]

[0035] Where, d t This represents the score at the j-th center position, and σ represents the sigmoid activation function. t It is proportional to the distance between the predicted location and the center location.

[0036] The total loss of the tracking network is:

[0037] L = L cls +η1L reg +η2L cen

[0038] Where η1 and η2 are the set hyperparameters;

[0039] S3. Train the neural network constructed in S2 using the input samples from S1, specifically by: taking M... train The image is divided into matching image pairs and search image pairs as input to the neural network (X). v ,X t ) and (Z v Z t ), of which (Z v Z t The input for the matching branch is the video sequence M. train A randomly selected frame from the data; (X) v ,X t(X) is the input for the search branch. The selection method is as follows: after selecting the matching branch image, select from the image pairs following the matching branch image, denoted as (X). v ,X t The learning rate is set to 0.0005. The weights of the network are corrected by backpropagation of the loss function until the network basically converges, and a trained neural network is obtained.

[0040] S4. Use the trained neural network to track targets. Specifically, manually select the target to be tracked, generate a label for the first frame image, input the image sequence containing the label into the trained neural network, and obtain the position of the target center and the size of the target.

[0041] The beneficial effects of this invention are that, addressing the challenge of effectively modeling discriminative features during tracking of multimodal surveillance video data, it proposes a dual-stream attention method to focus on global features, thereby fully extracting and fusing both foreground and background features. In the feature extraction stage, a swin transformer is used to model global features. For each modality, features at different depths contain different information; therefore, features at different depths for each modality are fused, and interaction is performed between the matching image and the search image to fully extract the different features of the target. Simultaneously, in the tracking stage, to prevent the predicted target bounding box from drifting and losing the target, a center loss function is designed to improve the accuracy of target tracking. Attached Figure Description

[0042] Figure 1 This is a schematic diagram of the overall network model.

[0043] Figure 2 This is a schematic diagram of a feature extraction network. Detailed Implementation

[0044] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings and embodiments:

[0045] Example

[0046] This example includes the following steps:

[0047] S1: Data Preprocessing

[0048] A dataset was obtained by monitoring pedestrians and vehicles in various situations using surveillance video. The dataset contains 234 aligned RGB-T video sequences, each image measuring 630×460 pixels. 171 video sequence frames were used as the training set, and the remaining 63 video frames as the test set. The target in the first frame of each video was manually selected by using a bounding box to define the target, thus obtaining the label for the first frame. The input training samples are denoted as... i = 1, 2, ..., N j N represents the number of samples in the j-th class. Represents visible light data. Infrared data is represented. Sample labels are denoted as... i = 1, 2, ..., N j This corresponds to the input sample. To better utilize contextual information, the image can be cropped to retain the background information around the target bounding box. The pixel containing contextual information is defined as p = (w + h) / 4, meaning the matched image contains twice the contextual information of the target.

[0049] w′=w+2p

[0050] h′=h+2p

[0051] Therefore, the image is cropped to obtain a new sample: i = 1, 2, ..., N j M train_new The size is w′×h′. To accommodate the input size of the Siamese network, M is... train_new Scale to the appropriate size A:

[0052] a×w′×h′=A

[0053] A indicates that the input size of the matching image is 128×128 or the input size of the search image is 256×256.

[0054] At this point, only the label of the first frame is known, so the first frame image pair is denoted as (Z) as the matching branch. v Z t The subsequent image pairs to be tracked are used as search branches, denoted as (X). v ,X t );

[0055] S2: Constructing a neural network:

[0056] S2.1: Feature Extraction Network:

[0057] like Figure 2 As shown, the feature extraction network consists of the first three stages of the Swin transformer, which are three transformer blocks. Each transformer block includes a Swin transformer block and patchmerging. The parameters of the Swin transformer are shown in the table.

[0058] Table 1 Parameter settings for the feature extraction network

[0059]

[0060]

[0061] Final output image

[0062] S2.2: Multi-scale feature fusion interaction

[0063] like Figure 1 As shown, the feature extraction network contains three stages, and the outputs of the three stages in the search branch are as follows: (X 1V ,X 1T ),(X 2V ,X 2T ),(X 3V ,X 3T Similarly, the outputs of the three stages of the matching branch are: (Z) 1V Z 1T ),(Z 2V Z 2T ),(Z 3V Z 3T First, the two modalities in each stage are fused:

[0064] X1 = cat(X v1 X t1 )

[0065] X2 = cat(X) v2 X t2 )

[0066] X3 = cat(X v3 X t3 )

[0067] Similarly, the multimodal data are fused in the matching branch to obtain Z1, Z2, and Z3.

[0068] Applying the correlation calculation network corr to Z = (Z1, Z2, Z3) and X = (X1, X2, X3) yields the similarity matrix S. i :

[0069] S i =corr(Z) i ,X i ), i = 1, 2, 3

[0070] Here, we use depthwise convolution to calculate the similarity (corr), and the output is a similarity matrix. B i C represents the batch size of the network. i H represents the output dimension. i and W iThis represents the width and height of the matrix.

[0071] By concatenating the three similarity matrices, a similarity matrix S containing multi-scale information is obtained:

[0072] S = cat(S) i )

[0073] S = downsample(S)

[0074] Here, downsample means downsampling, which projects S onto a dimension space suitable for tracking.

[0075] S2.3: Tracking Network

[0076] A classification and regression subnetwork is used for prediction. Each point (i,j) on the feature response map S corresponds to multiple anchor boxes of different sizes, which serve as the benchmark for target tracking prediction.

[0077] Tracking involves three branches, or three loss functions, to constrain the target:

[0078] The classification loss uses cross-entropy loss to predict the center position of the target.

[0079]

[0080] Among them, y t Let y represent the true label of the anchor box for the t-th sample. t =1 indicates that the anchor frame represents the foreground; m t This indicates the probability that the anchor frame is foreground.

[0081] The regression loss is calculated using the IOU loss, which predicts the distances from the four edges of the bounding box to the center point:

[0082]

[0083] in,

[0084]

[0085] bbox represents the size of the prediction box, expressed as the distance between the four borders and the center.

[0086] The center loss is calculated using cross-entropy loss to suppress center point drift.

[0087]

[0088] The total loss is:

[0089] L = L cls +η1L reg +η2L cen

[0090] Here, η1 and η2 are the set hyperparameters.

[0091] S3: Train the constructed neural network

[0092] The input data is image pairs i = 1, 2, ..., N j The data is divided into inputs for a two-stream network: (X) v ,X t ) and (Z v Z t Load the parameters of an existing model pre-trained on a large dataset, and finally calculate the loss function through the three networks in step S2. Train for 1000 epochs with a learning rate of 0.0005, and adjust the network weights through backpropagation of the loss function until the network basically converges.

[0093] S4: Use the trained network to track the surveillance video:

[0094] The target to be tracked is manually selected, and a label is generated for the first frame image. The image sequence containing the label is first input into a dual-stream attention feature extraction network, and features are extracted from images of different modalities simultaneously. After multiple layers of feature extraction, feature maps of different scales are obtained. The feature maps at corresponding depths are fused to integrate information from different modalities. The correlation between the feature maps of the two-stream branches is calculated, the location of the target center is determined by classification, and the size of the target is determined by regression.

[0095] Ultimately, the recognition accuracy on the dataset was 0.571.

Claims

1.A method for multi-modal surveillance video object tracking based on dual-stream attention mechanism, characterized in that, The method comprises the following steps: S1, screening matched visible-infrared video pairs from the monitored video data, saving the video pairs as image pairs according to the same frame rate to obtain input samples; Select the image representing the first frame in the image pair as the matching image, and frame the target to be tracked in the matching image with a rectangular box, and the subsequent frame image as the search image; the input sample is denoted as N j denotes the number of the jth sample, denotes the visible light data, denotes the infrared data; the sample label is denoted as denotes the visible light data label, denotes the infrared data label; S2, constructing a neural network, including a double-flow feature extraction network, a multi-modal fusion network and a tracking network; The double-flow feature extraction network comprises two branches with the same structure, namely a matching branch and a search branch, the matching branch and the search branch are used for feature extraction of visible light images and infrared images respectively, and the feature extraction is specifically performed by using three cascaded transformer blocks; The multi-modal fusion network is used for fusing the characteristics extracted by the dual-flow feature extraction network, specifically: defining the outputs of three transformer blocks of the matching branch in the dual-flow feature extraction network as (X 1V , X 1T ), (X 2V , X 2T ), (X 3V , X 3T ), and defining the outputs of three transformer blocks of the search branch as (Z 1V , Z 1T ), (Z 2V , Z 2T ), (Z 3V , Z 3T ); the multi-modal fusion network obtains the outputs of each transformer block and fuses them to obtain: X1= cat(X v1 , X t1 ) X2 = cat(X v2 , X t2 ) X3 = cat(X v3 , X t3 ) Z1= cat(Z v1 , Z t1 ) Z2 = cat(Z v2 , Z t2 ) Z3 = cat(Z v3 , Z t3 ) Wherein, cat represents channel fusion of the image; The correlation calculation is performed again to obtain a similarity matrix S i : S i = corr(Z i , X i ), i = 1, 2, 3 wherein corr denotes a similarity computation, i.e. matching image Z i as a convolution kernel on X i the process of a convolution operation; The similarity matrix is spliced to obtain a similarity matrix S containing multi-scale information: S = cat(S i ) S = downsample(S) wherein downsample denotes a down-sampling, projecting S into a dimension space suitable for tracking, The tracking network comprises three branches, namely: a classification loss branch for predicting the center position of the target; wherein y t represents the anchor box real label of the t-th sample, y t = 1 indicates that the anchor box represents foreground; m t represents the probability that the anchor box is foreground; a regression loss branch for predicting the distance from the four edges of the target box to the center point: Wherein, bbox represents the size of the predicted box, which is represented by the distance of four edge frames and the center, (i, j) represents a point on S; a center loss branch for suppressing center point drift: where d t represents the score of the jthcenter position, σrepresents a sigmoid activation function, c t and the predicted position is proportional to the distance of the center position. The total loss of the tracking network is: L = L cls + η1L reg + η2L cen Wherein, η1 and η2 are set hyperparameters; S3, training the neural network constructed by S2 with the input samples of S1, specifically, M train is divided into matching image pairs and search image pairs as the input (X v , X t ) and (Z v , Z t ) of the neural network, wherein (Z v , Z t ) is the input of the matching branch, which is a certain frame randomly selected from the video sequence M train ; (X v , X t ) is the input of the search branch, which is selected in the following manner: after the matching branch image is selected, an image pair after the matching branch image is selected, denoted as (X v , X t ); the learning rate is set to 0.0005, the weights of the network are corrected through back propagation of the loss function until the network converges basically, and a trained neural network is obtained; S4, using the trained neural network for target tracking, specifically: manually selecting a target to be tracked, generating a label of a first frame image, inputting an image sequence containing the label into the trained neural network to obtain the position of the target center and the size of the target.

Images