A Cross-Mirror Target Tracking Method Driven by Spatiotemporal Communication Data

Abstract

The present invention discloses a cross-camera target tracking method driven by spatiotemporal communication data. The method first obtains the coding sequence of the tracking target template image and the search area respectively. A three-stage hybrid attention mechanism then fuses the target image and video frame to obtain the corresponding feature sequence. The probability estimation then obtains the target tracking result for a single-camera video. The residual network of the attention module then obtains the enhanced features of the disappeared target and the video frame, calculates the similarity between the two, and finally determines the target tracking result based on the similarity. The present invention is used to monitor video data in real time. It calls a single camera video in conjunction with the communication information of the tracking target, and continuously tracks abnormal targets in conjunction with videos from other cameras around a specific area until the target leaves the area. This method achieves cross-camera target tracking, thereby providing timely warnings and protecting the personal safety and information security of personnel in key areas.

Description

Technical Field

[0001] The present invention belongs to the field of image processing, in particular to the field of target tracking, and specifically relates to a cross-camera target tracking method driven by spatiotemporal communication data. Background Art

[0002] Object tracking is a fundamental task in computer vision and has been successfully deployed in various applications such as human-computer interaction and visual surveillance. Current popular trackers typically consist of several components to accomplish the tracking task: (1) a backbone that extracts general features of the tracked target and the search region (i.e., the area of interest to be monitored); (2) an integration module that allows information communication between the tracked target and the search region for subsequent target localization; and (3) a task-specific head that accurately locates the target and estimates its bounding box.

[0003] In real-world scenarios, it is difficult to design a simple and effective end-to-end tracker due to factors such as the different scales of the input image, deformation of the target, occlusion, and confusion of similar objects. Recently, researchers have applied the attention-based Transformer to the tracking field due to its global and dynamic modeling capabilities, and have achieved good performance. However, Transformer-based trackers still rely on convolutional neural networks (CNNs) for general feature extraction and only apply attention operations in the latter high-level abstract representation space. Since CNNs use local convolution kernels and lack global modeling capabilities, the application of the self-attention module in the entire tracking model is limited.

[0004] In recent years, multi-shot target tracking has been increasingly used in the field of tracking models. The current mainstream method is to use pedestrian re-identification technology to achieve cross-camera tracking. This technology is mainly divided into image-to-image and image-to-video person re-identification. It aims to retrieve pedestrians who are the same as the query target from an image- or video-based library. Its main principle is to regard cross-camera tracking as a cross-modal retrieval task and learn common potential embedding information from image and video modalities to measure the similarity between pedestrians in the video and the query target.

[0005] However, due to the large gap between image and video modalities and the use of redundant feature learning on all video frames, in scenes with densely distributed cameras, the target may be obscured in one camera but fully observed in another, which means that the partial visibility and indistinguishability of pedestrians are not so serious. Therefore, redundant feature learning will reduce the efficiency and accuracy of the model. Summary of the Invention

[0006] Purpose of the invention: The purpose of the present invention is to address the deficiencies in the prior art and provide a cross-camera target tracking method driven by spatiotemporal communication data. The present invention uses a hybrid attention mechanism to couple general feature extraction and target information, thereby obtaining more refined feature information and having global modeling capabilities; a complementary residual learner is used to obtain the most significant features in the video frame to reduce redundant matching information, while mining the potential complementary information between video frames to enhance feature representation, thereby reducing matching errors during cross-camera matching.

[0007] Technical solution: The present invention provides a cross-camera target tracking method driven by spatiotemporal communication data, comprising the following steps:

[0008] Step S1: Track the target template image T i and the current search area (i.e., the current video frame) S are cut into sub-pictures of fixed size (for example, a target template picture can be divided into N sub-pictures), and the convolution operation is used to perform basic encoding on each sub-picture to obtain the corresponding encoding sequence t j and s j , where the value range of j is changed according to the input target template image size and video frame size; then the basic coding sequence t j and s j Separate and reconstruct, and finally use linear projection operation to obtain the query, key and value vector groups corresponding to the tracking target template image and the search area t ,k t ,v t >、 s ,k s ,v s >

[0009] Among them, T i represents the i-th tracking target template image, i∈{1,2,...,n}; q t ,k t ,v t To track the query, key and value of the target template image, q s ,k s ,v s The query, key, and value for searching images;

[0010] Step S2: Use the three-stage hybrid attention mechanism to extract features from the input coding sequence

[0011] In the first stage, the key and value vectors of the target template image and the search area are spliced separately, and then the basic mixed attention operation is used to obtain the mixed feature F1 after the target template image and the search area are fused;

[0012] ​​In the second stage, the mixed features obtained in the first stage are resized through separable convolution and then input into the mixed attention mechanism to obtain the feature sequence F2 of the second stage;

[0013] In the third stage, the feature sequence F2 of the second stage is resized again through separable convolution, and multiple mixed attention operations are performed to obtain the final feature sequence F3 of the third stage;

[0014] Step S3: First, the feature sequence F3 is separated and reconstructed to obtain the feature sequence of the search area, and the prediction head based on the full convolution operation is used to locate the bounding box of the tracking target to obtain the detection result; then, based on the prediction confidence, an online update is performed, and the next search area of the next video frame is used as the search area of step S1, and the tracking target in the video is re-tracked until the target in the search area disappears;

[0015] Step S4: Utilize mobile phone positioning to obtain the location of the missing target (e.g., mobile phone positioning can be used) and retrieve partial videos from the three nearest cameras. Simultaneously, the spatiotemporal communication information (location information, communication information, and internet access information) of the missing target and the image of the missing target are fused using an attention mechanism. Initial features are extracted from the retrieved video frames using a residual network, thereby obtaining frame-level features of the three camera video sequences and the initial features of the target after feature fusion.

[0016] Step S5: Convolutionally map the video frame-level features containing temporal complementary information into a feature map, and use a multi-head attention mechanism to obtain salient features on the feature map. Then, activation and convolution operations are used to obtain the redundant salient regions activated in the initial features, thereby mining the remaining complementary regions of the current video frame and capturing the remaining complementary information. Next, a residual network is used to obtain the current frame's supplementary features from the complementary information, thereby obtaining enhanced features of the current video frame. Finally, the image of the disappeared target is regarded as a video frame sequence, and its corresponding enhanced features are obtained in the same way.

[0017] Step S6: Calculate the similarity between the acquired enhanced feature vector and the template image of the disappeared target, and set the cosine similarity threshold based on experience. If the maximum similarity is lower than the threshold, it means that the disappeared target has left the key area. Otherwise, obtain the bounding box of the pedestrian with the maximum similarity and the camera number, and return the video sequence of the camera and the target template image to step S1 to continuously track the target.

[0018] Furthermore, the specific process of step S1 is as follows:

[0019] S1.1. Obtain tracking target template images T respectively i and the basic coding sequence t of the search area S j and sj ;

[0020] Each sub-image is encoded using C convolutional layers with 1 zero-padding layer, 4 strides, and 7 kernel sizes. The formula is as follows:

[0021]

[0022] in, H t and W t are the height and width of the tracking target template image, H s and W s Represent the height and width of the search area, t j Indicates the encoding of the target template image, s j Indicates the encoding of the search area, Conv 7×7 represents a convolution with a kernel size of 7;

[0023] Then the basic coding sizes of the target sub-images are obtained as follows: The basic encoding size of the search sub-image is C is the number of channels (value is 64);

[0024] Then, the tracking target template image T i The basic encoding of N target sub-images and the basic encoding of the sub-picture of the search area Expand and splice along the channel direction to obtain a size of The coding sequence obtained by splicing is separated and reconstructed to obtain a two-dimensional feature image of the target coding sequence and the search sequence and

[0025] S1.2. Obtain the query, key, and value vectors of the target template image and search area respectively

[0026] First, for each two-dimensional feature image Perform a deep separation convolution operation DConv, and then perform a matrix flattening operation on each feature map of the target template image and the search area And the linear projection operation Finally, we get the corresponding query, key, and value. The specific formula is as follows:

[0027]

[0028] Furthermore, the detailed process of extracting features using the hybrid attention mechanism in step S2 is as follows:

[0029] Step S2.1: k tand k in the search area s Splice to get the key vector k m =Concat(k t ,k s ), set the v of the target template image t and search area v s Splice to get the value vector v m =Concat(v t ,v s ), Concat represents the concatenation operation; the subscript m here is only used to distinguish the symbols of variable parameters;

[0030] Step S2.2: Use the hybrid attention mechanism to fuse the target template image with the search area. The specific formula is as follows:

[0031]

[0032] d represents the dimension of the key, and the superscript T represents the transpose of the corresponding vector. Represents the activation function, Atten t and Atten s They are the attention maps of the target template image and the search area, that is, extracting their respective features and the correlation between the two;

[0033] Step S2.3: Extract the feature sequence by connecting the target template image and the attention map of the search area through a multi-layer perception mechanism in three stages. The extraction formula is:

[0034]

[0035] Among them, F represents the features extracted from the target template image and the search area using a mixed attention operation, Concat represents the splicing operation, Represents a multilayer perceptron network;

[0036] First, use a mixed attention operation to extract the size of The first stage feature sequence F1; C is the number of channels, for example, the value is 64;

[0037] Then, F1 is sequentially separated and reconstructed to obtain the two-dimensional feature maps of the target and search area, and then 3C convolutions with a kernel size of 3, a step size of 2, and a zero-filling layer of 1 are used to obtain the two-dimensional feature maps of the target and search area. and The encoding sequence is flattened and concatenated and then input into the hybrid attention mechanism. Four hybrid attention operations are performed to obtain the feature sequence F2 of the second stage, whose size is

[0038] Finally, F2 is sequence-separated and reconstructed to obtain the two-dimensional feature maps of the target and search area, and the corresponding encoding sequence is obtained using 6C convolution kernels. After flattening and splicing, it is input into the hybrid attention mechanism for 16 hybrid attention operations to obtain a size of The final feature sequence of the third stage is F3.

[0039] Furthermore, the specific process of obtaining the single camera target detection result in step S3 is as follows:

[0040] First, the feature sequence F3 is split to obtain the search area feature sequence, and the size is reconstructed as Feature map

[0041] Then, Input to a fully convolutional network consisting of L convolutional layers, batch normalization layers, and ReLU activation functions, and obtain two probability maps:

[0042]

[0043] Among them, P a (x,y) represents the vertex probability map at the upper left of the bounding box, P b (x,y) represents the vertex probability map at the bottom right of the bounding box, Conv represents the convolution operation, and BN represents batch normalization;

[0044] Next, the predicted bounding box coordinates are obtained by calculating the expected value of the probability distribution of the two vertices and As shown in the following formula:

[0045]

[0046] The area within these two vertices is the result of target detection;

[0047] Finally, an online updating strategy is adopted to obtain the tracking results.

[0048] Furthermore, the specific method of online updating strategy to obtain tracking results is:

[0049] For the target template image and the attention map of the search area, a multi-layer perceptron network and a Sigmoid function are used to obtain the final score, and then the part with a score greater than 0.5 is selected as the final result and stored in the target template library; the updated target template image and the next frame of video are used as the initial target template image and search area in step S1, and the abnormal target in the video is tracked again until the target disappears in the search area, that is, the target leaves the search area of the camera.

[0050] Furthermore, the detailed process of step S4 includes the following:

[0051] Step S4.1: Obtain fusion features:

[0052] Make loc ,γ net ,γ com They represent the corresponding features of the extracted target’s location information, Internet access information, and communication information, respectively. i Represents the template image of the i-th disappeared target, and the calculation formula for feature fusion using the attention mechanism is as follows:

[0053]

[0054] Among them, α, β are adjustable parameters, Ω t ,Ω γ ,Ω θ is the learnable weight, b A ,b θ is the bias vector, t j is the vector T i An element of θ j is an element of the weight vector θ, The template image representing the i-th disappeared target contains the fusion features of spatiotemporal communication information;

[0055] Step S4.2: Extract initial features using the residual network:

[0056] make Indicates the front of the extraction Frame video sequence, the first four residual layers of the residual network Resnet50 are used to extract the initial features of the video sequence and the disappeared target. The frame-level features of the video sequence are The initial features of the target after feature fusion are The specific formula is as follows:

[0057]

[0058] Among them, h, w and c are each feature The height, width and channel size of , κ represents the sequence number of the camera to which the video sequence belongs, represents the total number of video sequences, and g is the index of the video frame.

[0059] Furthermore, the process of obtaining the enhanced features in step S5 is as follows:

[0060] make represents the initial features of the g-th frame video with camera number κ, represents the enhanced features of the g-1th frame video, which contains the frame-level features of the g-1th frame and the time complementary information of the previous g-1 frames;

[0061] First, the enhanced features Projection into feature map And use the multi-head attention mechanism to generate the corresponding attention map

[0062] Then, the Softmax function is used to obtain the similarity between the salient features of the first g-1 video frames and the initial features of the g-th video frame to obtain the remaining complementary information of the g-th frame video

[0063]

[0064] Conv1 and Conv2 represent different convolutional layers, BN is the batch normalization layer, and Relu is the activation function. is the feature reconstruction operation, represents the salient features of the g-1th video frame with camera number κ, represents the convolution operation, Represents the remaining complementary information of the g-th frame video of camera number κ;

[0065] Finally, the residual network is used to Get the supplementary features of the g-th frame video of the camera And combine it with the g-1th frame-level enhanced feature Integrate to obtain the enhanced features of the g-th frame video

[0066]

[0067] And use the same method to calculate the enhanced features of the disappeared target template image

[0068] Furthermore, the specific method of cross-camera target tracking in step S6 is:

[0069] First, frame-level enhancement features Perform global average pooling to obtain the feature vector of each pedestrian image in the video frame represents the number of pedestrians detected in the g-th frame video of camera number κ;

[0070] Then, the enhanced features of the disappeared target template image Enhanced feature vector of pedestrian targets in video frames Perform channel normalization and similarity calculation:

[0071]

[0072] in represents the similarity between the disappeared target and the pedestrian image in camera κ;

[0073] Next, we set the cosine similarity threshold to 0.5 based on experience, and use the similarity to get the similarity ranking between the pedestrian image and the disappeared target template image.

[0074] Finally, the camera with the pedestrian image with the greatest similarity is selected to continue tracking the disappeared target. That is, the target image is obtained as a new target template image using the pedestrian features corresponding to the length, width and center position coordinates of the bounding box in the video frame, and the corresponding numbered camera video sequence and the new target template image are returned to step 1 to continuously track the target. When the similarity between the pedestrian image and the target template in all video frames is lower than the threshold, it means that the disappeared target has left the key area.

[0075] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0076] (1) The present invention effectively utilizes the correlation between the target and the search area, so that the extracted features are more targeted at the corresponding tracking target and capture more identification features for specific targets.

[0077] (2) Since the present invention adopts a prediction head based on full convolution operation, it can predict the target area, which significantly improves the accuracy and robustness of the tracking algorithm.

[0078] (3) The present invention combines the spatiotemporal communication data of the abnormal target (i.e., the target's mobile phone Internet data, communication data, and location information, etc.) to track it across the camera, thereby improving the tracking speed and accuracy.

[0079] (4) Due to the use of a complementary residual learner, the present invention can not only obtain the temporal complementary information between video frames, thereby suppressing the common salient information to achieve the goal of balancing efficiency and accuracy, but also thoroughly explore the potential complementary information between video frames to enhance the feature representation of pedestrians, thereby reducing the matching error during cross-camera matching. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 It is a flow chart of the arrangement of the present invention;

[0081] Figure 2 Schematic diagram of the hybrid attention mechanism in the embodiment;

[0082] Figure 3 Schematic diagram of the complementary residual learning module in the embodiment. DETAILED DESCRIPTION

[0083] The technical solution of the present invention is described in detail below, but the protection scope of the present invention is not limited to the embodiments.

[0084] The present invention uses the communication information of abnormal targets in the monitored key search area to roughly locate the tracking target to call the real-time video of a nearby single camera, and uses a three-stage hybrid attention mechanism to extract and fuse the features between the target image and the video frame, obtains the detection result through probability estimation, and uses an online update strategy to obtain the target tracking result in the single camera video; then, the disappearing target template is fused with its corresponding spatiotemporal communication information through the attention module, and the position of the disappearing target is obtained by using mobile phone positioning and calling the video of the three cameras with the smallest distance; the fused features of the disappearing target and the video are input into the complementary residual learner for feature enhancement and matching, and then the enhanced features are calculated with the target features for similarity; finally, the pedestrian image with the highest similarity and greater than the threshold is found, and its bounding box and the camera number to which it belongs are returned for continued tracking until the disappeared target is not matched in the queried video, and it is determined that the target has left the key area.

[0085] In short, the present invention can be used to monitor video data in real time, call a single camera video in combination with the communication information of the tracking target, and continuously track abnormal targets in combination with other camera videos around a specific area until the target leaves the area, realizing cross-camera target tracking, thereby achieving timely early warning and protecting the personal safety and information security of people in key areas.

[0086] like Figure 1 As shown, the overall process of cross-camera target tracking of the present invention is: first, obtain the target template and the video sequence of a single camera; then, target tracking, that is, using a multi-stage hybrid attention mechanism to realize single-camera target tracking; then pedestrian re-identification is performed, that is, when the target leaves the field of view of the current camera, the disappeared target template and its spatiotemporal communication information are fused through the attention module, and the complementary residual learning module is used to obtain enhanced features from the multi-camera video, and its similarity with the disappeared target is calculated; secondly, it is determined whether the similarities are all greater than the threshold. If they are all greater than the threshold, the lens to which the pedestrian image with the highest similarity belongs is found and the target template and the video of the camera are returned. If they are not all greater than the threshold, it means that the target has left the key area and the cross-camera tracking is ended.

[0087] Example:

[0088] This embodiment takes monitoring abnormal targets around a key building as an example. Here, a specific range centered on the building is divided into a key area, and the newly added spatiotemporal communication data and its corresponding targets in the key area are regarded as abnormal signals and abnormal targets.

[0089] The specific steps of cross-camera target tracking in this embodiment are as follows:

[0090] Step 1: First, the target template image and the search area (i.e., the video frame) are cut into sub-images of fixed size, and the images are basically encoded using convolution operations. Specifically, C convolution layers with 1 zero-padding layer, 4 strides, and 7 kernel sizes are used to encode the i-th tracking target template image T i And the search area S is encoded to obtain the basic coding sequence t j and s j .

[0091] Then, the target template image T i The basic coding of the N target sub-images and the basic coding of the sub-images of the search area are expanded and spliced along the channel direction, and the obtained coding sequence is separated and reconstructed to obtain the two-dimensional feature images of the target sequence and the search sequence and

[0092] Finally, a depthwise separable convolution operation is performed on each feature map, and each feature map of the target template image and the search area is flattened and processed through linear projection to obtain the required query, key, and value vectors:

[0093]

[0094] Among them, q t 、k t and v t Represents the query, key and value of the target template image respectively, q s 、k s and v s Represents the query, key and value of the search area respectively. DConv represents depth-wise separable convolution. means flattening the matrix, Represents a linear projection operation.

[0095] Step 2: Utilize Figure 2 A three-stage hybrid attention mechanism is obtained to extract features from the input encoding sequence.

[0096] First, the target template image is concatenated with the key and value vectors of the search area to obtain k m and v m , and use the hybrid attention mechanism to fuse the target template image with the search area:

[0097]

[0098] d represents the dimension of key, superscript T represents the transpose of the corresponding vector, Softmax represents the activation function, Atten tand Atten s They are the target template image and the search area attention map.

[0099] Next, the target template image and the attention map of the search area are spliced together through a multi-layer perceptron. The splicing formula is:

[0100] The target template image and the search area are extracted using a mixed attention operation to obtain the feature F1, the size of which is

[0101] Then, F1 is sequentially separated and reconstructed to obtain the two-dimensional feature maps of the target and search area, and then 3C convolutions with a kernel size of 3, a step size of 2, and a zero-filling layer of 1 are used to obtain the two-dimensional feature maps of the target and search area. and The encoding sequence is flattened and concatenated and then input into the hybrid attention mechanism. Four hybrid attention operations are performed to obtain the feature sequence F2 of the second stage, whose size is

[0102] Finally, similar operations are performed to extract the feature maps of the third stage, that is, F2 is sequence-separated and reconstructed to obtain the two-dimensional feature maps of the target and search area, and the corresponding coding sequence is obtained using 6C convolution kernels, which are flattened and spliced and then input into the hybrid attention mechanism for 16 hybrid attention operations to obtain a size of The final feature sequence F3.

[0103] Step 3: To obtain the tracking results of a single camera, first split the feature sequence F3 output in the third stage to obtain the search area feature sequence and reconstruct it into a feature map Then, it is input into a fully convolutional network consisting of L convolutional layers, batch normalization layers, and ReLU activation functions to obtain two probability maps:

[0104] P a (x,y),P b (x,y)=Conv(BN(Relu(T)))

[0105] P a (x,y) represents the vertex probability map at the upper left of the bounding box, P b (x,y) represents the vertex probability map at the bottom right of the bounding box. Next, the predicted bounding box coordinates are obtained by calculating the expected value of the vertex probability distribution and The area between the two vertices here is the result of target detection, as shown in the following formula:

[0106]

[0107] Finally, an online update strategy is used to obtain tracking results. Specifically, the attention map of the predicted results is first obtained. Then, a multi-layer perceptron network and a sigmoid function are used to obtain the final score. The results with scores greater than 0.5 are selected as the final results and stored in the target template library. The updated target template image and the next frame of video are then used as the initial target template image and search area in step 1. The abnormal target in the video is tracked again until the target disappears from the search area, that is, the target leaves the camera's search area.

[0108] Step 4: To improve the accuracy of cross-camera tracking, the target's spatiotemporal communication information is used as auxiliary information for pedestrian re-identification, that is, the position of the disappeared target is obtained by using the mobile phone positioning and the front view of the three nearest cameras is retrieved. Frame video (that is, first locate the geographical location when the tracking target disappears, and then call the three nearest camera videos near the geographical location), and at the same time, the target's spatiotemporal communication information (location information, communication information and Internet information) and the template image of the disappeared target are transmitted through Figure 3 The attention mechanism in

[15] is used to perform feature fusion.

[0109] Then, the first four residual layers of the residual network Resnet50 are used to compare the disappearing target template image with the front images of the three cameras. Perform initial feature extraction on video frames to obtain frame-level features of the video sequence and the initial features of the target template

[0110] Step 5: Use the redundant salient areas on the feature map to mine the remaining complementary information of the current video frame, and use the residual network to obtain supplementary features to enhance the features of the current video frame. To obtain the remaining complementary information, this embodiment uses the enhanced features of the g-1 frame Projection And use the multi-head attention mechanism to generate the corresponding attention map

[0111]

[0112] Then, the Softmax function is used to obtain the similarity between the salient features of the first g-1 video frames and the initial features of the g-th video frame, thereby obtaining the remaining complementary information of the g-th frame video

[0113]

[0114] Here, is the feature reconstruction operation, represents the salient features of the g-1th video frame with camera number κ, Represents the initial features of the g-th frame video with camera number κ.

[0115] Finally, the residual network is used to The supplementary features of the g-th frame video of the camera are obtained and integrated with the enhanced features of the g-1th frame to obtain the enhanced features of the g-th frame

[0116]

[0117] in represents the supplementary features of the g-th frame video of the κ-th camera, Represents the enhanced features of the g-1th frame video of the κth camera, and obtains the enhanced features of the disappeared target template image in the same way

[0118] Step 6: Use similarity calculation to calculate the similarity between the obtained enhanced feature vector and the template image of the disappeared target, and set the cosine similarity threshold based on experience to achieve cross-camera target tracking.

[0119] First, frame-level enhancement features Perform global average pooling to obtain the feature vector of each pedestrian image in the video frame It represents the number of pedestrians detected in the g-th frame video of camera number κ.

[0120] Then, the enhanced features of the disappeared target template image Enhanced feature vector of pedestrians in the video frame Perform channel normalization and calculate the similarity Θ κ : here It represents the similarity between the disappeared target and the pedestrian image in camera κ.

[0121] Next, the cosine similarity threshold is set to 0.5 based on experience, and the similarity ranking between the pedestrian image and the disappeared target template image is obtained according to the size of the similarity.

[0122] Finally, the camera with the most similar pedestrian image is selected to continue tracking the missing target. This involves using the pedestrian's features, including the length, width, and center coordinates of the bounding box in the video frame, to obtain the target image as a new target template image. The corresponding camera video sequence and the new target template image are then returned to step 1 to continue tracking the target. When the similarity between the pedestrian image and the target template image in all video frames falls below a threshold, the missing target has left the focus area.

[0123] In summary, the multi-stage hybrid attention mechanism employed in this invention effectively overcomes the low target recognition accuracy problem caused by local convolution in existing methods using convolutional neural networks. This not only enables single-shot target tracking, but also utilizes a complementary residual learning module to effectively address the low efficiency of person re-identification (PRI) due to redundant feature learning between video frames, further enabling cross-shot tracking. By monitoring key areas, the target is roughly located based on its communication information to enable single-shot tracking using a single nearby camera. Person re-identification technology is then used to locate the missing target in other cameras and continuously track it, achieving fast and accurate cross-shot tracking of the target.

Claims

1. A cross-camera target tracking method driven by spatiotemporal communication data, characterized by: The following steps are involved: Step S1: Track the target template image T i The current search area S is cut into sub-images of fixed size, and the convolution operation is used to perform basic encoding on each sub-image to obtain the corresponding encoding sequence t j and s j ; Then the basic coding sequence t j and s j Separate and reconstruct, and finally use linear projection operation to obtain the query, key and value vector groups corresponding to the tracking target template image and the search area t ,k t ,v t >、 s ,k s ,v s >​​ Among them, q t ,k t ,v t To track the query, key and value of the target template image, q s ,k s ,v s The query, key, and value for the search area; Step S2: Use the three-stage hybrid attention mechanism to extract features from the input coding sequence In the first stage, the key and value vectors of the target template image and the search area are spliced separately, and then the basic mixed attention operation is used to obtain the first stage feature sequence F1 after the target template image and the search area are fused; In the second stage, the first stage feature sequence F1 is resized through separable convolution and then input into the hybrid attention mechanism to obtain the second stage feature sequence F2; In the third stage, the second-stage feature sequence F2 is resized through separable convolution, and multiple mixed attention operations are performed to obtain the final third-stage feature sequence F3; Step S3: First, the feature sequence F3 is separated and reconstructed to obtain the feature sequence of the search area, and the prediction head based on the full convolution operation is used to locate the bounding box of the tracking target to obtain the detection result; then, based on the prediction confidence, an online update is performed, and the next frame of the video is used as the search area of step S1, and the tracking target in the video is re-tracked until the target in the search area disappears; Step S4: Use the mobile phone to locate the missing target and retrieve partial videos from the three nearest cameras. Simultaneously, the spatiotemporal communication information of the missing target and the image of the missing target are fused using an attention mechanism. The initial features of each of the retrieved video frames are extracted using a residual network, resulting in frame-level features of the three camera video sequences and the initial features of the missing target after feature fusion. Step S5: Convolutionally map the video frame-level features containing temporal complementary information into a feature map, and use a multi-head attention mechanism to obtain salient features on the feature map. Then, activation and convolution operations are used to obtain the redundant salient regions activated in the initial features, thereby mining the remaining complementary regions of the current video frame and capturing the remaining complementary information. Next, a residual network is used to obtain the current frame's supplementary features from the complementary information, thereby obtaining enhanced features of the current video frame. Finally, the image of the disappeared target is regarded as a video frame sequence, and its corresponding enhanced features are obtained in the same way. Step S6: Calculate the similarity between the acquired enhanced feature vector and the template image of the disappeared target, and set the cosine similarity threshold based on experience. If the maximum similarity is lower than the threshold, it means that the disappeared target has left the key area. Otherwise, obtain the bounding box of the pedestrian with the maximum similarity and the camera number, and return the video sequence of the camera and the target template image to step S1 to continuously track the target.

2. The cross-camera target tracking method based on spatiotemporal communication data drive according to claim 1, characterized in that: The specific process of step S1 is: S1.

1. Obtain tracking target template images T respectively i And the basic coding sequence t of the search picture S within the video frame j and s j ; Each sub-image is encoded using C convolutional layers with 1 zero-padding layer, 4 strides, and 7 kernel sizes. The formula is as follows: in, H t and W t are the height and width of the tracking target template image, H s and W s Respectively represent the height and width of the search image, t j Indicates the encoding of the target template image, s j Indicates the encoding of the search image, Conv 7×7 represents the convolution with a kernel size of 7; T i represents the i-th tracking target template image, i∈{1,2,...,n}; Then the basic coding sizes of the target sub-images are obtained as follows: The basic encoding size of the search sub-image is C is the number of channels; Then, the tracking target template image T i The basic encoding of N target sub-images and the basic encoding of the sub-picture of the search area Expand and splice along the channel direction to obtain a size of The coding sequence obtained by splicing is separated and reconstructed to obtain a two-dimensional feature image of the target coding sequence and the search sequence and S1.

2. Obtain the query, key, and value vectors of the target template image and the search image respectively First, for the two-dimensional feature image Perform a deep separation convolution operation DConv, and then perform a matrix flattening operation on each feature map of the target template and the search image And the linear projection operation , and finally get the corresponding query, key and value. The specific formula is as follows:

3. The cross-camera target tracking method based on spatiotemporal communication data drive according to claim 1, characterized in that: The detailed process of extracting features using the hybrid attention mechanism in step S2 is as follows: Step S2.1: k t and k in the search area s Splice to get the key vector k m =Concat(k t ,k s ), set the v of the target template image t and search area v s Splice to get the value vector v m =Concat(v t ,v s ), Concat represents the concatenation operation; Step S2.2: Use the hybrid attention mechanism to fuse the target template image with the search area. The specific formula is as follows: d represents the dimension of the key, and the superscript T represents the transpose of the corresponding vector. Represents the activation function, Atten t and Atten s They are the attention maps of the target template image and the search area; Step S2.3: Extract the feature sequence by connecting the target template image and the attention map of the search area through a multi-layer perception mechanism in three stages. The extraction formula is: Among them, F represents the features extracted from the target template image and the search area using a mixed attention operation, Concat represents the splicing operation, Represents a multilayer perceptron network; First, use a mixed attention operation to extract the size of The first stage feature sequence F1, C is the number of channels; H t and W t are the height and width of the tracking target template image, H s and W s Respectively represent the height and width of the search image; Then, F1 is sequentially separated and reconstructed to obtain the two-dimensional feature maps of the target and search area, and then 3C convolutions with a kernel size of 3, a step size of 2, and a zero-filling layer of 1 are used to obtain the two-dimensional feature maps of the target and search area. and The encoding sequence is flattened and concatenated and then input into the hybrid attention mechanism. Four hybrid attention operations are performed to obtain the second-stage feature sequence F2, whose size is Finally, F2 is sequence-separated and reconstructed to obtain the two-dimensional feature maps of the target and search area, and the corresponding encoding sequence is obtained using 6C convolution kernels. After flattening and splicing, it is input into the hybrid attention mechanism for 16 hybrid attention operations to obtain a size of The third stage feature sequence F3.

4. The cross-camera target tracking method based on spatiotemporal communication data drive according to claim 1, characterized in that: The specific process of obtaining the single camera target detection result in step S3 is: First, the feature sequence F3 is split to obtain the search area feature sequence, and the size is reconstructed as Feature map , C is the number of channels; Then, Input to a fully convolutional network consisting of L convolutional layers, batch normalization layers, and ReLU activation functions, and obtain two probability maps: Among them, P a (x,y) represents the vertex probability map at the upper left of the bounding box, P b (x,y) represents the vertex probability map at the bottom right of the bounding box, Conv represents the convolution operation, and BN represents batch normalization; Next, the predicted bounding box coordinates are obtained by calculating the expected value of the probability distribution of the two vertices and As shown in the following formula: Finally, an online updating strategy is adopted to obtain the tracking results.

5. The cross-camera target tracking method based on spatiotemporal communication data drive according to claim 4, characterized in that: The specific method of online updating strategy to obtain tracking results is: For the target template image and the attention map of the search area, a multi-layer perceptron network and a Sigmoid function are used to obtain the final score. Then, the part with a score greater than 0.5 is selected as the final result and stored in the target template library. The updated target template image and the next frame of video are used as the initial target template image and search area in step S1, and the abnormal target in the video is tracked again until the target disappears in the search area, that is, the target leaves the search area of the camera.

6. The cross-camera target tracking method based on spatiotemporal communication data drive according to claim 1, characterized in that: The detailed process of step S4 includes the following: Step S4.1: Obtain fusion features: Use Υ loc ,Υ net ,Υ com The calculation formula for representing the corresponding features of the target's location information, Internet access information, and communication information in the spatiotemporal communication message, and then using the attention mechanism to fuse the features, is as follows: Among them, α, β are adjustable parameters, Ω t ,Ω γ ,Ω θ is the learnable weight, b A ,b θ is the bias vector, t j is the vector T i An element of θ j is an element of the weight vector θ, The template image representing the i-th disappeared target contains the fusion features of spatiotemporal communication information; Step S4.2: Extract initial features using the residual network: make Indicates the front of the extraction Frame video sequence, the first four residual layers of the residual network Resnet50 are used to extract the initial features of the video sequence and the disappeared target. The frame-level features of the video sequence are The initial target features after feature fusion are , the specific formula is as follows: Among them, h, w and c are each feature The height, width and channel size of , κ represents the sequence number of the camera to which the video sequence belongs, represents the total number of video sequences, and g is the index of the video frame.

7. The cross-camera target tracking method based on spatiotemporal communication data drive according to claim 1, characterized in that: The process of obtaining the enhanced features in step S5 is as follows: make represents the initial features of the g-th frame video with camera number κ, Represents the enhanced features of the g-1th frame video, which contains the frame-level features of the g-1th frame and the time complementary information of the previous g-1 frames; h, w and c are each feature height, width, and channel size; First, the enhanced features Projection into feature map And use the multi-head attention mechanism to generate the corresponding attention map Then, the Softmax function is used to obtain the similarity between the salient features of the first g-1 video frames and the initial features of the g-th video frame to obtain the remaining complementary information of the g-th frame video Conv1 and Conv2 represent different convolutional layers, BN is the batch normalization layer, and Relu is the activation function. is the feature reconstruction operation, represents the salient features of the g-1th video frame with camera number κ, represents the convolution operation, Represents the remaining complementary information of the g-th frame video of camera number κ; Finally, the residual network is used to Get the supplementary features of the g-th frame video of the camera And combine it with the g-1th frame-level enhanced feature Integrate to obtain the enhanced features of the g-th frame video And use the same method to calculate the enhanced features of the disappeared target template image 8. The cross-camera target tracking method based on spatiotemporal communication data drive according to claim 1, characterized in that: The specific method of cross-lens target tracking in step S6 is: First, frame-level enhancement features Perform global average pooling to obtain the feature vector of each pedestrian image in the video frame represents the number of pedestrians detected in the g-th frame video of camera number κ; Then, the enhanced features of the disappeared target template image Enhanced feature vector of pedestrian targets in video frames Perform channel normalization and similarity calculation: in represents the similarity between the disappeared target and the pedestrian image in camera κ; Next, we set the cosine similarity threshold based on experience, and use the similarity to get the similarity ranking between the pedestrian image and the disappeared target template image. Finally, the camera with the pedestrian image with the greatest similarity is selected to continue tracking the disappeared target. That is, the target image is obtained as a new target template image using the pedestrian features corresponding to the length, width and center position coordinates of the bounding box in the video frame, and the corresponding numbered camera video sequence and the new target template image are returned to step 1 to continuously track the target. When the similarity between the pedestrian image and the target template image in all video frames is lower than the threshold, it means that the disappeared target has left the key area.

Images