Pedestrian cross-border head tracking method and system based on multi-modal dynamic feature fusion

Abstract

The application discloses a pedestrian cross-border head tracking method and system based on multi-modal dynamic feature fusion, belongs to the technical field of computer vision, and aims to solve the technical problem of how to improve the accuracy and efficiency of pedestrian detection and the stability of cross-border head tracking. By adopting advanced backbone networks, an adaptive anchor box mechanism and cross-scale feature fusion technology, the detection capability for small targets and high occlusion rate pedestrian targets is improved. Intelligent caching strategies and model fusion methods reduce the repeated calculation of instance segmentation and improve the efficiency of dynamic pedestrian segmentation. The combination of various feature extraction and enhancement technologies, including optimized Mamba, Transformer and multi-granularity feature fusion, enhances the purity, discriminability and adaptability to occlusion and view angle changes of features.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, specifically to a method and system for tracking pedestrians across head boundaries based on multimodal dynamic feature fusion. Background Technology

[0002] Existing pedestrian detection, segmentation, and tracking technologies suffer from several drawbacks, including insufficient detection capabilities for small targets, poor adaptability to complex scenes, inaccurate feature extraction, and low stability in cross-camera tracking. For instance, traditional target detection models are prone to missing small targets, instance segmentation is computationally inefficient in dynamic video streams, feature extraction struggles to effectively handle occlusion and viewpoint changes, and cross-camera tracking exhibits unsatisfactory recall rates when pedestrian occlusion rates are high.

[0003] Improving the accuracy and efficiency of pedestrian detection, as well as the stability of cross-border head tracking, are technical problems that need to be solved. Summary of the Invention

[0004] The technical objective of this invention is to address the above-mentioned shortcomings by providing a method and system for pedestrian cross-border head tracking based on multimodal dynamic feature fusion, thereby solving the technical problems of how to improve the accuracy and efficiency of pedestrian detection and the stability of cross-border head tracking.

[0005] In a first aspect, the present invention provides a method for pedestrian cross-border head tracking based on multimodal dynamic feature fusion, comprising the following steps:

[0006] Multi-scale pedestrian detection and instance segmentation: Multi-scale feature extraction is performed on the input video image using the SwinTransformer network structure. The extracted feature maps are dynamically fused based on the improved FPN network structure with dynamic weights. Adaptive anchor boxes are generated on the generated fused feature map. RoIAlign is used to extract features from candidate boxes and output a fixed-size feature map as candidate box features. The candidate boxes are then segmented using the SAM2 model to generate pedestrian segmentation masks. The importance score of the cache is calculated based on IoU, area, and time decay, and the cache content is dynamically updated or cleared.

[0007] Pedestrian Feature Extraction and Fusion: Based on candidate box features and pedestrian segmentation masks, continuous feature extraction is performed using lightweight Mamba to output continuous variation features of pedestrian appearance and gait. Local and global features are extracted using lightweight Mamba, and multi-granular feature interaction and fusion are performed using LiteTransformer to output fused features that combine details and overall representation. Global features and multiple local features are differentiated based on the fused features. An attention mechanism is used to dynamically allocate the weights of local features and concatenate the global features to generate pedestrian features.

[0008] Cross-border head tracking: Pedestrian detection is performed on input multi-camera video frames using an optimized EfficientDet-D4 model. The BiFPN++ module enhances multi-scale feature fusion, outputting detection box images including coordinates, confidence scores, and detection box IDs. The detection box images are then compared with candidate boxes for complementary verification. HRNet keypoint detection and perspective transformation are used to align the pose of the detection box images. Using the pose-aligned detection box images as input, a ResNet-50 model extracts basic features, which are then fused with pedestrian features to generate ReID features. These ReID features are bound to the detection box IDs and stored in a cache. The cosine similarity of the ReID features and the IoU overlap of the detection boxes are calculated. The consistency score between the predicted speed and the actual speed is calculated using LSTM. The cosine similarity of the ReID features, the IoU overlap of the detection boxes, and the consistency score between the predicted and actual speeds are weighted and fused to obtain a comprehensive similarity score. A threshold is then used to determine whether the pedestrians are the same person.

[0009] As a preferred method, multi-scale pedestrian detection and instance segmentation includes the following steps:

[0010] Multi-scale feature extraction: The input video image is divided into image blocks through the SwinTransformer network structure, and multi-scale feature extraction is performed through linear embedding layer and hierarchical Transformer module, outputting feature maps with different downsampling ratios and semantic levels, namely F3, F4 and F5;

[0011] Dynamic feature fusion: Based on the improved FPN network structure, feature maps F3, F4, and F5 are fused in a top-down and bottom-up manner. During the fusion process, the fusion weights are dynamically adjusted according to the pedestrian scale distribution of the current frame to generate fused feature maps P3, P4, and P5.

[0012] Adaptive anchor box generation: Assign a hierarchical index to each fused feature map, calculate a dynamic adjustment coefficient based on the difference between the pedestrian size in the current frame and the average size of the training set, and generate anchor boxes with multiple aspect ratios;

[0013] Feature pooling: Combine RoIAlign to extract features from candidate boxes. During feature extraction, the candidate boxes are mapped to the corresponding feature maps, floating-point coordinates are retained, the RoI region is divided into sub-regions, floating-point coordinates are sampled in each sub-region, bilinear interpolation is used to calculate the feature values ​​of the sampled points, and they are aggregated into a fixed-size feature map as the candidate box feature map.

[0014] SAM2 Cache Update: Taking the candidate bounding box feature map as input, the SAM2 model outputs a pedestrian instance segmentation mask, and calculates the cache importance score by combining the IoU between the current frame and the previous frame, the segmentation region area and the timestamp. The cache content is dynamically updated or cleared based on the cache management mechanism.

[0015] Preferably, the adaptive anchor box generation includes the following steps:

[0016] Determine the hierarchical index: Assign a hierarchical index to each fused feature map;

[0017] Calculate the dynamic adjustment coefficient: Based on the difference between the pedestrian size in the current frame and the average size of the training set, calculate the dynamic adjustment coefficient. Where the numerator is the sum of the areas of the first N pedestrian boxes, and the denominator is the product of the average area of ​​the training set and N;

[0018] Calculate the anchor frame dimensions: For each level k, calculate the reference area of ​​the anchor frame for that level using the following formula:

[0019]

[0020] Among them, S k The anchor frame size (area) for the k-th FPN level;

[0021] S base As the benchmark scale;

[0022] γ is the scaling factor;

[0023] w i h i The width and height of the first N pedestrian bounding boxes detected in the current frame;

[0024] The sum of the areas of the first N pedestrian boxes;

[0025] The product of the average pedestrian area in the N·avg_area training set and N;

[0026] Generate multi-aspect ratio anchor boxes: based on S for each level k This generates three commonly used aspect ratio anchor frames to cover pedestrians of different shapes at that level.

[0027] Multi-scale object detection: Multi-scale hierarchical feature extraction is performed on the input image through the SwinTransformer network structure, and multiple feature maps are output. Based on the improved FPN structure with dynamic weights, the multiple feature maps are fused and multiple fused feature maps are output.

[0028] Candidate box generation: For each fused feature map, an anchor box adapted to the corresponding scale of pedestrians is generated on the fused feature map to obtain adaptive candidate boxes of different scales. The candidate boxes are mapped to floating-point coordinates through RoIAlign, each candidate box is divided into multiple sub-regions, and bilinear interpolation sampling is performed. The sampled points are aggregated into features of a fixed size as candidate box features.

[0029] For each candidate bounding box, the importance score of the current frame segmentation result is calculated based on IoU, region area and time decay, and high-value frames are selectively cached.

[0030] Preferably, during SAM2 cache updates, the features of all pedestrian candidate boxes in the current frame are obtained. Using these candidate box features as input, combined with the floating-point coordinates of the candidate boxes in the original image, a pixel-level binary mask generated by the ASM2 model that only covers the pedestrian region within the candidate box is used as the pedestrian segmentation mask. The IoU value (IoU) between the current frame segmentation result and the same type of pedestrian in the previous frame is then calculated. t Retrieve the maximum area A of the pedestrian segmentation region from the historical cache. max Given the current frame's timestamp t, calculate the cache importance score for the current frame using the following formula:

[0031]

[0032] Among them, S t Assign an importance score to the current frame buffer; the higher the score, the greater the value of the current frame buffer.

[0033] α and β are trainable parameters: α controls the IoU. t The larger the α value, the more emphasis is placed on the consistency of the two-frame segmentation; β controls the contribution to the scoring. The larger the β value, the more important the large region segmentation results are in terms of contribution to the score.

[0034] IoU t The IoU value of the segmentation result;

[0035] A t The area of ​​the segmented region in the current frame;

[0036] A max This represents the maximum area of ​​the historical cache segment.

[0037] λ is the time decay coefficient;

[0038] t is the timestamp of the current frame.

[0039] e -λt For time decay term,

[0040] α·IoU t For IoU consistency terms, IoU t The higher the score, the greater the reference value of the current frame segmentation result for subsequent frames, and the higher the score for this contribution.

[0041] For the percentage of the segmented region, A max The closer to 1, the richer the feature information contained in the segmentation result, and the higher the score of this contribution;

[0042] (1-α-β)·e -λt () represents the time decay term; the smaller t is, the more e -λt The closer to 1, the higher the score for this contribution; conversely, for frames that have not been updated for a long time, the score drops rapidly, reflecting the time decay logic.

[0043] S is obtained by weighted summation of the consistency term, the segmentation region proportion term, and the time decay term. t If S t If the value exceeds the threshold, the current frame segmentation result is considered high-value, and this will include the initial segmentation mask, segmentation region boundary coordinates, pedestrian instance unique ID, frame number / timestamp t, and A. t The complete segmentation information is stored in the cache management unit. If S t If the value of the segmented frame is less than or equal to the threshold, the current frame is considered low-value, the cache is not updated, and the result of this frame is not reused first in subsequent frame processing. Furthermore, the cache management unit periodically checks the cached frames according to the time decay term: frames with a large cache time t are directly evicted; frames with a medium t and S... t Within a predetermined range, the cache weight of frames is reduced, and cached results with higher weights are prioritized for reuse, i.e., the S segmentation information is reused first. t Higher, timestamp-updated frames.

[0044] As a preferred method, pedestrian feature extraction and fusion includes the following steps:

[0045] By performing structured pruning and quantization operations on Mamba, a lightweight Mamba is obtained.

[0046] Candidate bounding box features and pedestrian segmentation masks are concatenated into multimodal input features. These multimodal input features are then normalized. The normalized multimodal input features are used as input, and continuous feature extraction is performed using lightweight Mamba. The output is a pedestrian appearance continuity feature vector F. cont ;

[0047] Based on pedestrian segmentation masks, the pedestrian region is divided into multiple local regions: head, upper body, and lower body. Each local region is input into lightweight Mamba for feature extraction, resulting in local features for each region. These local features are then concatenated to form a complete local feature F. l ;

[0048] The candidate bounding box features are subjected to global average pooling. The candidate bounding box features after global average pooling are then fused with the global area and shape information of the segmentation mask, and input into lightweight Mamba to obtain the global feature F. g And adjust F through 1×1 convolution. l or F g The number of channels;

[0049] With global feature F g As input, it uses LiteTransformer to perceive local details and outputs optimized global features F. g_opt and F l Segmented input to a lightweight Mamba SSM layer, enhancing local details, outputting F. l_opt Based on the constructed interaction matrix, F g_opt and F l_opt Fusion, outputting multi-granularity fused features F fused ;

[0050] From fusion feature F fused Extracting global features F from the middle g and local features F of each part l i To determine the effectiveness of each local region, the global feature F is used. g Using the query and local features as the key, the weights of the local features corresponding to each local region are calculated. After weighted fusion of the local features, the global features are concatenated to generate the final pedestrian feature F. out .

[0051] As a preferred method, cross-border head tracking includes the following steps:

[0052] For the EfficientDet-D4 model, the optimal structural parameters are searched through Bayesian optimization, and the BiFPN++ module is introduced into the EfficientDet-D4 model to obtain an optimized EfficientDet-D4 model. The BiFPN++ module performs bidirectional fusion of multiple hierarchical feature maps P3, P4, P5, P6 and P7 output by the Backbone in the EfficientDet-D4 model from top to bottom and bottom to top, and dynamically adjusts the fusion contribution through learnable weight coefficients.

[0053] The multi-camera video frames of the input optimized EfficientDet-D4 model are preprocessed, and normalization, size adaptation and channel rearrangement are performed through the preprocessing.

[0054] The preprocessed multi-camera video frames are input into the optimized EfficientDet-D4 model, and the output is a detection box image including coordinates, confidence score and detection box ID. The detection box ID is generated in the format of shot number, video frame number and detection box number.

[0055] The predicted output detection boxes are detected and filtered. The filtering rules include basic filtering and high occlusion protection. In basic filtering, the detection boxes are filtered based on confidence. In high occlusion protection, the detection box images are filtered by combining the mapping table between the segmented regions and candidate boxes in the cache. If the proportion of pedestrian pixels exceeds a predetermined value, the detection box image is retained.

[0056] Using the bounding box image as input, HRNet is used to detect pedestrian key points and calculate the perspective transformation matrix to correct non-frontal poses into standardized frontal images.

[0057] A standardized frontal image is input into a ResNet-50 to extract basic features. These features are then jointly trained using ArcFace, Triplet, and Circle loss functions. The basic features are concatenated with pedestrian features and compressed into a 512-dimensional feature vector using a 1×1 convolution, which serves as the ReID feature F. final The ReID feature is bound to the detection box ID and stored in the cache;

[0058] The detection box image is compared with the ReID feature F. final In addition to speed data pairing, the cosine similarity formula is used to calculate the ReID feature similarity between the two detection boxes in the candidate pair, calculate the IoU overlap of the detection boxes, output the predicted speed based on the pre-trained LSTM model, calculate the consistency score between the predicted speed and the actual speed, and weight and fuse the ReID feature cosine similarity, the IoU overlap of the detection boxes and the consistency score between the predicted speed and the actual speed to obtain the comprehensive similarity.

[0059] It records pedestrian ID, detection box, ReID features, speed and matching status, supports trajectory prediction and missing target location completion, and cleans up and archives trajectories that have not been matched for a long time.

[0060] As a preferred configuration, the LSTM model is set up as follows: input the velocity sequence and normalized position sequence from frame t-4 to t-1, and measure the velocity change Δv in frame t. x and Δv y Calculate the prediction speed v for frame t+1. x t+1=v x ,t+Δv x v y t+1=v y ,t+Δv y ;

[0061] Get the actual speed (x) of the detection box in frame t+1. real ,y real ), calculate the Euclidean distance between the predicted speed and the actual speed:

[0062]

[0063] Substitute the Gaussian function to calculate the consistency score:

[0064]

[0065] Where σ is an empirical parameter.

[0066] In a second aspect, the present invention provides a pedestrian cross-border head tracking system based on multimodal dynamic feature fusion, comprising a multi-scale pedestrian detection and instance segmentation module, a pedestrian feature extraction and fusion module, and a cross-border head tracking module, for implementing pedestrian cross-border head tracking through a pedestrian cross-border head tracking method based on multimodal dynamic feature fusion as described in any of the first aspects.

[0067] The pedestrian cross-border head tracking method and system based on multimodal dynamic feature fusion of the present invention have the following advantages:

[0068] 1. Improve detection accuracy: By adopting advanced backbone networks, adaptive anchor box mechanisms and cross-scale feature fusion, the detection capability for small targets and pedestrian targets with high occlusion rates is improved;

[0069] 2. Enhanced segmentation efficiency: Intelligent caching strategies and model fusion methods reduce redundant calculations in instance segmentation and improve the efficiency of dynamic pedestrian segmentation;

[0070] 3. Improved feature quality: The combination of multiple feature extraction and enhancement techniques, including optimized Mamba, Transformer and multi-granularity feature fusion, enhances the purity, discriminativeness and adaptability to occlusion and viewpoint changes of features;

[0071] 4. Enhanced tracking stability: Multimodal matching strategies and advanced trajectory prediction models improve the stability and accuracy of cross-camera tracking, reducing missed detections and false detections. Attached Figure Description

[0072] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0073] The invention will be further described below with reference to the accompanying drawings.

[0074] Figure 1 This is a flowchart of a pedestrian cross-border head tracking method based on multimodal dynamic feature fusion, as described in Example 1. Detailed Implementation

[0075] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments are not intended to limit the present invention. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0076] This invention provides a method and system for pedestrian cross-border head tracking based on multimodal dynamic feature fusion, which addresses the technical problems of improving the accuracy and efficiency of pedestrian detection and the stability of cross-border head tracking.

[0077] Example 1:

[0078] This invention provides a method for pedestrian cross-border head tracking based on multimodal dynamic feature fusion, comprising three steps: multi-scale pedestrian detection and instance segmentation, pedestrian feature extraction and fusion, and cross-border head tracking.

[0079] Step S100: Multi-scale pedestrian detection and instance segmentation: Multi-scale feature extraction is performed on the input video image using the SwinTransformer network structure. The extracted feature maps are dynamically fused based on the improved FPN network structure with introduced dynamic weights. Adaptive anchor boxes are generated on the generated fused feature map. In conjunction with RoIAlign, feature extraction is performed on the candidate boxes, and a fixed-size feature map is output as the candidate box features. The candidate boxes are segmented using the SAM2 model to generate pedestrian segmentation masks. The cache importance score is calculated based on IoU, area, and time decay, and the cache content is dynamically updated or cleared.

[0080] As a specific implementation of multi-scale pedestrian detection and instance segmentation, the following steps are included:

[0081] (1) Multi-scale feature extraction: The input video image is divided into image blocks by the SwinTransformer network structure, and multi-scale feature extraction is performed by the linear embedding layer and the hierarchical Transformer module. The output feature maps with different downsampling ratios and semantic levels are F3, F4 and F5, respectively.

[0082] (2) Dynamic feature fusion: Based on the improved FPN network structure, feature maps F3, F4, and F5 are fused in a top-down and bottom-up manner. During the fusion process, the fusion weights are dynamically adjusted according to the pedestrian scale distribution of the current frame to generate fused feature maps P3, P4, and P5.

[0083] (3) Adaptive anchor box generation: Assign a hierarchical index to each fused feature map, calculate the dynamic adjustment coefficient based on the difference between the pedestrian size in the current frame and the average size of the training set, and generate multi-aspect ratio anchor boxes;

[0084] (4) Feature pooling: Combine RoIAlign to extract features from candidate boxes. During feature extraction, the candidate boxes are mapped to the corresponding feature maps, floating-point coordinates are retained, the RoI region is divided into sub-regions, floating-point coordinates are sampled in each sub-region, bilinear interpolation is used to calculate the feature values ​​of the sampled points, and they are aggregated into a fixed-size feature map as the candidate box feature map.

[0085] (5) SAM2 cache update: Taking the candidate box feature map as input, the pedestrian instance segmentation mask is output through the SAM2 model, and the cache importance score is calculated by combining the IoU between the current frame and the previous frame, the area of ​​the segmented region and the timestamp. The cache content is dynamically updated or cleared based on the cache management mechanism.

[0086] This example involves steps 1.1 adaptive anchor box generation, 1.2 feature pooling, and 1.3 SAM2 cache update.

[0087] Adaptive anchor box generation includes the following:

[0088] (1) SwinTransformer hierarchical feature extraction: First, the basic feature extraction is completed by SwinTransformer. The three hierarchical feature maps (denoted as F3, F4, and F5) output by Stage 3, Stage 4, and Stage 5 of SwinTransformer ensure that F3 / F4 / F5 meet the requirements of subsequent fusion. The specific process is as follows: After the original image is input, it is divided into image blocks of fixed size by the PathPartition layer; the segmented image blocks are mapped into feature vectors with consistent dimensions by the LinearEmbedding layer; the feature vectors are sequentially entered into the improved Transformer module of Stage 1-5, where: Stage 1 directly processes the feature vectors output by LinearEmbedding; before being input into the Transformer module, Stage 2-4 are downsampled by the PathMerging layer (the feature map size is halved and the number of channels is doubled), and finally output F3 (Stage 3), F4 (Stage 4), and F5 (Stage 5).

[0089] (2) Improved FPN feature fusion (enhanced multi-scale feature representation): Based on the bottom-up and top-down logic optimization of conventional FPN, the focus is on solving the adaptation problem between Transformer features and anchor box generation. The process is as follows: Top-down feature upsampling: Upsample F5 (the highest layer with the strongest semantics) to the same size as F4 through bilinear interpolation, and then add it to F4 element-wise (fusion of high-level semantics and mid-level details) to obtain the fused feature map P4; Secondary fusion: Upsample P4 to the same size as F3, and add it to F3 to obtain the fused feature map P3; Dynamic weight adjustment (core improvement): During the fusion process, F3 / F4 / F5 are given dynamic weights (the weights are determined by the pedestrian scale distribution in the current frame. When the proportion of small-scale pedestrians is high, the weight of F3 is increased, and when the proportion of large-scale pedestrians is high, the weight of F5 is increased), instead of the fixed weights of conventional FPN, to enhance the target scale adaptability.

[0090] (3) Adaptive Anchor Box Generation: On the three fused feature maps P3, P4, and P5 output by the improved FPN, anchor boxes adapted to the corresponding scale of pedestrians are generated respectively. The specific calculation logic is as follows: Determine the hierarchical index: Assign a hierarchical index k to the three fused feature maps (k=1 for P3, k=2 for P4, and k=3 for P5); Calculate the dynamic adjustment coefficient: Calculate the adjustment term based on the difference between the pedestrian size in the current frame and the average size of the training set. (The numerator is the sum of the areas of the first N pedestrian boxes, and the denominator is the product of the average area of ​​the training set and N); Substitute into the formula to calculate the anchor box scale: For each level k, calculate the reference area of ​​the anchor box at that level using the following formula:

[0091]

[0092] Among them, S k The anchor frame size (area) for the k-th FPN level;

[0093] S base As the benchmark scale;

[0094] γ is the scaling factor;

[0095] w i h i The width and height of the first N pedestrian bounding boxes detected in the current frame;

[0096] The sum of the areas of the first N pedestrian boxes;

[0097] The product of the average pedestrian area in the N·avg_area training set and N;

[0098] Generate multi-aspect ratio anchor boxes: based on S for each level kGenerate three commonly used aspect ratio anchor frames to cover pedestrians of different shapes (such as standing pedestrians and bending pedestrians) at this level.

[0099] Step 1.2 Feature Pooling: Traditional RoIPooling requires forcibly quantizing the floating-point coordinates of candidate boxes on the feature map to integers when processing candidate boxes. This leads to a positional offset between the candidate box features and the real pedestrian target (especially significant for small-scale pedestrians and pedestrians in edge regions), thus reducing the accuracy of subsequent detection and segmentation. RoIAlign, however, avoids this problem by using unquantized coordinate mapping and bilinear interpolation sampling: it retains the floating-point coordinates of candidate boxes on the feature map without forcibly quantizing them to integers; for non-integer feature points corresponding to the floating-point coordinates, it calculates their feature values ​​through bilinear interpolation, ensuring the positional accuracy and continuity of feature extraction.

[0100] The purpose of RoIAlign is to accurately extract fixed-size features from the adaptive candidate boxes of different scales (from distant pedestrians to nearby pedestrians) generated in step 1.1. This avoids feature loss due to coordinate quantization for small-scale pedestrians and ensures the positional accuracy of features for large-scale pedestrians, providing a high-quality feature foundation for subsequent instance segmentation mask prediction and pedestrian classification. From candidate boxes to fixed-size features:

[0101] (1) Candidate box coordinate mapping (from original image to feature map): Since the candidate box coordinates are defined based on the original image, they need to be mapped to the coordinate system of the fused feature map of its level (the key is to combine the downsampling rate and retain floating-point numbers);

[0102] (2) The RoI region is divided into uniform sub-regions: In order to achieve fixed-size output, the RoI region needs to be uniformly divided into sub-regions that match the output size;

[0103] (3) Sampling and bilinear interpolation calculation within sub-regions: Sampling point selection: Four sampling points are evenly selected within each sub-region (2 rows and 2 columns, such as near the top left, top right, bottom left, and bottom right corners of the sub-region). The coordinates of the sampling points are floating-point numbers. Bilinear interpolation calculation of the feature values ​​of the sampling points: Find the four nearest integer coordinate points around the sampling point; assign weights based on the distance between the sampling point and the four integer points, and sum the weighted values ​​to obtain the feature values ​​of the sampling point; The calculation of the feature values ​​of the sampling points can be visualized through the bilinear interpolation formula. For the sampling point of the (i,j)th sub-region within the sub-region, the relationship between its feature value and the feature values ​​of the surrounding integer coordinate points is as follows:

[0104]

[0105] Among them, f i,j The aggregated feature value representing the i-th row and j-th sub-region;

[0106] y represents the y-axis offset of the candidate box on the feature map;

[0107] i is the row index of the sub-region;

[0108] j is the column index of the sub-region;

[0109] (m,n) represents the offset of the sampling point within the sub-region;

[0110] Let be the eigenvalue of the feature map at (m,n).

[0111] (4) Sub-region feature aggregation and fixed-size output.

[0112] SAM2 Cache Update: SAM2 is a model with efficient instance segmentation capabilities. The core goal of its caching mechanism is to reduce the repeated calculation of pedestrian instance segmentation in the video stream, improve processing efficiency, and ensure the effectiveness of cached content through importance priority plus time decay.

[0113] The purpose of this caching mechanism is to filter and cache the segmentation information of high-value frames based on the fixed-size features of the candidate bounding boxes and the corresponding preliminary segmentation results output in step 1.2. (Segmentation information refers to the complete data set generated by SAM2 with the fixed-size features of the candidate bounding boxes as input, which can support the reuse of subsequent frames. Specifically, it includes three types of content: 1. Core segmentation results: pedestrian instance segmentation mask, segmentation region boundary coordinates; 2. Segmentation association identifier: pedestrian instance unique ID, frame sequence number / timestamp t; 3. Segmentation quality evaluation data: segmentation region area A) t IoU with the previous frame t In subsequent frame processing, cached results are prioritized, and recalculation is only performed on frames that are not covered by the cache or have insufficient scores. This significantly reduces the computational load of video stream processing and improves real-time performance while ensuring segmentation accuracy. The specific process is as follows:

[0114] (1) Obtain relevant data for current frame segmentation:

[0115] From step 1.2, obtain: fixed-size features of all pedestrian candidate boxes in the current frame (fixed-size features extracted separately for each candidate box to support subsequent segmentation calculations); and the preliminary segmentation mask corresponding to each candidate box (a pixel-level binary mask generated by SAM2 using the fixed-size features of a single candidate box as input, combined with the floating-point coordinates of the candidate box in the original image, covering only the pedestrian region within the candidate box; the size is consistent with the candidate box, 1 represents pedestrian pixels, and 0 represents background pixels within the candidate box; the generation process is: feature input adaptation → SAM2 Encoder encoding → Decoder mask generation → removing invalid results with a pedestrian pixel ratio <30%, used for subsequent IoU). t and A t (Calculation); IoU value between the current frame segmentation result and the same type of pedestrian in the previous frame. t(IoU t The closer the value is to 1, the higher the consistency of the segmentation between the two frames. The determination of "same type of pedestrians" requires multi-dimensional matching: ① Position continuity matching: calculate the position IoU between the boundary coordinates of the segmented region in the previous frame and the candidate box coordinates in the current frame. If it is >0.5, it is initially determined to be the same potential pedestrian; ② Appearance feature matching: extract the 128-dimensional appearance feature vector corresponding to the segmentation result of the previous frame and the appearance feature vector of the candidate box in the current frame, and calculate the cosine similarity. If it is >0.85, it is further confirmed; ③ Comprehensive determination: prioritize the condition that "position IoU>0.5 and appearance similarity>0.8", or position IoU<0.5 but appearance similarity>0.88 and timestamp t≤5, which can determine that they are the same pedestrians, and then calculate the IoU value of the segmentation mask of the two frames.

[0116] Retrieve from the cache management unit: the maximum area A of the pedestrian segmentation region in the historical cache. max (That is, the maximum area of ​​the segmented region among all the segmentation results in the historical cache, A when the first frame is processed) max = Area of ​​the segmented region in the current frame); Timestamp t of the current frame (with the first frame as t=0, used for subsequent time decay calculations);

[0117] (2) Calculate the importance score S of the current frame buffer. t Substitute the obtained parameters into the importance scoring function to calculate the buffer value of the current frame:

[0118]

[0119] Among them, S t Assign an importance score to the current frame buffer; the higher the score, the greater the value of the current frame buffer.

[0120] α and β are trainable parameters: α controls the IoU. t The larger the α value, the more emphasis is placed on the consistency of the two-frame segmentation; β controls the contribution to the scoring. The larger the β value, the more important the large region segmentation results are in terms of contribution to the score.

[0121] IoU t The IoU value of the segmentation result;

[0122] A t The area of ​​the segmented region in the current frame;

[0123] A max This represents the maximum area of ​​the historical cache segment.

[0124] λ is the time decay coefficient;

[0125] t is the timestamp of the current frame;

[0126] e -λt This is the time decay term.

[0127] IoU Consistency Term (α·IoU t ): The higher the IoU t (the more consistent the segmentation of two frames), the higher the reference value of the current frame's segmentation result for subsequent frames, and the higher the score contributed by this term;

[0128] Segmentation Area Ratio Term A max The closer it is to 1 (the larger the segmentation area of the current frame, such as a large pedestrian nearby), the richer the feature information contained in this segmentation result, and the higher the score contributed by this term;

[0129] Time Decay Term ((1 - α - β)·e -λt ): The smaller t is (the closer the current frame is to the latest frame), the closer e -λt is to 1, and the higher the score contributed by this term; conversely, for frames that have not been updated for a long time (t large), the score of this term drops rapidly, reflecting the time decay logic;

[0130] The weighted sum of the three terms gives S t (α + β < 1, ensuring that the sum of the three weights is 1, and the score range is between 0 and 1);

[0131] (3) Determine whether to update the cache: If S t > 0.7 (exceeding the preset threshold): Determine that the segmentation result of the current frame is of high value, and store it (including complete segmentation information: preliminary segmentation mask, segmentation area boundary coordinates, pedestrian instance unique ID, frame number / timestamp t, A t ) in the cache management unit. If the current frame A t > A max , synchronously update A max to the current frame A t ; If S t ≤0.7, determine that the segmentation result of the current frame is of low value (such as poor segmentation consistency, too small area, or long frame time), do not update the cache, and do not preferentially reuse the result of this frame during subsequent frame processing;

[0132] (4) Clean up invalid cache: The cache management unit regularly checks the cached frames according to the time decay term: For frames with a relatively large cache time t (such as t > 30 frames) and S t dropping below 0.3, directly eliminate them (release cache space); For frames with medium t (such as 10 < t < 30) and S t between 0.3 and 0.7, reduce their cache weights (during subsequent frame processing, preferentially call cached results with higher weights, that is, preferentially reuse frames with higher S t and updated timestamps).

[0133] Step S200 Pedestrian Feature Extraction and Fusion: Based on candidate box features and pedestrian segmentation masks, continuous feature extraction is performed using lightweight Mamba to output continuous variation features of pedestrian appearance and gait. Local and global features are extracted using lightweight Mamba. LiteTransformer is used to perform multi-granular feature interaction and fusion of local and global features to output fused features that have both detailed and overall representations. Global features and multiple local features are differentiated based on the fused features. An attention mechanism is used to dynamically allocate the weights of local features and concatenate the global features to generate pedestrian features.

[0134] The specific implementation of pedestrian feature extraction and fusion includes the following steps:

[0135] (1) Perform structured pruning and quantization operations on Mamba to obtain lightweight Mamba;

[0136] (2) The candidate box features and pedestrian segmentation mask are concatenated into multimodal input features, and the multimodal input features are normalized. The normalized multimodal input features are used as input, and continuous features are extracted through lightweight Mamba to output the pedestrian appearance continuity feature vector F. cont ;

[0137] (3) Based on the pedestrian segmentation mask, the pedestrian region is divided into multiple local regions, namely head, upper body and lower body. Each local region is input into lightweight Mamba for feature extraction to obtain the local features corresponding to each local region. The local features corresponding to each local region are then concatenated into a complete local feature F. l ;

[0138] (4) Perform global average pooling on the candidate box features. After fusing the candidate box features after global average pooling with the global area and shape information of the segmentation mask, input the result into lightweight Mamba to obtain the global feature F. g And adjust F through 1×1 convolution. l or F g The number of channels;

[0139] (5) Using global features F g As input, it uses LiteTransformer to perceive local details and outputs optimized global features F. g_opt and F l Segmented input to a lightweight Mamba SSM layer, enhancing local details, outputting F. l_opt Based on the constructed interaction matrix, F g_opt and F l_opt Fusion, outputting multi-granularity fused features F fused ;

[0140] (6) From the fusion feature F fused Extracting global features F from the middle g and local features F of each part l i To determine the effectiveness of each local region, the global feature F is used. g Using the query and local features as the key, the weights of the local features corresponding to each local region are calculated. After weighted fusion of the local features, the global features are concatenated to generate the final pedestrian feature F. out .

[0141] This embodiment involves three steps: step 2.1 lightweight Mamba model, step 2.2 multi-granularity feature interaction, and step 2.3 on-demand fusion of multi-granularity features.

[0142] Step 2.1 Lightweight Mamba Model: The original Mamba model is an efficient sequence feature extraction model based on the Selective State Space Model (SSM), which is good at capturing long-range dependencies in data (such as appearance changes and gait trends of pedestrians in consecutive frames in a video). However, the original model has a large number of parameters and high computational cost, making it difficult to adapt to the real-time processing needs of edge devices (such as surveillance cameras and mobile terminals).

[0143] The core goal of lightweighting is to find a balance between reducing the number of parameters and computational cost and preserving the ability to capture continuous features of pedestrian appearance. This is achieved through two main techniques: structured pruning and quantization. Structured pruning removes redundant structures with low contribution (such as unimportant neural network channels and weight parameters) from the model, rather than pruning randomly, ensuring that the model structure is intact and can perform normal inference after pruning. Quantization converts the high-precision floating-point weights in the original model into low-precision values, reducing storage and computational complexity, while also reducing accuracy loss through quantization calibration.

[0144] The combination of the two can reduce the number of original Mamba parameters by 50% to 70% while maintaining the ability to capture pedestrian continuity features such as clothing texture and gait, providing efficient and high-quality feature input for subsequent multi-granularity feature interaction and dynamic weight fusion.

[0145] (1) Structured pruning: Important weights are selected and redundant parameters are removed using pruning formulas. The specific process is as follows:

[0146] Calculate the contribution index of the weight matrix: For the original Mamba weight matrix W, calculate the contribution index of each weight element W. i,j absolute value | W i,j | measures its contribution to feature extraction (the larger the absolute value, the higher the contribution); calculate the global maximum absolute value max(|W|) of the weight matrix as a reference benchmark for pruning.

[0147] Generate the pruned mask M: Generate a binary mask M according to the pruning formula, (I(·) is an indicator function, M is generated when the condition is met) i,j =1, M is not satisfied i,j =0):

[0148] M i,j =I(|W i,j |>τ·max(|W|))

[0149] Perform structured pruning: via element-wise multiplication:

[0150] Prune(W) = W⊙M

[0151] (⊙ represents the Hadamard product). Reset the redundant weights to zero to obtain the pruned weight matrix W. Pruned

[0152] (2) Quantization operation: Based on pruning, reduce the storage and computation costs of weights. The specific process is as follows:

[0153] Quantization Range Calibration: Input calibration dataset (samples containing pedestrian features), run the pruned Mamba model, and calculate the weight matrix W. Pruned The minimum and maximum activation values ​​of each layer are determined by the minimum value min_val and the maximum value max_val. A symmetric quantization strategy is used to determine the quantization mapping relationship: for example, mapping a 32-bit floating-point number [min_val, max_val] to an 8-bit integer [-127, 127].

[0154] Weighting and computational quantization: The pruned weight matrix W... Pruned Converting from 32-bit floating-point numbers to 8-bit integers reduces storage usage by 75%; adapting to model computation units: converting floating-point matrix multiplication in the SSM layer to integer matrix multiplication, and ensuring feature accuracy and reducing quantization errors during inference through quantization inversion (integer → floating-point).

[0155] (3) Lightweight Mamba Feature Extraction: Based on the lightweight model, the input features from the first step are processed, and a continuous feature vector is output:

[0156] Feature Input and Preprocessing: The candidate bounding box fixed-size features (output of step 1.2) and segmentation mask features (output of step 1.3) output from the first step are concatenated into multimodal input features; the input features are normalized (e.g., by subtracting the mean or dividing by the standard deviation) to adapt to the input requirements of Mamba.

[0157] Continuity feature extraction: Input features are processed through a lightweight Mamba SSM layer to capture the spatial continuity of clothing texture (such as the distribution of texture on a person's shirt within a frame) and the temporal continuity of gait (such as the movement trend of a person's legs in consecutive frames); the output is a pedestrian appearance continuity feature vector F.cont (512-dimensional), this vector contains long-range dependency features of pedestrians and is the core input for subsequent multi-granularity feature interaction (step 2.2).

[0158] Step 2.2 Multi-granularity feature interaction: In pedestrian feature extraction, multi-granularity refers to the hierarchical feature dimensions of local and global features. Local features focus on fine-grained details of pedestrians (such as backpack style, hairstyle, and shoe color), while global features focus on the overall representation of pedestrians (such as height, body type, gait, and overall clothing style). Both have limitations when used alone: ​​using only local features is prone to feature failure due to occlusion (such as an obscured backpack), while using only global features easily loses key distinguishing information (such as the difference in hairstyle between two pedestrians of similar body type).

[0159] The core objective of step 2.2 is to establish an interactive relationship between local and global features through the efficient LiteTransformer architecture, allowing features to simultaneously contain fine-grained details and overall representations. The specific implementation logic is as follows:

[0160] LiteTransformer: Processes global features to enhance their ability to perceive local details (e.g., associating global body shape features with knapsack location information). At the same time, it reduces computational complexity and memory usage by simplifying the self-attention calculation of Transformer (e.g., reducing the number of attention heads and feature dimensions). Feature interaction matrix construction: The global features processed by LiteTransformer are fused with the local features processed by Mamba. Then, LayerNorm (layer normalization) and Dropout (random deactivation) are used to optimize the feature distribution, finally obtaining multi-granularity fused features, which provides a more comprehensive feature foundation for subsequent dynamic weight fusion (step 2.3).

[0161] (1) Preprocessing of local and global features:

[0162] Local feature partitioning extraction: Based on the pedestrian instance segmentation mask in step 1.3 (pedestrian instance segmentation mask: refers to the mask obtained from the cached filtering in step 1.3). t The complete pedestrian pixel-level segmentation result, retaining the same size as the original image after >0.7), is generated by optimizing the initial mask using the SAM2 model, covering the complete region of a single pedestrian in the entire frame (accurately distinguishing pedestrians from background pixels). The pedestrian region is divided into three local regions: head (top to shoulder), upper body (shoulder to waist), and lower body (waist to feet). These are input into the lightweight Mamba (model from step 2.1) to extract 128-dimensional features for the head, 256-dimensional features for the upper body, and 128-dimensional features for the lower body. These features are then concatenated into a complete F-segmentation model. l (512 dimensions);

[0163] The relationship and difference between the pedestrian instance segmentation mask used in step 2.2 and the preliminary segmentation mask in step 1.3: Relationship: Both are generated by SAM2, and their core input is the fixed-size feature of the candidate box in step 1.2. They are also bound to the unique ID of the same pedestrian instance. The former is the result of the latter through "caching filtering + boundary expansion + pixel completion". Difference: The former's size is different from the original... Figure 1 High precision, complete coverage of pedestrian areas, and high accuracy (S t >0.7), used for precise partitioning and feature extraction; the latter is the same size as the candidate box, covers only a local area, and may contain low-precision results, used in step 1.3 to calculate IoU. t and A t .

[0164] Global Feature Refinement: The fixed-size candidate box features from step 1.2 are transformed into 512-dimensional vectors through global average pooling. After fusing with the global area and shape information of the segmentation mask from step 1.3, the vectors are input into lightweight Mamba to obtain the global feature F. g (512 dimensions);

[0165] Dimension alignment validation: Ensure F g (512 dimensions) and F l (512-dimensional) Consistent dimensions provide a foundation for subsequent feature addition (if dimensions do not match, adjust F through 1×1 convolution). l or F g (Number of channels).

[0166] (2) LiteTransformer processes global features F g By simplifying self-attention calculation, global features are associated with local details. The specific process is as follows:

[0167] F g The data is split into Q, K, and V dimensions (each 512 dimensions), and the self-attention weights are calculated.

[0168] The weights are multiplied by V and processed by an FFN network to output the optimized global feature F. g_opt (512-dimensional, enhancing the perception of local details).

[0169] (3) Mamba optimizes local features:

[0170] F l Segmented input is used to create a lightweight Mamba SSM layer, enhancing local details (such as backpack outline and hairstyle texture), and the output is F. l_opt (512 dimensions).

[0171] 4) Feature fusion and optimization:

[0172] Construct the interaction matrix H = F g_opt +Fl_opt After Dropout(0.1) to prevent overfitting and LayerNorm normalization, the multi-granularity fused feature F is output. fused (512 dimensions).

[0173] Step 2.3 On-demand fusion of multi-granular features: Based on the effectiveness of key point detection in identifying pedestrian body regions (e.g., if the head is occluded, the weight of the head local features is low; if the upper body is clear, the weight of the upper body is high), dynamic weights are assigned to each local feature through an attention mechanism, and then spliced ​​and fused with global features to finally output high-quality features that are occluded and adaptable to changes in viewing angle, which can be directly used for cross-camera tracking.

[0174] (1) Feature segmentation and key point-assisted judgment:

[0175] From step 2.2 F fused Extracting global features F from the middle g (512 dimensions) and local features (F l1 128 dimensions, F l2 256 dimensions, F l3 128 dimensions);

[0176] The key point detection module outputs the validity of each region (e.g., missing head key points → low validity of the head region, complete upper body key points → high validity of the upper body), providing a reference for weight allocation.

[0177] (2) Attention weight calculation (dynamic weight allocation):

[0178] Generate query vector Q: Q = F g W q (W q Given a 512×512 learnable matrix, let F g Mapped to a query vector (dimension 512);

[0179] Generate key vector K i : (W k For the learnable matrix of the corresponding dimension, such as F l1 Using a 128×512 matrix, each local feature is mapped to a key vector (each with a dimension of 512);

[0180] Calculate the weights: (d k =64, scaling factor, to avoid gradient vanishing; w i The region w represents the region with high effectiveness, where the weights of the three local features sum to 1. i big).

[0181] (3) Dynamic fusion and splicing:

[0182] Local feature weighting: (head w1F) l1 + Upper body w2F l2 +lower body w3F l3 (512 dimensions);

[0183] Concatenate with global features: (⊕ represents feature splicing, 512+512=1024 dimensions).

[0184] Step S300: Cross-border head tracking: Pedestrian detection is performed on the input multi-camera video frames using the optimized EfficientDet-D4 model. The BiFPN++ module is used to enhance multi-scale feature fusion, outputting a detection box image including coordinates, confidence score, and detection box ID. The detection box image is then compared with candidate boxes for complementary verification. The detection box image is then aligned to the pose using HRNet keypoint detection and perspective transformation. Using the pose-aligned detection box image as input, basic features are extracted using the ResNet-50 model. These basic features are then fused with pedestrian features to generate ReID features. The ReID features are bound to the detection box ID and stored in a cache. The cosine similarity of the ReID features and the IoU overlap of the detection boxes are calculated. The consistency score between the predicted speed and the actual speed is calculated using LSTM. The cosine similarity of the ReID features, the IoU overlap of the detection boxes, and the consistency score between the predicted speed and the actual speed are weighted and fused to obtain a comprehensive similarity score. A threshold is then used to determine whether the pedestrian is the same person.

[0185] The specific implementation of cross-border head tracking includes the following steps:

[0186] (1) For the EfficientDet-D4 model, the optimal structural parameters are searched by Bayesian optimization, and the BiFPN++ module is introduced into the EfficientDet-D4 model to obtain the optimized EfficientDet-D4 model. The BiFPN++ module performs bidirectional fusion of multiple hierarchical feature maps P3, P4, P5, P6 and P7 output by the Backbone in the EfficientDet-D4 model from top to bottom and bottom to top, and dynamically adjusts the fusion contribution through learnable weight coefficients.

[0187] (2) Preprocess the multi-lens video frames of the input optimized EfficientDet-D4 model, and perform normalization, size adaptation and channel rearrangement operations through preprocessing.

[0188] (3) Input the preprocessed multi-lens video frames into the optimized EfficientDet-D4 model and output a detection box image including coordinates, confidence and detection box ID. The detection box ID is generated in the format of lens number, video frame number and detection box number.

[0189] (4) Detect and filter the detection boxes in the predicted output. The filtering rules include basic filtering and high occlusion protection. In basic filtering, the detection boxes are filtered based on confidence. In high occlusion protection, the detection box images are filtered by combining the mapping table between the segmented regions and candidate boxes in the cache. If the proportion of pedestrian pixels exceeds the predetermined value, the detection box images are retained.

[0190] (5) Using the detection box image as input, HRNet is used to detect pedestrian key points and calculate the perspective transformation matrix to correct non-frontal poses into standardized frontal images.

[0191] (6) Input the standardized frontal image into ResNet-50 to extract basic features, and train it jointly with three loss functions: ArcFace, Triplet, and Circle. After concatenating the basic features with the pedestrian features, compress them into a 512-dimensional final feature vector through 1×1 convolution as the ReID feature F. final The ReID feature is bound to the detection box ID and stored in the cache;

[0192] (7) Combine the detection box image with the ReID feature F final In addition to speed data pairing, the cosine similarity formula is used to calculate the ReID feature similarity between the two detection boxes in the candidate pair, calculate the IoU overlap of the detection boxes, output the predicted speed based on the pre-trained LSTM model, calculate the consistency score between the predicted speed and the actual speed, and weight and fuse the ReID feature cosine similarity, the IoU overlap of the detection boxes and the consistency score between the predicted speed and the actual speed to obtain the comprehensive similarity.

[0193] (8) Record pedestrian ID, detection box, ReID features, speed and matching status, support trajectory prediction and missing target location completion, and clean up and archive long-term unmatched trajectories.

[0194] This embodiment involves step 3.1 EfficientDet optimization, step 3.2 ReID feature enhancement, and step 3.3 multimodal matching.

[0195] Step 3.1 EfficientDet Optimization (Highly Robust Pedestrian Detection): EfficientDet is an efficient object detection model based on EfficientNet and FPN (Feature Pyramid). D4 is its medium-scale version (balancing accuracy and speed). The core of Step 3.1 is to improve the robustness of pedestrian detection under different camera angles by optimizing model parameters through Bayesian optimization and introducing the BiFPN++ cross-scale fusion module. In particular, it addresses the problems of missed detection of small targets (such as pedestrians in the distance) and low accuracy in pedestrian detection with high occlusion (occlusion rate above 50%), providing a foundation of accurate pedestrian detection bounding boxes for subsequent ReID feature extraction and trajectory association.

[0196] (1) Bayesian optimization of hyperparameters: Define the optimization objective function:

[0197] argmax (r,d,w) AP(r,d,w)-0.5·Latency(r,d,w)

[0198] Among them, AP (average precision) measures detection accuracy, Latency measures inference speed, and 0.5 is the speed weight (balancing accuracy and speed).

[0199] Hyperparameter search: By iteratively searching in the parameter space (r: 2-4 layers, d: 256-512 channels, w: 0.75-1.25 coefficients) using a Bayesian optimization algorithm (such as a Gaussian process model), the optimal parameter combination that maximizes the objective function is found (e.g., r=3, d=512, w=1.0). This parameter combination will be directly used for the subsequent inference configuration of the BiFPN++ module and the EfficientDet-D4 model, ensuring an AP improvement of more than 10% for small objectives.

[0200] (2) BiFPN++ enhances cross-scale feature fusion: Input features: Input the 5-level feature maps (P3-P7, corresponding to 1 / 8-1 / 128 downsampling) output by the Backbone of EfficientDet-D4 (such as EfficientNet-B4) into BiFPN++, where P3 corresponds to the detail features of small objects (pedestrians in the distance) and P7 corresponds to the semantic features of large objects (pedestrians in the foreground);

[0201] Bidirectional fusion logic: Top-down fusion: High-level features (P7, semantically strong) are upsampled to the same size as P6 through bilinear interpolation, and then added element-wise to P6 after channel alignment (fusion of high-level semantics and mid-level features) to obtain the fused feature map P6'; P6' is further upsampled and fused with P5 to obtain P5', which is then passed to P3 to generate P3' (enhancing the detailed features of small targets); Bottom-up fusion: The fused P3' is downsampled to the same size as P4 through a convolution with a stride of 2, and fused with P4 to obtain P4'; P4' is further downsampled and fused with P5' to obtain P5', which is then passed to P7 to generate P7 (enhancing the combined features of large target semantics and details); Weighted feature fusion: For the input features of each fusion node (such as the fusion node of P6 and P7, the fusion node of P3' and P4), the weights are dynamically calculated using the Sigmoid function, formula:

[0202]

[0203] Among them, s i The feature importance score (the model learns through 1×1 convolution + ReLU, ranging from -5 to 5); e: natural constant; n is the total number of input features (e.g., n = 2 when P6 and P7 are fused); w is the P3 feature of the highly occluded target. i Improves performance by 20%-30%, enhances edge details in occluded areas, and ensures a 15% increase in AP for highly occluded targets.

[0204] (3) Pedestrian detection and screening:

[0205] Detection box ID generation logic: For each frame output detection box, generate a unique ID in the format of "shot number-frame number-detection box number" (e.g., the 3rd detection box in frame 100 of shot A has the ID "A-100-003") to ensure that each detection box has a unique identifier.

[0206] Model Input Preprocessing: The core input of the EfficientDet-D4 model is the raw video frames captured in real time by multiple cameras (e.g., 1920×1080 pixels, 3-channel RGB image). Preprocessing is required first: ① Normalization: Pixel values ​​are reduced from [0,255] to [0,1] by dividing by 255 to eliminate the influence of brightness differences; ② Size Adaptation: The LetterBoxresize strategy (a scaling strategy that keeps the aspect ratio of the original image unchanged) is adopted. When the image is adjusted to the standard input size of the model (e.g., 640×640), the shape distortion of targets such as pedestrians due to stretching / compression is avoided (e.g., a pedestrian originally 1.8m is stretched to 2.5m), thus ensuring the accuracy of subsequent detection box localization; ③ Channel Rearrangement: The RGB channel order is adjusted to the BGR format required by the model to complete the preprocessing and obtain a standard input image of 640×640.

[0207] The logic related to the aforementioned steps is as follows: the input original frame shares the same source data as the first step (multi-scale detection and segmentation) and the second step (feature extraction); the detection box in this step is complementary to the candidate box in the first step 1.2 (when the occlusion rate is >60%, the deviation rate of the detection box in this step is <5%), and it also serves as the benchmark for cropping the pedestrian region in step 3.2.

[0208] Detection and filtering execution: Based on the optimized EfficientDet-D4 model, pedestrian detection is performed on the preprocessed multi-camera frames, outputting the detection box (coordinates (x1, y1, x2, y2), corresponding to the original frame coordinate system) and confidence score (range 0-1, reflecting the probability that the detection box contains a pedestrian) for each pedestrian; Filtering rules: ① Basic filtering: retain detection boxes with confidence > 0.5 and remove false detections (such as misidentifying trees as pedestrians); ② High occlusion protection: for detection boxes with occlusion rate > 50% but confidence 0.4-0.5, the corresponding segmentation mask is queried in conjunction with the "segmentation region-detection box ID mapping table" output in step 1.3 (SAM2 cache update). If the pedestrian pixel ratio is > 40%, it is retained to ensure that the recall rate of high occlusion targets is improved to 85% (about 60% before optimization).

[0209] Step 3.2 ReID Feature Enhancement: The core of ReID (Person Re-identification) is to match the same pedestrian under different camera angles using pedestrian appearance features. Step 3.2 focuses on enhancing the discriminative power of ReID features through joint training with multiple loss functions and perspective transformation alignment. This addresses the feature mismatch caused by differences in camera viewpoints (e.g., front → side) and changes in pedestrian pose, making the features of the same pedestrian more compact and the features of different pedestrians more distinct.

[0210] (1) Detection box perspective transformation alignment:

[0211] Pre-trained HRNet keypoint detection:

[0212] Model foundation: HRNet maintains high-resolution features through multi-resolution branches (128×64, 64×32, 32×16), and is pre-trained on the COCOKeypoints and MPII datasets, with AP>0.85;

[0213] Fine-tuning and adaptation: Fine-tuning was performed using 10,000+ samples from the detection box cropping in step 3.1. The bottom 80% of the network was frozen, and the top keypoint prediction layer was trained. After 10 rounds, the AP of 10 keypoints (head: left and right ears; upper body: shoulders and waist; lower body: hips and ankles) reached 0.92, and the localization error was <2 pixels.

[0214] Detection execution: Using the detection box in step 3.1 as the boundary, input the key point coordinates output by HRNet and filter out key points with confidence scores <0.7.

[0215] Perspective transformation matrix calculation: Based on the key points of a frontal upright pedestrian (horizontal spacing of 30 pixels for the head and vertical spacing of 80 pixels for the upper body), the 3×3 matrix M is solved by the least squares method, with a correction rate of >95%.

[0216] Pose alignment: The perspective transformation matrix M is applied to the original pedestrian image within the detection box. Non-standard pose images such as side view, tilt, and bending are mapped to a standardized frontal upright image (256×128 pixels, pixel values ​​normalized to [0,1], image center coincides with pedestrian torso center) through bilinear interpolation algorithm, avoiding feature extraction deviation caused by pose differences (after alignment, the cross-camera feature similarity of the same pedestrian is improved from 0.4-0.5 to 0.85-0.95).

[0217] (2) ReID feature extraction and training with multiple loss functions:

[0218] Feature extraction: The frontal upright normalized image output from step 1) is input into the ResNet-50 backbone network (the last fully connected layer of the original network is removed, and the output of the Conv5 layer is retained). The Conv5 layer outputs a 7×4×2048 dimension feature map, which is compressed into a 2048-dimensional original feature vector by global average pooling (GAP); then optimized by a 1×1 convolutional layer (output channel number = 512) and a BatchNorm layer, and finally normalized by L2 (magnitude = 1) to obtain a 512-dimensional ReID feature vector;

[0219] Multiple loss function calculation (training phase):

[0220] ArcFaceLoss The inter-class separation is enhanced by the angular boundary m, and the calculation formula is as follows:

[0221]

[0222] Where, θ yi It is the angle between the current feature and the center of the target class, s = 64 (feature scaling factor), m = 0.5 (angle boundary);

[0223] TripletLoss The formulas for minimizing intra-class distance and maximizing inter-class distance are:

[0224]

[0225] Where, d anchor,positive Let d be the Euclidean distance between the anchor point and the positive sample. anchor,negative The Euclidean distance between the anchor point and the negative sample is α = 0.3 (marginal value);

[0226] CircleLoss The formulas for optimizing intra-class clustering and inter-class separation are as follows:

[0227]

[0228] Where N is the batch sample size; logit pos The similarity score for positive sample pairs; logit neg Δpos is the similarity score for negative sample pairs; Δpos is the target similarity threshold for positive sample pairs; Δneg is the target similarity threshold for negative sample pairs; γ is the scaling factor.

[0229] Total loss function:

[0230]

[0231] Among them, 0.5 and 0.3 are loss weights, which have been experimentally verified to improve feature matching accuracy by 18%;

[0232] Model optimization: The Adam optimizer was used, and the model was trained iteratively for 100 rounds until the validation set loss was <0.15. The parameters were then frozen for real-time inference.

[0233] (3) Real-time ReID feature output:

[0234] Multimodal feature fusion: The 1024-dimensional multi-granularity fused feature F output from the second step is fused. out The feature F is concatenated with the 512-dimensional basic ReID features extracted by ResNet-50 and compressed to 512 dimensions through a 1×1 convolution to obtain the final feature F. final (Including gait continuity and local details, scene similarity is improved by 15%-20% with changes in viewpoint);

[0235] Feature binding: Generating F final At the same time, store the "detection box ID-F" synchronously. final "The key-value pair mapping table is synchronized to step 3.3 in real time through Redis caching to resolve the gaps in the association between the detection box and the feature;

[0236] Feature Output: Multi-lens normalized image input model, outputting a 512-dimensional F-axis. final (Eigenvalue ∈ [-1, 1], modulus = 1), cross-camera similarity of the same pedestrian > 0.8.

[0237] Step 3.3, Multimodal Matching (Cross-Shot Target Association): The core challenge of cross-shot target association is that single-modal information is easily affected by scene interference: relying solely on ReID appearance features will cause matching failure due to pedestrians changing clothes or occlusion; relying solely on IoU position matching cannot cover non-overlapping areas of the camera's field of view; relying solely on motion trajectory will cause trajectory breakage due to pedestrians changing direction or stopping.

[0238] The solution in step 3.3 is to integrate three modalities: appearance (ReID), location (IoU), and motion (LSTM trajectory). By calculating the comprehensive similarity through weighted average, the advantages of each modality are complemented: ReID features ensure the appearance matching accuracy of non-overlapping areas, IoU supplements the location verification of overlapping areas, and LSTM trajectory solves the trajectory continuity problem when switching shots. Finally, a unique ID is assigned to each pedestrian to achieve continuous tracking across shots.

[0239] (1) Data preprocessing and single-modal score calculation:

[0240] Data pairing:

[0241] Basic association of detection boxes: The detection boxes output in step 3.1 (including ID, coordinates, and confidence score) are bound to the F set in step 3.2. final Speed ​​data (speed calculation: system frame rate 25 frames / second, time difference Δt = 0.04 seconds), (Unit: pixels per second)

[0242] Pairing rules: When lenses overlap, they are paired according to the center distance of the detection frame < 50 pixels; when they do not overlap, they are paired according to the time sequence; only candidate pairs with a frame number difference ≤ 5 are retained (to avoid mismatches caused by time discrepancies).

[0243] Pairing objective: Reduce computing power consumption by 60% and ensure that single-modal computing focuses on high-probability candidate pairs.

[0244] Single-modal computation (parallel): ReID similarity S ReID The ReID feature similarity between two detection boxes in a candidate pair is calculated using the cosine similarity formula, which is:

[0245]

[0246] Where f1 and f2 are the ReID feature vectors bound to the two detection boxes, and the result ranges from 0 to 1 (1 is completely similar and 0 is completely dissimilar). If the pedestrian occlusion rate corresponding to any detection box is >50%, the similarity result is multiplied by 0.5 (to weaken the influence of occlusion and blur features and avoid mismatches).

[0247] IoU score S IoU The intersection-union ratio (IUU) of the detection boxes is calculated only for candidate pairs of "overlapping shots". The formula is as follows:

[0248]

[0249] Box1 and Box2 are two detection boxes, with a result range of 0-1 (1 for complete overlap, 0 for no overlap); the candidate "non-overlapping shot" is due to no overlap in the field of view, S IoUSet it directly to 0, and it will not participate in location verification.

[0250] Motion Consistency S motion Motion state verification is implemented based on a pre-trained LSTM model. The specific process is as follows:

[0251] LSTM model configuration: Input the velocity sequence and normalized position sequence (5×4 feature matrix) from frame t-4 to t-1, predict the velocity change Δv in frame t. x Δv y Calculate the prediction speed v for frame t+1. x t+1=v x ,t+Δv x v y t+1=v y ,t+Δv y (To predict the velocity of frame t+1 from frame t, satisfying real-time requirements);

[0252] Consistency calculation: First, obtain the actual velocity (x) of the detection box in frame t+1. real ,y real Calculate the Euclidean distance between the predicted velocity and the actual velocity (using the coordinates of frame t and frame t+1).

[0253]

[0254] Substitute the Gaussian function to calculate the consistency score:

[0255]

[0256] σ is an empirical parameter, set to 2 (verified by over 100,000 motion samples, this value can make S...). motion The value is 1 when d=0 and drops to 0.5 when d=3; when the pedestrian is stationary (coordinate change < 5 pixels for 3 consecutive frames), S motion =0.8.

[0257] (2) Multimodal fusion and ID association:

[0258] Overall similarity calculation:

[0259] S total =0.5S ReID +0.3S IoU +0.2S motion

[0260] Weight setting basis: 5-fold cross-validation (Market-1501, DukeMTMC-reID datasets) shows that this combination achieves a matching accuracy of 92.3% and a false match rate of 4.8% (the problem of similarity exceeding the range caused by the original weight and 1.5 is solved).

[0261] ID allocation and conflict resolution:

[0262] Basic allocation logic: If S total If the value is greater than 0.6, it is determined to be the same pedestrian, and the existing ID is used; if there is no ID, a new unique ID is assigned according to the format "Ped_XXX" (such as "Ped_001").

[0263] One-to-many conflict handling: When a single detection box is matched with multiple detection boxes S total When the value is greater than 0.6, select S. total The highest pairing (such as S for A1 and B1) total =0.85 (preferred correlation);

[0264] Exception handling mechanism: If S total If the value is ≤0.6, the person is determined to be a different pedestrian. If there is no ID detection box, a new ID is assigned. If there is an existing ID detection box, the original ID is maintained.

[0265] (3) Trajectory Maintenance:

[0266] Tracking records are stored according to the sequence "Pedestrian ID → Camera Number → Frame Number → Core Data". Core data includes the detection box coordinates and F-axis coordinates. final Speed, scores for each modality, matching status (success / prediction), and support for trajectory backtracking and model optimization;

[0267] Prediction and Cleanup:

[0268] Position prediction: When the current camera does not detect a target, LSTM is invoked to predict the position based on historical trajectories (format: (x pred ,y pred (, confidence level)), synchronize to the next shot; the distance between the predicted position and the center of the detection box is <30 pixels and S total Prioritize association when the distance is >0.6, but for distances <30 pixels, S total Mark "Pending Confirmation" when the value is ≤0.6; Trajectory cleanup rules: If the ID has no associated detection box for 250 consecutive frames (10 seconds), mark the trajectory as "End", archive the data (retain for 30 days) and release resources to avoid memory occupation.

[0269] This embodiment proposes a pedestrian tracking scheme that combines multi-scale detection, adaptive feature enhancement, and cross-camera correlation. It improves the detection rate of occluded targets by using SwinTransformer-FPN and a dynamic anchor box mechanism; employs a SAM2 caching strategy to reduce the computational overhead of video segmentation; designs a multi-granularity feature fusion module based on Mamba and LiteTransformer, and combines ArcFaceLoss to enhance ReID discriminativity; finally, it integrates detection boxes, appearance features, and motion trajectories through a multi-modal matching strategy.

[0270] Example 2:

[0271] The present invention discloses a pedestrian cross-border head tracking system based on multimodal dynamic feature fusion, comprising a multi-scale pedestrian detection and instance segmentation module, a pedestrian feature extraction and fusion module, and a cross-border head tracking module, which is used to realize pedestrian cross-border head tracking through the method disclosed in Example 1.

[0272] The present invention has provided a detailed description of the pedestrian cross-border head tracking method and system based on multimodal dynamic feature fusion. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for pedestrian cross-border head tracking based on multimodal dynamic feature fusion, characterized in that, Includes the following steps: Multi-scale pedestrian detection and instance segmentation: Multi-scale feature extraction is performed on the input video image using the SwinTransformer network structure. The extracted feature maps are dynamically fused based on the improved FPN network structure with dynamic weights. Adaptive anchor boxes are generated on the generated fused feature map. RoIAlign is used to extract features from candidate boxes and output a fixed-size feature map as candidate box features. The candidate boxes are then segmented using the SAM2 model to generate pedestrian segmentation masks. The importance score of the cache is calculated based on IoU, area, and time decay, and the cache content is dynamically updated or cleared. Pedestrian Feature Extraction and Fusion: Based on candidate box features and pedestrian segmentation masks, continuous feature extraction is performed using lightweight Mamba to output continuous variation features of pedestrian appearance and gait. Local and global features are extracted using lightweight Mamba, and multi-granular feature interaction and fusion are performed using LiteTransformer to output fused features that combine details and overall representation. Global features and multiple local features are differentiated based on the fused features. An attention mechanism is used to dynamically allocate the weights of local features and concatenate the global features to generate pedestrian features. Cross-border head tracking: Pedestrian detection is performed on input multi-camera video frames using an optimized EfficientDet-D4 model. The BiFPN++ module enhances multi-scale feature fusion, outputting detection box images including coordinates, confidence scores, and detection box IDs. The detection box images are then compared with candidate boxes for complementary verification. HRNet keypoint detection and perspective transformation are used to align the pose of the detection box images. Using the pose-aligned detection box images as input, a ResNet-50 model extracts basic features, which are then fused with pedestrian features to generate ReID features. These ReID features are bound to the detection box IDs and stored in a cache. The cosine similarity of the ReID features and the IoU overlap of the detection boxes are calculated. The consistency score between the predicted speed and the actual speed is calculated using LSTM. The cosine similarity of the ReID features, the IoU overlap of the detection boxes, and the consistency score between the predicted and actual speeds are weighted and fused to obtain a comprehensive similarity score. A threshold is then used to determine whether the pedestrians are the same person.

2. The pedestrian cross-border head tracking method based on multimodal dynamic feature fusion according to claim 1, characterized in that, Multi-scale pedestrian detection and instance segmentation includes the following steps: Multi-scale feature extraction: The input video image is divided into image blocks through the SwinTransformer network structure, and multi-scale feature extraction is performed through linear embedding layer and hierarchical Transformer module, outputting feature maps with different downsampling ratios and semantic levels, namely F3, F4 and F5; Dynamic feature fusion: Based on the improved FPN network structure, feature maps F3, F4, and F5 are fused in a top-down and bottom-up manner. During the fusion process, the fusion weights are dynamically adjusted according to the pedestrian scale distribution of the current frame to generate fused feature maps P3, P4, and P5. Adaptive anchor box generation: Assign a hierarchical index to each fused feature map, calculate a dynamic adjustment coefficient based on the difference between the pedestrian size in the current frame and the average size of the training set, and generate anchor boxes with multiple aspect ratios; Feature pooling: Combine RoIAlign to extract features from candidate boxes. During feature extraction, the candidate boxes are mapped to the corresponding feature maps, floating-point coordinates are retained, the RoI region is divided into sub-regions, floating-point coordinates are sampled in each sub-region, bilinear interpolation is used to calculate the feature values ​​of the sampled points, and they are aggregated into a fixed-size feature map as the candidate box feature map. SAM2 Cache Update: Taking the candidate bounding box feature map as input, the SAM2 model outputs a pedestrian instance segmentation mask, and calculates the cache importance score by combining the IoU between the current frame and the previous frame, the segmentation region area and the timestamp. The cache content is dynamically updated or cleared based on the cache management mechanism.

3. The pedestrian cross-border head tracking method based on multimodal dynamic feature fusion according to claim 2, characterized in that, The adaptive anchor box generation includes the following steps: Determine the hierarchical index: Assign a hierarchical index to each fused feature map; Calculate the dynamic adjustment coefficient: Based on the difference between the pedestrian size in the current frame and the average size of the training set, calculate the dynamic adjustment coefficient. Where the numerator is the sum of the areas of the first N pedestrian boxes, and the denominator is the product of the average area of ​​the training set and N; Calculate the anchor frame dimensions: For each level k, calculate the reference area of ​​the anchor frame for that level using the following formula: Among them, S k The anchor frame size (area) for the k-th FPN level; S base As the benchmark scale; γ is the scaling factor; w i h i The width and height of the first N pedestrian bounding boxes detected in the current frame; The total area of ​​the first N pedestrian boxes; The product of the average pedestrian area in the N·avg_area training set and N; Generate multi-aspect ratio anchor boxes: based on S for each level k Generate three commonly used aspect ratio anchor frames to cover pedestrians of different shapes at this level; Multi-scale object detection: Multi-scale hierarchical feature extraction is performed on the input image through the SwinTransformer network structure, and multiple feature maps are output. Based on the improved FPN structure with dynamic weights, the multiple feature maps are fused and multiple fused feature maps are output. Candidate box generation: For each fused feature map, an anchor box adapted to the corresponding scale of pedestrians is generated on the fused feature map to obtain adaptive candidate boxes of different scales. The candidate boxes are mapped to floating-point coordinates through RoIAlign, each candidate box is divided into multiple sub-regions, and bilinear interpolation sampling is performed. The sampled points are aggregated into features of a fixed size as candidate box features. For each candidate bounding box, the importance score of the current frame segmentation result is calculated based on IoU, region area and time decay, and high-value frames are selectively cached.

4. The pedestrian cross-border head tracking method based on multimodal dynamic feature fusion according to claim 1, characterized in that, During SAM2 cache updates, the features of all pedestrian candidate boxes in the current frame are obtained. Using these candidate box features as input, combined with the floating-point coordinates of the candidate boxes in the original image, a pixel-level binary mask generated by the ASM2 model that only covers the pedestrian region within the candidate box is used as the pedestrian segmentation mask. The IoU value (IoU) between the current frame segmentation result and the same type of pedestrian in the previous frame is then calculated. t Retrieve the maximum area A of the pedestrian segmentation region from the historical cache. max Given the current frame's timestamp t, calculate the cache importance score for the current frame using the following formula: Among them, S t Assign an importance score to the current frame buffer; the higher the score, the greater the value of the current frame buffer. α and β are trainable parameters: α controls the IoU. t The larger the α value, the more emphasis is placed on the consistency of the two-frame segmentation; β controls the contribution to the scoring. The larger the β value, the more important the large region segmentation results are in terms of contribution to the score. IoU t The IoU value of the segmentation result; A t The area of ​​the segmented region in the current frame; A max This represents the maximum area of ​​the historical cache segment. λ is the time decay coefficient; t is the timestamp of the current frame. e -λt For time decay term, α·IoU t For IoU consistency terms, IoU t The higher the score, the greater the reference value of the current frame segmentation result for subsequent frames, and the higher the score for this contribution. For the percentage of the segmented region, A max The closer to 1, the richer the feature information contained in the segmentation result, and the higher the score of this contribution; (1-α-β)·e -λt () represents the time decay term; the smaller t is, the more e -λt The closer to 1, the higher the score for this contribution; conversely, for frames that have not been updated for a long time, the score drops rapidly, reflecting the time decay logic. S is obtained by weighted summation of the consistency term, the segmentation region proportion term, and the time decay term. t If S t If the value exceeds the threshold, the current frame segmentation result is considered high-value, and this will include the initial segmentation mask, segmentation region boundary coordinates, pedestrian instance unique ID, frame number / timestamp t, and A. t The complete segmentation information is stored in the cache management unit. If S t If the value of the segmented frame is less than or equal to the threshold, the current frame is considered low-value, the cache is not updated, and the result of this frame is not reused first in subsequent frame processing. Furthermore, the cache management unit periodically checks the cached frames according to the time decay term: frames with a large cache time t are directly evicted; frames with a medium t and S... t Within a predetermined range, the cache weight of frames is reduced, and cached results with higher weights are prioritized for reuse, i.e., the S segmentation information is reused first. t Higher, timestamp-updated frames.

5. The pedestrian cross-head tracking method based on multimodal dynamic feature fusion according to claim 1, characterized in that, Pedestrian feature extraction and fusion includes the following steps: By performing structured pruning and quantization operations on Mamba, a lightweight Mamba is obtained. Candidate bounding box features and pedestrian segmentation masks are concatenated into multimodal input features. These multimodal input features are then normalized. The normalized multimodal input features are used as input, and continuous feature extraction is performed using lightweight Mamba. The output is a pedestrian appearance continuity feature vector F. cont ; Based on pedestrian segmentation masks, the pedestrian region is divided into multiple local regions: head, upper body, and lower body. Each local region is input into lightweight Mamba for feature extraction, resulting in local features for each region. These local features are then concatenated to form a complete local feature F. l ; Global average pooling is performed on the candidate bounding box features. The candidate bounding box features after global average pooling are then fused with the global area and shape information of the segmentation mask, and input into lightweight Mamba to obtain the global feature F. g And adjust F through 1×1 convolution. l or F g The number of channels; With global feature F g As input, it uses LiteTransformer to perceive local details and outputs optimized global features F. g_opt and F l Segmented input to a lightweight Mamba SSM layer, enhancing local details, outputting F. l_opt Based on the constructed interaction matrix, F g_opt and F l_opt Fusion, outputting multi-granularity fused features F fused ; From fusion feature F fused Extracting global features F from the middle g and local features F of each part l i To determine the effectiveness of each local region, the global feature F is used. g Using the query and local features as the key, the weights of the local features corresponding to each local region are calculated. After weighted fusion of the local features, the global features are concatenated to generate the final pedestrian feature F. out .

6. The pedestrian cross-border head tracking method based on multimodal dynamic feature fusion according to claim 1, characterized in that, Cross-border head tracking includes the following steps: For the EfficientDet-D4 model, the optimal structural parameters are searched through Bayesian optimization, and the BiFPN++ module is introduced into the EfficientDet-D4 model to obtain an optimized EfficientDet-D4 model. The BiFPN++ module performs bidirectional fusion of multiple hierarchical feature maps P3, P4, P5, P6 and P7 output by the Backbone in the EfficientDet-D4 model from top to bottom and bottom to top, and dynamically adjusts the fusion contribution through learnable weight coefficients. The multi-camera video frames of the input optimized EfficientDet-D4 model are preprocessed, and normalization, size adaptation and channel rearrangement are performed through the preprocessing. The preprocessed multi-camera video frames are input into the optimized EfficientDet-D4 model, and the output is a detection box image including coordinates, confidence score and detection box ID. The detection box ID is generated in the format of shot number, video frame number and detection box number. The predicted output detection boxes are detected and filtered. The filtering rules include basic filtering and high occlusion protection. In basic filtering, the detection boxes are filtered based on confidence. In high occlusion protection, the detection box images are filtered by combining the mapping table between the segmented regions and candidate boxes in the cache. If the proportion of pedestrian pixels exceeds a predetermined value, the detection box image is retained. Using the bounding box image as input, HRNet is used to detect pedestrian key points and calculate the perspective transformation matrix to correct non-frontal poses into standardized frontal images. A standardized frontal image is input into a ResNet-50 to extract basic features. These features are then jointly trained using ArcFace, Triplet, and Circle loss functions. The basic features are concatenated with pedestrian features and compressed into a 512-dimensional feature vector using a 1×1 convolution, which serves as the ReID feature F. final The ReID feature is bound to the detection box ID and stored in the cache; The detection box image is compared with the ReID feature F. final In addition to speed data pairing, the cosine similarity formula is used to calculate the ReID feature similarity between the two detection boxes in the candidate pair, calculate the IoU overlap of the detection boxes, output the predicted speed based on the pre-trained LSTM model, calculate the consistency score between the predicted speed and the actual speed, and weight and fuse the ReID feature cosine similarity, the IoU overlap of the detection boxes and the consistency score between the predicted speed and the actual speed to obtain the comprehensive similarity. It records pedestrian ID, detection box, ReID features, speed and matching status, supports trajectory prediction and missing target location completion, and cleans up and archives trajectories that have not been matched for a long time.

7. The pedestrian cross-border head tracking method based on multimodal dynamic feature fusion according to claim 6, characterized in that, The LSTM model is configured as follows: input the velocity sequence and normalized position sequence from frame t-4 to t-1, and measure the velocity change Δv in frame t. x and Δv y Calculate the prediction speed v for frame t+1. x t+1=v x ,t+Δv x v y t+1=v y ,t+Δv y ; Get the actual speed (x) of the detection box in frame t+1. real ,y real ), calculate the Euclidean distance between the predicted speed and the actual speed: Substitute the Gaussian function to calculate the consistency score: Where σ is an empirical parameter.

8. A pedestrian cross-border head tracking system based on multimodal dynamic feature fusion, characterized in that, It includes a multi-scale pedestrian detection and instance segmentation module, a pedestrian feature extraction and fusion module, and a cross-border head tracking module, which are used to realize cross-border head tracking of pedestrians through a multi-modal dynamic feature fusion-based pedestrian head tracking method as described in any one of claims 1-7.

Images