An AI vision-based target tracking method and system

Abstract

The application belongs to the technical field of image recognition, and provides a target tracking method and system based on AI vision, which comprises the following steps: original video frame acquisition; environment adaptive image enhancement; multi-target detection and multi-modal feature extraction; local optical flow motion estimation; multi-target trajectory association; occlusion processing and re-identification; trajectory output and result analysis; the method sets a physical-depth learning cascade defogging model, dynamically switches light / thick fog processing paths through dark channel mean, fuses atmospheric scattering physical priori and U-Net residual correction, has an environment adaptive perception architecture, breaks through the failure bottleneck of traditional single model under sudden fog concentration, constructs a trajectory cognition system coupled with appearance-motion-geometry, designs a dynamic cost matrix and a feature cache pool, solves the ID switching problem caused by similar target aggregation and long-time occlusion, and improves the accuracy of target tracking in an occlusion environment.

Description

Technical Field

[0001] This application relates to the field of image recognition technology, and in particular to a target tracking method and system based on AI vision. Background Technology

[0002] AI-based vision-based target tracking is a dynamic analysis method that combines deep learning, computer vision, and pattern recognition technologies. It aims to capture target objects in video or image sequences in real time and continuously track their position, behavior, and attributes across consecutive frames. This method is widely used in traffic management, intelligent security, and autonomous driving.

[0003] In urban traffic monitoring scenarios, the system needs to track the trajectories of multiple targets (such as pedestrians or vehicles) in real time. However, the following technical bottlenecks exist:

[0004] Environmental interference: Sudden changes in fog concentration (visibility < 50 meters) lead to image blurring and decreased contrast. In traditional target tracking methods, feature extraction is prone to failure, resulting in a high rate of missed detections.

[0005] Target occlusion: When a large truck occludes a small car or a vehicle briefly occludes a pedestrian, the target's appearance information is lost, resulting in a high re-identification error rate.

[0006] ID switching issues: When pedestrians with similar appearances (such as groups of students in school uniforms) are too close together, accurate identification is difficult, and traditional Hungarian algorithm tracking methods have a high matching error rate.

[0007] Therefore, it is necessary to propose a target tracking method and system based on AI vision. Summary of the Invention

[0008] The purpose of this application is to propose a target tracking method and system based on AI vision to solve at least one technical problem mentioned in the background art.

[0009] To address the aforementioned technical problems, this application provides an AI vision-based target tracking method, employing the following technical solution:

[0010] An AI vision-based target tracking method includes the following steps:

[0011] 100. Raw video acquisition: Acquires continuous RGB video streams through monitoring equipment, synchronously obtains thermal infrared image sequences, and outputs raw video frames and thermal infrared frames;

[0012] 200. Environmental Adaptive Image Enhancement: The original video frame is subjected to rain and fog intensity determination. According to the rain and fog intensity, when the rain and fog intensity is low, a physical model is used to remove fog. When the rain and fog intensity is high, the original video frame and thermal infrared frame are input into the dual-branch U-Net, the residual fog map and temperature feature map are fused, and the final enhanced image is output to provide high-definition input for subsequent detection.

[0013] 300, Multi-target detection and multi-modal feature extraction; Based on the final enhanced image, target detection and multi-modal feature extraction are performed. The improved YOLOv7-tiny is used to detect targets, generate detection boxes, extract the appearance features, motion features and geometric features of the image within the detection boxes, and output the multi-modal feature set and fused features after lightweight attention fusion, so as to achieve accurate target localization and feature description, and provide multi-dimensional features for trajectory association;

[0014] 400, corner detection and Lucas-Kanade sparse optical flow calculation are performed on the image within the detection box, and the set of corner motion vectors is output to estimate the local motion trend of the target and provide motion basis for trajectory prediction;

[0015] 500, Multi-target trajectory association; Generate historical trajectories for detected targets, predict target trajectories, fuse appearance similarity, motion consistency, position distance, thermal infrared features of the current frame detection box and thermal infrared features of historical trajectories, construct a dynamic cost matrix to match the current detection box and the predicted trajectory, output the matching matrix, and realize target ID allocation and trajectory update based on the matching matrix;

[0016] 600, Occlusion Handling and Re-identification: Based on the target detection and feature matching results, it is determined whether the target is occluded. Kalman correction and feature caching are performed on occluded targets. After the occlusion is removed, the target ID is associated through appearance-motion-infrared joint matching, and the corrected trajectory state is output to maintain the trajectory continuity of the occluded target.

[0017] 700, Trajectory Output and Result Analysis: Calculates ID switching rate and trajectory continuity for the final trajectory set, detects speed abrupt changes or stagnation events, and outputs quantitative indicators and alarm signals.

[0018] Preferably, in step 200, the dark channel mean is calculated for the original video frame to determine the rain and fog intensity, and the image is initially defogging based on the rain and fog degradation model. If the rain and fog intensity is high, the image is residually corrected to further eliminate residual fog effects.

[0019] Preferably, in step 300, the final enhanced image is downsampled by Backbone and enhanced through the SEBlock enhancement channel to strengthen target features, decode the detection box coordinates, and output a set of detection boxes after confidence filtering and non-maximum suppression to achieve target localization.

[0020] Preferably, in step 400, corner point motion vectors with an angle deviation greater than 45° are identified as outlier vectors, and outlier vectors are removed to reduce motion estimation error.

[0021] Preferably, in step 500, based on historical state positions, the Kalman algorithm predicts the target state, calculates the covariance through state transition and observation correction, and calculates the distance between the predicted position and the actual position based on the predicted state and the covariance to obtain the motion consistency S. m This is used to measure the consistency of motion between the current detection box and historical trajectories.

[0022] Preferably, in step 500, the similarity between the apparent features of the current frame detection box and the features stored in the historical trajectory is calculated to obtain the apparent similarity, and the Euclidean distance between the centroid of the current detection box and the trajectory prediction position is calculated to obtain the positional distance.

[0023] Preferably, in step 600, for partially occluded targets, the detection box is retained, and the position is corrected using Kalman filtering and optical flow vector to reduce positioning deviation;

[0024] For a completely occluded target, the trajectory position is maintained based on Kalman tracing, and the position is continuously recursively calculated using the state transition matrix.

[0025] Preferably, in step 600, the mean values ​​of appearance features and thermal infrared features are cached in a feature cache pool to resist appearance distortion caused by occlusion. After the occlusion is removed, the current detection box is judged to be the same target as the historical trajectory by four decision-making factors: appearance similarity, motion consistency, spatiotemporal constraints, and thermal infrared consistency.

[0026] To address the aforementioned technical problems, this application also provides an AI vision-based target tracking system, comprising:

[0027] The video acquisition module is used to acquire continuous video streams and obtain raw video frames I. t And thermal infrared frames, transmitted to the image enhancement module;

[0028] Image enhancement module: for the original video frame I t Rain and fog intensity are determined, and based on the rain and fog intensity, the original video frame I is analyzed using a joint physics-deep learning approach. t Perform dehazing processing and output the final enhanced image J. t ', Transmitted to the target detection module;

[0029] Target detection module; for the final enhanced image J t Perform object detection and generate a set of detection boxes {B}. t The data is then transmitted to the feature extraction module and the trajectory association module.

[0030] Feature extraction module: Extracts detection box B t Apparent features f of internal images app Motion characteristics I x I y Geometric features f geo Combined with thermal infrared features, lightweight attention fusion is used to output a multimodal feature set {f}. app I x I y f geo The fusion features are transmitted to the local optical flow motion estimation module and the trajectory association module;

[0031] Local optical flow motion estimation module; based on motion feature I x I y For detection box B t Corner detection and Lucas-Kanade sparse optical flow calculation are performed on the internal image, and the set of corner motion vectors is output. Transmitted to the trajectory association module;

[0032] Trajectory association module: Generates ID and historical trajectory T for detected targets. k Based on multimodal features and corner motion vectors and historical trajectory T k Kalman blotting predicts the target state and outputs a matching matrix X. ij According to X ij Perform trajectory matching and association; if matching fails, create a new ID and transmit it to the status judgment module, trajectory update module and occlusion handling module.

[0033] Status determination module: based on detection box B t and historical trajectory T k Determine whether the target is occluded and output the target status label (including visible, partially occluded, and completely occluded). If occlusion is determined, trigger the occlusion processing module.

[0034] Occlusion processing module: Performs Kalman correction and feature caching on occluded targets, generates a feature cache pool, and after the occlusion is removed, associates the target ID through appearance-motion joint matching, outputs the corrected trajectory status, outputs the recovered trajectory ID or new ID, and transmits it to the trajectory update module.

[0035] The trajectory update module updates the trajectory state based on the matching results from the occlusion handling module and outputs a set of trajectories {T} with IDs. k The data is then transmitted to the closed-loop optimization module.

[0036] Loop optimization module: Based on the trajectory set {T kThe system calculates the ID switching rate and trajectory continuity, outputs parameter tuning instructions, and transmits them to the trajectory target detection module and the association module to achieve closed-loop optimization.

[0037] Preferably, the image enhancement module performs dark channel mean calculation on the original video frames to determine the rain and fog intensity, performs preliminary defogging processing on the image based on the rain and fog degradation model, and performs residual correction on the image when the rain and fog intensity is high to further eliminate residual fog effects.

[0038] The trajectory association module calculates the similarity between the apparent features of the current frame detection box and the features stored in the historical trajectory, and obtains the apparent similarity S. a Calculate the Euclidean distance between the centroid of the current detection box and the predicted trajectory position to obtain the position distance D. ij Fusion of apparent similarity S a Motion consistency S m Location distance D ij The cost matrix C is dynamically constructed using the thermal infrared features of the current frame detection box and the thermal infrared features of historical trajectories. ij According to the dynamic cost matrix C ij and matching matrix X ij The Hungarian algorithm is used to find the optimal association and output the matching results between the detection box and the trajectory.

[0039] The state determination module determines whether the target is occluded based on target detection and feature matching. The determination indicators include the intersection-union ratio of the current detection box and the position box of the trajectory in the previous frame, the confidence of the detection box, and the feature matching result.

[0040] The beneficial effects of this invention are as follows:

[0041] This application provides an AI vision-based target tracking method, which sets up a cascaded physics-deep learning dehazing model. It dynamically switches between light and dense fog processing paths and infrared feature extraction by using dark channel mean. It integrates atmospheric scattering physics priors and U-Net residual correction, and has an environment-adaptive perception architecture, breaking through the failure bottleneck of traditional single models under sudden fog concentration changes.

[0042] This method sets up a multimodal resilient tracking mechanism, constructs a trajectory cognition system with appearance-motion-geometry ternary coupling, designs a dynamic cost matrix and feature cache pool, solves the ID switching problem caused by similar target clustering and long-term occlusion, and improves the accuracy of target tracking in occluded environments.

[0043] This method drives online parameter self-tuning through dual indicators of ID switching rate (DDR) and trajectory continuity (TC), enabling the algorithm to evolve autonomously in complex scenarios and complete the parameter optimization closed loop. It couples four layers of capabilities—environmental perception, multimodal cognition, occlusion resilience, and closed-loop evolution—into a closed loop, achieving robust tracking in multi-hazard environments (fog / occlusion / deformation) for the first time. Attached Figure Description

[0044] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is an overall flowchart of the target tracking method provided in the embodiments of this application;

[0046] Figure 2 This is an exemplary system architecture diagram to which this application can be applied;

[0047] Figure 3 This is a flowchart of ID and trajectory provided in the embodiments of this application. Detailed Implementation

[0048] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0049] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0050] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0051] A target tracking method based on AI vision, comprising steps 100-700:

[0052] Step 100, Raw Video Frame Acquisition: Acquire a continuous RGB video stream using monitoring equipment, simultaneously obtain a thermal infrared image sequence, and output the raw video frame I. t and thermal infrared frame Q t .

[0053] Original video frame I t ∈R H×W×3 (H×W resolution, 3-channel RGB), using video frame acquisition technology, dynamic visual information of targets such as pedestrians and vehicles in the scene is obtained. The video stream frame rate is 25-30fps, and each frame is an original RGB image. The thermal infrared image Q... t The dimensions are H×W, and the single-channel grayscale value represents the temperature, providing the raw visual data required for tracking in subsequent processing.

[0054] Step 200, Environment-Adaptive Image Enhancement: Enhance the original video frame I t Rain and fog intensity are determined, and based on the rain and fog intensity, the original video frame I is analyzed using a joint physics-deep learning approach. t Perform dehazing processing and fuse the residual fog map R t Temperature characteristic map F temp Output the final enhanced image J t ', to provide high-definition input for subsequent detection.

[0055] In another embodiment, in step 200, the dark channel mean is calculated on the original video frame to determine the rain and fog intensity. Based on the rain and fog degradation model, the image is initially defogging. When the rain and fog intensity is high, the image is residually corrected to further eliminate residual fog effects.

[0056] Based on an atmospheric scattering physics model, the brightness of the real scene after defogging is restored by dividing the "original image - atmospheric light" by the transmittance. The processed result is used as input for multi-target detection, solving the problem of decreased detection accuracy caused by rain and fog. The specific processing method is as follows:

[0057] 201. Perform dark channel analysis on the original video frames to determine the intensity of rain and fog.

[0058] For the input image I(x) (where x represents pixels), it is divided into multiple local windows Ω(x) of size w×w (e.g., 15×15). Within each local window Ω(x), the minimum value of the RGB three channels is taken, and the dark channel value J is calculated. dark (x):

[0059] J dark (x)=min c∈{r,g,b} (min y∈Ω(x) I c (y))

[0060] Among them, I c (y) represents the intensity value of pixel y in channel c (r represents the red channel, g represents the green channel, and b represents the blue channel). The minimum value is taken within a local window to obtain the dark channel image J. dark .

[0061] Calculate the dark channel mean of the entire image.

[0062]

[0063] Where N is the total number of pixels in the image. The dark channel mean is used to determine the intensity of rain and fog, and to determine the specific strategy for environment-adaptive image enhancement.

[0064] If the dark channel mean is ≤0.3 (experimental threshold), it indicates low rain / fog intensity.

[0065] If the mean value of the dark channel is greater than 0.3, it indicates that the intensity of rain and fog is high.

[0066] 202. Based on the physical model, the image is dehazed to restore a clear image outline.

[0067] In 2021, a rain and fog image degradation model was used to simulate image degradation caused by light scattering in the atmosphere. The rain and fog image degradation model is as follows:

[0068] I t (x)=J t (x)T t (x)+A(1-T t (x))

[0069] Among them, J t (x) represents the true intensity of the pixel after dehazing, I t (x) represents the pixel intensity of the original rain and fog image, A represents the atmospheric light intensity (global background light, calculated by traversing the image pixels, sorting, and then cropping the top 0.1%), and T represents the atmospheric light intensity (global background light, calculated by traversing the image pixels, sorting, and then cropping the top 0.1%). t (x) represents the transmittance (0≤T≤1, the smaller the value, the denser the fog);

[0070] 2022, transmittance T t (x) Calculations are performed using the prior that "the dark channel of a hazy image approaches 0":

[0071]

[0072] Where Ω(x) is a 15×15 local window around pixel x, and ω=0.95 is a coefficient for preserving fog information and preventing over-enhancement; It is the minimum value after normalization of pixels within the window, eliminating the influence of absolute atmospheric light and reflecting the local fog concentration.

[0073] In 2023, preliminary real images J were derived by back-engineering from the rain and fog image degradation model. t (x):

[0074]

[0075] Where, ∈=10 -6 To prevent extremely small values ​​where the denominator is 0 (fixed empirical values).

[0076] If the dark channel mean is ≤0.3, the rain / fog intensity is low, and this J t (x) is the value of the final enhanced image J t 'Avoid over-processing;'

[0077] If the dark channel mean is >0.3, the rain / fog intensity is high, which is beneficial to J. t (x) performs deep learning enhancement processing, J t (x) is input into the deep learning enhancement module to improve the dehazing effect through residual correction.

[0078] 203, Deep Learning Enhancement (U-Net Residual Correction) provides high-resolution input for detection. In dense fog, the input is the original video frame I. t With thermal infrared frame Q t to U-Net.

[0079] In 2031, pre-trained U-Net will be used to correct residual errors in the model. U-Net is a semantic segmentation model based on a convolutional neural network (CNN), consisting of a shrinking path (downsampling part) and an expanding path (upsampling part). The shrinking path is used to extract image features, while the expanding path is used to restore image resolution and combine the features from the shrinking path to perform accurate pixel-level predictions.

[0080] RESIDE is a dataset used to study image dehazing algorithms. It was constructed by researchers by collecting images from real-world scenes and generating synthetic foggy images using simulation algorithms. RESIDE contains foggy images of different concentrations and their corresponding clear images, and is divided into different subsets such as training and testing sets, providing data support for the training and evaluation of dehazing algorithms.

[0081] The residual error of the physical model was corrected using a U-Net model fine-tuned on the RESIDE dataset (containing images of fog at different concentrations), and the image J... t Using (x) as input, the residual fog map R is calculated through forward propagation of the U-Net model. t :

[0082] R t =F UNet (J t :θ)R t This represents information about the remaining fog in the image, where F UNet This is the fine-tuned U-Net model, where θ represents the pre-trained weights.

[0083] 2032, Process thermal infrared images and extract temperature feature maps Ftemp ;

[0084] Spatial temperature features were extracted using 4 layers of 3×3 convolutional kernels.

[0085] F temp =Conv3×3(ReLU(Conv3×3(T) t )))

[0086] Output 256-channel temperature feature map F temp .

[0087] 2033, by analyzing the dehazed image J t Subtracting residual fog map R t , Fusion temperature feature map F temp The final enhanced image J is obtained as output. t ':

[0088] J t '=J t -λR t +γConvF temp

[0089] Where λ = 0.7 is the adaptive attenuation factor. Through U-Net fine-tuning, residual haze is further eliminated, improving image detail clarity. γ is the thermal infrared weight coefficient, and the feature dimensions are aligned through convolutional layers.

[0090] Step 300: Multi-target detection and multi-modal feature extraction; based on the final enhanced image J t Perform object detection and multimodal feature extraction to generate detection bounding box B. t Extract detection box B t Apparent features f of internal images app Motion characteristics I x I y Geometric features f geo and thermal infrared features f thermal Lightweight attention fusion is performed to output a multimodal feature set {f}. app I x I y f geo f thermal} and fusion features f fuesd This enables precise positioning and feature description of the target, providing multi-dimensional features for trajectory association.

[0091] In yet another embodiment, in step 300, the final enhanced image J is processed by Backbone. t Downsampling and SEBlock enhancement channels are used to strengthen target features, decode the detection box coordinates, and after confidence filtering and non-maximum suppression, output the detection box set {B}.t}, in order to achieve the positioning target.

[0092] 301. The multi-target detection method is as follows:

[0093] Generate detection bounding box B based on YOLOv7-tiny detection. t Detection box B t It is the rectangular bounding box used to locate the target (such as a pedestrian, vehicle, etc.) in the t-th frame of the image, represented by coordinates B. t =(x min ,y min ,x max ,y max Each B t This represents a detected target bounding box. Detection box B t Generate through the following steps:

[0094] 3011, Backbone encoding: This will finalize the enhanced image J. t Encoding as intermediate feature F in .

[0095] YOLOv7-tiny is a lightweight object detection model whose detection head is responsible for predicting the bounding box parameters (t) of the object. x , t y , t w , t h The YOLOv7-tiny backbone network consists of convolutional layers and pooling layers, and adds an SEBlock (Squeeze-and-Excitation Block) after each convolutional layer to improve the detection capability of small objects.

[0096] J t The formula for mapping from pixel space to feature space is:

[0097] F in =Backbone(J t ')=Pool(Conv(J t '))

[0098] Obtain intermediate features F in The number of channels is C, and the size is H×W, where Conv is the convolution operation (extracting local features, such as edges and textures); Pool is the pooling operation (downsampling, compressing spatial dimensions, and retaining key features);

[0099] 3012, SEBlock channel enhancement: for F in Perform channel attention weighting and output detection features F out .

[0100] According to the SEBlock channel enhancement formula:

[0101] F out =σ(W2(σ(W1·GlobalPool(F) in )))⊙F in

[0102] Obtain the detection feature F out F out The detection features are channel-weighted and used for target localization; where σ is the Sigmoid activation function; W1 is a 1×1 convolutional layer (dimensionality reduced to C / r, reduction ratio r = 8); W2 is a 1×1 convolutional layer (dimensionality increased back to C); GlobalPool is global average pooling; F in ⊙ represents the intermediate feature map; ⊙ represents element-wise multiplication (channel weighting).

[0103] By adaptively adjusting channel weights, the feature response to small targets (such as pedestrians) is enhanced, reducing missed detections.

[0104] 3013, Anchor Frame Decoding:

[0105] b x =σ(t) x )+c x

[0106] b y =σ(t) y )+c y

[0107] b w =p w ·exp(t w )

[0108] b h =p h ·exp(t h )

[0109] Among them, t x , t y , t w , t h The bias and scaling factor predicted by the model (output from the regression head of YOLOv7-tiny); c x c y p represents grid coordinates; w p h σ represents the width and height of the anchor frame; σ is the activation function that limits the offset to the range of (0,1) to ensure that the center point is within the current grid.

[0110] After decoding, the center and width / height coordinates of the detection box are obtained (b x ,by ,b w ,b h Convert to bounding box coordinates:

[0111] x min =b x -b w / 2

[0112] x max =b x +b w / 2

[0113] y min =b y -b h / 2

[0114] y max =b y +b h / 2

[0115] Finally, the coordinates B of the detection box are obtained. t =(x min ,y min ,x max ,y max Each box includes a category label (e.g., pedestrians, vehicles), B t It can be used directly for subsequent tracking.

[0116] 3014, Confidence Filtering and Non-Maximum Suppression (NMS), output detection box set {B t}

[0117] The confidence score (cof) is the probability predicted by the model that a pedestrian exists within the current bounding box. YOLOv7-tiny outputs the raw confidence score, which is then converted into a probability value (conf) using the Sigmoid function, mapping the raw confidence score to [0,1].

[0118] conf = σ (original confidence score)

[0119] The higher the conf value, the greater the likelihood that there is a pedestrian inside the box.

[0120] Confidence filtering: Keep all detection boxes that satisfy conf≥0.5, and remove low confidence boxes (such as boxes with conf<0.5 are discarded).

[0121] NMS is used to remove overlapping redundant detection boxes and retain the most likely correct box. NMS is performed separately for each category (such as pedestrians and vehicles), and boxes from different categories do not inhibit each other.

[0122] NMS processes it as follows:

[0123] 30141, Sort the filtered detection boxes in descending order of conf;

[0124] 30142, select the box with the highest confidence level and add B. t Final result set;

[0125] 30143, calculate the intersection-union ratio (IoU) between this bounding box and the remaining bounding boxes:

[0126] IOU = Area of ​​intersection / Area of ​​union

[0127] Delete all boxes where IoU > 0.3;

[0128] 30144, Repeat the above steps until all boxes have been processed.

[0129] Through the above steps, from anchor box decoding to confidence filtering and NMS processing, an accurate and non-redundant set of detection boxes {B} is finally generated. t (Including categories) provides reliable input for subsequent tracking and re-identification.

[0130] 302. The multimodal feature extraction method is as follows:

[0131] For each detection box B t Image J within t Extract the following features: appearance features, motion features, and geometric features.

[0132] 3021, Apparent feature f app The extraction method is to extract the output of the global average pooling layer of ResNet50 (pre-trained on ImageNet), and then reduce the dimensionality to 512 dimensions through a 1×1 convolution:

[0133] f app =Conv1×1(GlobalPool(ResNet50(J t '(B t ))))

[0134] Where Conv1×1 is a 1×1 convolutional layer; GlobalPool is global average pooling; ResNet50 is a pre-trained ResNet50 model; J t '(B t ) is the detection box B t The image region within.

[0135] Output f app It is a 512-D vector that captures the appearance of pedestrians, such as clothing texture and body shape.

[0136] 3022, Motion Feature I x I yThe extraction method involves uniformly sampling four key points (corner points + center point) within the detection frame, and calculating the velocity vector using Lucas-Kanade optical flow, i.e., the directional gradient I in the x and y directions. x I y ;

[0137] I x =J t '(B t )*G x

[0138] I y =J t '(B t )*G y

[0139] Where * represents convolution; G x G y For the Sobel operator, calculate the gradients in the X and Y directions respectively:

[0140]

[0141] The output motion feature dimension is 8-D (4 points × 2 dimensions), reflecting the short-term motion trend of pedestrians.

[0142] 3023, geometric feature f geo The extraction method involves calculating the aspect ratio and centroid coordinates of the detection box.

[0143] f geo =(w / h,(x min +x max ) / 2,(y min +y max ) / 2)f geo Used to determine whether target shapes are consistent when associating trajectories. Here, w / h is the aspect ratio, which reflects the target shape; for example, w / h < 1 for pedestrians and w / h > 1 for vehicles, where w = x. max -x min h=y max -y min , representing the width and height of the detection box, respectively; centroid coordinates

[0144] c x =(x min +x max ) / 2, c y =(y min +y max ) / 2, z t It reflects the central location of the target.

[0145] Output geometric features f geoThe dimension is 3-D, describing the shape and position of pedestrians and distinguishing pedestrians from other targets. The centroid is used to assist in position matching.

[0146] 3024, thermal infrared signature f thermal :

[0147] For thermal infrared images, the feature map height H' and width W' are obtained through a convolutional downsampling process: Input thermal infrared image size H×W, after 4 layers of convolution, with stride=2 in each layer, we get H'=H / 16 and W'=W / 16.

[0148] The method for dynamically adjusting channel weights using the SENet structure is as follows:

[0149] 30241. Perform global average pooling on the feature map to obtain a channel statistics vector z with dimension c, where c is the number of channels. Global average pooling formula:

[0150]

[0151] Where H' is the height of the feature map, W' is the width of the feature map, i is the index in the height direction (from 1 to H'), j is the index in the width direction (from 1 to W'), c is the channel index, indicating that the c-th channel is currently being processed, and F (c) temp For the c-th channel of the temperature feature map (i.e., the feature map output from the previous convolution), F (c) temp (i,j) represents the feature value of the c-th channel at position (i,j), and z represents the compressed feature.

[0152] 30242 learns the correlation between channels through two fully connected layers and outputs a weight vector with the same number of input channels.

[0153] A fully connected layer is a type of neural network layer in deep learning, where each input neuron is connected to each output neuron. In the channel attention mechanism, two fully connected layers are used to generate channel weights.

[0154] The first fully connected layer (W1) reduces the input dimension from c to c / r (where r is the reduction ratio, taken as 16), and then passes it through the ReLU activation function. The first fully connected layer is derived from the design requirements of the channel attention mechanism, and is automatically learned through backpropagation.

[0155] The second fully connected layer (W2) restores the dimension from c / r to c, and then obtains the weight of each channel (between 0 and 1) through the Sigmoid activation function. The source of the first fully connected layer is the decompression requirement of the channel attention mechanism. It is jointly optimized with W1 to minimize the classification loss.

[0156] The first and second fully connected layers are two consecutive linear transformation layers in the channel attention module, used to learn the dependencies between channels. Fully connected layers and neural networks are existing deep learning techniques, so they will not be elaborated further.

[0157] Attention weight generation formula:

[0158] s=σ1(W2δ1(W1z))

[0159] Where s is the channel attention weight obtained after two fully connected layers and activation functions, σ1 is the Sigmoid activation function, which compresses the output to the (0,1) interval, W1 is the weight of the first fully connected layer, W2 is the weight of the second fully connected layer, which increases the output dimension of the first fully connected layer to the same number of channels as the original, and δ1 is the ReLU activation function.

[0160] Channel weighting:

[0161] F' temp =s⊙F temp

[0162] Among them, F' temp This is the weighted temperature distribution feature map, ⊙ represents channel-by-channel multiplication, F temp s represents the original temperature feature map, and s represents the channel attention weights.

[0163] By weighting attention through channels, the characteristics of high-temperature areas (such as human bodies / vehicle engines) can be amplified.

[0164] Global average pooling, according to the formula:

[0165]

[0166] Obtain the pooled feature vectors v1 and F'. temp (i,j) represents the temperature characteristic at position (i,j).

[0167] Dimensionality reduction using fully connected layers, according to the formula:

[0168] f thermal =Wv1

[0169] The final infrared feature f is obtained thermal , where W is the dimensionality reduction weight matrix with a size of 128×256, which is used as network parameters and is obtained through end-to-end training, and v1 is the pooling feature vector.

[0170] 30243, calculate the confidence level C of thermal infrared features. thermal ,formula:

[0171]

[0172] Calculate the temperature standard deviation for each pixel position k within the detection frame, and then calculate the average standard deviation of all pixels to obtain C. thermal C thermal Used to measure the temperature distinction between a target and the background, C thermal A larger value indicates a more significant temperature difference between the target and the background. Here, N1 is the total number of pixels within the current detection box. During optimization, if C... thermal <0.1, automatically increase the thermal infrared weight η.

[0173] Output multimodal feature set {f app I x I y f geo f thermal This data is used to provide the foundation for subsequent trajectory association and ID allocation.

[0174] 303, The lightweight attention fusion method is as follows:

[0175] f fuesd =α'·f app +β'·[I x ,I y ]+δ'·f geo +η·f thermal

[0176] Obtain the fusion feature f fuesd α' is the apparent weight, β' is the motion weight, δ' is the geometric weight, and η is the thermal infrared weight, focusing on the f of the occluded target. thermal Under normal lighting conditions, α' = 0.6, β' = 0.2, δ' = 0.1, η = 0.1; under foggy conditions, α' = 0.4, β' = 0.2, δ' = 0.1, η = 0.4; under complete darkness, α' = 0.1, β' = 0.1, δ' = 0.0, η = 0.8. fuesd In subsequent trajectory association, it is used to calculate the apparent similarity S. a It serves as a storage unit for the feature cache pool in re-identification and participates in the confidence calculation for weight tuning in closed-loop optimization.

[0177] Step 400, Local optical flow motion estimation; for detection box B t Corner detection and Lucas-Kanade sparse optical flow calculation are performed on the internal image, and the set of corner motion vectors is output. Estimate the local motion trend of the target to provide motion basis for trajectory prediction.

[0178] In another embodiment, in step 400, a corner motion vector with an angle deviation > 45° is identified as an outlier vector, and outliers are removed to reduce motion estimation error.

[0179] The method for estimating local optical flow motion is as follows:

[0180] 401, Corner Detection.

[0181] Within the detection box, the Shi-Tomasi algorithm is used to filter corner points based on the minimum eigenvalue of the Hessian matrix, locating key points with significant movement within the detection box, thus obtaining the corner point set {(x k ,y k The specific process is as follows:

[0182] 4011, Calculate the Hessian matrix within the detection box.

[0183] Within a local window W (e.g., 5×5) of the image within the detection box, gradients are aggregated to generate a Hessian matrix H:

[0184]

[0185] The trace tr(M) and determinant det(M) of H are respectively:

[0186] tr(M)=I x 2 +I y 2

[0187] det(M) = I x 2 I y 2 -(I x I y ) 2

[0188] 4012, Corner response calculation, determines whether a pixel is a corner.

[0189] Find the smallest eigenvalue λ of H as the corner response value R:

[0190] R = min(λ1, λ2)

[0191] The corner response value R determines which points will be selected as corner points. Here, λ1 and λ2 are the two smallest eigenvalues ​​of H, which are calculated by λ1+λ2=tr(M) and λ1λ2=det(M).

[0192] 4013, filter corner points.

[0193] A quality threshold of 0.01 is set using experiments or statistical methods. When R > 0.01, the point is identified as a corner point (a significant point of motion). Points with R > 0.01 are retained, and the set of corner points {(x k ,y k )}.

[0194] 402, Calculation of corner motion vectors.

[0195] First, the corner points detected by Shi-Tomasi are used as input, and then the motion vectors of these corner points between adjacent frames are calculated using Lucas-Kanade sparse optical flow. This allows for the estimation of the target's local motion. Sparse optical flow tracks only "meaningful" corner points, avoiding invalid calculations for flat regions (without grayscale changes), while Shi-Tomasi can efficiently filter out suitable corner points for tracking.

[0196] The method for calculating the corner motion vector is as follows:

[0197] 4021, Time gradient calculation.

[0198] The temporal gradient I is obtained by calculating the gray-level difference between images within the detection box of adjacent frames. t :

[0199] I t =J t '(B t )-J t-1 '(B t-1 )

[0200] Among them, J t-1 ' represents the detection box B in the previous frame. t-1 The image inside, with B t Alignment is crucial to avoid positional misalignment.

[0201] 4022, within the neighborhood W of the corner point, construct the optical flow equation:

[0202]

[0203] The corner motion vector is calculated. This reflects the displacement of corner points from frame t-1 to frame t, used to estimate the local motion trend of the target. Here, b represents the image gradient (Ii). x ,I y ) and time gradient (I t The sum of the products of ) reflects the direction and intensity of the brightness change:

[0204]

[0205] v x ,v y For pixel motion velocity vectors, This represents the velocity of a pixel in the x / y directions. Each corner point corresponds to one vector, and the corner point motion vector... It reflects the local motion trend of the target and is used to construct the target's motion characteristics.

[0206] Corner motion vector More sensitive than the center of mass velocity, it can capture local motion of the target (such as a pedestrian turning around), improving the accuracy of short-term trajectory prediction.

[0207] 4023, Exception Handling:

[0208] 40231, Outlier Removal: If the proportion of outlier vectors (vectors with a deviation of >45° from the historical direction) is >30%, use K-means clustering (number of clusters = 2) to remove outliers to avoid erroneous motion interfering with subsequent Kalman predictions;

[0209] 40232, Failure Switching: If there are less than 2 valid vectors, switch to subsequent Kalman predictions.

[0210] Finally, the set of corner motion vectors is output. And anomaly markers, corner motion vector set It reflects the direction and velocity of motion in the local area of ​​the target, providing motion characteristics for trajectory prediction and association.

[0211] Step 500: Multi-target trajectory association; generate historical trajectories T for detected targets. k Predict the target trajectory and construct a dynamic cost matrix C. ij Match the current detection box with the predicted trajectory and output the matching matrix X. ij According to the matching matrix X ij To achieve the goal of ID allocation and trajectory update.

[0212] In another embodiment, in step 500, the similarity between the apparent features of the current frame detection box and the features stored in the historical trajectory is calculated to obtain the apparent similarity S. a Calculate the Euclidean distance between the centroid of the current detection box and the predicted trajectory position to obtain the position distance D. ij Fusion of apparent similarity S a Motion consistency S m Location distance D ij The cost matrix C is dynamically constructed using the thermal infrared features of the current frame detection box and the thermal infrared features of historical trajectories. ij .

[0213] In multi-target tracking, multiple targets may be detected in the current frame (i.e., the current detection box), while the system has saved the historical trajectory T formed in the previous frame or earlier. k (Each trajectory corresponds to a continuously tracked target). Since the target may move, be occluded, or newly appear, it is necessary to determine whether the current detection box and the existing trajectory belong to the same target.

[0214] When a trajectory is detected for the first time and no existing trajectory is matched, the system assigns it a unique ID and initializes a new trajectory (i.e., the beginning of the "historical trajectory") starting from the current detection box.

[0215] ID (Identity) is a unique identifier for a target. It is used to associate the same target across frames and to distinguish different targets in a video sequence (such as different pedestrians or vehicles), ensuring the consistency of the identity of the same target during tracking.

[0216] When a detected target cannot be associated with any existing trajectory (existing ID), it is determined to be a new target, and a new unique ID is assigned to it.

[0217] Historical Trajectory T k It is the set of states of the target in previous frames. Historical Trajectory T k The generation steps are as follows:

[0218] 1. When the target is assigned a new ID (first successful association), initialize the historical trajectory T. k Record the state of the first frame;

[0219] 2. In each subsequent frame, if the target with that ID is successfully associated, add the current frame's state (such as bounding box, features) to T. k In the middle, update the trajectory;

[0220] 3. If the target is temporarily occluded (not detected but not exceeding the disappearance threshold), the trajectory T k The system predicts the location based on a motion model (such as Kalman filtering), stores it in the trajectory library, and associates it when it reappears.

[0221] The multi-target trajectory association method is as follows:

[0222] 501, trajectory prediction.

[0223] 5011, the state vector x of the target's motion state. t =[x,y,v x ,v y ] T Where position (x, y) and velocity (v) are... x ,v y ).

[0224] 5012, Based on the historical state position, initiate Kalman prediction to predict the next state of the target. formula:

[0225]

[0226] Where F is the state transition matrix, xt-1 The target state for frame t-1:

[0227]

[0228] Δt = 1 / 30s (frame interval), while simultaneously buffering historical appearance features. (first 5 frames f) app (mean), used for re-identification.

[0229] The predicted position of the target in frame t is used to compare with the actual position of the current detection box to determine whether they are the same target (the closer the predicted position is to the actual position, the higher the probability of association).

[0230] 5013, process noise covariance.

[0231] Based on the MOT17 pedestrian motion variance statistics process noise Q:

[0232] Q = diag([q1 2 ,q2 2 ,q3 2 ,q4 2 ])

[0233] Where q1, q2 = 0.1m, position noise; q3, q4 = 0.05m / s, velocity noise.

[0234] The covariance P at time t is calculated through state transition and observation correction. t :

[0235] P t =FP t-1 F T +Q

[0236] At the initial time (t=0), P0 is an artificially set initial covariance based on prior knowledge of the uncertainty of the initial state; for example, if the initial state is completely known, P0 is a zero matrix; if it is uncertain, the diagonal elements are set to larger variance values. t Used to reflect forecast uncertainty, ranging from (0, +∞), the smaller the range, the more consistent the forecast.

[0237] P t The uncertainty of state estimation can be described directly through numerical values:

[0238] Diagonal elements: represent the variance of a single state variable (such as position, velocity, etc.). The larger the variance, the greater the deviation between the estimated value and the true value of the variable, and the higher the uncertainty.

[0239] Off-diagonal elements: represent the covariance between different state variables, reflecting the correlation between variables (such as whether the estimation errors of position and velocity are correlated).

[0240] 502, Construct the dynamic cost matrix C ij .

[0241] Cost is a metric that measures the difference between two things. The smaller the difference, the lower the cost (the more likely they are the same target); the larger the difference, the higher the cost (the more likely they are different targets). Minimum cost matching is to find the set of matching relationships with the minimum total cost among all possible "box-trajectory" matching combinations, ensuring that each box is associated with the correct historical trajectory as much as possible.

[0242] Fusion appearance similarity S a Motion consistency S m Location distance D ij Calculate the cost matrix C ij .

[0243] Apparent similarity S a Measuring appearance consistency:

[0244]

[0245] S a Let S be the cosine similarity, with a range of [-1, 1]. a Values ​​greater than 0.8 are considered to be the same target. Where f (i) fused f is the fusion feature of the i-th detected target in the current frame; (j) fused The fusion features stored for historical trajectory j.

[0246] S a It measures the similarity between the appearance features of the current frame detection box and the features stored in the historical trajectory.

[0247] Motion Consistency S m The distance between the predicted location and the actual location is used to measure whether the motion pattern of the current detection box matches the motion trend of the historical trajectory.

[0248]

[0249] Among them, z t P represents the current position (centroid coordinates) of the detection box. t -1 To predict the covariance matrix P t The inverse matrix, H′ is the observation matrix, To predict the location of the detection box.

[0250] Spatial distance D ij The Euclidean distance (in pixels) between the centroid of the current detection box and the predicted trajectory position;

[0251] D ij =‖pos i -pred j ||2

[0252] Where, pos i Let be the centroid coordinates of the i-th current detection box. pred j The predicted centroid coordinates of the historical trajectory. ‖·‖2 represents the two-dimensional Euclidean distance (L2 norm).

[0253] Fusion dynamic weight output cost matrix C ij :

[0254] C ij =α(1-S a )+βS m +γD ij +ω‖f i thermal -f j thermal ||

[0255] Among them, S a To detect the apparent similarity between bounding box i and trajectory j; S m For motion consistency; f i thermal To detect the thermal infrared features of frame i, f j thermal Let be the thermal infrared features of trajectory j, and α, β, γ, and ω be weight coefficients with a sum of 1. The initial values ​​are α = 0.6, β = 0.3, and γ = 0.1, which are dynamically adjusted according to the degree of occlusion: when occlusion is > 40%, α = 0.4, β = 0.4, and γ = 0.2 to increase the weight of motion features; ω = 0.2 is the thermal infrared similarity weight.

[0256] 503, trajectory association.

[0257] C ij The matching cost between the i-th historical trajectory and the j-th current detection box (the smaller the value, the higher the matching degree) is used to solve the optimal association using the Hungarian algorithm (finding the matching pair with the minimum cost to achieve ID allocation).

[0258] Objective function:

[0259] min∑ i,j C ij X ij

[0260] Among them, X ij is a binary decision variable (0 or 1), indicating whether the detection box i matches the trajectory j.

[0261] Objective: Minimize the total matching cost ∑ i,j C ij X ij .

[0262] Constraints:

[0263] Each trajectory can be matched with at most one detection box: ∑ i X ij =1;

[0264] Each detection bounding box can match at most one trajectory: ∑ j X ij =1;

[0265] X ij ∈{0,1}, binary constraint.

[0266] By improving the Hungarian algorithm, the minimum cost matching is solved, and the matching relationship between the detection box and the historical trajectory is output.

[0267] X ij The solution process is as follows:

[0268] 5031, Cost Matrix Normalization:

[0269] Subtract the minimum value from each row: C ij =C ij -min k C ik ;

[0270] Subtract the minimum value from each column: C ij =C ij '-min k C kJ ';

[0271] Normalization amplifies cost differences, highlighting the optimal match.

[0272] 5032, Trial Assignment and Adjustment:

[0273] Cover all zero elements with the fewest possible horizontal / vertical lines;

[0274] If the number of lines is less than the matrix dimension, find the minimum uncovered value δ:

[0275] Subtract δ from uncovered elements;

[0276] Add δ at the intersection of lines;

[0277] Repeat until the number of lines equals the matrix dimension.

[0278] 5033, match confirmed:

[0279] Starting from a row / column containing only one zero, select X. ij =1;

[0280] Delete the row / column containing the zero, and recursively process the remaining matrix;

[0281] Output: Optimal matching matrix X ij .

[0282] The matching results are as follows:

[0283] X ij =1: This indicates a successful match. The current detection box i is associated with the existing trajectory j, which retains its ID. The trajectory status is updated (current position, motion vector, and features are added; the position is corrected using Kalman filtering, and the features are refreshed with the current value).

[0284] X ij =0 and no match: This indicates a matching failure. The current detection box i did not match any trajectory. A new ID is assigned to it (incrementing sequentially, such as 1, 2, 3...), and an initial historical trajectory T is created. k (Store the state of the first frame: position, features, motion vector);

[0285] X ij =0 and no match in column: This indicates a matching failure. Trajectory j did not match the detection box and is marked as occluded. Kalman prediction is enabled to maintain the trajectory. If there is no match for 3 consecutive frames, it is marked as disappeared (deleted or temporarily stored for 30 frames).

[0286] The output multi-target tracking results include the continuous trajectory of each target (including ID, timestamp, position, and velocity).

[0287] Step 600, Occlusion processing and re-identification: Based on the target detection and feature matching results, determine whether the target is occluded, perform Kalman correction and feature caching on the occluded target, and after the occlusion is removed, associate the target ID through appearance-motion-infrared joint matching, output the corrected trajectory state, and maintain the trajectory continuity of the occluded target.

[0288] In another embodiment, in step 600, for partially occluded targets, the detection box is retained, and the position is corrected using Kalman filtering and optical flow vector to reduce positioning deviation; for completely occluded targets, the trajectory position is maintained based on Kalman prediction, and the position is continuously recursively calculated using the state transition matrix F.

[0289] In yet another embodiment, in step 600, the apparent feature f is cached through a feature cache pool. app Mean and thermal infrared characteristics f thermal The mean value resists appearance distortion caused by occlusion. After occlusion is removed, the appearance similarity S is used to determine the result. a Motion consistency S m Spatiotemporal constraints S t Thermal infrared uniformity S hThe four decision-making methods determine whether the current detection frame belongs to the same target as the historical trajectory.

[0290] If no occlusion occurs, the target continues to be tracked based on the matching result output in step 500; if occlusion occurs, occlusion processing and re-identification are performed.

[0291] The occlusion handling and re-identification process is as follows:

[0292] 601, Occlusion detection.

[0293] In target tracking, occlusion determination is based on the results of target detection and feature matching, and is judged comprehensively using the following indicators:

[0294] 1. Detection box overlap: Calculate the intersection-union ratio (IOU) between the current detection box and the position box of the previous frame trajectory. IOU < 0.6 indicates partial occlusion, and IOU < 0.2 indicates complete occlusion.

[0295] 2. Confidence: The confidence of the detection box in the current frame has decreased by more than 30% compared with the historical average; (Occlusion will cause the target appearance to be incomplete, and the model's confidence in the detection box will decrease);

[0296] 3. Feature matching failed:

[0297] Apparent mismatch: Apparent feature similarity S a <0.8;

[0298] Motion inconsistency: Motion consistency score S m >3;

[0299] Spatiotemporal conflict: Occlusion time (time difference between the current frame and the previous frame) > 2.5s (corresponding to a maximum distance of 3m for a pedestrian moving at 1.2m / s).

[0300] When the above situation occurs, the occlusion handling process is triggered.

[0301] 602, Obscuring treatment.

[0302] 6021, Partial Occlusion: Preserve the detection box, correct its position using Kalman filtering and optical flow vectors, reduce positioning deviation, and update the features of the visible area.

[0303]

[0304] Obtain the corrected target state for the current frame. Where F is the state transition matrix, x t-1 For the target state of frame t-1, K t Let H' be the Kalman gain, H′ be the observation matrix, and z′ be the Kalman gain. t The measurement value for the current frame. The state is estimated a priori for frame t, and then predicted using the state in frame t-1:

[0305] When partially occluded, the detection bounding box may become inaccurate due to edge occlusion, requiring the fusion of optical flow information to enhance robustness. t Observational data extracted from the detection box / optical flow:

[0306]

[0307] in, The centroid of the current detection frame; α1 is the centroid for optical flow estimation, and α1 is the occlusion coefficient (when partially occluded, α1 = 0.3-0.5, reducing the weight of the detection box).

[0308]

[0309] Based on corner motion vector Calculate by averaging the components:

[0310] K t =P t H′ T (H′P t H′ T +R) -1

[0311] Among them, P t Let H' be the covariance, and H' be the observation matrix. T Let R be the transpose of H′, and let R be the measurement noise. When partially occluded, the noise of the detection box increases, and the diagonal elements of R become larger. For example, in a normal scene, R = diag([1,1]), and when occluded, R = diag([5,5]).

[0312] 6022, Complete Occlusion: Based on Kalman prediction to maintain trajectory position, cache the appearance features f of the first 5 frames. app The mean f mem app , and the thermal infrared features of the first 5 frames f thermal mean f mem thermal A feature cache pool is generated, which participates in the appearance similarity S during re-identification matching. a The calculations are used to resist visual distortions caused by occlusion (such as reflections from clothing).

[0313] Setting up a feature cache pool has the following characteristics:

[0314] Temporal smoothing: 5-frame mean filtering for transient noise;

[0315] Spatial adaptation: Only cache targets that completely obscure the memory, saving memory;

[0316] Cross-scene generalization: Improved success rate of re-identification in foggy / nighttime scenes.

[0317] While maintaining the trajectory, the position is continuously recursively calculated using the state transition matrix F. Continuous predicted locations are obtained.

[0318] 603, Re-identification and verification after occlusion is removed:

[0319] 6031. After the occlusion is removed, the target reappears. Calculate the similarity between the current detection box and the historical trajectory, using a four-way decision (apparent similarity S...). a Motion consistency S m Spatiotemporal constraints S t Thermal infrared uniformity S h Determine whether the current detection box belongs to the same target as the historical trajectory.

[0320]

[0321] Among them, f thermal f represents the thermal infrared features extracted from the current frame. mem thermal is the historical average of cached thermal infrared features (average of thermal infrared features in the first 5 frames), and ||·|| is the L2 norm.

[0322] Spatiotemporal constraints S t The duration of occlusion cannot be too long (e.g., <5 seconds), otherwise the trajectory will become invalid due to the accumulation of prediction errors, and the ID needs to be reinitialized.

[0323] ID decision is as follows: When S a >0.8, S m <3, satisfies the spatiotemporal constraints, and S h When the value is greater than 0.85, confirm that the current detection box is associated with the ID of the occluded target, restore the original ID of the occluded target, and update the trajectory status; otherwise, determine that the current detection box is a new target and assign a new global ID.

[0324] 6032, After the obscuration was lifted, the target did not appear, and the ID was released after a timeout.

[0325] Three-way decision re-identification is used to reduce the false match rate. It should be noted that S... a and S m The experimental threshold was optimized using DAVIS17. ij As an auxiliary verification condition, if D ij <30 pixels indicates that the spatial location is continuous and supports ID association.

[0326] Step 700, Trajectory Output and Result Analysis: For the final trajectory set {T} k}Calculate ID switching rate and trajectory continuity, detect speed abrupt changes or stagnation events, and output quantitative indicators and alarm signals.

[0327] As another example, the trajectory output and result analysis are as follows:

[0328] 701, based on the final set of historical trajectories {T k Output the ID association result of the occluded target and the updated trajectory status (including ID, position, and velocity).

[0329] 702. Calculate quantitative indicators, including tracking indicators ID switching rate (DDR), trajectory continuity (TC), and average confidence level of thermal infrared features.

[0330] DDR = Number of ID switching times / Total number of tracks × 100%

[0331] TC = Number of consecutive tracks / Total number of frames × 100%

[0332]

[0333] DDR reflects identity consistency, with a target value of ≤5%; TC reflects trace integrity, with a target value of ≥90%.

[0334] 703, when the velocity changes abruptly (∣v) t -v t-1 |>2 pixels / frame), Stasis (v t An abnormal alarm is triggered when the value is less than 0.1 pixels per frame and lasts for 5 seconds.

[0335] 704 achieves dual-indicator constraints through ID switching rate (DDR) and trajectory continuity (TC). The ID switching rate and trajectory continuity form a closed-loop feedback to guide parameter tuning and output the trajectory.

[0336] The parameter optimization closed loop is constructed based on ID switching rate and trajectory continuity as follows:

[0337] 7041, Calculate DDR and TC based on the tracking results;

[0338] 7042, if DDR ≤ 5% and TC ≥ 90%, keep the current parameters;

[0339] 7043, if DDR > 5% or TC < 90%, locate the problematic component and adjust the relevant parameters:

[0340] DDR > 5%: The problem lies in the trajectory association in step 500; increase the cost matrix C. ijThe motion weight β in the model is used to enhance motion consistency and reduce apparent mismatches.

[0341] TC < 90%: The problem lies in the occlusion handling in step 600, which involves expanding the feature cache pool window (by caching the appearance features of the first 5 frames). app The average value is expanded to the average value of the first 8 frames in the cache to extend the re-identification time tolerance;

[0342] DDR > 5% or TC < 90% or Three consecutive decreases, adjusting the lightweight attention fusion feature f fused η in:

[0343]

[0344] Where η0 is the initial thermal infrared weight, C ref These are reference values ​​for different scenarios (0.15 for sunny days, 0.08 for foggy days, and 0.25 for nighttime).

[0345] DDR increases and TC decreases: The problem lies in trajectory association in step 500, mismatch of seemingly similar targets (such as groups of students in school uniforms), and reducing the cost matrix C. ij The apparent weight α in the equation;

[0346] DDR increases while TC remains normal: The problem lies in feature extraction in step 300. Fog / low light causes apparent feature degradation, reducing the cost matrix C. ij The apparent weight α in the equation;

[0347] DDR is normal and TC is reduced: The problem lies in the occlusion handling in step 600, and the re-identification threshold S. a Excessive S value leads to ID recovery failure; reduce S value. a Threshold;

[0348] Local TC drop: The problematic part is the optical flow estimation in step 400, motion estimation, abnormal interference trajectory prediction, tightening the outlier threshold (the outlier vector judgment deviation is reduced from 45° to 30°) to suppress abnormal motion interference.

[0349] 7044, After adjusting the parameters, re-execute the tracing.

[0350] Specifically, in foggy conditions, when an urban traffic monitoring system tracks the trajectories of multiple pedestrian targets in real time, the tracking method is as follows:

[0351] First, for the original video frame I t The fog density is adaptively determined through the dark channel, and dehazing is performed to enhance image quality, outputting the final enhanced image J. t '; Secondly, regarding J tThe improved YOLOv7-tiny is used to detect targets, combining extracted multimodal features (appearance features, motion features, and geometric features) to output a set of detection boxes {B}. t}, to achieve precise target positioning; for detection box B t Corner detection and Lucas-Kanade sparse optical flow calculation are performed on the internal image to estimate the local motion trend of the target. Then, the detection box and trajectory are associated using a dynamic cost matrix (integrating appearance consistency, motion consistency and spatial distance) and the Hungarian algorithm. For occluded targets, the mean of appearance features is cached in a feature cache pool to resist appearance distortion caused by occlusion. The detection box is re-identified through a three-way decision (appearance similarity, motion consistency and spatiotemporal constraints) to maintain the continuity of the target ID. Finally, the parameters are fine-tuned in a closed loop based on the ID switching rate (DDR) and trajectory continuity (TC) indicators to achieve autonomous optimization of the algorithm in complex scenes.

[0352] This solution integrates four layers of capabilities—environmental perception, multimodal cognition, occlusion resilience, and closed-loop evolution—in a closed loop to achieve robust tracking under multi-hazard environments (fog / occlusion / deformation).

[0353] In yet another embodiment, in step 500, the Kalman scattering method predicts the target state based on historical state positions. The covariance P is calculated through state transition and observation correction. t According to the predicted state Covariance P t Calculate the distance between the predicted position and the actual position to obtain the motion consistency S. m This is used to measure the consistency of motion between the current detection box and historical trajectories.

[0354] The entire process follows a closed loop of "enhancement → detection → optical flow → correlation → occlusion handling → optimization" to achieve stable tracking of multiple targets in harsh environments. The calculation results of each formula serve subsequent steps (such as the dark channel mean determining the enhancement strategy, optical flow vector assisting trajectory prediction, and cost matrix supporting ID allocation), ensuring tracking accuracy and real-time performance.

[0355] Based on the same inventive concept as the AI ​​vision-based target tracking method provided in the embodiments of this application, the embodiments of this application also provide an AI vision-based target tracking system. If there is anything unclear about the content in the system embodiments, please refer to the corresponding content in the method embodiments.

[0356] An AI vision-based target tracking system includes:

[0357] The video acquisition module is used to acquire continuous video streams and obtain raw video frames I. t And thermal infrared frames, transmitted to the image enhancement module;

[0358] Image enhancement module: for the original video frame I t Rain and fog intensity are determined, and based on the rain and fog intensity, the original video frame I is analyzed using a joint physics-deep learning approach. t Dehazing is performed, the residual haze map and temperature feature map are fused, and the final enhanced image J is output. t ', Transmitted to the target detection module;

[0359] Target detection module; for the final enhanced image J t Perform object detection and generate a set of detection boxes {B}. t The data is then transmitted to the feature extraction module and the trajectory association module.

[0360] Feature extraction module: Extracts detection box B t Apparent features f of internal images app Motion characteristics I x I y Geometric features f geo and thermal infrared characteristics f thermal Lightweight attention fusion outputs a multimodal feature set {f} app I x I y f geo} and fusion features f fuesd The data is transmitted to the local optical flow motion estimation module and the trajectory association module;

[0361] Local optical flow motion estimation module; based on motion feature I x I y For detection box B t Corner detection and Lucas-Kanade sparse optical flow calculation are performed on the internal image, and the set of corner motion vectors is output. Transmitted to the trajectory association module;

[0362] Trajectory association module: Generates ID and historical trajectory T for detected targets. k Based on multimodal features and corner motion vectors and historical trajectory T k Kalman blotting predicts the target state and outputs a matching matrix X. ij According to X ij Perform trajectory matching and association; if matching fails, create a new ID and transmit it to the status judgment module, trajectory update module and occlusion handling module.

[0363] Status determination module: based on detection box B t and historical trajectory T k Determine whether the target is occluded and output the target status label (including visible, partially occluded, and completely occluded). If occlusion is determined, trigger the occlusion processing module.

[0364] Occlusion processing module: Performs Kalman correction and feature caching on occluded targets, generates a feature cache pool, and after the occlusion is removed, associates the target ID through appearance-motion-infrared joint matching, outputs the corrected trajectory status, outputs the recovered trajectory ID or new ID, and transmits it to the trajectory update module.

[0365] The trajectory update module updates the trajectory state based on the matching results from the occlusion handling module and outputs a set of trajectories {T} with IDs. k The data is then transmitted to the closed-loop optimization module.

[0366] Loop optimization module: Based on the trajectory set {T k The system calculates the ID switching rate and trajectory continuity, outputs parameter tuning instructions, and transmits them to the trajectory target detection module and the association module to achieve closed-loop optimization.

[0367] As another embodiment, the image enhancement module calculates the dark channel mean of the original video frame, determines the rain and fog intensity, performs preliminary defogging processing on the image based on the rain and fog degradation model, and performs residual correction on the image when the rain and fog intensity is high, further eliminating residual fog effects.

[0368] The trajectory association module calculates the similarity between the apparent features of the current frame detection box and the features stored in the historical trajectory, and obtains the apparent similarity S. a Calculate the Euclidean distance between the centroid of the current detection box and the predicted trajectory position to obtain the position distance D. ij Fusion of apparent similarity S a Motion consistency S m Location distance D ij The cost matrix C is dynamically constructed using the thermal infrared features of the current frame detection box and the thermal infrared features of historical trajectories. ij According to the dynamic cost matrix C ij and matching matrix X ij The Hungarian algorithm is used to find the optimal association and output the matching results between the detection box and the trajectory.

[0369] The state determination module determines whether the target is occluded based on target detection and feature matching. The determination indicators include the intersection-union ratio of the current detection box and the position box of the trajectory in the previous frame, the confidence of the detection box, and the feature matching result.

[0370] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, the purpose of providing these embodiments is to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application.

Claims

1. An AI vision-based target tracking method, characterized in that, The method comprises the following steps: A continuous RGB video stream is collected by monitoring equipment, a thermal infrared image sequence is synchronously acquired, and original video frames and thermal infrared frames are output; Rain and fog intensity is determined for the original video frames, and according to the rain and fog intensity, when the rain and fog intensity is low, a physical model is used for defogging, and when the rain and fog intensity is high, the original video frames and the thermal infrared frames are input into a double-branch U-Net, residual fog images and temperature feature maps are fused, and finally enhanced images are output, providing high-definition input for subsequent detection; Target detection and multi-modal feature extraction are performed based on the finally enhanced images, an improved YOLOv7-tiny is used to detect targets, a detection frame is generated, apparent features, motion features, geometric features and thermal infrared features of an image in the detection frame are extracted, after lightweight attention fusion, a multi-modal feature set and a fusion feature are output, precise positioning and feature description of the target are realized, and multi-dimensional features are provided for track association; Corner point detection and Lucas-Kanade sparse optical flow calculation are performed on the image in the detection frame, a set of corner point motion vectors is output, and local motion trend of the target is estimated, providing a motion basis for track prediction; A historical track of the detected target is generated, the target track is predicted, thermal infrared features of the current frame detection frame and thermal infrared features of the historical track are fused, a dynamic cost matrix is constructed to match the current detection frame and the predicted track, a matching matrix is output, and ID allocation and track updating of the target are realized according to the matching matrix; Whether the target is occluded is determined based on the target detection and feature matching results, Kalman correction and feature caching are performed on the occluded target, after the occlusion is removed, target ID is associated through apparent-motion-infrared joint matching, a corrected track state is output, and track continuity of the occluded target is maintained; ID switching rate and track continuity of the final track set are calculated, a speed mutation or stagnation event is detected, and a quantitative index and an alarm signal are output.

2. The method of claim 1, wherein, Dark channel mean calculation is performed on the original video frames, rain and fog intensity is determined, and preliminary defogging processing is performed on the image based on a rain and fog degradation model, residual correction is performed on the image in the case of high rain and fog intensity, and residual fog effects are further eliminated.

3. The method of claim 1, wherein, The final enhanced image is down-sampled through a Backbone and enhanced channels through an SEBlock, target features are strengthened, detection frame coordinates are decoded, a set of detection frames is output after confidence filtering and non-maximum suppression, and the positioning target is realized.

4. The method of claim 1, wherein, Corner point motion vector angle deviation greater than 45° is determined as an outlier vector, and the outlier vector is removed to reduce motion estimation error.

5. The method of claim 1, wherein, The historical state position is used to predict the target state through Kalman, the covariance is calculated through state transition and observation correction, the distance between the predicted position and the actual position is calculated according to the predicted state and the covariance, the motion consistency is obtained, and the motion consistency between the current detection frame and the historical track is measured.

6. The method of claim 5, wherein, The similarity between the apparent features of the current frame detection frame and the features stored in the historical track is calculated, the apparent similarity is obtained, and the Euclidean distance between the center of the current detection frame and the predicted position of the track is calculated, the position distance is obtained.

7. The method of claim 1, wherein, For partially occluded targets, the detection frame is retained, the position is corrected through Kalman filtering and optical flow vector, and the positioning deviation is reduced; For the completely occluded target, the trajectory position is maintained based on Kalman prediction, and the position is recursively updated with a state transition matrix.

8. The method of claim 7, wherein, The mean value of the apparent feature and the mean value of the thermal infrared feature are cached through a feature cache pool, so as to resist the apparent distortion caused by occlusion. After the occlusion is removed, whether the current detection frame belongs to the same target as the historical trajectory is determined through four decision-making branches of apparent similarity, motion consistency, space-time constraint and thermal infrared consistency.

9. An AI vision-based target tracking system, characterized in that, The method comprises the following steps: a video acquisition module, configured to acquire a continuous video stream, obtain original video frames and thermal infrared frames, and transmit the original video frames and the thermal infrared frames to an image enhancement module; the image enhancement module: performing rain and fog intensity determination on the original video frames, performing de-fog processing on the original video frames through physical-deep learning combination according to the rain and fog intensity, fusing residual fog maps and temperature feature maps, and outputting final enhanced images to a target detection module; the target detection module; performing target detection on the final enhanced images to generate a detection frame set, and transmitting the detection frame set to a feature extraction module and a trajectory association module; the feature extraction module: extracting apparent features, motion features, geometric features and thermal infrared features of images in the detection frames, performing lightweight attention fusion, outputting a multi-modal feature set and a fusion feature, and transmitting the multi-modal feature set and the fusion feature to a local optical flow motion estimation module and the trajectory association module; the local optical flow motion estimation module: performing corner detection and Lucas-Kanade sparse optical flow calculation on images in the detection frames based on the motion features, and outputting a corner motion vector set to the trajectory association module; the trajectory association module: generating an ID and a historical trajectory for a detected target, predicting a target state based on multi-modal features, corner motion vectors and the historical trajectory, outputting a matching matrix, performing trajectory matching and association according to the matching matrix, creating a new ID when the matching fails, and transmitting the new ID to a state judgment module, a trajectory update module and an occlusion processing module; the state judgment module: judging whether a target is occluded according to a detection frame and a historical trajectory, outputting a target state label, and the target state label including visible, partially occluded and completely occluded, and triggering the occlusion processing module when the occlusion is determined; the occlusion processing module: performing Kalman correction and feature caching for an occluded target, generating a feature cache pool, associating a target ID through apparent-motion joint matching after the occlusion is removed, outputting a corrected trajectory state, outputting a recovered trajectory ID or a new ID, and transmitting the recovered trajectory ID or the new ID to the trajectory update module; the trajectory update module: updating a trajectory state according to a matching result of the occlusion processing module, outputting a trajectory set with an ID, and transmitting the trajectory set with the ID to a loop optimization module; the loop optimization module: calculating an ID switching rate and a trajectory continuity according to the trajectory set, outputting a parameter optimization instruction, and transmitting the parameter optimization instruction to the trajectory target detection module and the association module to realize closed-loop optimization.

10. The system of claim 9, wherein, The image enhancement module performs dark channel mean value calculation on the original video frames, determines rain and fog intensity, performs preliminary de-fog processing on the images based on a rain and fog degradation model, and performs residual correction on the images in the case of high rain and fog intensity, so as to further eliminate residual fog effects. The trajectory association module calculates the similarity between the apparent features of the current frame detection box and the stored features of the historical trajectory, obtains an apparent similarity, calculates the Euclidean distance between the current detection box centroid and the trajectory predicted position, obtains a position distance, fuses the apparent similarity, motion consistency, position distance, thermal infrared features of the current frame detection box and the thermal infrared features of the historical trajectory, dynamically constructs a cost matrix, and according to the dynamic cost matrix and the matching matrix, the Hungarian algorithm is used to solve the optimal association, and the matching result of the detection box and the trajectory is output. The state judgment module judges whether the target is blocked based on target detection and feature matching, and the judgment indexes include the intersection over union of the current detection box and the position box of the previous frame trajectory, the detection box confidence and the feature matching result.

Images