Multi-camera collaborative single-target tracking method and device

Abstract

The invention discloses a multi-camera collaborative single-target tracking method and device, belongs to the technical field of computer vision target tracking, and is suitable for scenes such as intelligent monitoring and public safety. The method comprises the following steps: constructing a camera space matrix and defining a view field range; the recognition robustness is improved by fusing the target face and the body features; tracking and constructing a target trajectory based on the fusion features; a missing frame is reconstructed or a future frame is predicted through a forward diffusion-reverse generation model, and trajectory coherence is optimized in combination with space-time constraints; probability graph loss optimization of a fixation point area is realized by using an encoder and a saliency prediction module, and finally a video conforming to fixation point constraints is jointly generated. The device comprises a video frame acquisition and preprocessing module, a feature extraction and fusion module, a target tracking module, a missing frame reconstruction and future frame prediction module, a fixation point positioning module, a joint optimization module and a video output module, and all the modules cooperatively realize end-to-end tracking. According to the method, the shielding scene recognition capability is enhanced through multi-modal feature fusion, the problem of track breakage is solved by means of a diffusion model, the monitoring requirement of a specific area is met in combination with fixation point optimization, the precision, continuity and practicability of target tracking in a multi-camera environment are effectively improved, and the method has remarkable technical advantages and application value.

Description

Technical Field

[0001] The present invention relates to a multi-camera coordinated single-target tracking method and device, which belongs to target tracking technology in computer vision and is applicable to the fields of intelligent monitoring, public safety, traffic management, etc. Background Art

[0002] Tracking a single target in a multi-camera environment is a key research direction in computer vision, widely used in intelligent surveillance, traffic management, public safety, and other fields. However, existing multi-camera target tracking technologies still have many shortcomings in handling target occlusion, target loss, and analyzing the location of target disappearance, making them difficult to meet the practical needs of complex scenarios. First, the problem of target trajectory consistency in multi-camera environments is a long-standing challenge. Due to differences in the field of view and perspective of different cameras, the target's trajectory may become disjointed or distorted when switching from one camera to another. Second, the rationality of the target is also a key challenge. The target's trajectory must conform to physical laws and spatiotemporal constraints. However, in real-world scenarios, targets may experience unreasonable trajectory jumps or disappearances due to occlusion, blind spots, or changes in motion direction. Furthermore, the problem of determining target occlusion and loss is particularly prominent in multi-camera environments. When a target is absent from a particular camera, existing technologies struggle to distinguish whether it is due to occlusion or loss. Finally, analyzing the location of target disappearance is also a shortcoming of existing technologies. When a target disappears from a particular camera, existing technologies often cannot accurately analyze the specific location and cause of its disappearance. Summary of the Invention

[0003] Purpose of the invention: In order to overcome the shortcomings of the prior art, the present invention provides a multi-camera collaborative single target tracking method and device, which solves the problems of trajectory breakage, feature matching errors, improper missing frame processing and inaccurate gaze point analysis in the prior art by constructing a multi-camera spatial matrix, fusing target features to construct the target trajectory, introducing a diffusion model to reconstruct missing frames and jointly optimizing the gaze point. It has the advantages of improving the consistency of the target trajectory, enhancing the accuracy of target state judgment and optimizing the rationality of the trajectory, and can achieve high-precision tracking, trajectory optimization and specific video generation of targets in complex scenarios.

[0004] Technical solution: To achieve the above purpose, the technical solution adopted by the present invention is:

[0005] A multi-camera collaborative single target tracking method includes the following steps:

[0006] Step 1: Input the video streams collected by each camera, extract the background features of each camera, and then identify the field of view of each camera. Combined with the field of view of each camera, a spatial matrix of each camera within the monitoring area is constructed to provide a spatial constraint basis for subsequent cross-camera trajectory association, which is used to solve the problem of trajectory interruption when switching between multiple perspectives.

[0007] Step 2: Input the target image and extract the target's facial feature vector. Based on the target's facial feature vector, identify the target in the video stream of each camera and extract the target's body feature vector. Perform feature fusion on the target's facial feature vector and body feature vector to obtain a fused feature vector. By fusion, the risk of tracking interruption caused by failure of a single feature is reduced.

[0008] Step 3: Track the target in the video streams of each camera based on the fused feature vector and construct the target trajectory within the monitoring area. A multi-camera collaboration mechanism is introduced when constructing the target trajectory based on the fused feature. The spatiotemporal correlation across cameras can help solve the target blind spot coverage problem.

[0009] Step 4: Analyze the missing frames of each camera based on the target trajectory and spatial matrix, add noise to the missing frames through the forward diffusion method, and then reconstruct the missing frames or predict future frames through the inverse generation method. Use the trajectory similarity algorithm to calculate the correlation of the target trajectory in different cameras, and introduce spatiotemporal constraints and trajectory smoothing algorithms to optimize the coherence and rationality of the target trajectory in different cameras.

[0010] Step 5: Input the fixation point location and specify the time window. The encoder and saliency prediction module output the fixation point saliency probability map, and optimize the fixation point probability map loss based on the true fixation point probability map.

[0011] Step 6: Jointly optimize the inverse generation process loss and the gaze point probability map loss to generate a video with the specified gaze point position within the specified time window. Joint optimization can achieve the video generation requirements of specific gaze points while ensuring the rationality of the target trajectory, improving the practicality of the system in scenarios such as public safety.

[0012] The method of the present invention can solve the problems of target trajectory fragmentation and confusion in a multi-camera environment, unreasonable trajectory caused by occlusion or loss of the target, and inaccurate analysis of the target disappearance position, and can generate a video with a coherent trajectory and specific gaze point that conforms to spatiotemporal constraints.

[0013] Specifically, in Step 2, the target image is input and the target's facial feature vector F0 is extracted; based on the target's facial feature vector F0, the target is identified in the video stream of each camera and the target's body feature vector G0 is extracted; the facial feature vector F0 and the body feature vector G0 are fused by the feature fusion module to obtain the fused feature vector F (0) The two inputs of the feature fusion module are first subjected to layer normalization to eliminate the dimensional differences between different feature dimensions and ensure the stability of the fusion process. Then, the cross-attention mechanism is used to realize the interactive perception of facial features and body features, so that the two features form a complementary relationship in the spatial dimension, effectively solving the problem of single feature failure in occlusion scenes. Finally, the multi-layer perceptron is used to adjust the channels and redistribute the channel weights according to the importance of the fused features to improve the discrimination of the target representation. The feature fusion process can be expressed as:

[0014] F (0) =MLP(CrossAttn(L(F0),L(G0)))

[0015] Where: L represents layer normalization, CrossAttn represents cross attention mechanism, MLP represents multi-layer perceptron; gradient descent method is used to optimize the fusion feature vector F (0) ,The feature fusion parameters are dynamically adjusted through back propagation, so that the fused feature vector not only retains the effective information of the original features, but also adapts to the spatiotemporal constraints of multi-camera collaborative tracking.

[0016] This step solves tracking problems in complex scenarios from two perspectives: target recognition and feature optimization. First, by extracting facial feature vectors from the input target image, the target's biometric characteristics can be accurately captured, providing a benchmark for subsequent cross-camera tracking. Based on these facial feature vectors, the target is identified in each video stream and its body feature vectors are extracted, expanding from static features to dynamic features and enhancing adaptability to changes in target posture. Fusion features can address issues such as target trajectory fragmentation, feature matching errors, and unstable feature extraction under occlusion caused by insufficient single feature recognition capabilities during target tracking in a multi-camera environment, thereby improving the effectiveness and accuracy of target tracking.

[0017] Specifically, in Step 1, the spatial matrix of each camera in the monitoring area is expressed as R={R xy}, R xy Indicates the camera sequence whose field of view includes the (x, y) position; in Step 3, the target trajectory in the monitoring area is represented by S = {S1, S2, ..., S t ,…},S trepresents the position of the target at time t, t = 1, 2, ...; in Step 4, based on the spatial matrix R and the target trajectory S, the video frames that should contain the target in the field of view at time t are analyzed, and the video frames that should contain the target and actually contain the target are judged as normal frames, and a normal frame set N at time t is constructed. t , the video frames that should contain the target but do not actually contain the target are judged as missing frames, and the missing frame set D at time t is constructed t ; Normal frame set N t and the reconstructed frame set C t Together they constitute the set A of all frames at time t t ; Use all frame sets A at time t-1 t-1 And the normal frame set N at time t t For the missing frame set D at time t t Perform target reconstruction and obtain the reconstructed frame set C at time t t ; Use all frame sets A at time t-1 t-1 For all frames A at time t t Perform target reconstruction to obtain the predicted frame set P at time t t .

[0018] This step, by establishing a spatiotemporal correlation model between the spatial matrix and the target trajectory, can solve the problem of trajectory interruption caused by the temporary disappearance or occlusion of the target in multi-camera scenarios. First, by mapping the field of view range of the spatial matrix, the spatial coverage logic of each camera in the monitoring area is determined, providing physical constraints for the spatiotemporal analysis of the target trajectory. Secondly, by modeling the time series of the target trajectory, the target position at consecutive moments is matched with the spatial matrix to determine whether the camera video frame where the target should appear actually contains the target, thereby separating normal frames from missing frames. By dynamically combining the normal frame set and the reconstructed frame set to form all frame sets, the missing frames are reversely reconstructed based on historical data and predicted to generate future frames based on the forward and backward correlation of the time series, ensuring the integrity and continuity of the target trajectory in the spatiotemporal dimension. This missing frame discrimination mechanism based on spatiotemporal constraints, combined with the joint optimization method of multi-moment frame sets, can effectively distinguish between the true disappearance of the target and temporary occlusion, avoiding trajectory misjudgment caused by the missing of a single frame data.

[0019] Specifically, in Step 4, the missing frame set D is obtained by forward diffusion. t Add noise to the missing frames in the video. If the video frame of the i-th camera at time t is a missing frame The video frame of the i-th camera at time t-1 is Video frame Add Gaussian noise and get the noise frame of the i-th camera at time t: Among them, σ 2 (t) and σ2 (t-1) represents the noise variance at time t and time t-1 respectively, I represents the identity matrix; N represents the Gaussian distribution; the forward diffusion method is simplified to a stochastic differential equation:

[0020]

[0021] Where: f(t) represents the drift term, which is used to describe the video frame The deterministic change of g(t) represents the sound intensity; w represents the standard Brownian motion;

[0022] All missing frames Corresponding noise frame Constructing the noise frame set Q t , concatenate normal frame set N t and noise frame set Q t Get Collection To B t Reverse generation to obtain E t ;like Then A t =P t =E t ;like Using N t Replace E t The video frames with the same camera index in A are obtained t ; Simplify the inverse generation method to a stochastic differential equation:

[0023]

[0024] Where: g 2 (t) represents the square of the noise intensity, which is used to ensure the stability of the inverse generation; log p(E t |B t ) represents the logarithm of the conditional probability distribution; It is an additional correction term generated in reverse, which is used to correct the evolution direction of the data so that the data gradually moves towards the reconstructed frame set E t Close; g(t)dw is the diffusion term generated in reverse, representing random noise.

[0025] This step addresses the missing target trajectory problem in multiple cameras by combining noise injection with inverse generation. A forward diffusion method adds Gaussian-distributed noise to the missing frames. Stochastic differential equations are used to model the effects of deterministic changes and random noise, forming a noisy frame set. Subsequent processing can exploit the noise distribution characteristics for data recovery. Normal and noisy frames are concatenated and fed into the inverse generation process. By designing an inverse stochastic differential equation that includes a drift term and a correction term, the stability of the generation process is ensured while the correction term guides the data evolution toward the target reconstruction. Specifically, the gradient of the logarithm of the conditional probability distribution is used as the correction term, allowing the generation process to incorporate spatiotemporal constraints to correct trajectory deviations. The squared noise intensity term ensures the convergence of the inverse diffusion process through mathematical constraints. Regarding the decision logic between missing and predicted frames, the dynamic replacement of frames with the same camera index achieves collaborative optimization of multi-camera data. Ultimately, the reconstructed video frames conform to physical laws while maintaining spatiotemporal coherence with trajectories in adjacent time windows.

[0026] Specifically, a diffusion model (such as DDPM) is used to train the reverse generation process, with B t As a condition, learn based on the noise frame set Q t Reconstruct missing frames or predict future frames; minimize the reconstruction loss via gradient descent.

[0027] Specifically, in Step 5, the gaze point position is input and a time window is specified. The video stream reconstructed by each camera is intercepted according to the time window. The gaze point position and the video stream are input into the gaze point positioning module. First, the encoder is used to extract deep features of each video stream and fuse them into multi-level spatial-temporal features to obtain the key visual area in the video content at the gaze point position and generate a multi-scale feature map. Then, the saliency prediction module is used to calculate the gaze point saliency probability map of the multi-scale feature map, and the binary cross entropy loss between the gaze point saliency probability map and the true gaze point probability map is calculated to optimize the gaze point probability map loss.

[0028] This step improves the accuracy of video content reconstruction in the fixation area by integrating spatiotemporal features with multi-scale analysis. First, the fixation point location and time window are input, limiting the spatiotemporal scope to key areas and time periods where the target is likely to be present, avoiding redundant global computation. The encoder performs deep feature extraction on the multi-camera reconstructed video stream, fusing and generating multi-level spatiotemporal features. This integrates complementary perspective information from different cameras, overcoming the problem of feature loss caused by target occlusion under a single perspective. The design of generating multi-scale feature maps captures detailed features and contextual information of the fixation area at different resolutions, adapting to possible scale changes of the target within the time window. The saliency prediction module combines the visual attention mechanism with probabilistic modeling by calculating saliency probability maps from the multi-scale feature maps to distinguish the target area from background interference. A binary cross-entropy loss is used to compare the predicted results with the true probability maps. Backpropagation is then used to optimize fixation point localization accuracy, ensuring that the reconstructed video content more closely matches the visual saliency patterns of targets in real scenes. This addresses the problem of biased fixation area prediction caused by target trajectory interruptions or occlusions.

[0029] A multi-camera collaborative single target tracking device, characterized by comprising a video frame acquisition and preprocessing module, a feature extraction and feature fusion module, a target tracking module, a missing frame reconstruction and future frame prediction module, a gaze point positioning module, a joint optimization module, and a video output module;

[0030] The video frame acquisition and preprocessing module extracts the background features of each camera from the video stream collected by each camera, and then identifies the field of view of each camera; based on the field of view of each camera, it constructs a spatial matrix of each camera in the monitoring area;

[0031] The feature extraction and feature fusion module extracts the target's facial feature vector; identifies the target in the video stream of each camera based on the target's facial feature vector, and extracts the target's body feature vector; performs feature fusion on the target's facial feature vector and body feature vector to obtain a fused feature vector;

[0032] The target tracking module tracks the target in the video stream of each camera based on the fused feature vector and constructs the target trajectory in the monitoring area;

[0033] The missing frame reconstruction and future frame prediction module analyzes the missing frames of each camera based on the target trajectory and spatial matrix, adds noise to the missing frames through the forward diffusion method, and then reconstructs the missing frames or predicts future frames through the inverse generation method. It uses a trajectory similarity algorithm to calculate the correlation of target trajectories in different cameras, and introduces spatiotemporal constraints and a trajectory smoothing algorithm to optimize the coherence and rationality of target trajectories in different cameras.

[0034] The fixation point localization module inputs the fixation point location and specifies the time window, outputs the fixation point saliency probability map through the encoder and saliency prediction module, and performs fixation point probability map loss optimization based on the fixation point true probability map;

[0035] The joint optimization module performs a weighted summation of the inverse generation process loss and the gaze point probability map loss, and supervises the training of the end-to-end model consisting of the video frame acquisition and preprocessing module, the feature extraction and feature fusion module, the target tracking module, the missing frame reconstruction and future frame prediction module, and the gaze point localization module through a gradient optimization method;

[0036] The video output module, given a target image and video streams from multiple cameras, reconstructs missing video frames or predicts future video frames through an inverse generation process; inputs a gaze point position and a specified time window, and generates a target trajectory that meets the gaze point constraints within the specified time window.

[0037] Specifically, the feature extraction and feature fusion module includes a facial feature extraction unit, a body feature extraction unit and a feature fusion unit; the facial feature extraction unit uses a pre-trained convolutional neural network (such as ResNet) to extract the target facial feature vector; the body feature extraction unit locates the target based on a target detection algorithm (such as YOLO), extracts features including bounding boxes and postures, and generates a body feature vector through a fully connected layer; the feature fusion unit includes two parallel layer normalization layers, a cross attention mechanism and a multi-layer perceptron.

[0038] Beneficial effects: The multi-camera collaborative single-target tracking method and device provided by the present invention have the following advantages over the existing technology: 1. Multimodal feature fusion: Through the cross-attention fusion of facial and body features, the recognition robustness in occlusion and low-resolution scenes is enhanced, and the risk of tracking interruption is reduced; 2. Spatiotemporal constraint trajectory optimization: Based on the spatial matrix and trajectory similarity algorithm, the breaks and unreasonable jumps of cross-camera trajectories are corrected to ensure spatiotemporal consistency; 3. Missing frame reconstruction and prediction: Combining forward diffusion and inverse generation models, the missing frame information can be effectively restored, the real disappearance and temporary occlusion can be distinguished, and the trajectory integrity can be improved; 4. Gaze point focus generation: Through saliency prediction and probability map loss optimization, video generation of specific areas is realized to meet the key area monitoring needs of scenarios such as public safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a schematic diagram of the implementation process of the present invention. DETAILED DESCRIPTION

[0040] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0041] A multi-camera collaborative single-target tracking method constructs a spatial matrix by analyzing the camera's field of view, providing a basic coordinate framework for multi-view collaboration. Feature fusion techniques are used to integrate the target's multimodal features, enhancing cross-camera recognition consistency. A diffusion model is introduced to perform noise addition and inverse generation on missing frames, enabling target trajectory prediction under occlusion. Specifically, the method first constructs a spatial matrix for each camera to determine the coverage relationship between their monitored areas. Upon inputting a target image, facial and body features are simultaneously extracted and cross-attentionally fused to form a highly discriminative fused feature. Based on this feature, continuous tracking is performed within the multi-camera video stream to generate an initial target trajectory. The system analyzes the frame data from each camera in real time, determines whether video frames that should contain the target are missing using the spatial matrix, applies forward diffusion noise addition to the missing frames, and reconstructs the motion state of the occluded target through conditional inverse generation. Finally, the inverse generation process is jointly optimized with the fixation probability map to generate a continuous tracking video that meets spatiotemporal constraints.

[0042] The specific implementation steps of the present invention are described in detail below.

[0043] Step 1: Input the video stream collected by each camera, extract the background features of each camera, and then identify the field of view of each camera; based on the field of view of each camera, construct the spatial matrix of each camera in the monitoring area.

[0044] The spatial matrix R refers to the topological structure that records the coverage of each camera's field of view. It can be achieved through geometric calibration combined with feature matching. It is used to determine the visibility relationship of the target between different cameras, provide a spatial basis for missing frame judgment, and solve the problem of trajectory breakage caused by perspective differences in traditional methods. The spatial matrix of each camera in the monitoring area is expressed as R = {R xy}, R xy A sequence of cameras whose field of view encompasses the (x, y) position.

[0045] Step 2: Input the target image and extract the target’s facial feature vector F0; identify the target in the video stream of each camera based on the target’s facial feature vector F0, and extract the target’s body feature vector G0; perform feature fusion on the facial feature vector F0 and the body feature vector G0 through the feature fusion module to obtain the fused feature vector F0. (0) .

[0046] The two inputs of the feature fusion module are firstly subjected to layer normalization, then feature fusion is performed through the cross-attention mechanism, and then channel adjustment is performed through the multi-layer perceptron:

[0047] F (0) =MLP(CrossAttn(L(F0),L(G0)))

[0048] Among them: L represents layer normalization processing, which is used to eliminate the differences in the numerical distribution of different features. Specifically, it can be implemented using the LayerNorm algorithm. By normalizing the mean and variance of the feature dimensions of each sample, the stability of the feature fusion process can be enhanced; CrossAttn represents the cross-attention mechanism, which takes the face feature vector F0 as the query vector and the body feature vector G0 as the key vector. After calculating the attention weight, the body feature vector G0 is weightedly aggregated to achieve dynamic fusion of cross-modal features; MLP represents the multi-layer perceptron; the gradient descent method is used to optimize the fusion feature vector F (0) , so that the fused features can more accurately reflect the comprehensive properties of the target, thereby improving the robustness of tracking.

[0049] The feature fusion module adopts a cross-attention mechanism, which implements cross-modal information interaction after processing feature distribution differences through layer normalization; cross-attention feature fusion enhances the cross-camera recognition consistency of target features, overcomes the defect that a single feature is easily affected by environmental interference, and can improve the accuracy of cross-camera target matching.

[0050] Step 3: Track the target in the video stream of each camera based on the fused feature vector and construct the target trajectory in the monitoring area.

[0051] The target trajectory refers to the position sequence of the target at continuous time points. It can be generated by combining the target detection algorithm with the Kalman filter tracking method. Its core function is to provide a time dimension motion trajectory benchmark for missing frame recognition. The target trajectory in the monitoring area is represented as S = {S1, S2, ..., S t ,…},S t Represents the position of the target at time t, t = 1, 2, ...

[0052] Step 4: Analyze the missing frames of each camera based on the target trajectory S and the spatial matrix R, add noise to the missing frames through the forward diffusion method, and then reconstruct the missing frames or predict future frames through the inverse generation method; use the trajectory similarity algorithm to calculate the correlation of the target trajectories in different cameras, and introduce spatiotemporal constraints and trajectory smoothing algorithms to optimize the coherence and rationality of the target trajectories in different cameras.

[0053] Based on the spatial matrix R and the target trajectory S, the video frames that should contain the target in the field of view at time t are analyzed, and the video frames that should contain the target and actually contain the target are judged as normal frames. The normal frame set N at time t is constructed. t , the video frames that should contain the target but do not actually contain the target are judged as missing frames, and the missing frame set D at time t is constructed t ; Normal frame set N t and the reconstructed frame set C tTogether they constitute the set A of all frames at time t t ; Use all frame sets A at time t-1 t-1 And the normal frame set N at time t t For the missing frame set D at time t t Perform target reconstruction and obtain the reconstructed frame set C at time t t ; Use all frame sets A at time t-1 t-1 For all frames A at time t t Perform target reconstruction to obtain the predicted frame set P at time t t Normal frame set N t and the missing frame set D t It refers to the classification of video frames based on the spatiotemporal matching results of the spatial matrix R and the target trajectory S. It can be implemented by using the inter-frame difference method and trajectory interpolation algorithm. Its function is to distinguish between surveillance clips in which the target appears normally and those in which it is abnormally missing, providing a data basis for subsequent reconstruction.

[0054] The missing frame set D is obtained by forward diffusion method. t Add noise to the missing frames in the video. If the video frame of the i-th camera at time t is a missing frame The video frame of the i-th camera at time t-1 is Video frame Add Gaussian noise and get the noise frame of the i-th camera at time t: Among them, σ 2 (t) and σ 2 (t-1) represents the noise variance at time t and time t-1 respectively, I represents the identity matrix; N represents the Gaussian distribution; the forward diffusion method is simplified to a stochastic differential equation:

[0055]

[0056] Where: f(t) represents the drift term, which is used to describe the video frame where g(t) represents the deterministic change of sound intensity and w represents the standard Brownian motion.

[0057] The forward diffusion method simulates the data degradation process by gradually adding Gaussian noise. Specifically, a stochastic differential equation is used to describe the noise addition rule, providing a reversible noise distribution for subsequent inverse generation.

[0058] All missing frames The corresponding noise frame Constructing the noise frame set Q t , concatenate normal frame set N t and noise frame set Q t Get Collection B t Reverse the condition to get Et , based on the space matrix R constrained marker E t Video frame with incorrect field of view, marked with E t The video frames that do not meet the constraints of adjacent video frames and historical target trajectories can be marked as abnormal frames and abnormal disappearance alarms can be issued; if Then A t =P t =E t ;like Using N t Replace E t The video frames with the same camera index in A are obtained t ; Simplify the inverse generation method to a stochastic differential equation:

[0059]

[0060] Where: g 2 (t) represents the square of the noise intensity, which is used to ensure the stability of the inverse generation; log p(E t |B t ) represents the logarithm of the conditional probability distribution; It is an additional correction term generated in reverse, which is used to correct the evolution direction of the data so that the data gradually moves towards the reconstructed frame set E t Close; g(t)dw is the diffusion term generated in reverse, representing random noise; the reconstruction loss is minimized by gradient descent.

[0061] The inverse generation method corrects the data evolution direction through conditional probability. Specifically, the correction term is used to adjust the diffusion path, which can effectively restore the motion trajectory of the occluded target and break through the limitations of traditional interpolation methods in complex scenes.

[0062] In this step, a spatiotemporal correlation model across cameras is established through a joint analysis of the spatial matrix R and the target trajectory S. This can accurately identify target loss caused by field of view switching, solve the problem of target trajectory interruption caused by field of view switching in a multi-camera environment, and achieve accurate identification and reconstruction of missing frames. At the same time, the combined reconstruction method of forward diffusion and reverse generation can more effectively restore the motion characteristics of the target in the cross-camera transition stage compared to the traditional interpolation algorithm. In addition, the prediction frame generation mechanism based on spatiotemporal constraints can effectively avoid the disappearance of the target trajectory in blind spots or transition areas, providing complete motion trajectory data support for subsequent gaze point positioning.

[0063] Step 5: Input the fixation point location and specify the time window. Output the fixation point saliency probability map through the encoder and saliency prediction module, and optimize the fixation point probability map loss based on the true fixation point probability map.

[0064] The gaze point position is input and a time window is specified. The video stream reconstructed by each camera is intercepted according to the time window. The gaze point position and video stream are input into the gaze point localization module. First, the encoder extracts deep features from each video stream and fuses them into multi-level spatial-temporal features to obtain the key visual area in the video content at the gaze point position and generate a multi-scale feature map. Then, the saliency prediction module calculates the gaze point saliency probability map of the multi-scale feature map, calculates the binary cross entropy loss between the gaze point saliency probability map and the true gaze point probability map, and performs gaze point probability map loss optimization.

[0065] A temporal window refers to a specific time segment of a video stream that is captured based on user needs. This can be achieved using a sliding window or a fixed window, focusing on target motion and visual attention areas within a specific time segment. The encoder is a neural network model used to extract deep features from a video stream. It can be implemented using a two-dimensional convolutional neural network (2D CNN) combined with a temporal attention mechanism, or a lightweight three-dimensional convolutional neural network (3D CNN). It captures spatiotemporal correlations in the video, providing a foundation for subsequent feature fusion. To reduce computational complexity, only short-term spatiotemporal features can be extracted within the temporal window. Multi-level spatial-temporal features are a combination of low-level detail features and high-level semantic features extracted from different network layers. These features can be implemented using skip connections or a feature pyramid structure, enhancing the representation of the fixation region at different scales. A multi-scale feature map is a collection of feature maps with different resolutions or receptive fields. These maps can be generated through downsampling and upsampling operations, capturing multi-granular information about the target at the fixation point. The saliency prediction module is a neural network module based on an attention mechanism. It can be implemented using a combination of spatial and channel attention, focusing on key areas of the video stream related to the fixation point.

[0066] Specifically, after inputting the gaze point location, data for the corresponding time period is extracted from the reconstructed video streams of each camera according to a specified time window. The extracted video stream is fed into the encoder, where single-frame spatial features are first extracted using a 2D CNN. A temporal attention mechanism is then introduced to fuse features from multiple frames within the time window, extracting joint features across the temporal and spatial dimensions. Skip connections are then used to fuse shallow features with deep features to form a multi-level spatio-temporal feature map. Subsequently, a feature map is extracted at the gaze point location, and multi-scale feature maps are generated through upsampling and downsampling, preserving spatial resolution while compressing the temporal dimension. The saliency prediction module performs a weighted fusion of the multi-scale feature maps, utilizing an attention mechanism to enhance the response of key regions, and generate a gaze point saliency probability map. This probability map is then compared pixel by pixel with the ground-truth probability map, and the model parameters are optimized using backpropagation using a binary cross-entropy loss to improve gaze point localization accuracy.

[0067] Step 6: Jointly optimize the inverse generation process loss and the gaze point probability map loss, train a multi-camera collaborative single target tracking model, and generate a video with the specified gaze point position within the specified time window.

[0068] Step 7: Given a target image and video streams from multiple cameras, input the gaze point location and specify a time window. Use the trained multi-camera collaborative single target tracking model to generate a target trajectory that meets the gaze point constraints within the specified time window.

[0069] A multi-camera collaborative single target tracking device for implementing the above method, comprising a video frame acquisition and preprocessing module, a feature extraction and feature fusion module, a target tracking module, a missing frame reconstruction and future frame prediction module, a gaze point positioning module, a joint optimization module, and a video output module;

[0070] The video frame acquisition and preprocessing module extracts the background features of each camera from the video stream collected by each camera, and then identifies the field of view of each camera; based on the field of view of each camera, it constructs a spatial matrix of each camera in the monitoring area;

[0071] The feature extraction and feature fusion module extracts the facial feature vector of the target; identifies the target in the video stream of each camera based on the facial feature vector of the target, and extracts the body feature vector of the target; performs feature fusion on the facial feature vector and body feature vector of the target to obtain a fused feature vector; the feature extraction and feature fusion module includes a facial feature extraction unit, a body feature extraction unit and a feature fusion unit; the facial feature extraction unit uses a pre-trained convolutional neural network to extract the target facial feature vector; the body feature extraction unit locates the target based on the target detection algorithm, extracts features including the bounding box and posture, and generates a body feature vector through a fully connected layer; the feature fusion unit includes two parallel layer normalization layers, a cross attention mechanism and a multi-layer perceptron.

[0072] The target tracking module tracks the target in the video stream of each camera based on the fused feature vector and constructs the target trajectory in the monitoring area;

[0073] The missing frame reconstruction and future frame prediction module analyzes the missing frames of each camera based on the target trajectory and spatial matrix, adds noise to the missing frames through the forward diffusion method, and then reconstructs the missing frames or predicts future frames through the inverse generation method. It uses a trajectory similarity algorithm to calculate the correlation of target trajectories in different cameras, and introduces spatiotemporal constraints and a trajectory smoothing algorithm to optimize the coherence and rationality of target trajectories in different cameras.

[0074] The fixation point localization module inputs the fixation point location and specifies the time window, outputs the fixation point saliency probability map through the encoder and saliency prediction module, and performs fixation point probability map loss optimization based on the fixation point true probability map;

[0075] The joint optimization module performs a weighted summation of the inverse generation process loss and the gaze point probability map loss, and supervises the training of the end-to-end model consisting of the video frame acquisition and preprocessing module, the feature extraction and feature fusion module, the target tracking module, the missing frame reconstruction and future frame prediction module, and the gaze point localization module through a gradient optimization method;

[0076] The video output module, given a target image and video streams from multiple cameras, reconstructs missing video frames or predicts future video frames through an inverse generation process; inputs a gaze point position and a specified time window, and generates a target trajectory that meets the gaze point constraints within the specified time window.

[0077] This case achieves the spatiotemporal consistency of multi-camera target trajectories, and can accurately infer the motion path of the target when it is partially occluded, significantly improving the robustness of cross-camera tracking. This case can also distinguish between physical occlusion and target loss, avoiding tracking interruptions caused by misjudgment, and can help determine the location where the target disappears.

[0078] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the above embodiments do not limit the present invention in any form, and any technical solutions obtained by equivalent replacement or equivalent transformation fall within the scope of protection of the present invention.

Claims

1. A multi-camera collaborative single target tracking method, characterized by: The steps include: Step 1: Input the video stream collected by each camera, extract the background features of each camera, and then identify the field of view of each camera; based on the field of view of each camera, construct the spatial matrix of each camera in the monitoring area; Step 2: Input the target image and extract the target's facial feature vector; identify the target in the video stream of each camera based on the target's facial feature vector and extract the target's body feature vector; perform feature fusion on the target's facial feature vector and body feature vector to obtain a fused feature vector; Step 3: Track the target in the video stream of each camera based on the fused feature vector and construct the target trajectory in the monitoring area; Step 4: Analyze the missing frames of each camera based on the target trajectory and spatial matrix, add noise to the missing frames through the forward diffusion method, and then reconstruct the missing frames or predict future frames through the inverse generation method. Use the trajectory similarity algorithm to calculate the correlation of the target trajectory in different cameras, and introduce spatiotemporal constraints and trajectory smoothing algorithms to optimize the coherence and rationality of the target trajectory in different cameras. Step 5: Input the fixation point location and specify the time window. The encoder and saliency prediction module output the fixation point saliency probability map, and optimize the fixation point probability map loss based on the true fixation point probability map. Step 6: Jointly optimize the inverse generation process loss and the gaze point probability map loss to generate a video with the specified gaze point position within the specified time window.

2. The multi-camera coordinated single target tracking method according to claim 1, characterized in that: In the Step 2, the target image is input and the target's facial feature vector F0 is extracted; based on the target's facial feature vector F0, the target is identified in the video stream of each camera and the target's body feature vector G0 is extracted; the facial feature vector F0 and the body feature vector G0 are fused by the feature fusion module to obtain the fused feature vector F (0) ; The two inputs of the feature fusion module are firstly subjected to layer normalization, then feature fusion is performed through the cross-attention mechanism, and then channel adjustment is performed through the multi-layer perceptron: F (0) =MLP(CrossAttn(L(F0),L(G0))) Where: L represents layer normalization, CrossAttn represents cross attention mechanism, MLP represents multi-layer perceptron; gradient descent method is used to optimize the fusion feature vector F (0) .

3. The multi-camera coordinated single target tracking method according to claim 1, characterized in that: In Step 1, the spatial matrix of each camera in the monitoring area is represented by R={R xy }, R xy Indicates the camera sequence whose field of view includes the (x, y) position; in Step 3, the target trajectory in the monitoring area is represented by S = {S1, S2, ..., S t ,…},S t represents the position of the target at time t, t = 1, 2, ...; in Step 4, based on the spatial matrix R and the target trajectory S, the video frames that should contain the target in the field of view at time t are analyzed, and the video frames that should contain the target and actually contain the target are judged as normal frames, and a normal frame set N at time t is constructed. t , the video frames that should contain the target but do not actually contain the target are judged as missing frames, and the missing frame set D at time t is constructed t ; Normal frame set N t and the reconstructed frame set C t Together they constitute the set A of all frames at time t t ; Use all frame sets A at time t-1 t-1 And the normal frame set N at time t t For the missing frame set D at time t t Perform target reconstruction and obtain the reconstructed frame set C at time t t ; Use all frame sets A at time t-1 t-1 For all frames A at time t t Perform target reconstruction to obtain the predicted frame set P at time t t .

4. The multi-camera coordinated single target tracking method according to claim 3, characterized in that: In Step 4, the missing frame set D is obtained by forward diffusion. t Add noise to the missing frames in the video. If the video frame of the i-th camera at time t is a missing frame The video frame of the i-th camera at time t-1 is Video frame Add Gaussian noise and get the noise frame of the i-th camera at time t: Among them, σ 2 (t) and σ 2 (t-1) represents the noise variance at time t and time t-1 respectively, I represents the identity matrix; N represents the Gaussian distribution; the forward diffusion method is simplified to a stochastic differential equation: Where: f(t) represents the drift term, which is used to describe the video frame The deterministic change of g(t) represents the sound intensity; w represents the standard Brownian motion; All missing frames Corresponding noise frame Constructing the noise frame set Q t , concatenate normal frame set N t and noise frame set Q t Get Collection To B t Reverse generation to obtain E t ;like Then A t =P t =E t ;like Using N t Replace E t The video frames with the same camera index in A are obtained t ; Simplify the inverse generation method to a stochastic differential equation: Where: g 2 (t) represents the square of the noise intensity, which is used to ensure the stability of the inverse generation; logp(E t |B t ) represents the logarithm of the conditional probability distribution; It is an additional correction term generated in reverse, which is used to correct the evolution direction of the data so that the data gradually moves towards the reconstructed frame set E t Close; g(t)dw is the diffusion term generated in reverse, representing random noise.

5. The multi-camera coordinated single target tracking method according to claim 4, characterized in that: Use the diffusion model to train the reverse generation process, with B t As a condition, learn based on the noise frame set Q t Reconstruct missing frames or predict future frames; The reconstruction loss is minimized via gradient descent.

6. The multi-camera coordinated single target tracking method according to claim 3, characterized in that: In Step 5, the gaze point position is input and a time window is specified. The video stream reconstructed by each camera is intercepted according to the time window. The gaze point position and the video stream are input into the gaze point positioning module. First, the encoder is used to extract deep features of each video stream and fuse them into multi-level spatial-temporal features to obtain the key visual area in the video content at the gaze point position and generate a multi-scale feature map. Then, the saliency prediction module is used to calculate the gaze point saliency probability map of the multi-scale feature map, and the binary cross entropy loss between the gaze point saliency probability map and the true gaze point probability map is calculated to optimize the gaze point probability map loss.

7. A multi-camera coordinated single target tracking device, characterized by: It includes video frame acquisition and preprocessing module, feature extraction and feature fusion module, target tracking module, missing frame reconstruction and future frame prediction module, gaze point positioning module, joint optimization module and video output module; The video frame acquisition and preprocessing module extracts the background features of each camera from the video stream collected by each camera, and then identifies the field of view of each camera; based on the field of view of each camera, it constructs a spatial matrix of each camera in the monitoring area; The feature extraction and feature fusion module extracts the target's facial feature vector; identifies the target in the video stream of each camera based on the target's facial feature vector, and extracts the target's body feature vector; performs feature fusion on the target's facial feature vector and body feature vector to obtain a fused feature vector; The target tracking module tracks the target in the video stream of each camera based on the fused feature vector and constructs the target trajectory in the monitoring area; The missing frame reconstruction and future frame prediction module analyzes the missing frames of each camera based on the target trajectory and spatial matrix, adds noise to the missing frames through the forward diffusion method, and then reconstructs the missing frames or predicts future frames through the inverse generation method. It uses a trajectory similarity algorithm to calculate the correlation of target trajectories in different cameras, and introduces spatiotemporal constraints and a trajectory smoothing algorithm to optimize the coherence and rationality of target trajectories in different cameras. The fixation point localization module inputs the fixation point location and specifies the time window, outputs the fixation point saliency probability map through the encoder and saliency prediction module, and performs fixation point probability map loss optimization based on the fixation point true probability map; The joint optimization module performs a weighted summation of the inverse generation process loss and the gaze point probability map loss, and supervises the training of the end-to-end model consisting of the video frame acquisition and preprocessing module, the feature extraction and feature fusion module, the target tracking module, the missing frame reconstruction and future frame prediction module, and the gaze point localization module through a gradient optimization method; The video output module, given a target image and video streams from multiple cameras, reconstructs missing video frames or predicts future video frames through an inverse generation process; inputs a gaze point position and a specified time window, and generates a target trajectory that meets the gaze point constraints within the specified time window.

8. The multi-camera coordinated single target tracking device according to claim 7, characterized in that: The feature extraction and feature fusion module includes a facial feature extraction unit, a body feature extraction unit and a feature fusion unit; the facial feature extraction unit uses a pre-trained convolutional neural network to extract the target facial feature vector; the body feature extraction unit locates the target based on the target detection algorithm, extracts features including bounding boxes and postures, and generates body feature vectors through a fully connected layer; the feature fusion unit includes two parallel layer normalization layers, a cross attention mechanism and a multi-layer perceptron.

Images