Multi-camera 3D reconstruction-driven target identity and trajectory localization method

Abstract

This invention relates to the fields of computer vision and 3D graphics technology, specifically disclosing a target identity and trajectory localization method driven by multi-camera 3D reconstruction. The method involves acquiring video sequences through a synchronous camera network and constructing an implicit representation model containing only static background structures to establish a stable 3D coordinate system. Then, target detection is performed on the real-time video stream, and the 2D coordinates of the detection points are back-projected and the accurate 3D spatial coordinates are calculated by querying the static model. Finally, the target's motion state, appearance, and behavioral characteristics are fused to perform multi-target tracking and identity association, generating a 3D motion trajectory with a consistent identity identifier.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and 3D graphics technology, specifically to a method for target identity and trajectory localization driven by multi-camera 3D reconstruction. Background Technology

[0002] With the rapid development of computer vision and intelligent monitoring technologies, 3D scene perception and target tracking based on multi-camera networks have become key technologies in fields such as security, autonomous driving, and intelligent transportation. Existing technologies typically employ structure-of-motion reconstructing (SCOR) or multi-view stereo vision methods to reconstruct the 3D structure of a scene from multi-view videos, then combine this with target detection and re-identification techniques to track the trajectory of a specific target in the reconstructed 3D space. However, the core of these methods lies in their heavy reliance on visual feature points extracted from video sequences for 3D reconstruction, and multi-target tracking largely depends on associating appearance features with short-term motion models in the 2D image domain, finally mapping them to 3D space using techniques such as triangulation. This approach performs reasonably well in ideal scenarios with static or sparse crowds, but its limitations become glaringly apparent when facing highly dynamic, high-density real-world scenarios with complex interactions, leading to a sharp decline in the reliability of localization and tracking results.

[0003] In high-density dynamic scenes (such as transportation hubs), the visual features relied upon for 3D reconstruction largely originate from the moving pedestrians themselves, leading to distortion and drift in the reconstructed 3D scene reference frame. Simultaneously, multi-target tracking and association strategies based on appearance similarity and local motion smoothness systematically generate "silent identity-trajectory exchange" to maintain trajectory continuity when targets undergo prolonged occlusion and interaction. Specifically, when two similar-looking targets (such as passengers A and B) move parallel to each other in dense crowds and occlude, the system may output two continuous and smooth trajectories in 3D space. However, the identity information carried by these trajectories is silently exchanged at a certain point in time (i.e., trajectory A' subsequently corresponds to passenger B, while B' corresponds to passenger A). This error is logically self-consistent within the system and difficult to detect and correct through conventional post-processing. However, it can cause fundamental misjudgments in applications such as security tracing, constituting a deep bottleneck restricting the high-reliability practical application of existing technologies. This invention is proposed to fundamentally solve this thorny problem. Summary of the Invention

[0004] The purpose of this invention is to provide a target identity and trajectory localization method driven by multi-camera 3D reconstruction to solve the problems mentioned above.

[0005] The objective of this invention can be achieved through the following technical solutions:

[0006] A multi-camera 3D reconstruction-driven target identification and trajectory localization method includes the following steps:

[0007] S1: Acquire a synchronized video sequence of the scene using multiple pre-positioned synchronized cameras;

[0008] S2: Extract a continuous frame sequence from the synchronized video sequence, and perform dense feature extraction on each frame image to obtain visual features including color, geometric structure and semantic information;

[0009] S3: Based on visual features, the scene is spatiotemporally modeled, and by introducing the time dimension and decoupling static and dynamic components, a time-independent implicit representation model of the static scene is generated.

[0010] S4: Perform target detection on the real-time video stream, back-project the two-dimensional positioning reference point of the detected target into the observation ray, calculate the intersection point of the ray and the static background structure by querying the implicit representation model of the static scene, and obtain the three-dimensional spatial coordinates of each target in each frame.

[0011] S5: Based on the three-dimensional spatial coordinate sequence of each target, and simultaneously integrate the target's motion state features, appearance features and behavior features, multi-target tracking and identity association are performed. The accuracy of identity recognition is enhanced by the complementarity of multi-modal features, and three-dimensional motion trajectories with consistent identity identifiers are generated for each target.

[0012] As a further aspect of the present invention: the process of acquiring the visual features is as follows:

[0013] A multi-scale feature extraction network is constructed, which extracts visual features at different abstraction levels from the input image frame through multiple feature extraction branches connected in parallel.

[0014] The first branch captures global geometric context information, the second branch preserves local color and texture details, and the third branch focuses on regions with semantic information.

[0015] Adaptive fusion of feature maps output from each branch is performed, and the contribution of features from different branches is dynamically adjusted through learnable weight parameters to generate a unified dense feature map containing color, geometric structure and semantic information.

[0016] By leveraging the spatiotemporal continuity between consecutive frames, a smoothing constraint is applied to the dense features at corresponding positions in adjacent frames to extract temporally stable visual features.

[0017] As a further aspect of the present invention: the spatiotemporal joint modeling of the scene based on visual features specifically includes:

[0018] A spatiotemporal coding network is constructed. The input of the spatiotemporal coding network is a multi-scale feature pyramid based on visual features, and the output is a spatiotemporal joint representation of the scene.

[0019] By designing a time-aware feature aggregation module, visual features in a continuous frame sequence are aligned and fused along the time dimension;

[0020] We weight the feature contributions of different time steps and spatial locations to preserve the stable structural features in the scene; we then deeply fuse the processed temporal features with the original spatial features to generate a joint representation that includes the spatiotemporal characteristics of the scene.

[0021] As a further aspect of the present invention: the generation of the time-independent static scene implicit representation model specifically includes:

[0022] Based on spatiotemporal joint representation, a static-dynamic decoupled network is constructed, which includes parallel static and dynamic branches. In the static branch, time invariance constraints are applied to ensure that the scene representation output by the static branch does not change over time.

[0023] In the dynamic branch, the changing components in the scene are extracted, and the spatial distribution range of the dynamic components is restricted; the purely static scene representation is gradually separated from the complete spatiotemporal representation, first processing the significantly static region, and then processing the dynamic-static boundary region;

[0024] The output parameters of the static branch are solidified into a lightweight implicit representation model of the static scene.

[0025] As a further aspect of the present invention: the step of back-projecting the two-dimensional positioning reference point of the detected target into an observation ray specifically includes:

[0026] Based on camera parameters, the two-dimensional positioning reference point of each detected target is converted into a three-dimensional observation ray originating from the camera optical center;

[0027] By analyzing the motion trajectory of the same target in adjacent frames, the ray direction is smoothed and optimized.

[0028] By utilizing the geometric property that the reverse rays of the same target under different viewpoints should be coplanar and intersect at a point in three-dimensional space, the initial projected rays are jointly optimized.

[0029] As a further aspect of the present invention: the analysis to obtain the three-dimensional spatial coordinates of each target in each frame specifically includes:

[0030] Adaptive sampling is performed along the observed ray in the implicit representation model of the static scene, and the precise intersection position of the ray and the scene surface is determined by evaluating the spatial occupancy probability of the sampling points.

[0031] First, potential intersection areas are quickly located at a coarse-grained level, and then the coordinates of the intersection points are accurately calculated at a fine-grained level.

[0032] By leveraging the smoothness of the target's motion between adjacent frames, the 3D coordinate estimation results of the current frame are optimized and corrected.

[0033] By comparing the consistency between the three-dimensional coordinates calculated from different perspectives, abnormal positioning results are eliminated, and finally accurate three-dimensional spatial coordinates are output.

[0034] As a further aspect of the present invention: the multi-target tracking and identity association based on the three-dimensional spatial coordinate sequence of each target, while simultaneously fusing the target's motion state features, appearance features, and behavioral features, specifically includes:

[0035] The three-dimensional spatial coordinate sequence of each target is represented as a node in the figure. The node attributes include the target's motion state characteristics, appearance characteristics, and behavior characteristics.

[0036] The motion consistency, appearance similarity, and behavior matching degree of target nodes between adjacent frames are taken into account.

[0037] Rapid coarse association is performed based on motion state features to filter candidate matching pairs, and then fine association is performed by fusing appearance features and behavioral features.

[0038] Finally, a graph optimization algorithm is used to solve for the optimal identity association result in the global spatiotemporal scope, ensuring the continuity of identity during the tracking process.

[0039] As a further aspect of the present invention: the generation of three-dimensional motion trajectories with consistent identity identifiers for each target specifically includes:

[0040] The weight ratios of motion state features, appearance features, and behavior features in identity determination are adaptively adjusted based on scene complexity and occlusion conditions.

[0041] By leveraging the target's historical movement patterns and current behavioral characteristics, we can predict its future state and enhance the temporal consistency of identity association.

[0042] When the results of multimodal feature discrimination are contradictory, the optimal identity determination result is selected through confidence assessment and spatiotemporal consistency verification.

[0043] Finally, through trajectory smoothing and post-processing, identity jumps are eliminated, and three-dimensional motion trajectories with spatiotemporal consistency are generated for each target.

[0044] The beneficial effects of this invention are:

[0045] (1) In high-density dynamic scenes (such as airports and train stations), traditional multi-camera 3D tracking methods suffer from coordinate system drift due to the reliance on feature points containing moving targets for 3D reconstruction. Furthermore, when multiple targets are occluded for extended periods and have similar appearances, a logically consistent but physically flawed "silent identity-trajectory exchange" is easily generated. This invention introduces a spatiotemporal semantic static prior to construct a time-independent, pure, static scene implicit representation model, providing a stable and unchanging world coordinate system for all calculations, thus eliminating reference system distortion at its source. Based on this, through deep fusion and global temporal optimization of multimodal features (motion, appearance, behavior), behavioral features (such as gait) decoupled from appearance are introduced into the data association decision-making process. Graph optimization algorithms are used for joint reasoning over long time periods, effectively identifying and correcting identity association errors that are unavoidable in traditional methods. This ensures high reliability of identity-trajectory binding even in extremely complex scenarios.

[0046] (2) Through a series of meticulous designs, this invention improves performance while ensuring the practicality of the solution. First, in terms of accuracy and robustness, a static scene representation model based on neural implicit fields and a ray tracing query mechanism are adopted, enabling 3D positioning accuracy to reach the centimeter level and making it insensitive to dynamic interference in the scene (such as pedestrian flow). Temporal consistency constraints in feature extraction and multimodal confidence evaluation in trajectory generation further enhance the overall solution's anti-interference capability. Second, in terms of efficiency, through hierarchical processing strategies (such as coarse-fine correlation in feature extraction and hierarchical sampling in coordinate analysis) and model solidification techniques (parameter pruning, quantization, and distillation), computational and storage overhead is reduced, meeting the needs of large-scale real-time monitoring applications. Attached Figure Description

[0047] The invention will now be further described with reference to the accompanying drawings.

[0048] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0049] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] Please see Figure 1 As shown, this invention is a target identity and trajectory localization method driven by multi-camera 3D reconstruction, comprising the following steps:

[0051] S1: Acquire a synchronized video sequence of the scene using multiple pre-positioned synchronized cameras;

[0052] S2: Extract a continuous frame sequence from the synchronized video sequence, and perform dense feature extraction on each frame image to obtain visual features including color, geometric structure and semantic information;

[0053] S3: Based on visual features, the scene is spatiotemporally modeled, and by introducing the time dimension and decoupling static and dynamic components, a time-independent implicit representation model of the static scene is generated.

[0054] S4: Perform target detection on the real-time video stream, back-project the two-dimensional positioning reference point of the detected target into the observation ray, calculate the intersection point of the ray and the static background structure by querying the implicit representation model of the static scene, and obtain the three-dimensional spatial coordinates of each target in each frame.

[0055] S5: Based on the three-dimensional spatial coordinate sequence of each target, and simultaneously integrate the target's motion state features, appearance features and behavior features, multi-target tracking and identity association are performed. The accuracy of identity recognition is enhanced by the complementarity of multi-modal features, and three-dimensional motion trajectories with consistent identity identifiers are generated for each target.

[0056] In S1, a synchronized video sequence of the scene is acquired through multiple pre-positioned synchronized cameras, specifically including:

[0057] Based on the actual spatial dimensions and specific monitoring requirements of the surveillance scene, determine the number and spatial distribution of cameras. A data acquisition network will be constructed using no fewer than four high-resolution industrial-grade cameras. The spatial distribution of these cameras will adhere to the following basic principles: any spatial point in the scene must be simultaneously covered by at least three cameras with different perspectives, and the overlap of the field of view angles between any two adjacent cameras should be maintained between 30% and 50%. All cameras will be equipped with global shutter sensors to completely eliminate the rolling shutter effect that may occur when capturing fast-moving objects. Camera lens selection will be based on the monitoring distance: for medium-to-close-range surveillance scenarios within 10 meters, a 6mm fixed-focus lens will be used; for long-range surveillance scenarios exceeding 10 meters, a 12mm fixed-focus lens will be used.

[0058] To achieve strict frame-level synchronized acquisition, a master-slave hardware synchronization scheme is adopted. One camera is designated as the master device, and the rest are slave devices. A dedicated synchronization signal cable transmits the frame trigger pulse signal generated by the master device to all slave devices. The frequency of this pulse signal is strictly consistent with the video acquisition frame rate, set to 30 frames per second. At the rising edge of each synchronization pulse, all cameras simultaneously begin the exposure process for a single frame image. This design ensures that all cameras capture scene images at the same physical moment at any given time. To verify the actual synchronization accuracy, a high-frequency flashing LED is placed in the scene and kept continuously running. By comparing the on / off state of this LED in all video streams to ensure complete consistency, the system synchronization error is confirmed to be less than 1 millisecond.

[0059] Precise calibration of camera parameters was achieved using a specially designed two-dimensional planar calibration board. This board contains an 8×6 black and white checkered pattern, with each square having an actual physical size of 30 mm × 30 mm. Within the field of view of each camera, the calibration board was displayed in at least 10 different poses and positions, ensuring that the image of the calibration board was clear and occupied a significant area of ​​the screen each time. Using the classic Zhang Zhengyou calibration method, the intrinsic parameter matrix and distortion coefficient set for each camera were calculated. The intrinsic parameter matrix is ​​a 3×3 matrix that fully describes the camera's core intrinsic properties such as focal length and principal point coordinates; the distortion coefficients are a coefficient vector containing three radial distortion parameters and two tangential distortion parameters, used to accurately correct various nonlinear distortions of the lens. After obtaining the internal parameters of all cameras, the calibration board is fixed in the scene, and images of all cameras relative to the calibration board are acquired synchronously. By solving the Perspective-n-Point geometry problem, the external parameters of each camera relative to the common world coordinate system are calculated, namely a 3×3 rotation matrix and a 3×1 translation vector, so as to achieve accurate spatial registration of all cameras in a unified world coordinate system.

[0060] Continuous acquisition time is no less than 5 minutes to ensure sufficient and diverse human movement and activity patterns in the scene are captured. The acquired raw video data is stored in a lossless compression format. In the preprocessing stage, geometric distortion correction is first performed on each frame using the calculated distortion coefficients to eliminate image distortion introduced by the lens. Subsequently, white balance and color consistency adjustments are performed to ensure consistent color reproduction across all cameras. This adjustment is achieved by photographing a 24-color standard color chart and establishing a 3×3 mapping matrix from the color space of each camera to the standard color space. Finally, a multi-view synchronized video sequence with strictly aligned timestamps and precisely corrected geometric and color information is generated. This sequence will serve as direct input data for the subsequent construction of the implicit representation model of the static scene.

[0061] In S2, a continuous frame sequence is extracted from the synchronized video sequence, and dense feature extraction is performed on each frame to obtain visual features including color, geometric structure, and semantic information, specifically including:

[0062] Design a neural network architecture with three parallel branches, each responsible for extracting visual features at different levels of abstraction. The first branch employs a combination of convolutional layers with a large receptive field, using five consecutive convolutional layers. The first three layers have a kernel size of 5×5, and the last two layers have a kernel size of 3×3. This design captures the global geometric context information of the image, focusing on extracting the overall layout and macroscopic structural features of the scene. The second branch uses a densely connected convolutional structure with eight convolutional layers. All convolutional kernel sizes alternate between 1×1 and 3×3, maintaining high feature resolution and specifically designed to preserve local color variations and subtle texture details. The third branch introduces a self-attention computation unit. It first extracts basic features through a 3×3 convolutional layer, then calculates the association weights between different locations in the feature map. Based on the weight distribution, it focuses on regions in the image with rich semantic information, such as scene elements like doors, windows, and fixtures.

[0063] In the first branch, a spatial pyramid pooling layer is added after the last convolutional layer. Four pooling windows of different scales are used to downsample the feature map. Then, the features at different scales are upsampled to the original size and concatenated to form multi-scale geometric context features. The second branch adds shortcut connections between every two convolutional layers to ensure effective gradient propagation. Channel attention is calculated in the last convolutional layer, and the learned weight vectors are used to recalibrate each feature channel, highlighting important color and texture features. The third branch generates an attention weight map by calculating the similarity score between each location in the feature map and all other locations. The original features are multiplied by the weight map to obtain enhanced semantic features. The similarity calculation uses a dot product method, and the result is processed using a normalized exponential function.

[0064] The feature maps output from the three branches are concatenated along the channel dimension to form a temporary fused feature. A learnable weight adjustment structure is designed. This structure first performs global average pooling on the temporary fused feature, compressing the feature map of each channel into a single value. Then, it learns the importance weights of each channel through two fully connected layers. The first fully connected layer reduces the number of channels to 1 / 4 of the original number, and the second fully connected layer restores the original number of channels. Finally, a non-linear activation function is used to map the output weights to a range of 0 to 1. The learned weights are multiplied channel by channel of the corresponding feature map to dynamically adjust the contribution of features from different branches. During the weight learning process, an additional weight distribution balancing constraint term is added to ensure that no single branch's features completely dominate the fusion result.

[0065] By leveraging the spatiotemporal continuity between consecutive video frames, additional constraints are imposed on the feature extraction process. For two adjacent frames, an optical flow estimation algorithm is first used to calculate the inter-frame motion field. Based on this motion field, the feature map of the previous frame is warped into the coordinate system of the current frame. Then, the difference between the warped features and the actual extracted features of the current frame is calculated using a pixel-wise mean squared error method. This difference is used as part of the loss function to optimize the parameters of the feature extraction network during training, prompting the network to extract feature representations that are insensitive to small motion changes. Simultaneously, a temporal smoothness constraint is constructed in the feature space, requiring that the distance between corresponding feature vectors in adjacent frames in the feature space should be less than a set threshold. This threshold is initially set to 0.1 and dynamically adjusted during the training process.

[0066] The training of the entire feature extraction network is divided into two stages. The first stage uses a large image classification dataset for pre-training to learn general visual feature representations. The second stage fine-tunes the network on synchronized video sequences of the target scene, using a weighted sum of reconstruction loss and temporal consistency loss as the optimization objective, with the reconstruction loss weight set to 1.0 and the temporal consistency loss weight set to 0.5. The optimization process uses stochastic gradient descent with momentum, a momentum parameter set to 0.9, and an initial learning rate of 0.001, which decays to 50% every 10 training epochs. A weight decay strategy is employed during training to prevent overfitting, with a weight decay coefficient set to 0.0001. This carefully designed training strategy ensures that the final extracted visual features possess both rich semantic information and good spatiotemporal consistency.

[0067] Post-processing is performed on the dense feature maps output by the network. First, the feature vectors at each spatial location are normalized to a magnitude of 1 to eliminate the influence of brightness variations on feature representation. Then, the mean and variance of all feature vectors in the feature map are calculated, and these statistics are used to standardize the feature distribution, ensuring that features extracted from different images have similar distribution characteristics. Finally, to reduce storage and computational overhead, the feature maps are appropriately downsampled with a downsampling factor of 2, improving processing efficiency while preserving important information. Through these processing steps, dense feature maps of uniform size are obtained. These feature maps simultaneously encode color, geometric structure, and semantic information, providing reliable input for subsequent construction of implicit representation models for static scenes.

[0068] In S3, spatiotemporal joint modeling of scenes is performed based on visual features. By introducing the time dimension and decoupling static and dynamic components, a time-independent implicit representation model of the static scene is generated, specifically including:

[0069] First, a spatiotemporal coding network is constructed, employing an encoder-decoder structure containing 12 convolutional layers and 3 deconvolutional layers. The encoder uses 6 convolutional layers, each with a 3×3 kernel size and a stride of 2, reducing the input feature map size to 1 / 64 of its original size through layer-by-layer downsampling. The decoder uses 3 deconvolutional layers, each with a 4×4 kernel size and a stride of 2, progressively restoring the spatial resolution of the feature map. The network input is a multi-scale feature pyramid constructed based on visual features. This pyramid contains feature maps at four different scales, with the largest scale feature map having a spatial size of 1 / 2 of the original image and the smallest scale having a spatial size of 1 / 16 of the original image. Each scale feature map contains 256 channels of visual features.

[0070] The time-aware feature aggregation module employs a bidirectional recurrent neural network structure, comprising forward and backward processing directions. Forward processing uses a 3-layer gated recurrent unit with a hidden layer dimension of 512; backward processing uses the same network structure. At each time step, the visual features of the current frame are fused with the hidden state of the previous time step, with information flow controlled by update and reset gates. The update gate uses the sigmoid function to map input values ​​between 0 and 1, and the reset gate uses the same calculation method. During feature alignment, deformable convolution based on optical flow is used, adjusting the sampling position of the convolution kernel at each spatial location according to the optical flow vector to ensure accurate feature alignment across different time steps.

[0071] The feature contribution weighting process employs a spatiotemporal attention map containing two independent branches: a temporal attention branch and a spatial attention branch. The temporal attention branch calculates the importance weight for each time step by performing global average pooling on the feature sequences from 16 consecutive time steps, followed by learning the weight distribution through two fully connected layers. The first fully connected layer has an output dimension of 64, and the second fully connected layer has the same output dimension as the number of time steps. The spatial attention branch calculates the importance weight for each spatial location by compressing the number of feature channels to 1 using a 1×1 convolution, and then generating a spatial weight map using a sigmoid function. Finally, the temporal and spatial weights are multiplied to obtain the complete spatiotemporal attention map.

[0072] The feature fusion stage employs a progressive fusion strategy, divided into three levels. The first level of fusion concatenates the temporally aggregated features with the original spatial features, then reduces the number of channels to 512 using a 1×1 convolution. The second level of fusion introduces residual connections, adding the first-level fusion result to the input features to ensure no important information is lost. The third level of fusion uses a feature pyramid structure to upsample and concatenate the fused features at different scales, ultimately generating a joint representation containing the spatiotemporal characteristics of the scene. This joint representation contains 512 channels, with a spatial size one-quarter that of the original image.

[0073] The specific implementation process of generating implicit representation models for static scenes:

[0074] Based on spatiotemporal joint representation, a static-dynamic decoupled network is constructed, comprising two parallel branches. The static branch uses eight convolutional layers, each with a 3×3 kernel size, maintaining the feature map spatial dimensions unchanged. Special constraint design ensures the output does not change over time. The dynamic branch employs a similar network structure, but uses dilated convolutions in the last three layers with dilation coefficients of 2, 4, and 8 to increase the receptive field and better capture motion changes in the scene.

[0075] In the static branch, time invariance constraints are implemented in two ways. The first is to apply a consistency loss to the output of the same spatial point at different time steps during training, calculating the variance of the output features over five consecutive time steps, requiring this variance to be less than a threshold of 0.05. The second is to add an instance normalization layer to the network structure to remove time-dependent statistics from the features. In the dynamic branch, sparsity constraints restrict the spatial distribution of dynamic components. An L1 regularization term is added to the loss function, with a weight coefficient set to 0.01, causing the dynamic features to approach zero in most regions.

[0076] Static-dynamic separation employs a progressive strategy, completed in three stages. The first stage handles salient static regions using a predefined staticity scoring function, calculated based on the temporal stability of features, with a stability threshold set to 0.8. The second stage addresses the boundary between static and dynamic regions using a conditional random field model, iteratively optimizing to determine the static probability of each pixel. The third stage performs fine-grained separation using an adversarial training strategy, training a discriminator network to distinguish between static and dynamic features, and improving separation performance by minimizing the adversarial loss.

[0077] The model parameter solidification process consists of four steps. First, parameter pruning is performed, removing connections with weights whose absolute values ​​are less than 0.001. Next, quantization is performed, converting 32-bit floating-point parameters to 8-bit integers while adding quantization error compensation terms. Then, knowledge distillation is performed, using the original large network to guide the training of a lightweight network, with the distillation temperature parameter set to 3. Finally, model compression is performed, using low-rank decomposition to decompose the fully connected layer parameter matrix into the product of two smaller matrices, with a compression ratio set to 0.5.

[0078] Network training and optimization:

[0079] The training process consists of three stages. The first stage trains the spatiotemporal coding network, using a weighted sum of reconstruction loss and temporal consistency loss as the objective function, with a reconstruction loss weight of 1.0 and a temporal consistency loss weight of 0.5. The second stage trains the static-dynamic decoupling network, adding static consistency loss and dynamic sparsity loss to the original loss function, with weights of 0.8 and 0.3 respectively. The third stage performs joint fine-tuning, readjusting the weights of all loss functions using an adaptive weight adjustment strategy, dynamically adjusting the weights based on the convergence of each loss term.

[0080] The optimizer uses the Adam algorithm, with a first-order moment decay factor of 0.9 and a second-order moment decay factor of 0.999. The initial learning rate is set to 0.001, and it decays to 0.7 every 20 training epochs. The batch size is set to 8, and gradient clipping is used to prevent gradient explosion. The gradient threshold is set to 1.0. The total number of training epochs is set to 200, and training is terminated early if the loss function value decreases by less than 1% for 10 consecutive epochs.

[0081] Model validation and tuning:

[0082] A complete verification index system was established, including six indicators such as static performance score, reconstruction quality, and temporal consistency. The static performance score is calculated by measuring the coefficient of variation of the model output at different time steps, which must be less than 0.1. Reconstruction quality is evaluated using peak signal-to-noise ratio (PSNR), which must be greater than 30 dB. Temporal consistency is evaluated by calculating the optical flow consistency error between adjacent frames, which must be less than 0.5 pixels.

[0083] In S4, target detection is performed on the real-time video stream. The two-dimensional positioning reference points of the detected targets are back-projected as observation rays. The intersection points of the rays and the static background structure are calculated by querying the implicit representation model of the static scene. The three-dimensional spatial coordinates of each target in each frame are then obtained. Specifically, this includes:

[0084] First, a deep learning-based object detection network is used to process the real-time video stream. This network consists of three main parts: a backbone network, a feature fusion network, and a detection head. The backbone network uses a residual network structure with 50 convolutional layers. With an input image size of 1920×1080, the output feature map size is 60×34, and the number of channels is 1024. The feature fusion network uses a feature pyramid structure, upsampling and concatenating feature maps from different levels of the backbone network to form a multi-scale feature map with 256 channels. The detection head uses three parallel branches to predict the target bounding box, class confidence, and keypoint location, respectively.

[0085] After acquiring the target detection results, the bottom midpoint of the bounding box of each detected target is selected as a 2D localization reference point. Based on the intrinsic and extrinsic parameter matrices obtained from camera calibration, the 2D localization reference point is converted into a 3D observation ray. The intrinsic parameter matrix is ​​a 3×3 matrix containing focal length and principal point coordinate parameters; the extrinsic parameter matrix is ​​a 4×4 transformation matrix containing rotation and translation parameters. The specific transformation process first transforms the image coordinates to the camera coordinate system using the inverse of the intrinsic parameter matrix, and then transforms them to the world coordinate system using the inverse of the extrinsic parameter matrix, obtaining the direction vector originating from the camera's optical center.

[0086] The observation ray smoothing optimization employs a Kalman filter-based method, using six state variables to describe the target motion, including three-dimensional position and velocity. The state transition matrix uses a uniform motion model, and the observation matrix establishes the relationship between the two-dimensional image coordinates and the three-dimensional state. Through prediction and update steps, the direction vector of the observation ray is filtered, with a smoothing coefficient set to 0.8, effectively reducing the impact of detection noise.

[0087] A multi-view geometric joint optimization was used to establish the objective function, which included three constraints: a coplanar constraint requiring that observation rays from different cameras should lie on the same plane; an intersection constraint requiring that these rays intersect at a single point in 3D space; and a motion smoothing constraint requiring that the localization results of adjacent frames should be continuous and smooth. The weights of the coplanar constraint, intersection constraint, and motion smoothing constraint were set to 1.0, 1.2, and 0.6, respectively. The Levenberg-Marquardt method was used to solve for the optimal solution, with 50 iterations and a convergence threshold of 0.001.

[0088] The specific implementation process of three-dimensional spatial coordinate analysis:

[0089] Adaptive sampling is performed along the observation ray in the implicit representation model of the static scene. The sampling process is divided into two stages. The first stage involves uniform sampling, starting from 0.5 meters away from the camera's optical center along the ray direction and ending at 50 meters, with sampling intervals of 0.1 meters, for a total of 495 sampling points. The second stage involves fine sampling in the initially determined intersection area, with the sampling interval reduced to 0.01 meters. The number of sampling points is dynamically adjusted according to the size of the area, with a maximum of 200 sampling points.

[0090] Spatial occupancy probability assessment uses a pre-trained static scene implicit representation model to calculate the spatial occupancy probability value for each sampled point. This model receives the 3D coordinates of the sampled point as input, processes them through eight fully connected layers (each containing 256 neurons), uses the ReLU activation function, and finally outputs a probability value between 0 and 1 using the sigmoid function. A probability value greater than 0.5 indicates that the point is occupied by the scene surface; a higher probability value indicates that the point is more likely to be located on the scene surface.

[0091] The hierarchical localization strategy first rapidly identifies potential intersection areas at a coarse-grained level. This coarse-grained assessment uses low-resolution sampling at 0.5-meter intervals, calculating the spatial occupancy probability of each sampling point and identifying continuous regions with probability values ​​exceeding a threshold of 0.3. Then, at a fine-grained level, dense sampling is performed within the identified potential regions at 0.01-meter intervals. A probability distribution curve is fitted using cubic spline interpolation to find the location with the highest probability value, which is then used as the precise intersection point coordinates.

[0092] Motion smoothness constraints utilize the target's motion characteristics between adjacent frames to establish a motion model that corrects the coordinate estimation for the current frame. A second-order motion model is used, including three state variables: position, velocity, and acceleration. By performing polynomial fitting on the coordinate sequence of the previous five frames, the target position in the current frame is predicted. The prediction result is then weighted and fused with the observation result, with the weight coefficients dynamically adjusted according to the prediction error, ranging from a maximum weight of 0.7 to a minimum weight of 0.3.

[0093] Multi-view consistency verification compares 3D coordinates calculated from different camera viewpoints, requiring that the coordinate differences of the same target under different viewpoints be less than a set threshold. The spatial coordinate difference threshold is set to 0.1 meters, and the angular difference threshold is set to 5 degrees. When the difference between the calculation result of a certain viewpoint and other viewpoints exceeds the threshold, the observation of that viewpoint is marked as an anomaly, and its weight is reduced or it is directly removed in subsequent calculations. Consistency verification is achieved by calculating the mean and standard deviation of coordinates under all viewpoints, with the standard deviation threshold set to 0.05 meters.

[0094] Coordinate optimization and output:

[0095] A complete coordinate optimization process is established, comprising three optimization stages. The first stage performs local optimization, using the Gauss-Newton method to minimize the reprojection error, with 20 iterations. The second stage performs global optimization, considering the motion continuity between adjacent frames, and jointly optimizing the coordinate estimates for 10 consecutive frames using a graph optimization method. The third stage performs post-processing, using median filtering to remove outliers, with a filtering window size of 5 frames.

[0096] The final output 3D spatial coordinates include three components: X, Y, and Z, with centimeter-level accuracy. The coordinate range is determined based on the actual scene size: 0-50 meters in the X-axis direction, 0-30 meters in the Y-axis direction, and 0-10 meters in the Z-axis direction. The output frequency is consistent with the video stream frame rate, outputting 30 sets of coordinate data per second. Each coordinate point is accompanied by a confidence score, calculated based on reprojection error and multi-view consistency, with a score range of 0-1. Coordinate results greater than 0.8 are considered high-confidence localizations.

[0097] In S5, multi-target tracking and identity association are performed based on the three-dimensional spatial coordinate sequences of each target, while simultaneously fusing the target's motion state features, appearance features, and behavioral features. The accuracy of identity determination is enhanced through the complementarity of multi-modal features, generating three-dimensional motion trajectories with consistent identity identifiers for each target. Specifically, this includes:

[0098] The specific implementation process of multi-target tracking and identity association:

[0099] First, a spatiotemporal graph model is constructed, representing the 3D spatial coordinates of each target at each time step as a node in the graph. Each node contains three types of attribute features: motion state features, including 3D position coordinates, velocity vector, and acceleration vector, totaling 9 dimensions; appearance features, using a 512-dimensional feature vector extracted from the target region image via a convolutional neural network; and behavioral features, containing 15 dimensions, calculated by analyzing the target's 3D skeletal keypoint sequence. Edges between nodes represent the relationships between targets at different time steps, and the edge weights are calculated based on feature similarity.

[0100] Motion consistency calculation employs a multi-dimensional approach. Position consistency is calculated using Euclidean distance, considering the 3D positional changes of the target between two consecutive frames, with a distance threshold of 2 meters. Velocity consistency is calculated using the cosine similarity of velocity vectors, with a threshold of 0.85. Acceleration consistency is determined by analyzing acceleration trends, requiring the rate of acceleration change between adjacent frames to be less than 3 meters per second. 3 Cosine similarity is used to calculate appearance similarity. The dot product is calculated after normalizing the 512-dimensional appearance feature vector, and the similarity threshold is set to 0.7.

[0101] The association process employs a hierarchical strategy. The coarse association stage uses the Hungarian algorithm for rapid matching based on motion state features, retaining only candidate pairs with a motion consistency score higher than 0.8. The fine association stage integrates appearance and behavioral features, with the appearance feature weight initially set at 0.6 and the behavioral feature weight at 0.4. For each candidate pair, a comprehensive association score is calculated using the formula: motion consistency score multiplied by 0.5, appearance similarity score multiplied by 0.3, and behavioral matching score multiplied by 0.2. An association is confirmed only when the comprehensive score exceeds 0.75.

[0102] The graph optimization algorithm employs an inference method based on maximum a posteriori probability. An optimization window containing 30 consecutive frames of observation data is established, constructing a probabilistic graphical model encompassing all target nodes. The optimal identity association is solved using an iterative optimization method. Each iteration includes two steps: first, fixing other variables, optimizing the continuity of the identity of a single target; then, fixing the identity association, optimizing the smoothness of the motion trajectory. The optimization process uses the belief propagation algorithm, with 50 iterations and a convergence threshold of 0.01.

[0103] The specific implementation process of generating three-dimensional motion trajectories:

[0104] The adaptive weight adjustment mechanism dynamically updates feature weights based on scene complexity and occlusion levels. Scene complexity is assessed using target density; when the number of targets exceeds 20, it is defined as a high-complexity scene. In this case, the motion state feature weight increases to 0.6, the appearance feature weight decreases to 0.25, and the behavior feature weight remains at 0.15. Occlusion is assessed using target visibility; when a target is occluded by more than 50%, the appearance feature weight automatically decreases by 0.2, while the motion state feature weight increases accordingly.

[0105] Temporal consistency enhancement uses a Long Short-Term Memory (LSTM) network to predict the future state of a target. The network input contains the target's trajectory over the past 10 frames, and the output is a state prediction for the next 5 frames. The network structure consists of three hidden layers, each with 128 neurons, using the tanh activation function. The prediction results are fused with the current observations, and the fusion weights are adaptively adjusted based on the prediction error: when the prediction error is less than 0.1 meters, the prediction weight is set to 0.7; when the prediction error is between 0.1 and 0.3 meters, the weight linearly decreases to 0.3; when the prediction error is greater than 0.3 meters, only the observed data is used.

[0106] A confidence assessment system is established to resolve multimodal feature conflicts. The confidence score for motion state features is calculated based on trajectory smoothness, by fitting a motion curve to a position sequence of five consecutive frames. A confidence score of 1.0 is achieved when the residual is less than 0.05 meters, and linearly decreases to 0.3 when the residual is between 0.05 and 0.2 meters. The confidence score for appearance features is calculated through feature quality assessment, using a combination of feature vector magnitude and stability. The confidence score for behavioral features is based on the detection quality assessment of skeletal keypoints. When conflicts arise between features, the feature with the highest confidence score is selected as the primary criterion, but at least two features must have a confidence score exceeding 0.6 to make a judgment.

[0107] The trajectory post-processing consists of three main steps. First, the trajectory is smoothed using a Kalman filter. The state vector includes position, velocity, and acceleration, with the observation noise covariance set to 0.01 and the process noise covariance set to 0.001. Next, identity jump detection is performed. By analyzing abrupt changes in the feature vectors of the trajectory, a possible identity jump is identified when the cosine similarity of appearance features between adjacent frames is less than 0.5 and the difference in behavioral features exceeds 0.3. Finally, trajectory repair is performed. For short-duration (less than 5 frames) tracking interruptions, cubic spline interpolation is used to fill in the missing trajectory segments.

[0108] Trajectory quality assessment and output:

[0109] A trajectory quality evaluation index system was established, comprising four dimensions: the continuity index requires that the number of trajectory interruptions be less than 5% of the total number of frames, and the smoothness index requires that the rate of change of acceleration be less than 5 m / s.3 The consistency metric requires an identity feature change rate of less than 0.1 frames, while the integrity metric requires an effective trajectory length of more than 30 frames. Each metric has a weight of 0.25, and trajectories with a comprehensive score greater than 0.8 are considered high-quality trajectories.

[0110] The final output 3D motion trajectory includes the following data: the trajectory number uses a globally unique identifier, the timestamp is accurate to milliseconds, the 3D spatial coordinates are accurate to centimeters, the motion state includes velocity and acceleration vectors, the identity features include appearance feature vectors and behavioral feature vectors, and the confidence score ranges from 0 to 1. The trajectory data is stored in time series format, with 30 trajectory points per second, supporting both real-time streaming output and batch processing modes.

[0111] The working principle of this invention is as follows: Multiple synchronously deployed cameras acquire scene video sequences, which are then calibrated and preprocessed. A multi-branch neural network is then constructed to extract dense visual features containing color, geometric structure, and semantic information from the video frames. Based on these features, a time-independent static scene implicit representation model is generated through spatiotemporal joint modeling and static-dynamic component decoupling. Subsequently, target detection is performed on the real-time video stream, and the two-dimensional detection points are back-projected and their precise three-dimensional spatial coordinates are calculated by querying the static scene model. Finally, the motion state, appearance, and behavioral features of the targets are fused for multi-target tracking and identity association, generating high-precision three-dimensional motion trajectories with consistent identity identifiers for each target. This method effectively solves the problems of reference frame drift and silent identity-trajectory exchange by separating a pure static reference frame from the dynamic scene and organically combining multimodal features, achieving stable and accurate tracking and positioning of multiple targets in complex scenes.

[0112] The foregoing has provided a detailed description of one embodiment of the present invention, but this description is merely a preferred embodiment and should not be construed as limiting the scope of the invention. All equivalent variations and modifications made within the scope of the claims of this invention should still fall within the patent coverage of this invention.

Claims

1. A target identity and trajectory localization method driven by multi-camera 3D reconstruction, characterized in that, Includes the following steps: S1: Acquire a synchronized video sequence of the scene using multiple pre-positioned synchronized cameras; S2: Extract a continuous frame sequence from the synchronized video sequence, and perform dense feature extraction on each frame image to obtain visual features including color, geometric structure and semantic information; S3: Spatiotemporal joint modeling of the scene based on visual features, and generating a time-independent implicit representation model of the static scene by introducing the time dimension and decoupling static and dynamic components. The spatiotemporal joint modeling of the scene based on visual features specifically includes: A spatiotemporal coding network is constructed. The input of the spatiotemporal coding network is a multi-scale feature pyramid based on visual features, and the output is a spatiotemporal joint representation of the scene. By designing a time-aware feature aggregation module, visual features in a continuous frame sequence are aligned and fused along the time dimension; We weight the feature contributions of different time steps and spatial locations to preserve the stable structural features in the scene; we then deeply fuse the processed temporal features with the original spatial features to generate a joint representation that includes the spatiotemporal characteristics of the scene. The generation of a time-independent static scene implicit representation model specifically includes: Based on spatiotemporal joint representation, a static-dynamic decoupled network is constructed, which includes parallel static and dynamic branches. In the static branch, time invariance constraints are applied to ensure that the scene representation output by the static branch does not change over time. In the dynamic branch, the changing components in the scene are extracted, and the spatial distribution range of the dynamic components is restricted; the purely static scene representation is gradually separated from the complete spatiotemporal representation, first processing the significant static region, and then processing the dynamic-static boundary region; The output parameters of the static branch are solidified into a lightweight implicit representation model of the static scene; S4: Perform target detection on the real-time video stream, back-project the two-dimensional positioning reference point of the detected target into the observation ray, calculate the intersection point of the ray and the static background structure by querying the implicit representation model of the static scene, and obtain the three-dimensional spatial coordinates of each target in each frame. S5: Based on the three-dimensional spatial coordinate sequence of each target, and simultaneously integrate the target's motion state features, appearance features and behavior features, multi-target tracking and identity association are performed. The accuracy of identity recognition is enhanced by the complementarity of multi-modal features, and three-dimensional motion trajectories with consistent identity identifiers are generated for each target.

2. The target identity and trajectory localization method driven by multi-camera 3D reconstruction according to claim 1, characterized in that, The process of acquiring the visual features is as follows: A multi-scale feature extraction network is constructed, which extracts visual features at different abstraction levels from the input image frame through multiple feature extraction branches connected in parallel. The first branch captures global geometric context information, the second branch preserves local color and texture details, and the third branch focuses on regions with semantic information. Adaptive fusion of feature maps output from each branch is performed, and the contribution of features from different branches is dynamically adjusted through learnable weight parameters to generate a unified dense feature map containing color, geometric structure and semantic information. By leveraging the spatiotemporal continuity between consecutive frames, a smoothing constraint is applied to the dense features at corresponding positions in adjacent frames to extract temporally stable visual features.

3. The target identity and trajectory localization method driven by multi-camera 3D reconstruction according to claim 1, characterized in that, The step of back-projecting the two-dimensional positioning reference point of the detected target into an observation ray specifically includes: Based on camera parameters, the two-dimensional positioning reference point of each detected target is converted into a three-dimensional observation ray originating from the camera optical center; By analyzing the motion trajectory of the same target in adjacent frames, the ray direction is smoothed and optimized. By utilizing the geometric property that the reverse rays of the same target under different viewpoints should be coplanar and intersect at a point in three-dimensional space, the initial projected rays are jointly optimized.

4. The target identity and trajectory localization method driven by multi-camera 3D reconstruction according to claim 1, characterized in that, The analysis yields the three-dimensional spatial coordinates of each target in each frame, specifically including: Adaptive sampling is performed along the observed ray in the implicit representation model of the static scene, and the precise intersection position of the ray and the scene surface is determined by evaluating the spatial occupancy probability of the sampling points. First, potential intersection areas are quickly located at a coarse-grained level, and then the coordinates of the intersection points are accurately calculated at a fine-grained level. By leveraging the smoothness of the target's motion between adjacent frames, the 3D coordinate estimation results of the current frame are optimized and corrected. By comparing the consistency between the three-dimensional coordinates calculated from different perspectives, abnormal positioning results are eliminated, and finally accurate three-dimensional spatial coordinates are output.

5. The target identity and trajectory localization method driven by multi-camera 3D reconstruction according to claim 1, characterized in that, The multi-target tracking and identity association based on the three-dimensional spatial coordinate sequence of each target, while simultaneously fusing the target's motion state features, appearance features, and behavioral features, specifically includes: The three-dimensional spatial coordinate sequence of each target is represented as a node in the figure. The node attributes include the target's motion state characteristics, appearance characteristics, and behavior characteristics. The motion consistency, appearance similarity, and behavior matching degree of target nodes between adjacent frames are taken into account. Rapid coarse association is performed based on motion state features to filter candidate matching pairs, and then fine association is performed by fusing appearance features and behavioral features. Finally, a graph optimization algorithm is used to solve for the optimal identity association result in the global spatiotemporal scope, ensuring the continuity of identity during the tracking process.

6. The target identity and trajectory localization method driven by multi-camera 3D reconstruction according to claim 1, characterized in that, The generation of three-dimensional motion trajectories with consistent identity identifiers for each target specifically includes: The weight ratios of motion state features, appearance features, and behavior features in identity determination are adaptively adjusted based on scene complexity and occlusion conditions. By leveraging the target's historical movement patterns and current behavioral characteristics, we can predict its future state and enhance the temporal consistency of identity association. When the results of multimodal feature discrimination are contradictory, the optimal identity determination result is selected through confidence assessment and spatiotemporal consistency verification. Finally, through trajectory smoothing and post-processing, identity jumps are eliminated, and three-dimensional motion trajectories with spatiotemporal consistency are generated for each target.

Images