Multi-camera cooperative target relay tracking and three-dimensional positioning system

Abstract

The present application relates to the technical field of intelligent visual monitoring, in particular to a target relay tracking and three-dimensional positioning system based on multi-camera cooperation, which comprises a video acquisition module, a target analysis module and a monocular tracking module. Image capture devices arranged at multiple positions continuously acquire monitoring video streams and transmit them to an image processing device. The device analyzes the video streams in real time, identifies candidate target frames, class labels and confidence scores, filters out low-confidence target frames according to a preset threshold, determines initial detection targets and assigns unique identity labels. The system extracts the deep convolutional neural network feature vector of the target, matches it with the features of the historical trajectory library, and matches the target with the original trajectory identity. If the target is not matched, a new trajectory is generated, forming a sequence of frame-by-frame trajectory points with identity labels. This system can reduce the interference of invalid detection data, avoid target identity confusion, and maintain the continuity and stability of the tracking trajectory within the field of view of a single camera.

Description

Technical Field

[0001] This invention relates to the field of intelligent visual surveillance technology, and in particular to a multi-camera collaborative target relay tracking and three-dimensional positioning system. Background Technology

[0002] In existing multi-camera visual surveillance scenarios, image capture devices deployed at different spatial locations each acquire video streams and transmit them to image processing equipment. The image processing equipment performs routine target detection on a single video stream, directly outputting candidate target boxes, target category labels, and confidence scores. It does not filter the detection results based on confidence thresholds; the detection process only performs preliminary extraction of target information, without standardizing the definition of valid targets or assigning independent identification identifiers to detected targets. Target tracking within the field of view of a single camera relies on inter-frame positional information to complete trajectory association, without employing deep convolutional neural networks to extract feature vectors or using feature similarity matching to achieve trajectory identity continuity.

[0003] Existing detection methods fail to remove low-confidence invalid target boxes, and redundant detection data continuously interferes with subsequent recognition and tracking processes. Furthermore, the lack of a unified standard for identifying initial detected targets easily leads to target identity confusion. Monocular tracking relies solely on location association to complete trajectory connections; when the target state changes, the trajectory identity cannot be stably preserved, easily resulting in trajectory interruptions and identity confusion, and failing to form a frame-by-frame trajectory point sequence bound to a fixed identity. This invention addresses the problems of lack of confidence filtering in target detection, lack of unique initial target identification, and the absence of depth feature matching in monocular tracking, thereby optimizing the state stability of target tracking in multi-camera scenarios. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies by proposing a multi-camera collaborative target relay tracking and three-dimensional positioning system.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: a multi-camera collaborative target relay tracking and three-dimensional positioning system, comprising: The video acquisition module consists of multiple image capture devices deployed in different spatial locations that continuously acquire surveillance video streams and transmit the surveillance video streams to the image processing device. The target parsing module is used to analyze the monitoring video stream from each image capture device in real time, identify all candidate target boxes and their corresponding target category labels and confidence scores, filter out candidate target boxes with confidence scores lower than the preset filtering threshold according to the preset filtering threshold, determine the candidate target boxes that meet the requirements as the initial detection targets, and assign a unique target identity to each initial detection target. The monocular tracking module extracts feature vectors based on deep convolutional neural networks for all initially detected targets within the field of view of each image capture device. It then performs similarity matching between these feature vectors and feature vectors in the historical trajectory database of the image capture device. Targets that successfully match inherit their original trajectory identity, while targets that fail to match form new target trajectories with new target identity. This completes independent tracking of targets within the field of view of each image capture device and generates a frame-by-frame trajectory point sequence with target identity.

[0006] As a further aspect of the present invention, the system further includes: An identity transfer module is used to transfer the target identity between adjacent image capture devices; For any two adjacent image capture devices with overlapping fields of view, establish an association for target identity transfer; Within the overlapping area, from the frame-by-frame trajectory point sequence with target identification of the first image capture device, the target that appears in the most recent preset time period and is moving toward the center of the overlapping area is extracted as the target to be transmitted. For each target to be transmitted, the feature vectors of the target are extracted from the last few frames of images before it leaves the field of view of the first image capture device, forming the first feature vector set of the target. Within the field of view of the second image capture device, newly emerging targets in the overlapping area during the preset time period are extracted, and feature vectors of each newly emerging target in the first few frames of images when it appears are extracted to form a second feature vector set of potential targets. Calculate the average cosine similarity between the first feature vector set and the second feature vector set of all potential targets. Determine the pair of targets with the highest average cosine similarity that exceeds a preset transmission threshold as the same physical target. Assign the target identity identifier carried by the target to be transmitted in the first image capture device to the corresponding potential target in the second image capture device to complete the target identity transmission across cameras.

[0007] As a further aspect of the present invention, the system further includes: The three-dimensional coordinate calculation module, after completing the target identity transfer, obtains the pixel coordinates of the same physical target in the image plane of any two different image capture devices within the same timestamp or a very small time difference. By calling the preset intrinsic parameter matrix and distortion coefficients of each image capture device, the pixel coordinates are subjected to distortion removal processing to obtain coordinates in the normalized camera coordinate system. By combining the extrinsic parameter matrix between the two image capture devices, which includes a rotation matrix and a translation vector, the three-dimensional spatial coordinates of the physical target at the current time are solved from the corresponding pixel coordinate pairs after distortion correction using the principle of triangulation. The calculated three-dimensional spatial coordinates are bound to the corresponding target identity identifier and timestamp to form a three-dimensional trajectory point of the target, and recorded in the global target trajectory database.

[0008] As a further aspect of the present invention, the step of using the triangulation principle to solve for the three-dimensional spatial coordinates of the physical target at the current time from the corresponding pixel coordinate pairs after distortion correction includes: Based on the world coordinate system, and according to the projection matrix of the first image capture device, a projection equation is established from the three-dimensional spatial coordinates to the distorted pixel coordinates of the image plane of the first image capture device. Based on the world coordinate system, and according to the projection matrix of the second image capture device, a projection equation is established from the three-dimensional spatial coordinates to the image plane of the second image capture device to remove the distorted pixel coordinates. The two sets of projection equations are combined to form an overdetermined linear system of equations about the three components of the three-dimensional spatial coordinates. The singular value decomposition algorithm is used to solve the overdetermined linear equation system to obtain the optimal three-dimensional spatial coordinate solution in the least squares sense. The optimal three-dimensional spatial coordinate solution is validated for rationality, and erroneous solutions with negative depth values ​​or exceeding the preset effective physical space range are eliminated.

[0009] As a further aspect of the present invention, the system further includes: The trajectory optimization module is used to smooth and predict trajectories in the global target trajectory database. From the global target trajectory database, extract all three-dimensional trajectory points of the same target under consecutive timestamps according to the target identity identifier, forming the original three-dimensional trajectory sequence of the target; For the original three-dimensional trajectory sequence, a motion model is established by applying the Kalman filter algorithm to estimate and filter the position and velocity state of the target in three-dimensional space, and a smoothed three-dimensional trajectory point sequence is obtained. The motion model established based on the Kalman filter algorithm uses the estimated state vector from the previous moment as the initial value to predict the target's three-dimensional spatial position in the next moment and generate predicted three-dimensional trajectory points. The smoothed 3D trajectory point sequence and the predicted 3D trajectory points are output together as a high-confidence target 3D motion trajectory.

[0010] As a further aspect of the present invention, a motion model is established by applying a Kalman filter algorithm to the original three-dimensional trajectory sequence, and the position and velocity state of the target in three-dimensional space are estimated and filtered to obtain a smoothed three-dimensional trajectory point sequence, including: The coordinates of the target in the three-dimensional space in the original three-dimensional trajectory sequence are used as the observation vector of the system; Establish a state model of the Kalman filter system with the target's position and velocity as the state vector, where the state vector contains three-dimensional position components and three-dimensional velocity components; The state vector of the Kalman filter is initialized with the position information of the first three-dimensional trajectory point in the original three-dimensional trajectory sequence, and the velocity component is initialized to zero. Define the state transition matrix that describes the motion law of the target, and set the system process noise covariance matrix and the observation noise covariance matrix; For each subsequent three-dimensional trajectory point in the original three-dimensional trajectory sequence, the prediction step of the Kalman filter algorithm is executed, and the state vector and state estimate covariance at the current moment are predicted based on the state estimate value at the previous moment using the state transition matrix. The update step of the Kalman filter algorithm is executed, the Kalman gain is calculated, and then the predicted state vector and the state estimation covariance are corrected by combining the observation values ​​of the current three-dimensional trajectory points to obtain the optimal state estimate at the current time. The position components are extracted from the optimal state estimate obtained after each update step and used as smoothed three-dimensional spatial coordinates. These coordinates are then arranged in chronological order to form a smoothed three-dimensional trajectory point sequence.

[0011] As a further aspect of the present invention, the system further includes: The tracking relay trigger module is used to determine the continuity of tracking at the edge of the camera network coverage. Real-time monitoring of the high-confidence target's three-dimensional motion trajectory corresponding to each target identity identifier, and calculation of the target's motion direction and speed; Predict the target's movement path in the next time period and determine whether the predicted movement path will enter the physical coverage boundary area of ​​the current camera network; If it is predicted that the device will enter the physical coverage boundary area, then search for other image capture devices adjacent to the physical coverage boundary area that are not connected to the current cooperative network. If there is an image capture device that is not connected to the current collaborative network, a tracking relay request is generated. The tracking relay request includes the target's appearance feature vector, the last known three-dimensional spatial coordinates, and the direction and speed of movement. The tracking relay request is sent to the system to which the image capture device that is not connected to the current collaborative network belongs, so as to trigger a target identity matching and tracking relay process on a larger scale.

[0012] As a further aspect of the present invention, the system further includes: The re-identification module is used to handle the re-identification of a target that reappears after a brief disappearance; When the number of frames in which a target disappears from the field of view of an image capture device due to occlusion exceeds a preset threshold, the target's identity is marked as temporarily lost. The field of view of the image capture device is continuously monitored, and when a new initial target is detected in a subsequent frame, the feature vector of the new target is calculated. The feature vector of the new target is compared with the historical set of feature vectors of all targets in a temporarily lost state stored before they disappeared within a preset time window. If the maximum similarity between the feature vector of the new target and the feature vector history set of a target that is temporarily lost exceeds the re-identification threshold, then the new target is determined to be a reappearance of the temporarily lost target, and the original target identity is assigned to the new target to restore its continuous tracking trajectory. If the similarity between the feature vector of the new target and the historical feature vector set of all temporarily lost targets does not exceed the re-identification threshold, then a new target identity is assigned to the new target.

[0013] As a further aspect of the present invention, the system further includes: The data processing module is used to remove outliers and interpolate data to complete the 3D trajectory points. For the sequence of three-dimensional trajectory points under the same target identity in the global target trajectory database, the velocity and acceleration are calculated in chronological order; Set velocity thresholds and acceleration thresholds, and mark three-dimensional trajectory points whose instantaneous velocity exceeds the velocity threshold or whose instantaneous acceleration exceeds the acceleration threshold as suspected anomalies; For each suspected anomaly point, check the spatial continuity of its adjacent trajectory points. If the distance between the suspected anomaly point and the points before and after it is much greater than the reasonable displacement calculated based on the velocity of the adjacent points, then the suspected anomaly point is determined to be an abnormal three-dimensional trajectory point and is removed from the sequence. After removing the abnormal three-dimensional trajectory points, the coordinates and timestamps of the valid three-dimensional trajectory points before and after them are used to generate complete three-dimensional spatial coordinates on the missing timestamps using a cubic spline interpolation algorithm, which are then inserted into the target's three-dimensional trajectory point sequence.

[0014] As a further aspect of the present invention, the system further includes: The behavior pattern learning module collects high-confidence three-dimensional motion trajectories of a large number of targets over a long period of time from the global target trajectory database, after outlier removal and interpolation completion processing. For each of the three-dimensional motion trajectories of the high-confidence target, the geometric and kinematic features of the trajectory are extracted. The features include the total trajectory length, average velocity, velocity variance, main direction of motion, location of the dwell point, and dwell time. Using an unsupervised clustering algorithm, cluster analysis is performed on the vector set composed of the geometric and kinematic features of all trajectories to automatically discover different trajectory pattern categories; The geometric and kinematic features of the newly generated high-confidence target 3D motion trajectory are matched with the discovered trajectory pattern categories, and the most similar trajectory pattern label is assigned to them to achieve online classification of target behavior.

[0015] Compared with the prior art, the advantages and positive effects of the present invention are as follows: The system analyzes the monitoring video streams from each image capture device in real time, identifying candidate bounding boxes, corresponding target category labels, and confidence scores. Candidate bounding boxes with confidence scores below a preset filtering threshold are eliminated, and compliant bounding boxes are identified as initial detection targets. Each initial detection target is assigned a unique target identity identifier. Invalid candidate bounding boxes with low confidence are directly filtered out, reducing interference from redundant detection data on target recognition. The definition of initial detection targets more closely aligns with actual monitoring needs, and each valid detection target has an independent identity identifier, preventing target identity confusion during the detection phase.

[0016] For the initial detected target within the field of view of a single camera, a deep convolutional neural network feature vector is extracted. This feature vector is then compared with the feature vectors in the historical trajectory database of the corresponding image capture device. Targets that successfully match retain their original trajectory identity, while targets that fail to match generate new trajectories with new identity identifiers. This process completes independent monocular tracking and generates a frame-by-frame sequence of trajectory points with target identity identifiers. The similarity matching of deep convolutional features replaces the traditional position-association tracking method. When a target moves across frames within the field of view of a single camera, its identity identifier remains stable, and there are no identity switching or trajectory breaks during trajectory generation. The frame-by-frame sequence of trajectory points is always bound to the corresponding target identity, maintaining the trajectory continuity of monocular tracking. Attached Figure Description

[0017] Figure 1 This is a timing diagram of the multi-camera collaborative target relay tracking and three-dimensional positioning system described in this invention; Figure 2 The flowchart for solving three-dimensional coordinates. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0019] In the description of this invention, it should be understood that the terms "length," "width," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientation or positional relationships, are based on the orientation or positional relationships shown in the accompanying drawings and are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, in the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0020] See Figure 1 The video acquisition module includes multiple image capture devices deployed in different spatial locations. These devices continuously capture video streams of the monitored area, forming a monitoring video stream, which is then transmitted to the backend image processing device. The target parsing module runs within the image processing device. It analyzes the monitoring video stream from each image capture device in real time, identifies all candidate target boxes in the video frames using a built-in target detection algorithm, and generates a corresponding target category label and confidence score for each candidate target box. Based on a preset filtering threshold, the system filters out candidate target boxes with confidence scores lower than the threshold, and determines the candidate target boxes that meet the requirements as the initial detection targets within the field of view of the image capture device. At the same time, each initial detection target is assigned a unique digital identifier as a target identity identifier. The monocular tracking module works independently for each image capture device. It extracts the feature vector of each initially detected target based on a deep convolutional neural network and performs similarity matching between the feature vector of the target in the current frame and the feature vector in the historical trajectory database maintained by the image capture device. If the match is successful, the target inherits the original target identity of its matched historical trajectory, realizing cross-frame association. If the match fails, a new target trajectory is created for the target with a new target identity. Through the above process, the system completes independent target tracking within each image capture device and outputs a frame-by-frame trajectory point sequence with the target identity.

[0021] In one embodiment of the present invention, the identity transfer module is the core component of multi-camera collaborative target relay tracking. The multi-camera collaborative target relay tracking and 3D positioning system includes multiple image capture devices deployed in space. The monitoring fields of image capture device one and image capture device two have a physically overlapping area. The system establishes an identity transfer association between image capture device one and image capture device two, meaning the system knows that the fields of view of these two image capture devices are adjacent and share a portion of their monitoring range. The identity transfer module continuously receives a frame-by-frame trajectory point sequence with target identity identifiers from image capture device one, and filters out targets that meet the conditions for transfer from the sequence. The filtering conditions include that the target appeared in the sequence within the most recent preset time period, and its movement direction is moving towards the center of the overlapping area between image capture device one and image capture device two. The preset time period can be set to the number of video frames corresponding to the past 2 seconds.

[0022] In some embodiments, the identity transfer module performs feature extraction for each selected target to be transferred. The module extracts feature vectors from the last N frames of images of the target before it leaves the field of view of image capture device one, and uses the average or direct set of these N feature vectors as the first feature vector set of the target to be transferred, where N is a preset positive integer. In a specific implementation, the identity transfer module synchronously processes the video stream of image capture device two. Within the field of view of image capture device two, the module monitors all newly detected targets that appear in the overlapping area within the same preset time period, and marks each newly appearing target as a potential target. For each potential target, the module extracts feature vectors from the first M frames of images when it appears in the field of view of image capture device two, forming the second feature vector set of this potential target, where M is another preset positive integer.

[0023] Optionally, the identity transfer module calculates the matching degree between the first feature vector set and the second feature vector set of each potential target. The matching degree is obtained by calculating the average cosine similarity. In a specific implementation, it is assumed that the first feature vector set of a target to be transferred contains n feature vectors, represented as a set. ={ , ,..., The second feature vector set of a potential target contains m feature vectors, denoted as a set. ={ , ,..., }, then the mean cosine similarity between the first feature vector set and the second feature vector set is Calculated using the following formula:

[0024] in: Represents the first eigenvector set. 1 eigenvector Represents the first eigenvector in the second eigenvector set. 1 eigenvector, symbol " " represents the dot product operation of vectors, and Let these represent the magnitudes of the vectors, and the summation and traversal methods, respectively. From 1 to , From 1 to Mean cosine similarity The value range is [-1, 1]. The closer the value is to 1, the more similar the appearances of the targets represented by the two feature vector sets are. It can be understood that the formula calculates the cosine similarity between each vector in the first feature vector set and each vector in the second feature vector set, and takes the average of all combinations, thus obtaining a robust measure of inter-set similarity.

[0025] In some embodiments, for a target to be transferred, the identity transfer module calculates the mean cosine similarity between its first feature vector set and the second feature vector sets of all potential targets within the field of view of the image capture device two, resulting in a similarity list. The module searches for the maximum value in the similarity list. If this maximum mean cosine similarity value exceeds a preset transfer threshold, such as 0.85, the identity transfer module determines that the target to be transferred that generated this maximum mean cosine similarity and the potential targets are the same physical target. The identity transfer module then performs an identity identification transfer operation, assigning the target identity carried by the target to be transferred in the image capture device one to the successfully matched potential target in the image capture device two, thereby completing the cross-camera target identity transfer from the image capture device one to the image capture device two, so that the target can maintain identity consistency when crossing the field of view of different cameras.

[0026] In one embodiment of the present invention, the three-dimensional coordinate calculation module starts working after the identity transfer module completes the target identity transfer. The module acquires the position information of the same physical target in images acquired by two different image capture devices at the same time or with a very small difference. This position information is represented in the form of pixel coordinates. See also Figure 2The 3D coordinate calculation module calls the pre-calibrated and stored intrinsic parameter matrix and distortion coefficients of image capture device one to perform distortion correction on the pixel coordinates from image capture device one, obtaining coordinates in the normalized camera coordinate system of image capture device one. Similarly, the module calls the intrinsic parameter matrix and distortion coefficients of image capture device two to perform distortion correction on the corresponding pixel coordinates from image capture device two, obtaining coordinates in the normalized camera coordinate system of image capture device two. The system has pre-calibrated the spatial relationship between image capture device one and image capture device two. This relationship is described by an extrinsic parameter matrix, which includes a rotation matrix and a translation vector. The 3D coordinate calculation module combines the two pairs of distorted pixel coordinates and the extrinsic parameter matrix, using the principle of triangulation to solve for the 3D spatial coordinates of the physical target in the world coordinate system.

[0027] In some embodiments, the 3D coordinate calculation module needs to ensure time synchronization when acquiring pixel coordinates. The module searches for the two trajectory points with the closest timestamps under the same physical target identity from the timestamp sequences of image capture device one and image capture device two. When the difference between the two timestamps is less than a minimum time difference threshold, the module determines that the pixel coordinates of these two trajectory points can be used for 3D calculation. Optionally, the intrinsic parameter matrix includes parameters such as the focal length and principal point coordinates of the image capture device, and the distortion coefficients include radial distortion and tangential distortion parameters. The distortion removal process uses an iterative optimization algorithm to map the distorted pixel coordinates to an ideal distortion-free pixel plane, thereby eliminating the impact of lens distortion on the accuracy of subsequent 3D calculation.

[0028] In practical implementation, the 3D coordinate calculation module uses the principle of triangulation to calculate the 3D spatial coordinates. The specific process is as follows: Using the world coordinate system as a unified reference system, based on the projection matrix of image capture device one, a projection equation can be established from the target's 3D spatial coordinates to the distorted pixel coordinates on the image plane of image capture device one. Simultaneously, based on the projection matrix of image capture device two, a projection equation can be established from the target's 3D spatial coordinates to the distorted pixel coordinates on the image plane of image capture device two. Combining these two sets of projection equations forms a 3D spatial coordinate system for the target. The overdetermined linear equation system. The projection equations can be expressed in the form of:

[0029] in: This represents the distorted pixel coordinates on the image plane of a certain image capture device. This represents the 3x4 projection matrix of the image capture device (obtained by multiplying the intrinsic and extrinsic parameter matrices). This represents the three-dimensional spatial coordinates of the target in the world coordinate system. It is a non-zero scaling factor. For image capture device one and image capture device two, the above equations can be derived separately, by eliminating the scaling factor. , get about , , A system of linear equations.

[0030] In some embodiments, the three-dimensional coordinate solution module uses the singular value decomposition algorithm to solve the above-mentioned overdetermined linear equation system, obtaining the optimal three-dimensional spatial coordinate solution in the least squares sense. Optionally, the module performs a validity check on the obtained optimal 3D spatial coordinate solution, including determining the depth value. Whether it is a positive number, and how to determine the coordinates in three-dimensional space. Whether it is within a preset effective physical space range, such as a monitorable space area with dimensions of 100 meters, 50 meters, and 10 meters respectively. If the coordinates are negative or exceed the effective physical space range, the 3D coordinate calculation module will determine that the calculation has failed and discard the erroneous solution.

[0031] It is understandable that the projection matrix Integrating the internal imaging geometry of image capture devices and the external spatial pose relationships between different image capture devices, it serves as a bridge connecting two-dimensional pixel coordinates and three-dimensional world coordinates. In practical implementation, the three-dimensional coordinate calculation module will calculate the three-dimensional spatial coordinates that have successfully passed the rationality check. The target identity identifier and corresponding timestamp of the two pixel coordinates used to generate the coordinate are bound together to form a complete three-dimensional trajectory point record of the target. The three-dimensional coordinate calculation module writes this three-dimensional trajectory point record into the system's global target trajectory database for subsequent trajectory analysis and application modules to call.

[0032] In one embodiment of the present invention, the trajectory optimization module processes the trajectories in the global target trajectory database. The module retrieves data from the database according to the target's identity, extracting all three-dimensional trajectory points belonging to the same target identity under consecutive timestamps. These chronologically ordered three-dimensional trajectory points constitute the target's original three-dimensional trajectory sequence. The trajectory optimization module receives the original three-dimensional trajectory sequence as input and applies a Kalman filter algorithm to it. The Kalman filter algorithm is used to establish a motion model of the target and estimate and filter the target's position and velocity state in three-dimensional space, obtaining a smoothed three-dimensional trajectory point sequence through filtering. Based on the motion model established by the Kalman filter algorithm, the trajectory optimization module performs a prediction function. Using the estimated state vector from the previous moment as the initial value, the module extrapolates forward to predict the target's three-dimensional spatial position at the next moment and generates predicted three-dimensional trajectory points. Finally, the trajectory optimization module outputs the smoothed three-dimensional trajectory point sequence and the predicted three-dimensional trajectory points together. The output data is defined as a high-confidence target three-dimensional motion trajectory.

[0033] In some embodiments, the trajectory optimization module uses the target's coordinates in three-dimensional space from the original three-dimensional trajectory sequence as the observation vector of the Kalman filter system. The module establishes a state model of the Kalman filter system with the target's position and velocity as state vectors, where the state vector contains six components: a three-dimensional position component and a three-dimensional velocity component. The trajectory optimization module initializes the state vector of the Kalman filter. The position component of the state vector is initialized with the position information of the first three-dimensional trajectory point in the original three-dimensional trajectory sequence, and the velocity component of the state vector is initialized to a zero vector. The trajectory optimization module defines a state transition matrix describing the target's motion law. The state transition matrix is ​​constructed based on a uniform motion model or a uniformly accelerated motion model. Simultaneously, the trajectory optimization module sets the system process noise covariance matrix and the observation noise covariance matrix. The parameters of these two covariance matrices are configured according to the target's motion uncertainty and three-dimensional coordinate observation error in the actual scene.

[0034] In its implementation, the trajectory optimization module iteratively executes two core steps of the Kalman filter algorithm for each subsequent 3D trajectory point in the original 3D trajectory sequence: the prediction step and the update step. In the prediction step, the trajectory optimization module uses a predefined state transition matrix to predict the current state vector and state estimate covariance based on the optimal state estimate from the previous time step. In the update step, the trajectory optimization module calculates the Kalman gain and then corrects the state vector and state estimate covariance obtained in the prediction step by combining the observed values ​​of the current 3D trajectory points, thus obtaining the optimal state estimate for the current time step. After each update step, the trajectory optimization module extracts the position component from the optimal state estimate. This position component represents the smoothed 3D spatial coordinates. Arranging the smoothed 3D spatial coordinates in chronological order constitutes the smoothed 3D trajectory point sequence.

[0035] Optionally, the state-space model of the Kalman filter algorithm can be expressed as the following equation:

[0036] in: Indicates that the system is in The state vector at time step, the state vector Includes three-dimensional position and three-dimensional velocity components. Represents the state transition matrix. The system process noise is represented by the following formula: . Indicates in The observation vector at time t, in this implementation, the observation vector These are the three-dimensional spatial coordinates read from the global target trajectory database. Represents the observation matrix. Represents the observation noise, and its covariance matrix is: The trajectory optimization module's computation process is based on the observation vectors. Recursively estimate the optimal state vector .

[0037] It is understandable that the Kalman gain is a dynamically calculated matrix, which determines whether the predicted or observed values ​​should be trusted more during the update step. In some embodiments, the system process noise covariance matrix... and observation noise covariance matrix Adjustments are needed based on the actual application scenario; larger ones The value indicates that the model assumes significant uncertainty in the system process, which makes the filter place more trust in the observations. A value indicating high noise in the observed data will make the filter trust the predicted values ​​more. Optionally, the trajectory optimization module generates the predicted 3D trajectory points independently of the smoothing process. The module outputs the smoothing result at the current time while utilizing the optimal state estimate at the current time. and state transition matrix Through formula Calculate the state prediction value at the next time step. and from the state prediction value The positional components are extracted as predicted 3D trajectory points. Ultimately, the high-confidence target 3D motion trajectory contains a smooth sequence of historical 3D trajectory points and at least one predicted 3D trajectory point in the future, providing coherent motion state information for subsequent behavior analysis or relay tracking triggering.

[0038] In one embodiment of the present invention, the tracking relay triggering module and the re-identification module work together to handle the tracking continuity of the target at the edge of the camera network and the identity recovery problem after the target temporarily disappears. The tracking relay triggering module monitors the high-confidence three-dimensional motion trajectory of each target identity identifier in the global target trajectory database in real time. The module calculates the instantaneous motion direction and velocity of the target based on the continuous three-dimensional trajectory points in the trajectory. Based on the target's position, velocity, and direction at the current moment, the tracking relay triggering module predicts the target's motion path in the next preset time period. The module determines whether the predicted motion path will enter the predefined physical coverage boundary area of ​​the current camera network. The physical coverage boundary area is a polygonal area in three-dimensional space that marks the boundary of the camera network's monitoring range. If the predicted motion path will enter the physical coverage boundary area, the tracking relay triggering module searches for spatial locations adjacent to the physical coverage boundary area and queries whether there are other image capture devices not connected to the current collaborative network. Image capture devices not connected to the current collaborative network refer to independent camera devices that are physically adjacent but do not participate in the current system's collaborative tracking calculation.

[0039] In some embodiments, the tracking relay triggering module predicts the target's motion path using a linear extrapolation model, assuming the target is within the current time period. The interior maintains uniform linear motion, with the direction of motion being a unit vector. This indicates that the speed is... The target's current location is Then predict the location From the formula The calculation shows that, among which It is the preset length of the future time period. This is the speed calculated so far. It is a normalized motion direction vector. It is the last known three-dimensional spatial coordinate vector. If the predicted position... If the target's coordinates are located inside or at the edge of the physical coverage boundary area, the tracking relay triggering module determines that the target is about to leave the current monitoring network. Optionally, when it is confirmed that there is an adjacent image capture device that is not connected to the current cooperative network, the tracking relay triggering module generates a structured tracking relay request. The tracking relay request includes the target's appearance feature vector and the target's last known three-dimensional spatial coordinates. Target's direction of movement With speed magnitude The tracking relay trigger module sends the tracking relay request to the independent system to which the image capture device is not connected to the current collaborative network through a preset network interface, triggering a target identity matching and tracking relay process on a larger scale.

[0040] In practice, the re-identification module handles situations where a target temporarily disappears from the field of view of a single image capture device due to occlusion and then reappears. When the number of consecutive video frames in which a target disappears from the field of view of an image capture device due to occlusion exceeds a preset threshold, the re-identification module marks the target's identity as temporarily lost. The re-identification module continuously monitors the field of view of this image capture device. When the target parsing module of the image capture device detects a new initially detected target in subsequent video frames, the re-identification module calculates the feature vector of the new target. The re-identification module compares the similarity of this new target's feature vector with the historical feature vector set stored before the disappearance of all targets in a temporarily lost state within a preset time window. The historical feature vector set contains the feature vector set of multiple frames of images from a period of time before the target disappeared.

[0041] In some embodiments, the re-identification module calculates the similarity between the new target feature vector and a set of historical feature vectors of a lost target as follows: Let the feature vector of the new target be... The historical feature vector set of a target that is temporarily lost contains There are 1 historical feature vectors, denoted as . Re-identification module calculation The cosine similarity with each historical feature vector in the set is used, and the maximum value is taken as the similarity value for this comparison. Table 1 below illustrates an exemplary comparison process: Table 1: Comparison of the similarity between the historical feature vector sets of candidate new targets and lost targets

[0042] The table above shows three candidate new targets appearing in the field of view of the image capture device. The re-identification module calculates the maximum similarity between the feature vector of each candidate new target and the historical feature vector sets of two temporarily lost targets. It can be understood that the re-identification module performs the above calculation for each candidate new target, obtaining a list of similarities with all temporarily lost targets. If the feature vector of a candidate new target has a maximum similarity to the historical feature vector set of a temporarily lost target... If the new target exceeds a preset re-identification threshold, the re-identification module determines that it is a reappearance of the corresponding temporarily lost target. The re-identification module assigns the original target identity to the new target and restores its continuous tracking trajectory. Optionally, if the similarity between the feature vector of a new target and the historical feature vector set of all temporarily lost targets is... If none of the values ​​exceed the re-identification threshold, the re-identification module instructs the system to assign a new target identity to this new target and begin tracking it as a completely new target. The logic of the tracking relay trigger module and the re-identification module together ensures the continuity of target tracking in both spatial relay and temporally discontinuous scenarios.

[0043] In one embodiment of the present invention, the data processing module cleans and improves the three-dimensional trajectory points in the global target trajectory database. The data processing module extracts data from the global target trajectory database grouped according to target identity identifiers, and sorts the three-dimensional trajectory point sequences under the same target identity identifier by timestamp to form an ordered sequence. The data processing module calculates the instantaneous velocity and instantaneous acceleration of the target based on the coordinate difference and time interval between adjacent three-dimensional trajectory points in the ordered sequence. The system sets a positive velocity threshold and a positive acceleration threshold. The data processing module traverses each three-dimensional trajectory point in the ordered sequence. If the instantaneous velocity of a certain three-dimensional trajectory point exceeds the velocity threshold or the instantaneous acceleration exceeds the acceleration threshold, the data processing module marks this three-dimensional trajectory point as a suspected anomaly. For each marked suspected anomaly, the data processing module checks its spatial continuity. The module calculates the spatial distance between the suspected anomaly and its preceding valid 3D trajectory point, and also calculates the spatial distance between the suspected anomaly and its following valid 3D trajectory point. Based on the velocity and time interval of the preceding valid 3D trajectory point, a reasonable displacement range is calculated. If the distance between the suspected anomaly and the points before and after it is much greater than the calculated reasonable displacement range, the data processing module determines that the suspected anomaly is an abnormal 3D trajectory point and removes it from the sequence.

[0044] In some embodiments, after removing abnormal 3D trajectory points, the ordered sequence will have data gaps at the corresponding timestamps. The data processing module uses the coordinates and timestamp information of the valid 3D trajectory points before and after the abnormal 3D trajectory points, and employs a cubic spline interpolation algorithm to generate complete 3D spatial coordinates at the missing timestamps. The data processing module then uses the calculated interpolated coordinates... The data is inserted into the target's 3D trajectory point sequence to form a continuous and complete trajectory. In practice, the behavior pattern learning module collects long-term trajectory data that has undergone outlier removal and interpolation completion processing. It also extracts high-confidence 3D motion trajectories of a large number of targets over a long period from the global target trajectory database. These high-confidence 3D motion trajectories are output by the trajectory optimization module. The behavior pattern learning module extracts features from each input high-confidence 3D motion trajectory. The extracted geometric and kinematic features include total trajectory length, average velocity, velocity variance, main direction of motion, location of dwell points, and dwell time. Dwell points are determined by analyzing the spatial clustering of multiple consecutive points in the trajectory and the duration of velocities below a threshold. The main direction of motion can be obtained through vector or principal component analysis of the trajectory's start and end points.

[0045] In some embodiments, the behavior pattern learning module uses an unsupervised clustering algorithm to analyze all extracted feature vector sets. Each feature vector consists of multiple geometric and kinematic feature values ​​of a trajectory. Optionally, the unsupervised clustering algorithm can employ K-means clustering, DBSCAN clustering, or a Gaussian mixture model. The behavior pattern learning module automatically groups trajectories with similar features into the same group through cluster analysis, thereby discovering different trajectory pattern categories hidden in the data. Each discovered trajectory pattern category represents a common target motion behavior pattern, such as "rapidly passing through in a straight line," "circular wandering," or "stopping at a fixed point and then leaving." When the system generates a new high-confidence target 3D motion trajectory, the behavior pattern learning module extracts the same geometric and kinematic features of this new trajectory online and constructs a feature vector. The behavior pattern learning module calculates the similarity or distance between the new trajectory feature vector and the center points or category models of each discovered trajectory pattern category. The behavior pattern learning module assigns the new trajectory the trajectory pattern label with the highest similarity or closest distance, achieving online automatic classification of target behavior.

[0046] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments that can be applied to other fields. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A multi-camera coordinated target relay tracking and three-dimensional positioning system, characterized in that, The system includes: The video acquisition module consists of multiple image capture devices deployed in different spatial locations that continuously acquire surveillance video streams and transmit the surveillance video streams to the image processing device. The target parsing module is used to analyze the monitoring video stream from each image capture device in real time, identify all candidate target boxes and their corresponding target category labels and confidence scores, filter out candidate target boxes with confidence scores lower than the preset filtering threshold according to the preset filtering threshold, determine the candidate target boxes that meet the requirements as the initial detection targets, and assign a unique target identity to each initial detection target. The monocular tracking module extracts feature vectors based on deep convolutional neural networks for all initially detected targets within the field of view of each image capture device. It then performs similarity matching between these feature vectors and feature vectors in the historical trajectory database of the image capture device. Targets that successfully match inherit their original trajectory identity, while targets that fail to match form new target trajectories with new target identity. This completes independent tracking of targets within the field of view of each image capture device and generates a frame-by-frame trajectory point sequence with target identity.

2. The multi-camera collaborative target relay tracking and three-dimensional positioning system according to claim 1, characterized in that, The system also includes: An identity transfer module is used to transfer the target identity between adjacent image capture devices; For any two adjacent image capture devices with overlapping fields of view, establish an association for target identity transfer; Within the overlapping area, from the frame-by-frame trajectory point sequence with target identification of the first image capture device, the target that appears in the most recent preset time period and is moving toward the center of the overlapping area is extracted as the target to be transmitted. For each target to be transmitted, the feature vectors of the target are extracted from the last few frames of images before it leaves the field of view of the first image capture device, forming the first feature vector set of the target. Within the field of view of the second image capture device, newly emerging targets in the overlapping area during the preset time period are extracted, and feature vectors of each newly emerging target in the first few frames of images when it appears are extracted to form a second feature vector set of potential targets. Calculate the average cosine similarity between the first feature vector set and the second feature vector set of all potential targets. Determine the pair of targets with the highest average cosine similarity that exceeds a preset transmission threshold as the same physical target. Assign the target identity identifier carried by the target to be transmitted in the first image capture device to the corresponding potential target in the second image capture device to complete the target identity transmission across cameras.

3. The multi-camera collaborative target relay tracking and three-dimensional positioning system according to claim 2, characterized in that, The system also includes: The three-dimensional coordinate calculation module, after completing the target identity transfer, obtains the pixel coordinates of the same physical target in the image plane of any two different image capture devices within the same timestamp or a very small time difference. By calling the preset intrinsic parameter matrix and distortion coefficients of each image capture device, the pixel coordinates are subjected to distortion removal processing to obtain coordinates in the normalized camera coordinate system. By combining the extrinsic parameter matrix between the two image capture devices, which includes a rotation matrix and a translation vector, the three-dimensional spatial coordinates of the physical target at the current time are solved from the corresponding pixel coordinate pairs after distortion correction using the principle of triangulation. The calculated three-dimensional spatial coordinates are bound to the corresponding target identity identifier and timestamp to form a three-dimensional trajectory point of the target, and recorded in the global target trajectory database.

4. The multi-camera collaborative target relay tracking and three-dimensional positioning system according to claim 3, characterized in that, The method of using triangulation to solve for the three-dimensional spatial coordinates of the physical target at the current time from the corresponding pixel coordinate pairs after distortion correction includes: Based on the world coordinate system, and according to the projection matrix of the first image capture device, a projection equation is established from the three-dimensional spatial coordinates to the distorted pixel coordinates of the image plane of the first image capture device. Based on the world coordinate system, and according to the projection matrix of the second image capture device, a projection equation is established from the three-dimensional spatial coordinates to the image plane of the second image capture device to remove the distorted pixel coordinates. The two sets of projection equations are combined to form an overdetermined linear system of equations about the three components of the three-dimensional spatial coordinates. The singular value decomposition algorithm is used to solve the overdetermined linear equation system to obtain the optimal three-dimensional spatial coordinate solution in the least squares sense. The optimal three-dimensional spatial coordinate solution is validated for rationality, and erroneous solutions with negative depth values ​​or exceeding the preset effective physical space range are eliminated.

5. The multi-camera collaborative target relay tracking and three-dimensional positioning system according to claim 3, characterized in that, The system also includes: The trajectory optimization module is used to smooth and predict trajectories in the global target trajectory database. From the global target trajectory database, extract all three-dimensional trajectory points of the same target under consecutive timestamps according to the target identity identifier, forming the original three-dimensional trajectory sequence of the target; For the original three-dimensional trajectory sequence, a motion model is established by applying the Kalman filter algorithm to estimate and filter the position and velocity state of the target in three-dimensional space, and a smoothed three-dimensional trajectory point sequence is obtained. The motion model established based on the Kalman filter algorithm uses the estimated state vector from the previous moment as the initial value to predict the target's three-dimensional spatial position in the next moment and generate predicted three-dimensional trajectory points. The smoothed 3D trajectory point sequence and the predicted 3D trajectory points are output together as a high-confidence target 3D motion trajectory.

6. The multi-camera collaborative target relay tracking and three-dimensional positioning system according to claim 5, characterized in that, For the original three-dimensional trajectory sequence, a motion model is established using the Kalman filter algorithm to estimate and filter the position and velocity state of the target in three-dimensional space, resulting in a smoothed three-dimensional trajectory point sequence, including: The coordinates of the target in the three-dimensional space in the original three-dimensional trajectory sequence are used as the observation vector of the system; Establish a state model of the Kalman filter system with the target's position and velocity as the state vector, where the state vector contains three-dimensional position components and three-dimensional velocity components; The state vector of the Kalman filter is initialized with the position information of the first three-dimensional trajectory point in the original three-dimensional trajectory sequence, and the velocity component is initialized to zero. Define the state transition matrix that describes the motion law of the target, and set the system process noise covariance matrix and the observation noise covariance matrix; For each subsequent three-dimensional trajectory point in the original three-dimensional trajectory sequence, the prediction step of the Kalman filter algorithm is executed, and the state vector and state estimate covariance at the current moment are predicted based on the state estimate value at the previous moment using the state transition matrix. The update step of the Kalman filter algorithm is executed, the Kalman gain is calculated, and then the predicted state vector and the state estimation covariance are corrected by combining the observation values ​​of the current three-dimensional trajectory points to obtain the optimal state estimate at the current time. The position components are extracted from the optimal state estimate obtained after each update step and used as smoothed three-dimensional spatial coordinates. These coordinates are then arranged in chronological order to form a smoothed three-dimensional trajectory point sequence.

7. The multi-camera collaborative target relay tracking and three-dimensional positioning system according to claim 5, characterized in that, The system also includes: The tracking relay trigger module is used to determine the continuity of tracking at the edge of the camera network coverage. Real-time monitoring of the high-confidence target's three-dimensional motion trajectory corresponding to each target identity identifier, and calculation of the target's motion direction and speed; Predict the target's movement path in the next time period and determine whether the predicted movement path will enter the physical coverage boundary area of ​​the current camera network; If it is predicted that the device will enter the physical coverage boundary area, then search for other image capture devices adjacent to the physical coverage boundary area that are not connected to the current cooperative network. If there is an image capture device that is not connected to the current collaborative network, a tracking relay request is generated. The tracking relay request includes the target's appearance feature vector, the last known three-dimensional spatial coordinates, and the direction and speed of movement. The tracking relay request is sent to the system to which the image capture device that is not connected to the current collaborative network belongs, so as to trigger a target identity matching and tracking relay process on a larger scale.

8. The multi-camera collaborative target relay tracking and three-dimensional positioning system according to claim 1, characterized in that, The system also includes: The re-identification module is used to handle the re-identification of a target that reappears after a brief disappearance; When the number of frames in which a target disappears from the field of view of an image capture device due to occlusion exceeds a preset threshold, the target's identity is marked as temporarily lost. The field of view of the image capture device is continuously monitored, and when a new initial target is detected in a subsequent frame, the feature vector of the new target is calculated. The feature vector of the new target is compared with the historical set of feature vectors of all targets in a temporarily lost state stored before they disappeared within a preset time window. If the maximum similarity between the feature vector of the new target and the feature vector history set of a target that is temporarily lost exceeds the re-identification threshold, then the new target is determined to be a reappearance of the temporarily lost target, and the original target identity is assigned to the new target to restore its continuous tracking trajectory. If the similarity between the feature vector of the new target and the historical feature vector set of all temporarily lost targets does not exceed the re-identification threshold, then a new target identity is assigned to the new target.

9. The multi-camera collaborative target relay tracking and three-dimensional positioning system according to claim 3, characterized in that, The system also includes: The data processing module is used to remove outliers and interpolate data to complete the 3D trajectory points. For the sequence of three-dimensional trajectory points under the same target identity in the global target trajectory database, the velocity and acceleration are calculated in chronological order; Set velocity thresholds and acceleration thresholds, and mark three-dimensional trajectory points whose instantaneous velocity exceeds the velocity threshold or whose instantaneous acceleration exceeds the acceleration threshold as suspected anomalies; For each suspected anomaly point, check the spatial continuity of its adjacent trajectory points. If the distance between the suspected anomaly point and the points before and after it is much greater than the reasonable displacement calculated based on the velocity of the adjacent points, then the suspected anomaly point is determined to be an abnormal three-dimensional trajectory point and is removed from the sequence. After removing the abnormal three-dimensional trajectory points, the coordinates and timestamps of the valid three-dimensional trajectory points before and after them are used to generate complete three-dimensional spatial coordinates on the missing timestamps using a cubic spline interpolation algorithm, which are then inserted into the target's three-dimensional trajectory point sequence.

10. The multi-camera collaborative target relay tracking and three-dimensional positioning system according to claim 9, characterized in that, The system also includes: The behavior pattern learning module collects high-confidence three-dimensional motion trajectories of a large number of targets over a long period of time from the global target trajectory database, after outlier removal and interpolation completion processing. For each of the three-dimensional motion trajectories of the high-confidence target, the geometric and kinematic features of the trajectory are extracted. The features include the total trajectory length, average velocity, velocity variance, main direction of motion, location of the dwell point, and dwell time. Using an unsupervised clustering algorithm, cluster analysis is performed on the vector set composed of the geometric and kinematic features of all trajectories to automatically discover different trajectory pattern categories; The geometric and kinematic features of the newly generated high-confidence target 3D motion trajectory are matched with the discovered trajectory pattern categories, and the most similar trajectory pattern label is assigned to them to achieve online classification of target behavior.

Images