Abnormal target cross-lens linkage tracking method based on three-dimensional space mapping

Abstract

The application discloses an abnormal target cross-lens linkage tracking method based on three-dimensional space mapping, which comprises the following steps: acquiring a first camera video and identifying an abnormal target; mapping a target two-dimensional position into a first three-dimensional space position under a unified three-dimensional space coordinate system according to a projection relationship; predicting a time window when the target enters a candidate camera field of view based on the position, a motion state and a three-dimensional scene model; acquiring a candidate camera video on demand and detecting a candidate object in the time window; and matching according to multi-dimensional correlation criteria such as space-time and appearance, confirming the target and generating a global track. The application reduces the false matching rate by means of space-time constraint priority and appearance feature assistance, realizes dynamic optimal allocation of computing resources, and supports seamless cross-lens linkage tracking and privacy compliance.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent security, and in particular to a cross-camera cooperative tracking method for abnormal targets based on 3D spatial mapping. Background Art

[0002] With the development of smart city construction, video surveillance systems play a key role in preventing and combating illegal and criminal acts such as intrusion and movement of illegal personnel. Traditional security monitoring has transitioned from the early stage of manual monitoring to the stage of intelligent video analysis. In recent years, remarkable progress has been made in the detection and tracking technologies for illegal personnel. By using target detection, multi-target tracking and cross-camera tracking algorithms under a single camera, the identification and trajectory reconstruction of specific abnormal targets can be realized to a certain extent. However, in actual complex scenarios, illegal personnel often have high confrontational nature, and will deliberately avoid surveillance or use complex terrain to hide, which makes continuous cross-region and cross-camera tracking a research hotspot and difficulty in the current intelligent security field.

[0003] However, the existing cross-camera tracking technologies still have many deficiencies when dealing with the cooperative tracking of illegal personnel. First, most mainstream methods rely excessively on the similarity matching of 2D image features. When illegal personnel cross surveillance areas with non-overlapping fields of view, or when appearance features change drastically due to sudden illumination changes, occlusion and camouflage, target loss or identity drift is very likely to occur. Second, the traditional 2D pixel coordinate system cannot accurately reflect the real spatial position relationship in the physical world, resulting in the lack of a unified spatial scale benchmark for the cooperation between multiple cameras, making it difficult to achieve accurate trajectory prediction and collaborative positioning. Third, when processing video streams with a large range and multiple perspectives, existing systems consume huge computing resources and have high latency, which cannot meet the actual combat requirements for real-time, continuous and three-dimensional cooperative tracking of illegal personnel.

[0004] Therefore, how to use 3D spatial mapping to construct a unified physical coordinate association to solve the problems of spatial discontinuity and feature failure in cross-camera tracking is an urgent technical problem to be solved at present. Summary of the Invention

[0005] The purpose of this section is to outline some aspects of the embodiments of the present invention and briefly introduce some preferred embodiments. Simplifications or omissions may be made in this section, as well as in the abstract and the title of the invention of the present application, to avoid obscuring the purpose of this section, the abstract and the title of the invention, and such simplifications or omissions shall not be used to limit the scope of the present invention.

[0006] In view of the above existing problems, the present invention is proposed. Therefore, the present invention provides a cross-camera cooperative tracking method for abnormal targets based on 3D spatial mapping, so as to solve the problems raised in the background art.

[0007] To solve the above-mentioned technical problems, the present invention provides the following technical solution: a method for cross-camera linkage tracking of abnormal targets based on three-dimensional spatial mapping, comprising: Acquire video data captured by the first camera and identify target objects that match preset abnormal event types; The position information of the target object in the two-dimensional image of the first camera is mapped to a first three-dimensional spatial position in a unified three-dimensional spatial coordinate system according to a preset projection relationship. Based on the first three-dimensional spatial position, the motion state of the target object, and the pre-constructed three-dimensional scene model, predict the field of view of at least one candidate camera that the target object will enter, and determine the corresponding time window. Within the time window, acquire video data captured by the candidate camera and detect whether there is a candidate object; Based on preset cross-camera association criteria, the candidate object is matched with the target object. If the match is successful, the candidate object is confirmed as the target object, and a global trajectory covering the first camera and the candidate camera is generated.

[0008] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention maps targets to a unified three-dimensional physical space, constructing a method that prioritizes spatiotemporal constraints and is supplemented by appearance features. Candidate objects are filtered using physical and kinematic laws, and then comprehensively confirmed using non-sensitive appearance features. This effectively overcomes the problem of misjudgment caused by sudden changes in lighting across lenses, differences in perspective, occlusion, or target camouflage, resulting in different appearances of the same person or different people in the same appearance, thus reducing the probability of losing target tracking or identity drift.

[0009] 2. This invention abandons the traditional exhaustive search strategy of blindly comparing the entire network and introduces a path deduction algorithm based on a 3D scene model and the target's motion state. By predicting the candidate cameras where the target may appear and their dynamic time windows, an event-driven on-demand wake-up mechanism is adopted to selectively retrieve and analyze the video stream of a specific camera only within a specific time period. This targeted retrieval method avoids ineffective global video stream processing, achieves dynamic optimal allocation of system computing resources, effectively reduces system latency, and meets the needs of real-time continuous tracking of abnormal targets in actual combat.

[0010] 3. Furthermore, based on a unified three-dimensional spatial coordinate system, this invention can not only generate continuous three-dimensional global trajectories covering multiple lenses, but also utilize the target's current three-dimensional spatial coordinates and the known installation coordinates of the spherical camera to calculate the required yaw and pitch angles of the pan-tilt unit in real time through spatial geometric calculations. This directly drives the lens to automatically turn and optically zoom and focus, achieving a hardware-linked approach from trajectory tracking to detail locking. Simultaneously, the extraction of strong biometric features such as facial features is deliberately avoided throughout the tracking process. A temporary hash-based global identity identifier is used, coupled with an immutable audit log, enhancing security effectiveness while fully complying with current data security and privacy protection regulations. Attached Figure Description

[0011] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein: Figure 1 This is a flowchart illustrating the overall process of a cross-camera linkage tracking method for abnormal targets based on three-dimensional spatial mapping, according to an embodiment of the present invention. Detailed Implementation

[0012] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0013] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0014] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0015] Furthermore, in the description of this invention, it should be noted that the terms "upper," "lower," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. These terms are used solely for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. In addition, the term "first" is used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0016] Example 1

[0017] Reference Figure 1 This is the first embodiment of the present invention, which provides a method for cross-camera linkage tracking of abnormal targets based on three-dimensional spatial mapping, including: S1. Acquire video data captured by the first camera and identify the target object that matches the preset abnormal event type.

[0018] It should be noted that in real-world, complex security monitoring scenarios, the amount of video stream data is enormous. Performing cross-camera 3D tracking on all moving targets would consume huge amounts of computing resources and cause system latency, failing to meet the practical requirements of real-time, continuous, and coordinated tracking. Therefore, it is necessary to first utilize a lightweight artificial intelligence model for target detection and multi-target tracking at the 2D image level of a single camera (the first camera). This model should then be combined with spatial geometry and kinematic rules to filter out specific targets (i.e., target objects) exhibiting abnormal behavior. This reduces the search space for subsequent 3D spatial mapping and cross-camera matching, improving the overall operational efficiency of the video surveillance system.

[0019] Furthermore, the sequence of video frames captured in real time by the first camera is obtained. For each frame, it is used as input data and fed into a pre-trained deep convolutional neural network (CNN) object detection model.

[0020] Specifically, in this embodiment, the target detection model adopts an architecture based on a feature pyramid network and a multi-scale decoupling head. The internal data processing logic of the model is as follows: First, multi-scale visual features of the image are extracted through the backbone network. Then, the neck network uses a top-down and bottom-up bidirectional fusion mechanism to stitch together the shallow high-resolution features (which are beneficial for small target detection) and the deep high-semantic features (which are beneficial for anti-occlusion). Finally, based on the stitched and fused features, the detection head outputs the two-dimensional bounding box coordinates (including the center point coordinates and the width and height of the box), target category (e.g., vehicle, person, object, etc.) and confidence score of each target in the image.

[0021] It is worth noting that during the pre-training phase of this object detection model, a labeled dataset containing multiple scene monitoring perspectives was constructed. This dataset includes real 2D bounding boxes and category labels for various targets (such as people, vehicles, and objects). During training, a classification loss function (e.g., Focal Loss) was used to address class imbalance, and a bounding box regression loss function (e.g., GIoU Loss) was used to optimize the position of the predicted boxes. The network weights were iteratively updated using a backpropagation algorithm (e.g., the Adam optimizer) until the model's detection accuracy on the validation set reached a preset threshold.

[0022] Furthermore, in order to identify abnormal behaviors that require time span support (such as vehicles driving in the wrong direction or people being stranded), it is also necessary to establish the temporal trajectory of the target in the first camera's view.

[0023] Specifically, the two-dimensional bounding box output by the model and the target features are fed into the multi-target tracking algorithm module. This module uses motion prediction algorithms (such as Kalman filtering) to predict the possible position of the target in the next frame, and combines the target's appearance features with data association algorithms (such as the Hungarian algorithm) to match targets in previous and subsequent frames. When a match is successful, the system assigns a globally unique local tracking ID to each target in the image and records the center point coordinates of the target in each frame in chronological order, generating a continuous historical trajectory of the target in the two-dimensional image coordinate system.

[0024] Furthermore, based on the target categories and continuous historical trajectories generated above, the system extracts the kinematic features of the target in real time (such as speed, direction, and position changes) and performs a logical comparison with the triggering rules of preset abnormal event types.

[0025] Specifically, in this embodiment, the preset abnormal event types include the following five: (1) Target crossing the preset boundary (boundary crossing detection): One or more virtual boundary line segments are preset in the two-dimensional image of the first camera. The system extracts the coordinates of the center point of two consecutive frames in the target trajectory in real time and constructs the motion line between the two points. Through the geometric intersection algorithm, it is determined whether the motion line intersects with the preset virtual boundary line segment in space. If an intersection occurs, it indicates that the target has completed the penetration or boundary crossing behavior in physical space. The system triggers the boundary crossing anomaly and marks the target as a target object to be tracked.

[0026] (2) Reverse movement within a specified area: A legal travel direction vector is predefined within a specified area of ​​the screen. The system generates the actual movement direction vector of the target based on the starting and ending coordinates of the target within a set time window. Subsequently, the spatial angle between the actual movement direction vector and the legal travel direction vector is calculated. When this angle is greater than a preset reverse movement angle threshold (e.g., the angle is obtuse, indicating that the movement direction is opposite or seriously deviated), and the target's moving speed is greater than the stationary judgment threshold, the system determines that a reverse movement anomaly has occurred.

[0027] (3) Staying in a designated area for more than a preset time: A polygonal sensitive area is pre-defined in the two-dimensional image. The system uses a polygonal area inclusion algorithm (e.g., ray casting) to determine in real time whether the center point of the target enters the sensitive area. If the target enters, the system records the timestamp of its entry. In subsequent video frames, if the target remains within the area and the change in its position coordinates (displacement variance) is less than a preset wandering threshold, the system continues to calculate its dwell time. When the dwell time exceeds the preset maximum allowable dwell time, a dwelling anomaly is triggered.

[0028] (4) Target Abandoned Items: Combining background subtraction and target detection models, when the system detects a target such as a backpack / bag, it continuously monitors the position of its bounding box. If the bounding box of the item has a very high overlap (cross-over-union ratio) in multiple consecutive video frames, it indicates that the item is stationary. At this time, the system searches for personnel targets with historical trajectories intersecting with the item within a preset spatial radius, centered on the item. If no related personnel are found within a set time, the abandoned item is determined to be abnormal. The system not only treats the abandoned item as a target object, but also extracts and marks the person who initially put down the item as a target object through historical trajectory backtracking for subsequent cross-camera tracking.

[0029] (5) Multiple Targets Abnormally Gathering: For all personnel targets detected in the image, the system calculates the two-dimensional spatial straight-line distance between the center points of any two targets in real time. A gathering distance threshold representing the proximity of personnel is set. For any target in the image, the system counts the number of neighboring targets whose spatial distance is less than the threshold. When the number of neighboring targets of a certain target reaches or exceeds the preset abnormal gathering number threshold (e.g., 5 people), it is determined that an abnormal gathering event has occurred in the area, and the core members within the gathering group are marked as target objects.

[0030] It should be noted that through this step, the system can automatically extract target objects with high potential threats from massive amounts of normal background video streams, and output the position information and motion state of the target object in the two-dimensional image of the first camera, providing basic data input for subsequent mapping to a unified three-dimensional spatial coordinate system.

[0031] S2. The position information of the target object in the two-dimensional image of the first camera is mapped to a first three-dimensional spatial position in a unified three-dimensional spatial coordinate system according to a preset projection relationship.

[0032] It should be noted that traditional cross-camera tracking relies excessively on 2D image features, making it easy to lose track of targets when they cross non-overlapping fields of view or experience perspective distortion. The fundamental reason is that the 2D pixel coordinate system lacks depth information from the physical world and a unified spatial scale. To address this issue, this invention utilizes the inverse process of camera imaging, combined with prior 3D scene geometric constraints, to upgrade isolated 2D pixel coordinates and unify them into a global physical space. This eliminates the perspective distortion differences caused by varying installation heights and pitch angles of different cameras.

[0033] Furthermore, during the system initialization phase, a global unified three-dimensional spatial coordinate system (i.e., the world coordinate system) is pre-defined. Through camera calibration technology, the internal parameters (e.g., focal length, distortion coefficients) and external parameters (i.e., the actual physical position, installation height, and pitch / yaw angles of each camera in the world coordinate system) of the first camera and all candidate cameras are obtained. Simultaneously, based on the real geographic information of the monitored scene, a three-dimensional scene model is constructed. In this scene model, the ground elevation information, passable areas (e.g., roads, corridors), and physical obstacles (e.g., walls, green belts) in the physical space are defined.

[0034] Furthermore, the detection box of the target object in the 2D image of the first camera is obtained. Since the target object, such as a pedestrian or vehicle, moves on the ground, its reference point for contact with the physical world is located at its bottom. Therefore, in this invention, the geometric center of the detection box is not used directly. Instead, the horizontal center position of the detection box is calculated, and combined with the bottom edge of the detection box, the bottom center point of the detection box is extracted. This center point represents the contact position between the target object and the ground in the 2D image.

[0035] Furthermore, based on the reverse imaging principle of cameras, any pixel on a two-dimensional image actually corresponds to a line of sight ray originating from the optical center of the camera lens and projecting into the physical world in real three-dimensional space. Due to the lack of depth information, the specific position of the target on the ray cannot be determined solely by this ray. To obtain a uniquely identifiable three-dimensional coordinate, this invention introduces the physical ground plane constraint of the three-dimensional scene model. The extracted bottom center point is transformed into the corresponding line of sight ray, and the spatial intersection point of this ray with the ground plane in the three-dimensional scene model is calculated. This intersection point is the uniquely identifiable first three-dimensional spatial position of the target object in a unified three-dimensional spatial coordinate system. It should be noted that if the three-dimensional scene model contains multiple layers, such as stairs or a skybridge, the corresponding local terrain plane is automatically selected to perform intersection calculations with the ray based on the semantic region to which the target belongs in the two-dimensional image, thus adapting to complex terrain.

[0036] It should be noted that the process of determining this first three-dimensional spatial position is as follows: Let the coordinates of the bottom center point of the target in the two-dimensional image pixel coordinate system be... The intrinsic parameter matrix of the first camera is The extrinsic rotation matrix is The translation vector is According to the traditional pinhole camera model, its corresponding three-dimensional spatial ray equation satisfies: ,in Depth in camera coordinate system These are three-dimensional coordinates in the world coordinate system. This is combined with the equations of the ground plane in the 3D scene model (e.g., the equations of locally flat terrain). Substituting these values ​​into the above equation will uniquely solve for the target's first three-dimensional spatial position. .

[0037] Furthermore, in practical engineering applications, due to pixel-level jitter in the target detection bounding box (e.g., changes in the bottom edge of the box caused by a person stepping), and the unavoidable slight errors in camera calibration parameters, directly using a single calculated coordinate point as the absolute position can easily lead to matching failures in cross-camera matching due to minor deviations. Therefore, this embodiment generates a three-dimensional confidence region (i.e., the physical range within which the target is highly probable) after calculating the first three-dimensional spatial position.

[0038] Specifically, based on the principle of perspective, the farther a target is from the camera, the greater the deviation of its physical distance in three-dimensional space for a one-pixel deviation in the image. Based on this, the physical distance from the first three-dimensional spatial position to the optical center of the first camera is calculated. Then, an elliptical confidence region is generated centered on the first three-dimensional spatial position. The major axis of this ellipse follows the aforementioned line-of-sight ray direction, and its length is proportional to the physical distance (i.e., the greater the distance, the greater the error tolerance along the line-of-sight direction); the minor axis is perpendicular to the ray direction and is relatively shorter. Simultaneously, combined with the confidence score output by the target detection model in S1, the lower the score, the larger the overall range of the confidence region. Finally, a data set containing the first three-dimensional spatial position and its corresponding three-dimensional confidence region is output. It should be explained that this data set represents the current real-world spatial range of the target object.

[0039] S3. Based on the first three-dimensional spatial position, the motion state of the target object, and the pre-constructed three-dimensional scene model, predict the field of view of at least one candidate camera that the target object will enter, and determine the corresponding time window.

[0040] It should be noted that in traditional cross-camera tracking systems, when a target leaves the current camera's view, the system typically blindly wakes up all surrounding cameras for a full network comparison. This exhaustive search not only consumes a large amount of computing resources but also easily introduces distracting targets with similar appearances (e.g., unrelated people wearing the same color clothing), leading to false matches. Therefore, the solution in this invention adds spatiotemporal constraints to cross-camera tracking. That is, by deducing where the target can physically go and when it can arrive, the above exhaustive search is reduced to a targeted retrieval for specific cameras and within a specific time period, thereby improving the system's operating efficiency and matching accuracy.

[0041] Furthermore, based on the continuous historical trajectory generated in S1 and the unified three-dimensional spatial coordinate system mapped in S2, the instantaneous motion state of the target object when it leaves the field of view of the first camera (i.e., the vanishing point) is extracted.

[0042] Specifically, the initial three-dimensional spatial position of the target object before it disappears is obtained, and by calculating the three-dimensional coordinate displacement difference of the target object in several frames before it disappears, the scalar movement speed (i.e., how fast it moves) and the three-dimensional movement direction (i.e., the direction of travel) of the target object in three-dimensional space are extracted.

[0043] Furthermore, the aforementioned initial motion state parameters are input into a path deduction algorithm based on a 3D scene model. The deduction algorithm is performed using a discrete-time step simulation method: First, starting from the initial 3D spatial position, the ideal predicted position is calculated after a small time step based on the target's movement speed and direction. Then, a scene constraint correction mechanism is introduced: by querying a pre-built 3D scene model, it is determined whether the ideal predicted position falls within an obstacle or an impassable area. If not, the ideal predicted position is used as the actual predicted position for the next moment; if it falls within an obstacle, a correction vector along the obstacle's edge is generated based on the obstacle's geometric edge orientation, forcibly pulling the predicted position back to a nearby passable area, and adjusting the target's movement direction accordingly (this process aims to simulate the real physical behavior of a target encountering a wall or obstacle and circling around its edge).

[0044] It should be noted that through multiple iterations of the above-mentioned deduction algorithm, the system can generate one or more possible movement paths that conform to the actual terrain in the three-dimensional scene model.

[0045] Furthermore, after generating possible motion paths, it is necessary to determine which camera feeds the target will appear in next.

[0046] Specifically, in the pre-built 3D scene model, the field of view of each camera is modeled as a 3D view frustum (i.e., the 3D spatial range that the camera can see). Using spatial geometric intersection detection logic, the system checks whether each deduced possible motion path spatially intersects with the 3D view frustums of other cameras in the scene. If the possible motion path crosses the boundary of a camera's view frustum, it indicates that the target is highly likely to have entered that camera's monitoring range, and the system marks that camera as a candidate camera. It should be explained that the purpose of this process is to effectively eliminate invalid cameras that, although physically close, have their lenses facing away from the target's direction of movement.

[0047] Furthermore, after identifying the candidate cameras, it is necessary to calculate the time range within which the target appears in the view of the candidate camera.

[0048] Specifically, first, the actual path length of the target from its initial 3D spatial position to the candidate camera's view frustum boundary along the aforementioned possible motion paths is calculated. It is important to emphasize that this length is not the straight-line distance between two points, but rather the cumulative length along a curved path constrained by terrain. Subsequently, this actual path length is divided by the target's scalar velocity to obtain the base arrival time.

[0049] Furthermore, to improve the system's fault tolerance, considering that the target may accelerate, decelerate, or briefly move in close proximity within the monitoring blind zone, this invention does not use a single point in time, but instead constructs a dynamic time window based on the baseline arrival time. The upper and lower limits of this time window are set by comprehensively considering the following two compensation factors: Behavioral tolerance compensation: Based on the preset target type, an acceleration tolerance ratio and a deceleration tolerance ratio are assigned, thereby extending the base arrival time forward and backward; Positioning error compensation: Extract the spatial length (i.e., positioning error distance) of the three-dimensional confidence region generated in S2 along the target's movement direction. Divide this spatial length by the target's scalar movement velocity to calculate the time uncertainty caused by the positioning error. This time uncertainty is then added as a compensation time to the base arrival time, thereby relaxing the time window restriction. This effectively prevents missed detections caused by the target appearing outside the time window due to initial positioning deviation.

[0050] It should be noted that the final output contains a set of instructions including the candidate camera ID and the corresponding dynamic time window, which guides when and where to wake up and retrieve video data.

[0051] S4. Within the time window, acquire the video data captured by the candidate camera and detect whether there is a candidate object.

[0052] It should be noted that in traditional security monitoring systems, video streams from each camera are typically continuously transmitted to a central server for 24 / 7 analysis. This not only consumes a significant amount of network bandwidth but also results in a severe waste of computing resources. Therefore, this invention uses the candidate camera IDs output by S3 and dynamic time windows as trigger conditions, employing an event-driven, on-demand wake-up mechanism. Only during specific time periods when the target is highly likely to appear are the video streams from specific cameras retrieved and subjected to intensive computation, thereby achieving dynamic and optimal allocation of system computing resources while ensuring tracking continuity.

[0053] Furthermore, the system's backend scheduling module compares the current system time with the dynamic time window output by S3 in real time. To prevent missing the exact moment a target enters the frame due to network streaming delays, the system adds a pre-start buffer time (e.g., 2 to 3 seconds earlier) at the beginning of the time window. When the system time reaches this pre-start buffer time, it automatically sends a video stream retrieval request to the corresponding candidate camera and initiates a local multi-target tracking algorithm. This pre-start method ensures that when the target actually enters the prediction time window, the system has accumulated enough historical video frames (e.g., a trajectory sequence of more than half a second), enabling it to immediately calculate the accurate instantaneous speed and direction of motion the moment the target appears.

[0054] Furthermore, after acquiring video frames from candidate cameras, the system needs to determine whether the moving targets appearing in the frame are likely the target objects we want to track.

[0055] Specifically, the acquired video frames are input into the target detection model to identify all moving targets in the scene and output two-dimensional detection boxes for these targets.

[0056] Furthermore, the bottom center point of each target detection box (representing the target's ground point) is extracted. Combined with the intrinsic and extrinsic parameter matrices of the candidate camera and the ground plane constraints in the 3D scene model, the 3D coordinates of each moving target in the image are calculated in a unified 3D spatial coordinate system through inverse perspective projection. These moving targets that appear within the time window and whose category is consistent with the target object (e.g., all are people or vehicles) are defined as candidate objects.

[0057] Furthermore, in order to make subsequent judgments on the consistency of motion states, it is necessary to calculate the kinematic characteristics of the candidate object in the candidate camera frame.

[0058] Specifically, a multi-object tracking algorithm is used to locally track candidate objects within short video frames of candidate cameras, obtaining their 3D spatial position sequence over consecutive time steps. Then, the current video frame and a frame traced back at a specific interval (e.g., a historical frame from half a second ago) are selected as two observation points, and the 3D spatial displacement of the candidate object between these two observation points is calculated. Dividing this spatial displacement distance by the time difference between the two observations yields the instantaneous velocity of the candidate object; simultaneously, the direction of the 3D spatial line connecting the historical observation point to the current observation point is defined as the instantaneous motion direction of the candidate object.

[0059] Furthermore, considering that in complex scenarios, multiple candidate objects may simultaneously meet the conditions based solely on spatiotemporal relationships (e.g., people performing the same / similar actions), it is necessary to extract the appearance features of candidate objects to provide auxiliary confirmation data for subsequent appearance similarity judgment. Meanwhile, to meet privacy compliance requirements, this invention deliberately avoids extracting strong biometric features such as faces, instead extracting non-sensitive appearance features such as clothing color, texture, body shape, and carried items.

[0060] Specifically, a deep learning-based feature extraction model (using ResNet-50 as the backbone network) was constructed and applied. The input to this model is a candidate object image patch cropped from the original video frame based on the aforementioned two-dimensional detection boxes. Before input, the object image patch is uniformly scaled to a fixed resolution and normalized. The model's output is a floating-point feature vector. This feature vector does not contain specific semantic labels (e.g., red clothing), but rather represents the non-sensitive global appearance attributes of the candidate object in the latent space. Simultaneously, during the model pre-training phase, a metric learning mechanism (triple loss function) is employed. This forces the model to bring the feature vectors extracted from the same physical entity (even under different lighting and camera perspectives) closer together, while pushing the feature vectors of different physical entities further apart.

[0061] It should be noted that the training steps for this deep learning model are as follows: Construct a cross-camera pedestrian / vehicle re-identification dataset; in each training iteration, construct triplet samples. Anchor point samples For a target image captured by a certain camera, positive samples For images of the same target taken from other cameras, negative samples Given images of different targets from any camera, input the triples into the ResNet-50 backbone network to extract feature vectors and calculate the triple loss:

[0062] in, This represents the Euclidean distance between eigenvectors. The default boundary margin parameter is used; the network parameters are optimized by minimizing the loss of this triplet, so that the feature vector output by the model has illumination and viewpoint invariance across lenses.

[0063] S5. Based on the preset cross-camera association criteria, match the candidate object with the target object. If the match is successful, confirm the candidate object as the target object and generate a global trajectory covering the first camera and the candidate camera.

[0064] It should be noted that traditional cross-camera tracking techniques often rely heavily on the appearance features (such as clothing color and texture) of the target's two-dimensional image for similarity calculation. However, in real-world cross-camera scenarios, significant differences in lighting conditions, white balance settings, and shooting angles between different cameras can easily lead to misjudgments of the same person appearing as different people or vice versa. To address this issue, this invention uses the laws of physics and kinematics to filter candidate objects within a unified three-dimensional physical world. Only when the objective laws of the physical world are met is appearance feature introduced for final auxiliary confirmation, thereby reducing the system's false matching rate.

[0065] Furthermore, for each candidate object extracted in S4, its feature set is compared with the feature set of the target object when it leaves the first camera, and the following four criteria are calculated in sequence: Time window matching criterion (fast filtering layer): First, extract the timestamp of the candidate object appearing in the candidate camera's view, and determine whether the timestamp falls within the dynamic time window predicted by S3. This criterion is a hard logical threshold: if the appearance time of the candidate object is not within the predicted time window, it means that it violates the normal physical movement time of the target object (e.g., the target cannot cross an extremely long physical distance in a very short time), and the system will directly eliminate the candidate object, thereby avoiding subsequent complex calculations and saving system computing power; if it is within the time window, it proceeds to the next criterion.

[0066] Spatial reachability criterion (physical geometry layer): Under the premise of satisfying the time window, assess whether the current 3D spatial position of the candidate object is within a reasonable reachable range from the vanishing point of the target object. First, in the pre-built 3D scene model, the pathfinding algorithm calculates the actual shortest physical path length along the passable area from the vanishing point of the target to the appearance point of the candidate object. Then, based on the historical movement speed of the target object in the first camera before disappearing, multiply it by the time spent in the monitoring blind zone (i.e., the time difference between the appearance time of the candidate object and the disappearance time of the target), calculate the distance that the target should theoretically travel. In this process, the system compares the difference between the actual physical path length and the theoretical travel distance: the smaller the difference, the more the position of the candidate object conforms to the physical movement law of the target, and the higher the spatial reachability score; the larger the difference, the lower the score. At the same time, the error tolerance of this score is positively correlated with the size of the 3D confidence region generated in S2; the larger the confidence region, the higher the system's tolerance for distance differences.

[0067] Motion State Consistency Criterion (Kinematic Level): Based on the principle of inertia in physics, when a target crosses a monitoring blind spot in a short time, its motion direction and velocity usually exhibit a certain continuity. This is achieved by extracting the instantaneous motion direction vector and velocity magnitude of the candidate object calculated in S4 and comparing them with the historical direction vector and velocity magnitude of the target object. The difference in the directional angle and the difference in absolute velocity values ​​between the two in three-dimensional space are calculated respectively. The smaller the directional angle and the closer the velocity values, the better the continuity of the motion state, and the higher the corresponding motion state consistency score.

[0068] (4) Appearance Similarity Criterion (Feature Auxiliary Layer): Based on the above spatiotemporal and kinematic criteria, non-sensitive appearance features of candidate objects output by the deep learning model in S4 are extracted and the distance between them and the appearance features of the target object in the vector space is measured (cosine similarity is calculated). Since the feature vector has been stripped of illumination and viewpoint interference during the model training stage, the similarity score can objectively reflect the inherent consistency between the two in global attributes such as clothing and posture, and serve as the final auxiliary confirmation basis.

[0069] Furthermore, a dynamic weighted fusion strategy is adopted to sum the spatial accessibility score, motion state consistency score, and appearance similarity score in a weighted manner to obtain the final comprehensive matching degree.

[0070] Specifically, the system dynamically allocates weights based on the time difference the target spends within the blind zone. When the time difference is small (i.e., the blind zone is extremely short), the target is unlikely to undergo drastic changes in direction or clothing; in this case, the weights of spatiotemporal and kinematic criteria are automatically increased, while the weights of appearance criteria are decreased. Conversely, when the time difference is large, the uncertainty of the target's movement increases, so the weight of appearance feature criteria is appropriately increased. When the overall matching degree exceeds the preset matching confirmation threshold, the system determines that the match is successful and formally confirms the candidate object as the target object.

[0071] Furthermore, to comply with data security and privacy protection regulations, after confirming the target, this invention does not extract strong biometric features such as facial features. Instead, it generates a globally unique, temporary identity for the target object using a one-way hash function (hash algorithm), combined with the current timestamp and a random number. This identity is only valid during the current anomaly tracking period and is automatically destroyed after the event ends. Subsequently, the 3D trajectory sequence of the target object in the first camera, the inferred path in the blind zone (generated by interpolation based on the 3D scene model), and the 3D trajectory sequence in the candidate cameras are spatially stitched and smoothed to generate a continuous 3D global trajectory covering multiple shots. Simultaneously, the system's backend audit module will record the generation time, access nodes, and subsequent handling operations of this global trajectory in an immutable log for post-event security auditing. Finally, after generating the global trajectory and confirming the target's current 3D spatial position, the system automatically triggers coordinated hardware and software actions: On the one hand, the system scheduling module automatically pushes the real-time video footage of the candidate cameras that have captured the target object to the user terminal of the security personnel (such as a monitoring screen or mobile police terminal), and overlays and displays its three-dimensional global trajectory and abnormal event type on the screen.

[0072] On the other hand, if a PTZ-enabled dome camera is deployed within the monitoring scene, the system will extract the current 3D spatial coordinates of the target object and combine them with the known installation coordinates of the dome camera itself in a unified 3D spatial coordinate system. Utilizing the relative spatial positional relationship between the two, the system will perform spatial geometric calculations to determine in real-time the horizontal yaw angle and vertical pitch angle required for the dome camera to rotate. The calculated angle commands will then be sent to the dome camera's underlying control interface, driving the lens to automatically turn and optically zoom to focus on the target object, thereby achieving coordinated monitoring from trajectory tracking to detailed focus.

[0073] It should be noted that the specific solution process is as follows: Let the current three-dimensional spatial coordinates of the target object be... The three-dimensional coordinates of the optical center of the spherical camera are The relative displacement difference between the two in three-dimensional space is calculated as follows: , , The horizontal yaw angle is then calculated as follows: The vertical pitch angle is calculated as follows: The system converts the yaw and pitch angles into control commands and sends them to the gimbal.

[0074] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. An abnormal target cross-camera linkage tracking method based on three-dimensional space mapping, characterized in that, include: Acquire video data captured by the first camera and identify target objects that match preset abnormal event types; The position information of the target object in the two-dimensional image of the first camera is mapped to a first three-dimensional spatial position in a unified three-dimensional spatial coordinate system according to a preset projection relationship. Based on the first three-dimensional spatial position, the motion state of the target object, and the pre-constructed three-dimensional scene model, predict the field of view of at least one candidate camera that the target object will enter, and determine the corresponding time window. Within the time window, acquire video data captured by the candidate camera and detect whether there is a candidate object; Based on preset cross-camera association criteria, the candidate object is matched with the target object. If the match is successful, the candidate object is confirmed as the target object, and a global trajectory covering the first camera and the candidate camera is generated.

2. The cross-camera linkage tracking method for abnormal targets based on three-dimensional spatial mapping as described in claim 1, characterized in that, The preset abnormal event types include: The target crosses the preset boundary, travels against the flow of traffic in the designated area, stays in the designated area for more than the preset time, leaves behind items, or multiple targets gather abnormally.

3. The cross-camera linkage tracking method for abnormal targets based on three-dimensional spatial mapping as described in claim 1, characterized in that, The unified three-dimensional spatial coordinate system and the three-dimensional scene model are pre-constructed, including: The internal and external parameters of multiple cameras in the monitoring scene are calibrated to establish the unified three-dimensional spatial coordinate system. Based on the geometric structure information of the monitoring scene, a three-dimensional scene model is constructed, in which passable areas, restricted areas, blind spots, and the field of view coverage of each camera are defined.

4. The cross-camera linkage tracking method for abnormal targets based on three-dimensional spatial mapping as described in claim 1, characterized in that, Mapping the position information in the two-dimensional image to the first three-dimensional spatial position includes: Extract the detection box of the target object in the two-dimensional image and determine its projection point on the ground as two-dimensional position information; Using the intrinsic and extrinsic parameters of the first camera and the ground plane or multi-layer plane constraints in the three-dimensional scene model, the three-dimensional coordinates of the projection point in the unified three-dimensional spatial coordinate system are calculated, which are used as the first three-dimensional spatial position.

5. The cross-camera linkage tracking method for abnormal targets based on three-dimensional spatial mapping as described in claim 1 or 4, characterized in that, Mapping the position information in the two-dimensional image to the first three-dimensional spatial position further includes: A three-dimensional confidence region is generated for the first three-dimensional spatial location to characterize the positioning uncertainty, and the confidence region is used as the basis for calculation in the subsequent prediction and association criteria process.

6. The cross-camera linkage tracking method for abnormal targets based on three-dimensional spatial mapping as described in claim 1, characterized in that, Predicting the candidate cameras and determining the time window includes: Within the passable area defined by the three-dimensional scene model, the possible movement path of the target object in the future is deduced based on its current speed and direction. Determine which cameras' field of view coverage overlaps with the possible motion path to identify candidate cameras; Based on the length of the motion path and the speed of the target object, the estimated time to reach the field of view of each candidate camera is determined, and the time window is determined.

7. The cross-camera linkage tracking method for abnormal targets based on three-dimensional spatial mapping as described in claim 1, characterized in that, The cross-camera association criteria include: Spatial reachability criterion, used to determine whether the three-dimensional spatial position of the candidate object is within a reasonable reachable range starting from the first three-dimensional spatial position of the target object; The time window matching criterion is used to determine whether the occurrence time of the candidate object falls within the predicted time window; The motion state consistency criterion is used to determine whether the motion direction and speed of the candidate object are consistent with the motion state of the target object when it leaves the first camera.

8. The cross-camera linkage tracking method for abnormal targets based on three-dimensional spatial mapping as described in claim 7, characterized in that, The cross-camera association criteria also include: The appearance similarity criterion is used to extract non-sensitive appearance features of the target object and the candidate object for comparison. The comparison result serves as an auxiliary confirmation basis based on satisfying the spatial reachability, temporal window matching and motion state consistency criteria.

9. The cross-camera linkage tracking method for abnormal targets based on three-dimensional spatial mapping as described in claim 1, characterized in that, The method also includes a coordinated handling step: After the candidate object is confirmed as the target object, the video image of the candidate camera is automatically pushed to the user terminal, or the camera with pan-tilt function is controlled to turn to the three-dimensional spatial position after the target object is confirmed.

10. The cross-camera linkage tracking method for abnormal targets based on three-dimensional spatial mapping as described in claim 1, characterized in that, The method also includes a privacy compliance processing step: Generate a temporary or hashed global identity for the target object that is independent of specific biometric features and is valid during event tracking; The generation, access, and processing of the global trajectory are recorded for post-event auditing.

Images