A park intrusion prevention identification and tracking method based on face and gait fusion

Abstract

The application discloses a kind of based on the park invasion prevention identification and tracking method of face and gait fusion, belong to the park security technology field, method includes: S1, video frame is collected by multiple high-definition network camera;S2, target detection is carried out according to video frame, the trajectory of target is generated and local trajectory ID is distributed;S3, according to local trajectory ID, cross camera is associated global trajectory ID;S4, according to global trajectory ID, extract face feature and gait feature, and the identity of target is identified by the multimodal fusion of face and gait;S5, according to the target identity of identification, invasion discrimination and real-time early warning are carried out.The application is through the multimodal cascade identification technology of face and gait feature, the accuracy and robustness of target detection and identity determination are significantly improved, solve the problem that human cost is high, response lag and is susceptible to subjective fatigue influence in traditional park security scheme and is caused by manual patrol or monitor's naked eye inspection.

Description

Technical Field

[0001] This invention belongs to the field of campus security technology, specifically relating to a campus intrusion prevention identification and tracking method based on face and gait fusion. Background Technology

[0002] Traditional park security and intrusion prevention systems primarily rely on fixed-location surveillance cameras and manual patrols by security personnel. These systems typically trigger alarms through simple line-crossing detection or motion detection, exhibiting the following significant drawbacks: First, high false alarm rate: They are highly susceptible to interference from environmental factors such as shadows and changes in lighting. Second, limited identification capabilities: While advanced systems generally employ single facial recognition technology, in real-world park scenarios, intruders often wear masks or hats, or have their backs to the camera, making facial features undetectable and causing the system to malfunction. Third, lack of continuous cross-regional tracking capabilities: When a target moves between different camera views, traditional systems will identify the same target as multiple independent individuals, failing to establish a complete global activity trajectory and hindering effective data accumulation and early warning for frequently appearing "suspicious strangers."

[0003] Therefore, there is a need for a new type of anti-intrusion system that can adapt to complex obstructed environments, has the ability to track multiple cameras collaboratively, and can intelligently build databases and issue early warnings for strangers. Summary of the Invention

[0004] To address the aforementioned shortcomings in existing technologies, this invention provides a campus intrusion identification and tracking method based on face and gait fusion, which solves the problems of high labor costs, slow response, and susceptibility to subjective fatigue caused by relying on manual patrols or visual inspections by monitoring personnel in traditional campus security solutions.

[0005] To achieve the aforementioned objectives, the technical solution adopted by this invention is as follows: a campus intrusion prevention identification and tracking method based on face and gait fusion, comprising the following steps: S1. Capture video frames using multiple high-definition network cameras; S2. Detect targets based on video frames, generate target trajectories, and assign local trajectory IDs; S3. Associate global trajectory IDs across cameras based on local trajectory IDs; S4. Extract facial and gait features based on the global trajectory ID, and identify the target identity through multimodal fusion of face and gait; S5. Intrusion detection and real-time early warning based on the identified target identity.

[0006] Furthermore: In S1, the method for the high-definition network camera to capture video frames is as follows: Initialize the video stream access request, and sequentially call the standard OpenCV interface, FFmpeg backend and GStreamer backend to perform cascaded detection on different video decoding backends. If any backend successfully obtains a valid frame, the access is considered successful. If the request returns a port conflict error, the automatic port rewriting mechanism is used to attempt to replace the conflicting port in the URL with the default port for reconnection; at the same time, connection timeout and read timeout parameters are set to automatically repair port conflicts or connection abnormalities. Establish an asynchronous thread pool for read-write separation: The frame-capturing thread is responsible for continuously reading the raw pixel matrix from the RTSP stream or local camera. In real-time streaming mode, it does not sleep to clear the buffer and prevent latency backlog. The processing thread retrieves the latest frame from a fixed-length blocking queue and sends it to the algorithm inference layer. When the queue is full, the oldest frame is automatically discarded to ensure that the inference layer always retrieves the latest frame and avoids latency backlog caused by network jitter. When the continuous read failure exceeds 50 times, the reconnection mechanism is automatically triggered.

[0007] Furthermore, S2 includes the following sub-steps: S21. Use a deep convolutional neural network to perform human detection on video frames and obtain human target bounding boxes; S22. Input the bounding box into the multi-target tracker, predict the target's position in the next frame based on Kalman filtering, and associate it with the target using the color histogram algorithm to generate the target's trajectory and assign a local trajectory ID to the target. In S2, a memory safety management mechanism is adopted during the target detection process based on video frames: when calling the underlying recognition SDK for feature extraction, pointer lifetime binding technology is used to forcibly convert the image memory address to a c_ubyte pointer through ctypes and establish a feature buffer dictionary to ensure that the underlying C++ memory block will not be incorrectly released by the Python garbage collection mechanism in a multi-threaded concurrent environment.

[0008] Furthermore, S3 includes the following sub-steps: S31. Extract the appearance features of the current target and calculate the cosine similarity by combining the appearance features of the historical trajectory. The appearance features are obtained by using the HSV color histogram method: the human body area is cropped from the target bounding box, converted to the HSV color space, and the hue, saturation and brightness histograms are extracted and normalized and stitched into 24-dimensional appearance features. S32. Based on cosine similarity, a cross-camera and same-camera association strategy is adopted to associate the local trajectory ID with the global trajectory ID, and the target trajectory is merged into the historical trajectory.

[0009] Furthermore: In S32, the association strategies for cross-camera and same-camera connections are specifically as follows: S321. Cross-camera association: When the nearest camera of the historical trajectory is different from the current camera, if the cosine similarity is greater than the cross-camera matching threshold and the time interval is within the maximum time interval, the local trajectory ID is associated with the global trajectory ID corresponding to the historical trajectory. S322, Self-repair of breakpoints with the same camera: When the nearest camera of the historical trajectory is the same as the current camera, determine whether it is a continuation of the trajectory of the same target by combining the following three conditions: Condition 1, the bounding box IoU is greater than or equal to the self-repair IoU threshold; Condition 2, the cosine similarity is greater than or equal to the self-repair similarity threshold; Condition 3, the time interval is within the self-repair time window; When all three conditions are met, the local trajectory ID is associated with the global trajectory ID corresponding to the historical trajectory. S323. If the above matching fails, a new global trajectory ID is assigned to the local trajectory ID, and its appearance features are extracted and stored in the feature cache. The feature cache is updated smoothly using an exponential moving average to prevent interference from single-frame noise.

[0010] Furthermore: In S31, the cosine similarity of historical targets is calculated. The specific expression is: In the formula, For the current target appearance features, For the appearance characteristics of historical trajectories, As a stable term, Represents the inner product of eigenvectors. This indicates the Euclidean modulus.

[0011] Furthermore, S4 includes the following sub-steps: S41. Obtain the global trajectory ID and its corresponding BBox sequence from the single tracker through the logic controller, extract the full-body human image to obtain the gait image, and extract the facial region image to obtain the face image. S42. Trigger the face recognition process: Perform quality verification on the face image. When the face region size, clarity, and brightness meet the requirements, extract the face features. It is then compared with a facial feature database to obtain facial identifiers and the highest matching score. Matching score with the second highest .in, Used to indicate the degree of similarity between the current face and the optimal candidate face. Used to determine the distinguishability between the best candidate and the second-best candidate; Triggering the gait recognition process: Extracting gait features from gait images. The candidate identities are then compared with the gait feature database to obtain the original gait candidate identities. Optimal matching distance and suboptimal matching distance ; S43. Based on the highest matching score Matching score with the second highest Calculate the margin of face matching : In the formula, the face matching boundary Used to represent the degree of distinction between the best and second-best face candidates, based on the highest matching score. Marginalities of face matching The validity of a facial recognition identifier is determined based on the following two conditions: Condition 4: Highest matching score Greater than the effective face recognition threshold; Condition 5: Face matching margin The score is greater than the face boundary threshold, where the face boundary threshold is dynamically adjusted: when the highest matching score is reached... When the score is less than the high-score reference value, the face boundary threshold takes the standard default value; when the highest matching score is less than the reference value, the threshold takes the standard default value. When the score exceeds the high reference value, the facial boundary threshold should be appropriately lowered. The best-matching face identifier will be selected as the face candidate only if both of the above conditions are met. and the highest matching score As a confidence level of the face ; Based on the optimal matching distance and suboptimal matching distance Calculate the gait distinction margin : Based on the optimal matching distance Distinguishing the boundary with gait Calculate gait confidence : In the formula, This represents the baseline confidence level calculated from the optimal distance. This represents the correction factor obtained from the marginal conversion based on gait distinction. This represents the maximum distance threshold for gait. This represents the gait discrimination threshold; S44. Identify valid gait candidates through gait comparison: For each target trajectory, maintain a historical queue of gait comparison results. The queue stores candidate IDs, gait confidence scores, and gait discrimination margins generated from the most recent gait comparisons in chronological order. When new data is written to the queue, the system checks the data in reverse order from the most recent result, and counts the number of records that simultaneously meet the following three conditions as the consecutive hit count of the candidate identity: Condition 6, the candidate ID is consistent with the current candidate ID; Condition 7, the gait confidence is greater than or equal to the gait stability confidence threshold; Condition 8, the gait discrimination margin is greater than or equal to the corresponding margin threshold. Throughout the entire lifecycle of the target trajectory, a flag indicating whether a face has been detected is maintained. This flag is initialized to "No" when the trajectory is created. When a valid face region is successfully detected in any frame of the trajectory, the flag is permanently set to "Yes". Successfully detecting a valid face region in any frame of the trajectory requires meeting the following conditions simultaneously: Condition 9: At least one face is detected in the current frame via the SDK face detection interface, and the width and height of the bounding rectangle of the face are both no less than 50 pixels; Condition 10: The Laplacian variance is calculated as a sharpness index after Gaussian denoising of the face region, and this index is no less than a preset minimum sharpness threshold; Condition 11: The average pixel brightness of the face region is between 40 and 220, excluding invalid images that are too dark or overexposed. During the fusion determination phase, the current scene is divided into two categories based on the value of this flag: In a typical scenario: the flag is set to "Yes", the stable hit count threshold is 2, the confidence threshold is 0.65, and the marginal threshold is 0.08. Faceless scenario: The flag is "No", the stable hit threshold is 3 times, the confidence threshold is 0.73, the marginal threshold is 0.10, and the target must accumulate a predefined number of gait acquisition data within the recognition area; If the number of consecutive hits in a normal scene reaches the corresponding stable hit threshold, or the number of consecutive hits in a faceless scene reaches the corresponding stable hit threshold, then the gait candidate identity is marked as a valid gait candidate identity. S45. Based on facial recognition candidate identity Face confidence Effective gait candidate identity Gait confidence Distinguishing the boundary with gait And output the final identity according to the cascading sub-steps: S451. If both face candidate identities and valid gait candidate identities exist simultaneously, and both point to different individuals, then conflict resolution shall be performed: When the face confidence score is greater than or equal to the face high confidence threshold and the gait confidence score is greater than or equal to the gait high confidence threshold, the face candidate identity is adopted as the target identity. S452. If only face candidate identities exist or the face confidence level already meets the high confidence condition, then the following sub-rule applies: S4521. If the face confidence score is greater than or equal to the face high confidence threshold, and there are no valid gait candidate identities, then the face candidate identity shall be adopted as the target identity. S4522. If the face confidence is greater than or equal to the high confidence threshold for the face, but there are valid gait candidate identities pointing to different people and the gait confidence is less than 0.6, then the face candidate identity is adopted as the target identity when the face confidence further reaches the strict face threshold. S453. If the face confidence score does not reach the face high confidence threshold, but there are valid gait candidate identities and the gait confidence score is greater than or equal to the gait high confidence threshold, then the following sub-rule applies: S4531. Face Complete Loss Scenario: If no face is detected or the face confidence is 0, then the gait confidence must be greater than or equal to the faceless gait confirmation threshold, the gait distinction margin must be greater than or equal to the faceless margin threshold, the number of consecutive hits must be greater than or equal to the stable hit count threshold in the faceless scenario, and the target must have accumulated a predefined number of gait acquisition frames and duration within the recognition area; when the above conditions are met simultaneously, the valid gait candidate identity is adopted as the target identity; S4532. Scenarios where the face score is low but does not reach the high confidence threshold: If the face confidence score is between 0 and 0.90, and there is no valid gait candidate identity, or if the face identifier and the original gait candidate identity point to the same person, then the original gait candidate identity shall be adopted as the target identity. S454. If both face candidate identity and valid gait candidate identity exist and point to the same person, but neither individual modality reaches its respective high confidence threshold, then calculate the fusion score. : in, Indicates face weight, This represents the gait weight. When the fusion score reaches the fusion confirmation threshold, the candidate face identity is taken as the target identity. S455. If none of the above sub-steps are triggered, the trajectory of the current target remains unknown.

[0012] Furthermore: S5 specifically refers to: Virtual access control lines are preset in the images of each camera. When a target is detected crossing the access control line, the crossing event is first placed in the pre-alarm delay pool. During this delay period, the target's trajectory is periodically re-verified and fused every 0.2 seconds. This includes re-acquiring facial evidence from auxiliary cameras and extracting the latest gait features for comparison. The goal is to upgrade the target from an "unrecognized" state to an "recognized" state within this delay window to the maximum extent possible. After the delay expires, an event is output based on the target's final recognition status. If the target identity is a registered authorized person, it is determined to be authorized access, an authorization access event is generated and uploaded to the backend management platform via REST API; If the target is an unidentified person or a stranger, it is determined to be an illegal intrusion. A stable negative ID is assigned to the target, and a mapping relationship is established between the negative ID and the current global trajectory ID. An intrusion alarm event is generated and uploaded.

[0013] Compared with existing park security technologies based on single feature recognition or purely manual monitoring, the advantages of this invention are as follows: (1) By using multimodal cascaded recognition technology based on facial and gait features, the accuracy and robustness of target detection and identity determination are significantly improved. Traditional recognition schemes are prone to false alarms or missed alarms when faced with people wearing masks or hats, in insufficient lighting, or in non-positive postures. This scheme utilizes the uniqueness of gait recognition in long-distance, uncontrolled scenarios to complement high-precision facial micro-features. Even when an intruder is in a back view or their face is obscured, the macro-dynamic features of gait can still maintain continuous determination of the target's identity, solving the system failure problem caused by the lack of single recognition and ensuring the objectivity and integrity of security data.

[0014] (2) Based on the cross-camera collaboration and re-identification (ReID) mechanism, continuous and closed-loop management of global trajectories within the park is achieved. Existing technologies typically treat the images from different cameras as independent islands, resulting in a large number of redundant and fragmented trajectories when the same target moves across areas. This invention, through cross-camera global trajectory association and a self-repair mechanism for breakpoints within the same camera, can integrate discrete single-point tracking data into a unified global trajectory chain. This technical solution greatly reduces the labor intensity of manual video review and improves the response efficiency for tracing the source of intrusion behavior.

[0015] (3) Possessing the ability to manage the entire lifecycle of strangers and silently learn gait, the system's automated early warning efficiency is greatly enhanced. Unlike traditional systems that require pre-entry of whitelists, this invention achieves automatic perception, feature accumulation, and dynamic tracking of unknown individuals through a stable occupant ID for strangers and a false alarm correction ledger mechanism (assigning a negative ID to unidentified targets in S5, recording oldPerson / oldTrack after alarm upload, and sending a correction packet after subsequent identity confirmation). The system can automatically collect high-quality frame sequences in the background for gait model training, and can solidify the identity of frequently appearing suspicious individuals without manual intervention, thereby transforming the traditional "passive alarm" into "active prediction" based on historical data, significantly compensating for the shortcomings of human defense.

[0016] (4) By employing low-latency real-time stream capture, reconnection after interruption, and feature memory lifecycle binding mechanisms (corresponding to the S1 video stream acquisition thread, low buffering, and reconnection after interruption; _feature_buffers provides persistent protection for C++ feature pointers), the long-term stability of the system in industrial scenarios is ensured. For video stream packet loss or intermittent interruptions caused by network fluctuations in the campus, the system's designed reconnection strategy enables automatic error correction and second-level reconnection. Simultaneously, for AI inference under multi-threaded high concurrency, the system adopts low-level feature pointer lifecycle binding technology, fundamentally eliminating the risk of memory leaks and program crashes. This highly integrated hardware and software optimization scheme ensures the real-time performance and efficiency of the system when processing multiple high-definition monitoring channels, reduces redundant hardware performance requirements, and possesses strong applicability and economic benefits. Attached Figure Description

[0017] Figure 1 This is a flowchart of a campus intrusion prevention identification and tracking method based on face and gait fusion according to the present invention. Detailed Implementation

[0018] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0019] This invention upgrades traditional passive manual monitoring to an active anti-intrusion method based on multimodal machine vision. Addressing the shortcomings of traditional systems, such as target loss, identity verification failure, and inaccurate data when faced with masked individuals, face obstructions, individuals facing away from the camera, or switching between multiple cameras, this invention establishes a dual-modal identification method combining facial and gait features based on the principle of biometric feature consistency. This method can accurately, objectively, in real-time, continuously, and efficiently identify and track the global trajectory of unauthorized personnel within the park, minimizing potential security risks caused by missing features or human error.

[0020] like Figure 1 As shown, in one embodiment of the present invention, a campus intrusion prevention identification and tracking method based on face and gait fusion includes the following steps: S1. Capture video frames using multiple high-definition network cameras; S2. Detect targets based on video frames, generate target trajectories, and assign local trajectory IDs; S3. Associate global trajectory IDs across cameras based on local trajectory IDs; S4. Extract facial and gait features based on the global trajectory ID, and identify the target identity through multimodal fusion of face and gait; S5. Intrusion detection and real-time early warning based on the identified target identity.

[0021] To address the complex network environment of the campus, this invention employs a self-healing RTSP capture method. In S1, the method for the high-definition network camera to capture video frames is specifically as follows: Initialize the video stream access request, and sequentially call the standard OpenCV interface, FFmpeg backend and GStreamer backend to perform cascaded detection on different video decoding backends. If any backend successfully obtains a valid frame, the access is determined to be successful, thus solving the problem of "whether access is possible". If the request returns a port conflict error (e.g., port 8000 is in use), the automatic port rewriting mechanism is used to attempt to replace the conflicting port in the URL with a preset default port (e.g., 554) for reconnection; at the same time, connection timeout and read timeout parameters are set to automatically repair port conflicts or connection anomalies, in order to solve the problem of "how to recover after access failure"; Establish an asynchronous thread pool for read-write separation: The frame capture thread is responsible for continuously reading the raw pixel matrix from the RTSP stream or local camera. In real-time streaming mode, it does not sleep to clear the buffer and prevent latency backlog. The processing thread retrieves the latest frame from a fixed-length blocking queue (default queue capacity 30) and sends it to the algorithm inference layer. When the queue is full, the oldest frame is automatically discarded to ensure that the inference layer always retrieves the latest frame and avoids latency backlog caused by network jitter. When the continuous read failure exceeds 50 times, the reconnection mechanism is automatically triggered.

[0022] In this embodiment, an asynchronous decoupled architecture with read-write separation is adopted, configuring independent frame capture and processing threads for each video stream (RTSP stream or local camera). After starting, the frame capture thread continuously reads the raw image matrix. In real-time streaming mode, there is no sleep mode to empty the underlying buffer. If more than 50 consecutive read failures occur, automatic reconnection is triggered. When writing frames, if the queue is full, the oldest frame at the head of the queue is discarded before writing the current frame, ensuring that the queue contains only the latest data. The processing thread runs in parallel with the frame capture thread, retrieving frames from the queue in a blocking and waiting manner, and sequentially transferring them to the target detection, trajectory tracking, and identity recognition algorithm layers, implementing a fixed-length buffer mechanism. This mechanism ensures that the complex inference time in the backend does not block the frontend acquisition. Even in the event of network jitter or computing power fluctuations, long-term stable and extremely low-latency operation can be guaranteed by discarding redundant historical frames.

[0023] S2 includes the following steps: S21. Use a deep convolutional neural network to perform human detection on video frames and obtain human target bounding boxes (BBoxes). S22. Input the bounding box into the multi-target tracker, predict the target's position in the next frame based on Kalman filtering, and associate it with the target using the color histogram algorithm to generate the target's trajectory and assign a local trajectory ID to the target. In S2, a memory-safe management mechanism is employed during object detection based on video frames: when calling the underlying recognition SDK for feature extraction, pointer lifetime binding technology is used. Image memory addresses are forcibly cast to c_ubyte pointers using ctypes, and a feature buffer dictionary is established to ensure that, in a multi-threaded concurrent environment, the underlying C++ memory blocks are not incorrectly released by the Python garbage collection mechanism.

[0024] S3 includes the following steps: S31. Extract the appearance features of the current target and calculate the cosine similarity by combining the appearance features of the historical trajectory. The appearance features adopt the HSV color histogram method: the human body area is cropped from the target bounding box, converted into the HSV color space, and the hue (H), saturation (S) and lightness (V) histograms are extracted and normalized and stitched into 24-dimensional appearance features. In this embodiment, cosine similarity is defined as the inner product of the appearance features of the current target and the appearance features of the historical trajectory divided by the product of their respective moduli. S32. Based on cosine similarity, a cross-camera and same-camera association strategy is adopted to associate the local trajectory ID with the global trajectory ID, and the target trajectory is merged into the historical trajectory. The cross-camera and same-camera association strategy is as follows: S321. Cross-camera association: When the nearest camera of the historical trajectory is different from the current camera, if the cosine similarity is greater than the cross-camera matching threshold (default 0.5) and the time interval is within the maximum time interval (default 2.5 seconds), then the local trajectory ID is associated with the global trajectory ID corresponding to the historical trajectory. In this embodiment, the current camera refers to the camera that extracts the appearance features of the target trajectory of the current target; S322, Self-repair of breakpoints with the same camera: When the nearest camera of the historical trajectory is the same as the current camera, the following three conditions are considered to determine whether the trajectory is a continuation of the same target: Condition 1, the bounding box IoU is greater than or equal to the self-repair IoU threshold (default 0.3); Condition 2, the cosine similarity is greater than or equal to the self-repair similarity threshold (default 0.3); Condition 3, the time interval is within the self-repair time window (default 8.0 seconds); When all three conditions are met, the local trajectory ID is associated with the global trajectory ID corresponding to the historical trajectory. S323. If the above matching fails, a new global trajectory ID is assigned to the local trajectory ID, and its appearance features are extracted and stored in the feature cache. The feature cache is updated smoothly using an exponential moving average to prevent interference from single-frame noise.

[0025] In S31, the cosine similarity of historical targets is calculated. The specific expression is: In the formula, For the current target appearance features, For the appearance characteristics of historical trajectories, As a stable term, Represents the inner product of eigenvectors. Indicates the Euclidean modulus length; S4 includes the following steps: S41. Obtain the global trajectory ID and its corresponding BBox sequence from the single tracker through the logic controller, extract the full-body human image to obtain the gait image, and extract the facial region image to obtain the face image. In this embodiment, both gait images and face images undergo image enhancement processing: for gait images, size normalization (e.g., 128×64 pixels) and limited contrast adaptive histogram equalization (CLAHE) are performed to eliminate the impact of drastic lighting changes in the park on human silhouettes or appearance signatures.

[0026] For facial images, perform affine transformation alignment to ensure that facial features are in the standard feature extraction positions.

[0027] S42. Trigger the face recognition process: Perform quality verification on the face image. When the face region size, clarity, and brightness meet the requirements, extract the face features. It is then compared with a facial feature database to obtain facial identifiers and the highest matching score. Matching score with the second highest .in, Used to indicate the degree of similarity between the current face and the optimal candidate face. Used to determine the distinguishability between the best candidate and the second-best candidate; In this embodiment, the face image utilizes a high-precision algorithm engine to extract 1024-dimensional facial features. During this process, the system uses the "_feature_buffers" persistent memory buffer to lock the C++ pointer, preventing the memory from being garbage collected by Python in a multi-threaded environment.

[0028] Triggering the gait recognition process: Extracting gait features from gait images. The candidate identities are then compared with the gait feature database to obtain the original gait candidate identities. Optimal matching distance and suboptimal matching distance The smaller the distance, the closer the current gait characteristics are to the corresponding person in the database.

[0029] In this embodiment, gait images are input into a spatiotemporal convolutional network or a 3D-CNN to extract gait features. .

[0030] S43. Based on the highest matching score Matching score with the second highest Calculate the margin of face matching : In the formula, the face matching boundary This is used to represent the degree of distinction between the best and second-best face candidates. A larger face matching margin indicates a more significant advantage of the best candidate over the others, and a lower risk of misidentification. Further, based on the highest matching score... Marginalities of face matching The validity of a facial recognition identifier is determined based on the following two conditions: Condition 4: Highest matching score The similarity score of the best candidate must reach the minimum acceptable level if it is greater than the effective face recognition threshold. In this embodiment, the effective face recognition threshold is set to 0.78 by default. Condition 5: Face matching margin The value is greater than the face boundary threshold, where the face boundary threshold is dynamically adjusted. When the highest matching score When the score is less than the high score reference value (which is 0.92 by default), the face margin threshold is set to the standard default value (0.08 in this embodiment), which means that the score difference between the best candidate and the second best candidate is required to be at least 0.08, so as to fully suppress misidentification caused by factors such as differences in ID photo style; when When the value exceeds the high score reference value, the face edge threshold is appropriately lowered, and an appropriate lower value is selected (0.06 by default in this embodiment). The design basis for the dynamic adjustment strategy of the face marginal threshold is that when the optimal candidate score is already at an extremely high level, the credibility of the candidate being the correct person is already very high. At this time, appropriately reducing the requirement for the marginal difference can avoid false rejections caused by reasonable situations such as "single person database" or "high similarity between people in the database", thereby improving the recall rate of the system.

[0031] The system determines that the face recognition is valid only if both of the above conditions are met, and selects the best-matching face identifier as the face candidate. and the highest matching score As a confidence level of the face , It characterizes the credibility of the facial modality for the current candidate identity, and is used for subsequent multimodal fusion decision-making.

[0032] Based on the optimal matching distance and suboptimal matching distance Calculate the gait distinction margin : In the formula, gait distinguishes the boundary. This indicates the degree of distinction between the optimal gait candidate and the second-best gait candidate. Then, based on the optimal matching distance... Distinguishing the boundary with gait Calculate gait confidence : In the formula, This represents the baseline confidence level calculated from the optimal distance. This represents the correction factor obtained from the marginal conversion based on gait distinction. This represents the maximum distance threshold for gait, currently set to 13 by default. This represents the gait differentiation threshold, which is currently set to 0.08 by default.

[0033] S44. Identify valid gait candidates through gait comparison: To avoid misjudgments caused by a single gait retrieval, a historical queue of gait comparison results is maintained for each target trajectory. The queue stores the candidate IDs, gait confidence scores, and gait discrimination margins generated from the most recent gait comparisons in chronological order. The queue capacity is set to the larger of the required number of hits in normal scenarios and faceless scenarios (default 4) to ensure sufficient historical records for statistical analysis in any scenario.

[0034] When new data is written to the queue, the system checks the data in reverse order from the most recent result, continuously counting the number of records that simultaneously meet the following three conditions, which is used as the consecutive hit count of the candidate identity: Condition 6, the candidate ID is consistent with the current candidate ID; Condition 7, the gait confidence is greater than or equal to the gait stability confidence threshold; Condition 8, the gait discrimination margin is greater than or equal to the corresponding margin threshold (0.08 by default in normal scenarios, 0.18 by default in faceless scenarios).

[0035] Throughout the entire lifecycle of the target trajectory, a flag is maintained indicating whether a face has ever been detected. This flag is initialized to "no" when the trajectory is created; when the system's face detection module successfully detects a valid face region for the first time in any frame of the trajectory, the flag is permanently set to "yes".

[0036] In this scheme, the successful detection of a valid face region in any frame of the target trajectory requires the following conditions to be met simultaneously: Condition 9: At least one face is detected in the current frame through the SDK face detection interface, and the width and height of the bounding rectangle of the face are both not less than 50 pixels; Condition 10: After Gaussian denoising of the face region, the Laplacian variance is calculated as a sharpness index, and this index is not lower than the preset minimum sharpness threshold (default value 8.0); Condition 11: The average pixel brightness of the face region is between 40 and 220, excluding invalid images that are too dark or overexposed. When all the above conditions are met, the system determines that a valid face region has been successfully detected in the current frame.

[0037] During the fusion determination phase, the current scene is divided into two categories based on the value of this flag: In a typical scenario: if the flag is "Yes", it indicates that the target's face was captured at least once during this cross-camera passage. In this case, a relatively lenient threshold is used for gait stability determination—the stable hit count threshold is 2, the confidence threshold is 0.65, and the marginal threshold is 0.08.

[0038] In faceless scenarios: if the flag is "No" (meaning no face was detected throughout the trajectory), it indicates that the target is continuously facing away from the camera or has its face obscured. In this case, the gait stability determination automatically switches to a stricter threshold—a stable hit threshold of 3 times, a confidence threshold of 0.73, and a marginal threshold of 0.10; at the same time, a precondition is added: the target must accumulate a predefined amount of gait acquisition data within the recognition area (no less than 14 frames and a duration of no less than 1.6 seconds) to compensate for the identity uncertainty caused by the complete absence of a face.

[0039] If the number of consecutive hits in a normal scene reaches the corresponding stable hit threshold (default 2 times) or the number of consecutive hits in a faceless scene reaches the corresponding stable hit threshold (default 3 times), then the gait candidate identity is marked as a valid gait candidate identity. After meeting the above conditions, the original gait candidate identity is... as a candidate identity for effective gait The effective gait candidate identity refers to the number of records that simultaneously meet the following three conditions in the gait comparison history queue of the target trajectory, starting from the most recent comparison result and proceeding backwards (called the consecutive hit count).

[0040] S45. Based on facial recognition candidate identity Face confidence Effective gait candidate identity Gait confidence Distinguishing the boundary with gait The final identity is output by following the cascading sub-steps.

[0041] The final identity recognition process, based on face candidate identities, face confidence, effective gait candidate identities, gait confidence, and gait distinction boundaries, executes the following fusion decision rules in a chain from strong to weak priority: S451. If both a face candidate identity and a valid gait candidate identity exist simultaneously, and they point to different individuals, conflict resolution is performed. When the face confidence score is greater than or equal to the face high confidence threshold (default 0.90) and the gait confidence score is greater than or equal to the gait high confidence threshold (default 0.80), it is considered that both modalities have reached a high confidence level but there is a discrepancy. In this case, the face candidate identity is adopted first as the target identity because the face is the gold standard for identity authentication.

[0042] S452. If only face candidate identities exist or the face confidence level already meets the high confidence condition, then the following sub-rule applies: S4521. If the face confidence score is greater than or equal to the face high confidence threshold (default 0.90) and there are no valid gait candidate identities, then the face candidate identity is adopted as the target identity. S4522. If the face confidence is greater than or equal to the high confidence threshold for faces, but there are valid gait candidate identities pointing to different people and the gait confidence is less than 0.6, then the face candidate identity is required to be adopted as the target identity when the face confidence further reaches the strict face threshold (default 0.94) to prevent misidentification due to blurred faces.

[0043] S453. If the face confidence score does not reach the face high confidence threshold, but there are valid gait candidate identities and the gait confidence score is greater than or equal to the gait high confidence threshold, then the following sub-rule applies: S4531, Complete Face Loss Scenario: If no face is detected or the face confidence is 0, then the following conditions must be met: gait confidence greater than or equal to the faceless gait confirmation threshold (default 0.84), gait differentiation margin greater than or equal to the faceless margin threshold (default 0.18), consecutive hits greater than or equal to the stable hit count threshold in the faceless scenario in S44 (default 3 times), and the target must have accumulated a predefined number of gait acquisition frames (default 14 frames) and a duration (default 1.6 seconds) within the recognition area. When all the above conditions are met, the valid gait candidate identity is adopted as the target identity. S4532, Scenario where the face score is low but does not reach the high confidence threshold: The face confidence score is between 0 and 0.90, at which point the gait has independently reached a high confidence level. If there is no valid gait candidate identity (the face candidate identity has not passed the valid recognition judgment in S44), or the face identifier and the original gait candidate identity point to the same person, then the original gait candidate identity is adopted as the target identity. The logic of this branch is: the gait has formed a stable conclusion through strict multi-level screening, and the low score signal of the face, whether or not it is consistent with the gait, is not enough to overturn the gait judgment.

[0044] S454. If both face candidate identity and valid gait candidate identity exist and point to the same person, but neither individual modality reaches its respective high confidence threshold, then calculate the fusion score. : in, This indicates the face weight, currently set to 0.75 by default. This represents the gait weight, currently set to 0.25 by default. When the fusion score reaches the fusion confirmation threshold, the candidate face identity is used as the target identity; the current fusion confirmation threshold is set to 0.82 by default.

[0045] S455. If none of the above rules are triggered, the trajectory of the current target remains unknown, and the system waits for subsequent video frames to provide new identification evidence.

[0046] This invention replaces traditional single biometric detection with multimodal fusion recognition of face and gait. It leverages the robustness of gait recognition in long-distance, uncontrolled scenarios to reduce detection errors and improve the system's identity verification efficiency in complex environments.

[0047] S5 specifically refers to: Virtual access control lines (electronic fences) are preset in the view of each camera. When a target is detected crossing the access control line, the crossing event is first placed in a pre-alarm delay pool (default delay 1.0 second). During this delay, the target's trajectory is periodically rechecked and fused for confirmation every 0.2 seconds, including re-acquiring facial evidence from auxiliary cameras and extracting the latest gait features for comparison, maximizing the chance of upgrading the target from an "unrecognized" state to an "recognized" state within this delay window. After the delay expires, an event is output based on the target's final recognition status: If the target identity is a registered authorized person, it is determined to be authorized access, an authorization access event is generated and uploaded to the backend management platform via REST API; If the target is an unidentified person or a stranger, it is considered an unauthorized intrusion. A stable negative ID is assigned to the target, and a mapping relationship is established between this negative ID and the current global trajectory ID, generating an intrusion alarm event for upload. Subsequently, if the same global trajectory is associated with an authorized person through continuous face or gait recognition, a correction event is generated, correcting the intrusion record previously reported with a negative ID to an authorized person, thus achieving automatic correction of false alarms and avoiding manual intervention.

[0048] In the description of this invention, the above are merely preferred embodiments and are not intended to limit the scope of protection of this invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A campus intrusion prevention identification and tracking method based on face and gait fusion, characterized in that, Includes the following steps: S1. Capture video frames using multiple high-definition network cameras; S2. Detect targets based on video frames, generate target trajectories, and assign local trajectory IDs; S3. Associate global trajectory IDs across cameras based on local trajectory IDs; S4. Extract facial and gait features based on the global trajectory ID, and identify the target identity through multimodal fusion of face and gait; S5. Intrusion detection and real-time early warning based on the identified target identity.

2. The campus intrusion prevention identification and tracking method based on face and gait fusion according to claim 1, characterized in that, In S1, the method for capturing video frames by the high-definition network camera is as follows: Initialize the video stream access request, and sequentially call the standard OpenCV interface, FFmpeg backend and GStreamer backend to perform cascaded detection on different video decoding backends. If any backend successfully obtains a valid frame, the access is considered successful. If the request returns a port conflict error, the automatic port rewriting mechanism is used to attempt to replace the conflicting port in the URL with the default port for reconnection; at the same time, connection timeout and read timeout parameters are set to automatically repair port conflicts or connection abnormalities. Establish an asynchronous thread pool for read-write separation: The frame-capturing thread is responsible for continuously reading the raw pixel matrix from the RTSP stream or local camera. In real-time streaming mode, it does not sleep to clear the buffer and prevent latency backlog. The processing thread retrieves the latest frame from a fixed-length blocking queue and sends it to the algorithm inference layer. When the queue is full, the oldest frame is automatically discarded to ensure that the inference layer always retrieves the latest frame and avoids latency backlog caused by network jitter. When more than 50 consecutive read failures occur, the reconnection mechanism is automatically triggered.

3. The campus intrusion prevention identification and tracking method based on face and gait fusion according to claim 1, characterized in that, S2 includes the following steps: S21. Use a deep convolutional neural network to perform human detection on video frames and obtain human target bounding boxes; S22. Input the bounding box into the multi-target tracker, predict the target's position in the next frame based on Kalman filtering, and associate it with the target using the color histogram algorithm to generate the target's trajectory and assign a local trajectory ID to the target. In S2, a memory safety management mechanism is adopted during the target detection process based on video frames: when calling the underlying recognition SDK for feature extraction, pointer lifetime binding technology is used to forcibly convert the image memory address to a c_ubyte pointer through ctypes and establish a feature buffer dictionary to ensure that the underlying C++ memory block will not be incorrectly released by the Python garbage collection mechanism in a multi-threaded concurrent environment.

4. The campus intrusion prevention identification and tracking method based on face and gait fusion according to claim 1, characterized in that, S3 includes the following steps: S31. Extract the appearance features of the current target and calculate the cosine similarity by combining the appearance features of the historical trajectory. The appearance features are obtained by using the HSV color histogram method: the human body area is cropped from the target bounding box, converted to the HSV color space, and the hue, saturation and brightness histograms are extracted and normalized and stitched into 24-dimensional appearance features. S32. Based on cosine similarity, a cross-camera and same-camera association strategy is adopted to associate the local trajectory ID with the global trajectory ID, and the target trajectory is merged into the historical trajectory.

5. The campus intrusion prevention identification and tracking method based on face and gait fusion according to claim 4, characterized in that, In S32, the association strategies for cross-camera and same-camera connections are as follows: S321. Cross-camera association: When the nearest camera of the historical trajectory is different from the current camera, if the cosine similarity is greater than the cross-camera matching threshold and the time interval is within the maximum time interval, the local trajectory ID is associated with the global trajectory ID corresponding to the historical trajectory. S322, Self-repair of breakpoints with the same camera: When the nearest camera of the historical trajectory is the same as the current camera, the following three conditions are combined to determine whether it is a continuation of the trajectory of the same target: Condition 1, the bounding box IoU is greater than or equal to the self-repair IoU threshold; Condition 2, the cosine similarity is greater than or equal to the self-repair similarity threshold; Condition 3, the time interval is within the self-repair time window; When all three conditions are met, the local trajectory ID is associated with the global trajectory ID corresponding to the historical trajectory. S323. If the above matching fails, a new global trajectory ID is assigned to the local trajectory ID, and its appearance features are extracted and stored in the feature cache. The feature cache is updated smoothly using an exponential moving average to prevent interference from single-frame noise.

6. The campus intrusion prevention identification and tracking method based on face and gait fusion according to claim 4, characterized in that, In S31, the cosine similarity of historical targets is calculated. The specific expression is: In the formula, For the current target appearance features, For the appearance characteristics of historical trajectories, As a stable term, Represents the inner product of eigenvectors. This indicates the Euclidean modulus.

7. The campus intrusion prevention identification and tracking method based on face and gait fusion according to claim 1, characterized in that, S4 includes the following steps: S41. Obtain the global trajectory ID and its corresponding BBox sequence from the single tracker through the logic controller, extract the full-body human image to obtain the gait image, and extract the facial region image to obtain the face image. S42. Trigger the face recognition process: Perform quality verification on the face image. When the face region size, clarity, and brightness meet the requirements, extract the face features. It is then compared with a facial feature database to obtain facial identifiers and the highest matching score. Matching score with the second highest ;in, Used to indicate the degree of similarity between the current face and the optimal candidate face. Used to determine the distinguishability between the best candidate and the second-best candidate; Triggering the gait recognition process: Extracting gait features from gait images. The candidate identities are then compared with the gait feature database to obtain the original gait candidate identities. Optimal matching distance and suboptimal matching distance ; S43. Based on the highest matching score Matching score with the second highest Calculate the margin of face matching : In the formula, the face matching boundary Used to represent the degree of distinction between the best and second-best face candidates, based on the highest matching score. Marginalities of face matching The validity of a facial recognition identifier is determined based on the following two conditions: Condition 4: Highest matching score Greater than the effective face recognition threshold; Condition 5: Face matching margin The score is greater than the face boundary threshold, where the face boundary threshold is dynamically adjusted: when the highest matching score is reached... When the score is less than the high-score reference value, the face boundary threshold takes the standard default value; when the highest matching score is less than the reference value, the threshold takes the standard default value. When the score exceeds the high reference value, the facial boundary threshold should be appropriately lowered. The best-matching face identifier will be selected as the face candidate only if both of the above conditions are met. and the highest matching score As a confidence level of the face ; Based on the optimal matching distance and suboptimal matching distance Calculate the gait distinction margin : Based on the optimal matching distance Distinguishing the boundary with gait Calculate gait confidence : In the formula, This represents the baseline confidence level calculated from the optimal distance. This represents the correction factor obtained from the marginal conversion based on gait distinction. This represents the maximum distance threshold for gait. This represents the gait discrimination threshold; S44. Identify valid gait candidates through gait comparison: For each target trajectory, maintain a historical queue of gait comparison results. The queue stores candidate IDs, gait confidence scores, and gait discrimination margins generated from the most recent gait comparisons in chronological order. When new data is written to the queue, the system checks the data in reverse order from the most recent result, and counts the number of records that simultaneously meet the following three conditions as the consecutive hit count of the candidate identity: Condition 6, the candidate ID is consistent with the current candidate ID; Condition 7, the gait confidence is greater than or equal to the gait stability confidence threshold; Condition 8, the gait discrimination margin is greater than or equal to the corresponding margin threshold. Throughout the entire lifecycle of the target trajectory, a flag is maintained indicating whether a face has been detected. This flag is initialized to "No" when the trajectory is created. When a valid face region is successfully detected in any frame of the trajectory, the flag is permanently set to "Yes". Successfully detecting a valid face region in any frame of the trajectory requires meeting the following conditions simultaneously: Condition 9: At least one face is detected in the current frame via the SDK face detection interface, and the width and height of the bounding rectangle of the face are both no less than 50 pixels; Condition 10: The Laplacian variance is calculated as a sharpness index after Gaussian denoising of the face region, and this index is no less than a preset minimum sharpness threshold; Condition 11: The average pixel brightness of the face region is between 40 and 220, excluding invalid images that are too dark or overexposed. During the fusion determination phase, the current scene is divided into two categories based on the value of this flag: In a typical scenario: the flag is set to "Yes", the stable hit count threshold is 2, the confidence threshold is 0.65, and the marginal threshold is 0.

08. Faceless scenario: The flag is "No", the stable hit threshold is 3 times, the confidence threshold is 0.73, the marginal threshold is 0.10, and the target must accumulate a predefined number of gait acquisition data within the recognition area; If the number of consecutive hits in a normal scene reaches the corresponding stable hit threshold, or the number of consecutive hits in a faceless scene reaches the corresponding stable hit threshold, then the gait candidate identity is marked as a valid gait candidate identity. S45. Based on facial recognition candidate identity Face confidence Effective gait candidate identity Gait confidence Distinguishing the boundary with gait And output the final identity according to the cascading sub-steps: S451. If both face candidate identities and valid gait candidate identities exist simultaneously, and both point to different individuals, then conflict resolution shall be performed: When the face confidence score is greater than or equal to the face high confidence threshold and the gait confidence score is greater than or equal to the gait high confidence threshold, the face candidate identity is adopted as the target identity. S452. If only face candidate identities exist or the face confidence level already meets the high confidence condition, then the following sub-rule applies: S4521. If the face confidence score is greater than or equal to the face high confidence threshold, and there are no valid gait candidate identities, then the face candidate identity shall be adopted as the target identity. S4522. If the face confidence is greater than or equal to the high confidence threshold for the face, but there are valid gait candidate identities pointing to different people and the gait confidence is less than 0.6, then the face candidate identity is adopted as the target identity when the face confidence further reaches the strict face threshold. S453. If the face confidence score does not reach the face high confidence threshold, but there are valid gait candidate identities and the gait confidence score is greater than or equal to the gait high confidence threshold, then the following sub-rule applies: S4531. Face Complete Loss Scenario: If no face is detected or the face confidence is 0, then the gait confidence must be greater than or equal to the faceless gait confirmation threshold, the gait distinction margin must be greater than or equal to the faceless margin threshold, the number of consecutive hits must be greater than or equal to the stable hit count threshold in the faceless scenario, and the target must have accumulated a predefined number of gait acquisition frames and duration within the recognition area; when the above conditions are met simultaneously, the valid gait candidate identity is adopted as the target identity; S4532. Scenarios where the face score is low but does not reach the high confidence threshold: If the face confidence score is between 0 and 0.90, and there is no valid gait candidate identity, or if the face identifier and the original gait candidate identity point to the same person, then the original gait candidate identity shall be adopted as the target identity. S454. If both face candidate identity and valid gait candidate identity exist and point to the same person, but neither individual modality reaches its respective high confidence threshold, then calculate the fusion score. : in, Indicates face weight, This represents the gait weight. When the fusion score reaches the fusion confirmation threshold, the candidate face identity is taken as the target identity. S455. If none of the above sub-steps are triggered, the trajectory of the current target remains unknown.

8. The campus intrusion prevention identification and tracking method based on face and gait fusion according to claim 1, characterized in that, S5 specifically refers to: Virtual access control lines are preset in the images of each camera. When a target is detected crossing the access control line, the crossing event is first placed in the pre-alarm delay pool. During this delay period, the target's trajectory is periodically rechecked and fused for confirmation every 0.2 seconds. This includes re-acquiring facial evidence from auxiliary cameras and extracting the latest gait features for comparison. The goal is to upgrade the target from an "unrecognized" state to an "recognized" state within this delay window to the maximum extent possible. After the delay expires, an event is output based on the target's final recognition status. If the target identity is a registered authorized person, then it is determined to be authorized access, an authorization access event is generated and uploaded to the backend management platform via REST API; If the target is an unidentified person or a stranger, it is determined to be an illegal intrusion. A stable negative ID is assigned to the target, and a mapping relationship is established between the negative ID and the current global trajectory ID. An intrusion alarm event is generated and uploaded.

Images