Real-time face recognition tracking method based on multi-modal features and time sequence backtracking

Abstract

The invention discloses a real-time face recognition tracking method based on multi-modal features and time sequence backtracking. The method comprises the following steps: S1, carrying out frame-by-frame face detection on an input video stream, carrying out real-time tracking on detected face targets, and generating a track with a temporary track identifier for each face target; s2, extracting multi-modal features of a current frame face target in the track, wherein the multi-modal features at least comprise biological recognition features extracted based on a face recognition model and universal appearance features extracted based on a universal visual model; and S3, according to the quality evaluation result, determining a fusion weight of the biological recognition features and the general appearance features, and the like. According to the invention, high-precision, long-time-history and identity perception detection, tracking and attribute analysis can be carried out on the face target in the video stream.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a real-time face recognition and tracking method based on multimodal features and temporal backtracking. Background Technology

[0002] In recent years, deep learning has driven significant progress in face recognition and multi-object tracking technologies, and related methods have been widely applied in video analytics scenarios such as intelligent surveillance and smart retail. However, achieving high-precision, long-term, and identity-consistent face tracking and analysis in dynamic and complex real-world video streams still faces systemic challenges. Existing technologies mostly follow a paradigm that separates "detection-tracking" from "detection-recognition," and various methods have inherent limitations when dealing with practical conditions such as occlusion, pose changes, and lighting fluctuations. This reflects a deep-seated contradiction in the current technological system regarding feature utilization, module collaboration, and system optimization.

[0003] Mainstream tracking methods can be divided into three categories: motion model-based methods (such as SORT) rely on position prediction and geometric matching. Although computationally efficient, they are prone to identity switching in situations with occlusion or dense crowds; methods combining appearance features (such as DeepSORT) introduce Re-ID features to enhance matching robustness, but the computational cost is huge due to the need to extract depth features frame by frame, and the features themselves are prone to drift when pose and illumination change significantly, making it difficult to guarantee long-term association stability; end-to-end joint detection and tracking methods (such as FairMOT) improve consistency efficiency, but the model has high coupling and limited generalization ability, making it difficult to flexibly adapt to diverse scenarios. These methods collectively expose the problems of static feature utilization and lack of scene adaptation ability—that is, they cannot dynamically adjust feature selection and fusion strategies according to real-time conditions such as image quality and occlusion status, resulting in decreased discriminative power and tracking interruption in complex situations.

[0004] Furthermore, the existing system exhibits significant disconnects in system design and functional coordination. The tracking module typically lacks awareness of identity information, relying solely on visual or motion features to maintain the trajectory; the recognition module processes single-frame faces in isolation, failing to utilize the temporal context provided by tracking. This lack of information exchange between the two prevents the system from inferring the same target based on trajectory logic even when facial features are temporarily lost, and also prevents the system from using confirmed identities to inform tracking and association decisions, thus failing to form a closed loop of "identity awareness." In addition, the existing tracking process is essentially an open-loop system; once a mis-association of identity occurs, the error accumulates over time and cannot be corrected, resulting in low reliability of long-term trajectory data. Simultaneously, to maintain tracking or recognition performance, most systems still perform energy-intensive feature extraction and comparison on low-quality facial images, lacking a content-value-based computational resource scheduling mechanism, leading to significant efficiency waste.

[0005] In summary, while current face recognition tracking technology continues to improve in individual tasks such as detection and feature matching, it still faces overall bottlenecks when dealing with complex real-world scenarios due to the lack of a unified framework that coordinates instantaneous robust association, long-term identity consistency, and dynamic resource allocation. How to overcome the performance limitations caused by isolated optimization of individual modules under the existing paradigm and build a real-time face recognition tracking system that combines high accuracy, high stability, and high efficiency has become a key technical problem that urgently needs to be solved in this field. Summary of the Invention

[0006] In view of this, the purpose of this invention is to propose a real-time face recognition and tracking method based on multimodal features and temporal backtracking, which can perform high-precision, long-term, identity-aware detection, tracking, and attribute analysis of facial targets in video streams. This solution can be seamlessly integrated into various video AI applications, such as intelligent security monitoring platforms, smart retail customer flow analysis systems, intelligent video content review software, and various industry solutions requiring continuous tracking and analysis of the identities of people in videos.

[0007] According to one aspect of the present invention, a real-time face recognition and tracking method based on multimodal features and temporal backtracking is provided, the method comprising:

[0008] S1. Perform face detection frame by frame on the input video stream and track the detected face targets in real time, generating a trajectory with temporary trajectory markers for each face target;

[0009] S2. Extract the multimodal features of the face target in the current frame from the trajectory. The multimodal features include at least biometric features extracted based on the face recognition model and general appearance features extracted based on the general visual model. At the same time, perform quality assessment on the face image of the current frame to obtain the quality assessment result.

[0010] S3. Based on the quality assessment results, determine the fusion weight of the biometric features and the general appearance features, and calculate the association similarity between the current frame detection target and each existing trajectory based on the weighted fusion features; update the tracking status of each trajectory based on the association similarity, the tracking status includes confirmed status, unconfirmed status and lost status; wherein, when the association similarity of a trajectory is continuously higher than a first threshold, its tracking status is updated or maintained as confirmed status;

[0011] S4. For trajectories that are in a confirmed state and whose face image quality is higher than the second threshold, their biometric features are compared with the identity database to perform identity recognition or registration, and the successfully recognized identity tag is bound to the trajectory; when a trajectory is bound to a new identity tag in a subsequent frame, a time-series backtracking correction is triggered: using the new identity tag as the correction benchmark, the identity tags of the trajectory in the historical frames within the backtracking time window are recalculated, and the historical identity tags are replaced with consistent ones.

[0012] The real-time face recognition and tracking method described in the above technical solution achieves a systematic evolution from open-loop processing to closed-loop self-correction at the technical architecture level through the collaborative design of multimodal feature fusion and temporal backtracking correction. This effectively solves the problems currently faced in this field under complex dynamic scenarios. Traditional methods often treat face detection, tracking, and recognition as discrete stages. The tracking process lacks awareness of the target's identity, and the recognition stage is isolated from the temporal context, leading to frequent changes in identity identification under conditions such as occlusion and pose changes. Moreover, once an error occurs, it continues to accumulate along the trajectory. In addition, fixed feature fusion strategies and computational resource allocation lacking quality guidance also limit the balance between overall system efficiency and robustness. This solution, on the other hand, constructs a processing flow with adaptive adjustment and error correction capabilities by introducing a dynamic feature weight allocation mechanism based on quality assessment, a tracking state machine based on confidence management, and historical trajectory backtracking correction triggered by subsequent reliable recognition.

[0013] Specifically, this method first performs multimodal feature extraction and face image quality assessment simultaneously after completing frame-by-frame detection and initial tracking. The multi-dimensional quantitative scoring of quality here is not an isolated step, but rather serves as the dominant signal for subsequent feature fusion and recognition. During the association matching stage, the system dynamically adjusts the fusion weights of biometric features and general appearance features based on the quality score. For example, when the pose angle is too large or occlusion causes a decrease in the reliability of face recognition features, the system automatically increases the reliance on stable appearance features such as clothing and body shape. This establishes an adaptive expressive capability for complex scenes at the feature level, mitigating identity switching caused by the failure of a single feature. Simultaneously, the system maintains an identity confidence label for each trajectory with states such as "confirmed" and "unconfirmed," which migrates based on the weighted fusion similarity between consecutive frames. When a trajectory enters the "confirmed" state due to a consistently high matching degree, its identity recognition is solidified and used to constrain subsequent associations. This essentially embeds the identity recognition result into the tracking decision loop, achieving a paradigm shift from "identity-less tracking" to "identity-aware tracking."

[0014] Furthermore, this method coordinates identity recognition and historical trajectory cleaning through a conditional triggering mechanism. Only when a trajectory is in a "confirmed" state and the face quality score exceeds a set threshold is the computationally expensive biometric comparison and database registration initiated. This design avoids invalid recognition on low-quality frames, significantly improving system resource utilization. Crucially, when a trajectory obtains a new identity label with high confidence in subsequent frames, the system triggers a temporal backtracking correction process. This new label serves as a strong constraint, re-evaluating and overwriting the historical frame identities of the trajectory within a set time window. This mechanism overcomes the open-loop limitation of error propagation and uncorrectable errors in traditional tracking processes, enabling retrospective repair of identity jumps and ensuring the temporal consistency and reliability of long-term identity trajectories at the output end.

[0015] In summary, this solution organically coordinates three mechanisms: quality assessment-driven feature fusion, state machine-based identity closure, and backtracking correction to ensure historical consistency. This forms a complete technical system with self-adjustment and error correction capabilities. The system effectively unifies the three objectives of long-term tracking robustness, identity consistency, and computational efficiency.

[0016] In some embodiments, the quality assessment includes quantifying at least one of the following: sharpness, pose angle, lighting conditions, and degree of occlusion of the face image.

[0017] In the aforementioned technical solution, the specific design for facial image quality assessment involves quantifying and scoring at least one key factor, such as sharpness, pose angle, lighting conditions, and occlusion level. In traditional processing, image quality is typically used only as a pre-filtering condition or a post-hoc explanation factor, loosely coupled with the core recognition and tracking decision chain, leading to the system's inability to respond when features are unreliable. This solution, by introducing this multi-dimensional, quantifiable evaluation mechanism, essentially endows the system with the ability to "diagnose" the usability of facial images in real time. Sharpness score correlates with the precision of feature extraction, pose angle assessment affects the alignment and standardization of the facial model, lighting conditions determine the effectiveness of texture information, and occlusion level determines the proportion of missing information. These quantified outputs collectively constitute a comprehensive judgment on whether the current frame of the facial image can support reliable identity verification, providing a unified and objective input benchmark for all subsequent decisions that rely on image quality.

[0018] This evaluation scheme systematically integrates the system into a lightweight, real-time pipeline-embedded functional module, allowing its output to directly serve two downstream adaptive stages. First, in the dynamic feature fusion stage, these scores serve as the basis for assigning weights to biometric features and general appearance features. For example, when the pose angle score indicates excessive facial deflection, the system can explicitly reduce the weight of depth biometric features sensitive to frontal views, while increasing the weight of general appearance features relatively insensitive to viewpoint changes. This adjustment is based on image quality, rather than simply weakening all features due to poor quality. Second, in the stage triggering high-cost identity recognition and database comparison, the comprehensive quality score constitutes one of the triggering conditions alongside the "confirmed" tracking status. This ensures that the system only allocates computational resources for accurate recognition at critical moments when there is a quantitative grasp of the face image quality and a high level of confidence in identity tracking, thereby optimizing computational efficiency.

[0019] Therefore, this quality assessment design acts as an active sensor and scheduler within the closed-loop system. By providing structured diagnostic information, it enables the system to understand the specific challenges that may exist at the feature level and accordingly make evidence-based, automated switches between different strategies. This deep integration addresses the problem of image quality information being disconnected from the core algorithm flow in traditional methods.

[0020] In some embodiments, the fusion weight of the biometric feature and the general appearance feature is determined based on the quality assessment result, specifically:

[0021] Based on at least one quantitative indicator from the quality assessment results, the fusion weight coefficients of the biometric feature and the general appearance feature are calculated using a predefined weight mapping function or rule.

[0022] The weight mapping function or rule is configured such that: when the quality assessment result indicates that the face image is more suitable for identity determination, a higher weight coefficient is assigned to the biometric feature than to the general appearance feature; when the quality assessment result indicates that the face image is not suitable for identity determination, a higher weight coefficient is assigned to the general appearance feature than to the biometric feature.

[0023] In the aforementioned technical solution, the mechanism for dynamically determining the fusion weights of biometric features and general appearance features based on quality assessment results essentially constructs a closed-loop decision-making link from image quality perception to feature utility regulation, thereby achieving adaptability to complex scenes at the feature fusion level. Traditional multimodal fusion methods typically employ fixed weights or weighting methods based on simple heuristic rules (such as detection confidence), failing to fully consider the reliability differences of different feature modalities under different imaging conditions. For example, when the face is occluded or has a large angle of deflection, the discriminative power of biometric features extracted based on deep face models will significantly decrease; assigning them high weights will directly lead to incorrect association matching. Conversely, under high-quality frontal face conditions, over-reliance on general appearance features may introduce unnecessary ambiguity. This solution uses the quantitative output of quality assessment as the basis for weight allocation and designs explicit mapping rules—when the quality assessment characterizes the image as suitable for identity discrimination, higher weights are assigned to biometric features; conversely, higher weights are assigned to general appearance features—allowing the feature fusion strategy to dynamically adjust with changes in imaging conditions.

[0024] This dynamic weight allocation mechanism uses a predefined weight mapping function or rule to convert the values ​​of one or more specific quality indicators (such as pose angle and occlusion level) into coefficients that control the contribution of two types of features in similarity calculation. Essentially, this process establishes a lightweight, interpretable decision controller that dynamically configures the most suitable feature fusion strategy based on real-time perceived image conditions. For example, when the pose angle exceeds a threshold, the mapping rule can significantly reduce the weight of biometric features; when the occlusion score increases, it can linearly increase the weight of general appearance features. This real-time diagnostic adjustment makes feature-level information integration no longer blind but possesses targeted anti-interference capabilities.

[0025] Therefore, this dynamic weight allocation scheme improves the system's robustness to association under non-ideal imaging conditions. By suppressing the influence of unreliable features and enhancing the utility of alternative features, it effectively reduces identity switching caused by the failure of single-modal features. Furthermore, in high-quality frames, the system tends to use biometric features with strong discriminative power but potentially higher computational costs; in low-quality frames, it shifts to relying on relatively lightweight general appearance features, which is itself an optimization of computational efficiency.

[0026] In some embodiments, S3 further includes: generating a feature cache for each trajectory to store historical multimodal features; and comparing the fused features of the current frame with the historical features in the trajectory feature cache when calculating the association similarity.

[0027] In the aforementioned technical solution, the mechanism of establishing a feature cache for each trajectory to store historical multimodal features and comparing them with the features of the current frame during association essentially combines temporal modeling with the feature matching process. This addresses the dynamic changes and instantaneous degradation of the target's appearance during tracking caused by pose, lighting, and occlusion. Traditional tracking association methods are mostly based on instantaneous matching of the current frame detection result with the latest trajectory state. This "frame-to-point" matching mode is extremely sensitive to instantaneous noise in appearance features and cannot withstand feature "discontinuities" caused by short-term severe occlusion or drastic deformation. This solution constructs a short-term appearance memory model at the trajectory granularity by maintaining a dynamically updated feature cache for each trajectory. When calculating association similarity, the system does not simply compare the fused features of the current frame with a single "representative feature" of the trajectory (such as features from the previous frame or the mean feature), but rather performs a comprehensive comparison with a set of multimodal features from multiple historical moments stored in the cache. This design transforms association decisions from comparisons based on instantaneous snapshots to similarity assessments based on a historical sequence.

[0028] This feature caching and historical comparison mechanism significantly enhances the robustness and discriminative power of association decisions by introducing a controllable temporal context. The cache, acting as a prior knowledge base, preserves multimodal feature representations of the target across multiple quality states (e.g., clear frontal face, slight profile) over a past period. When the current frame's features are noisy or biased due to poor quality, multi-feature comparison with the historical cache provides more verification opportunities and consistency references, effectively resisting the accidental degradation of single-frame features. More importantly, this mechanism works in conjunction with dynamic weight allocation: the historical cache stores fused features weighted by quality assessment; therefore, the comparison process essentially measures the overall consistency between the current observation and a series of "quality-adaptive representations" in the trajectory's history. This enables the system to distinguish between "targets that are the same despite natural appearance changes" and "targets that are different despite similar appearances," making it particularly suitable for scenarios with periodic changes or intermittent occlusion of target appearance.

[0029] Therefore, this scheme implements a time-tolerant matching strategy in the association process by constructing and utilizing a temporal model of the trajectory's appearance. It reduces association jitter caused by single-frame quality fluctuations and enhances the trajectory's ability to maintain its appearance even in the face of brief mismatches. The feature buffer provides robust priors for association; its stored structured historical feature sequences also provide the necessary data foundation for subsequent temporal backtracking corrections to perform recalculation.

[0030] In some embodiments, when the correlation similarity of a trajectory remains above a first threshold, updating its tracking status or maintaining it as a confirmed status further includes:

[0031] For trajectories that are in a confirmed state, their associated identity tags will be used as prior information in the calculation of association similarity for subsequent frames.

[0032] In the aforementioned technical solution, the identity tag bound to the trajectory in the "confirmed" state is used as prior information to participate in subsequent association similarity calculations. Its essence lies in constructing a closed-loop feedback path from identity recognition to tracking decision-making, thereby achieving a paradigm shift from "data association based on appearance similarity" to "logical association incorporating identity cognition" at the system level. Traditional tracking association methods rely entirely on the similarity measurement of visual features (such as appearance and movement) between consecutive frames. This model is prone to misassociation when the target's appearance changes significantly or when similar interfering individuals exist. The fundamental reason is the lack of higher semantic level (i.e., "who") constraints and guidance in the decision-making process. This solution introduces the concept of "confirmed state," assigning a credibility identifier at the identity level to the trajectory, and injecting the specific identity tag bound to this state as strong prior knowledge into the association matching calculation of subsequent frames. This does not mean simply replacing feature comparison with identity tags, but rather using identity consistency as a new, highly weighted decision dimension, which, together with multimodal feature similarity, constitutes a comprehensive association criterion.

[0033] The core of this mechanism lies in achieving forward propagation of identity information and closed-loop decision-making over time. Once a trajectory is identified and marked with high confidence as a specific identity (such as "Person A"), the system's perception of the target changes from "a visual object that needs to be tracked" to "an individual with a known identity." In subsequent processing, for any new detected target that may be associated with this trajectory, the system, in addition to calculating the similarity between its appearance features and the trajectory's historical features, also assesses the degree of matching between its identity features (if extractable) and the registration features of "Person A," using this as a key prior. If the new target exhibits high identity similarity to "Person A," then even if its current appearance differs from the trajectory's historical cache due to occlusion or other reasons, the system can still maintain the association based on identity consistency. Conversely, if a new target is similar in appearance to the trajectory's historical cache, but its identity features clearly point to another known identity, "Person B," then the system can effectively reject this association based on the identity prior, thereby avoiding identity confusion. This design enables the tracking process to have identity logic verification capabilities, greatly enhancing the ability to distinguish between individuals with similar appearances.

[0034] Therefore, this identity prior feedback mechanism, by introducing identity semantic constraints, significantly improves long-term tracking accuracy in complex scenarios, especially when there are multiple similar-looking targets, and effectively suppresses identity switching. Furthermore, the system not only knows "who" the target is, but can also use this knowledge of "who" to better maintain tracking of it, forming a reinforcement loop of "identification assists tracking, and tracking consolidates identification".

[0035] In some embodiments, for trajectories that are in a confirmed state and whose facial image quality is higher than a second threshold, their biometric features are compared with an identity database for identity recognition or registration, and the successfully recognized identity tag is bound to the trajectory. Specifically:

[0036] If the similarity between the biometric feature and any registered feature in the identity database is higher than the third threshold, the identity recognition is deemed successful, and the corresponding registered identity tag is bound to the trajectory.

[0037] If the similarity between the biometric feature and all registered features in the identity database is lower than the third threshold, then after meeting the preset conditions, the biometric feature will be registered as a new identity in the database, and a new identity label will be assigned to the trajectory.

[0038] In the aforementioned technical solution, the specific rules for identity recognition or registration of trajectories in the "confirmed" state with facial image quality exceeding a second threshold essentially involve constructing a controlled interface for identity information storage and binding, coupled with both tracking confidence and image quality. Traditional face recognition systems often focus on single-frame detection and recognition, with their storage logic independent of the continuous tracking process. This makes them prone to invalid or duplicate identity registrations due to single-frame false detections, low-quality images, or brief interference, leading to database pollution and decreased subsequent recognition performance. Furthermore, the recognition process is often decoupled from the tracking trajectory, failing to effectively feed back and solidify the recognition results into the target's continuous trajectory. This solution, by setting explicit triggering conditions (state "confirmed" and quality exceeding a threshold) and decision rules (comparison based on a preset third threshold), strictly embeds identity recognition and registration into a high-quality, high-confidence tracking context. This makes the addition, retrieval, and binding of identity information a controlled, stateful closed-loop process within the entire tracking workflow.

[0039] This identity recognition and registration mechanism achieves high precision and reliability in identity information management through multi-level conditional filtering and judgment. First, the pre-established "confirmed status" and "high-quality threshold" ensure that samples triggering recognition not only have acceptable visual quality but also that their corresponding trajectories have been tracked and verified over a period of time, possessing spatiotemporal continuity. This significantly filters out interference from false detections, fleeting target appearances, or severely degraded frames. Second, in the core judgment stage, a binary decision based on a fixed threshold (the third threshold) is adopted: a successful match binds to an existing identity, forming an identity-based trajectory annotation; a complete mismatch only registers a new identity after meeting additional preset conditions (such as continuous appearance across multiple frames). This ensures the accuracy of identity binding, preventing misjudgments of trajectories that should match known identities as new individuals; furthermore, by setting stricter conditions for new user registration than simple successful recognition (requiring the meeting of additional preset conditions), it effectively prevents the database from easily expanding due to a single high-quality capture of unknown visitors or temporary personnel, maintaining the simplicity and effectiveness of the identity database.

[0040] In some embodiments, the new identity label is used as a correction benchmark to recalculate the identity label of the trajectory in historical frames within the backtracking time window, specifically:

[0041] Using the historical multimodal features stored within the backtracking time window and the new identity label as prior information, feature matching is re-executed.

[0042] In the aforementioned technical solution, the specific execution steps of the temporal backtracking correction in this solution involve utilizing historical multimodal features stored within the backtracking window and re-performing feature matching with newly acquired identity tags as prior information. The essence of this technology lies in constructing a strongly evidence-driven, recalculated trajectory history cleaning mechanism, thereby solving the problem of proactive detection and closed-loop correction of identity misjudgments. Traditional tracking systems have a unidirectional processing flow. Once an identity association decision is made based on instantaneous feature matching in a certain frame, this decision becomes the fixed history of the trajectory. Even if more reliable evidence is subsequently obtained, the system lacks the inherent ability to re-evaluate and update the previous decision. This open-loop characteristic leads to the continuous accumulation of identity errors, causing the credibility of identity tags on long-term trajectories to decay over time. The backtracking correction process defined in this solution overcomes the unidirectional problem: when the trajectory is bound to a new identity tag with high confidence in subsequent frames, this event is considered a clear correction signal. The system does not simply apply the new label to the current and future frames, but instead initiates a "time backtracking" process, using the new identity as a constraint, to recalculate all historical states of the trajectory within the backtracking time window.

[0043] This recalculation process utilizes historical multimodal feature raw data and, guided by new identity priors, completely re-executes the feature matching decision. Each historical frame within the backtracking window is treated as a new association problem with the known correct identity as a given condition. Since the identity relationship between the parties to be associated is now partially determined by the new identity labels (e.g., the trajectory is known to belong to "Person A"), the purpose of rematching is transformed into: in each historical frame, finding the detection target that best matches the identity features of "Person A" and verifying whether this target is compatible with other features of this trajectory (such as general appearance and motion continuity). This process fully utilizes the multimodal features stored in the historical cache and weighted by quality assessment, ensuring that the basis for re-determination is as sufficient as, or even more reliable than, the initial determination (because a stronger identity prior is introduced). In this way, the system can effectively identify and correct identity association errors caused by brief occlusion, severe pose changes, or interference from similar targets, generating a revised historical trajectory with consistent identity and logical consistency.

[0044] Therefore, this specific backtracking execution mechanism ensures that the identity labels of the output trajectory are not only passed forward on the timeline, but also backtracked and overwritten after new evidence is obtained, thereby maximizing the consistency of identity information over the global time range. This mechanism addresses the problem of error correction in long-term identity tracking, which is a post-processing approach in traditional tracking frameworks that can only perform local smoothing or interpolation.

[0045] According to another aspect of the present invention, a real-time face recognition and tracking system based on multimodal features and temporal backtracking is provided. Based on the above method, the system includes:

[0046] The face module is used to perform frame-by-frame face detection on the input video stream and track the detected face targets in real time, generating a trajectory with temporary trajectory markers for each face target;

[0047] The multimodal feature extraction and quality assessment module is used to extract multimodal features of the face target in the current frame of the trajectory. The multimodal features include at least biometric features extracted based on a face recognition model and general appearance features extracted based on a general visual model. At the same time, the module performs quality assessment on the face image of the current frame to obtain quality assessment results.

[0048] The dynamic identity association and state management module is used to determine the fusion weight of the biometric features and the general appearance features based on the quality assessment results, and calculate the association similarity between the current frame detection target and each existing trajectory based on the weighted fusion features; update the tracking status of each trajectory according to the association similarity, the tracking status includes confirmed status, unconfirmed status and lost status; wherein, when the association similarity of a trajectory is continuously higher than a first threshold, its tracking status is updated or maintained as confirmed status;

[0049] The identity recognition and temporal backtracking correction module is used to compare the biometric features of trajectories that are in a confirmed state and whose face image quality is higher than a second threshold with the identity database for identity recognition or registration, and to bind the successfully recognized identity tag to the trajectory. When a trajectory is bound with a new identity tag in a subsequent frame, temporal backtracking correction is triggered: using the new identity tag as the correction benchmark, the identity tags of the trajectory in the historical frames within the backtracking time window are recalculated, and the historical identity tags are replaced with consistent ones.

[0050] In order to better utilize the above methods, this application proposes a real-time face recognition and tracking system based on multimodal features and temporal backtracking. Each module corresponds to a step of the above methods, and its specific principles have been described above and will not be repeated here.

[0051] According to another aspect of the present invention, a real-time face recognition and tracking device based on multimodal features and temporal backtracking is provided, comprising:

[0052] At least one processor and a memory communicatively connected to said at least one processor;

[0053] The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method described above.

[0054] In the above technical solution, to better operate and process the method, the method is stored in memory, and the processor executes the stored method. It should be noted that the principle and effect of each step have been described above and will not be elaborated upon here.

[0055] According to another aspect of the present invention, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the above-described method.

[0056] In the above technical solution, to better operate and use the method, the method is stored in a computer-readable storage medium and implemented using a processor. It should be noted that the principle and effect of each step have been described above and will not be elaborated upon here. Attached Figure Description

[0057] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0058] Figure 1 This is a flowchart illustrating an embodiment of a real-time face recognition and tracking method based on multimodal features and temporal backtracking according to the present invention.

[0059] Figure 2 This is a schematic diagram of the workflow of an embodiment of a real-time face recognition and tracking method based on multimodal features and temporal backtracking according to the present invention;

[0060] Figure 3 This is a schematic diagram of an embodiment of a real-time face recognition and tracking system based on multimodal features and temporal backtracking according to the present invention. Detailed Implementation

[0061] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be particularly noted that the following embodiments are for illustrative purposes only and do not limit the scope of the invention. Similarly, the following embodiments are only some, not all, embodiments of the present invention, and all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0062] Example 1

[0063] Please see Figure 1 , Figure 2 A real-time face recognition and tracking method based on multimodal features and temporal backtracking, the method comprising:

[0064] S1. Perform face detection frame by frame on the input video stream and track the detected face targets in real time, generating a trajectory with temporary trajectory markers for each face target;

[0065] For example, to implement this step, this embodiment sets up three modules. Specifically:

[0066] Video input and preprocessing module: responsible for pulling video streams from one or more video sources (such as cameras, local video files), decoding them, and converting them into frame-by-frame image data.

[0067] Real-time detection and tracking module: Detects the position of all faces in each frame of the video in real time, performs short-term tracking on the detected faces, and assigns a temporary track ID to each face target. It adopts a "tracking-by-detection" strategy.

[0068] Face detector: Uses a lightweight and efficient deep learning detection model to output the bounding box of the face and the detection confidence (not the focus of this patent, so it will not be described in detail).

[0069] Target tracker: A self-developed face tracker is used (not the focus of this patent, so it will not be described in detail). This tracker receives the detection results of the current frame, uses a Kalman filter to predict the position of the existing trajectory in the current frame, and matches the predicted position with the actual detection box through IOU (Intersection over Union) and Hungarian algorithm, thereby achieving cross-frame association and maintaining the continuity of trajectory ID.

[0070] S2. Extract the multimodal features of the face target in the current frame from the trajectory. The multimodal features include at least biometric features extracted based on the face recognition model and general appearance features extracted based on the general visual model. At the same time, perform quality assessment on the face image of the current frame to obtain the quality assessment result.

[0071] In this embodiment, the quality assessment includes quantifying and scoring at least one of the following: the sharpness of the face image, pose angle, lighting conditions, and degree of occlusion.

[0072] For example, this embodiment performs deep analysis on each face image patch output by the tracker, extracts various features for association and recognition, and evaluates image quality.

[0073] Implementation Method: Face Quality Assessment (FQA): Input a face image patch, process it through a lightweight CNN model or a series of image processing algorithms, and output a face quality score. Evaluation metrics include, but are not limited to: blur, pose (i.e., pitch, yaw, and roll angles), illumination, and occlusion. Regarding how the face quality score is specifically obtained through algorithms and evaluation metrics, the core of this solution lies in clarifying its functional logic and implementation paradigm, rather than limiting it to a specific network structure. Its specific implementation can be based on mature technical paradigms; the key is to conduct reasonable mathematical modeling and supervised learning for the "quality assessment" task.

[0074] 1. Specific Algorithm Implementation Paradigm

[0075] The aforementioned multi-dimensional quality score can be implemented end-to-end using a lightweight convolutional neural network (CNN). The network takes a single face image region as input and its output layer is specifically designed to simultaneously or indirectly reflect multiple predefined quantification metrics.

[0076] Network design philosophy: This CNN is a feature extractor and regressor whose goal is to learn the mapping relationship between the pixels of the original face image and a set of standardized quality scores.

[0077] Output design (two common methods):

[0078] Multi-task learning: The network has multiple parallel output heads, each responsible for predicting a sub-indicator (such as Q_blur, Q_pose, Q_illum, Q_occ). Finally, the results are weighted and summed according to predetermined weights (such as 0.4, 0.3, 0.2, 0.1) to obtain the final Q_face.

[0079] Direct comprehensive scoring: The network directly outputs a comprehensive quality score Q_face. During training, the supervisory signal of this score integrates an ideal weighted combination of various sub-indicators, so its numerical change implicitly reflects the combined influence of multiple factors such as sharpness and pose.

[0080] 2. Evaluation Indicator Acquisition and Score Generation Process

[0081] The scores are learned from labeled data through supervised learning. The specific process is as follows:

[0082] Training data preparation: A large-scale face image dataset needs to be constructed, in which each face image is labeled with the ground truth values ​​of various quality sub-indicators (such as sharpness rating, pose angle, illumination score, occlusion ratio, etc.) by manual or automated tools. These ground truth values ​​are normalized to the [0,1] interval.

[0083] Model training:

[0084] If multi-task learning is adopted, the true values ​​of each sub-indicator are used as supervision signals, the loss (such as mean square error) of each task (such as regression task) is calculated separately, and then the weighted sum is used as the total loss for backpropagation.

[0085] If training a direct comprehensive scoring model, a comprehensive quality ground value needs to be synthesized for each image. For example, the ground values ​​of each sub-index can be weighted and summed directly using preset weights (i.e., Q_face_gt = 0.4 * Q_blur_gt + 0.3 * Q_pose_gt + ...), and this can be used as a supervision signal to train the network output to approximate the synthesized value.

[0086] Inference process: When deploying the application, a new face image is input and fed forward through the trained lightweight CNN to directly obtain its predicted, normalized overall quality score Q_face (and optional sub-index scores).

[0087] 3. Design Considerations and Explanations

[0088] This invention focuses on the functional definition of the quality assessment module (i.e., outputting a normalized quality score that conforms to multi-dimensional fusion logic) and its role in the system (used to control recognition triggering and feature fusion weights). The specific implementation can employ any lightweight CNN architecture that meets accuracy and efficiency requirements (such as variants of MobileNet and ShuffleNet), and the training method can be adjusted according to the data. Regardless of the output design and training method used, the final deployed model must satisfy a core functional logic—its output Q_face value can reliably distinguish whether an image is "suitable for identity recognition." That is, images with high scores should exhibit higher accuracy in subsequent identity comparisons, while images with low scores should be restricted from recognition operations by the system. Biometric feature extraction: A powerful pre-trained face recognition network is used to encode high-quality face image patches into a high-dimensional, highly recognizable identity feature vector.

[0089] Facial attribute analysis: A multi-task learning network is used to identify facial attributes such as gender, age group, whether glasses are worn, and facial expression. The intermediate layer features output by this network can also serve as auxiliary attribute feature vectors. For example, the facial attribute analysis module preferably employs a mature multi-task facial attribute recognition network, such as a multi-task learning model with a ResNet-50 convolutional neural network as its backbone. On a unified feature backbone, multiple output branches predict attributes such as gender, age group, whether glasses are worn, and expression category. Each branch is trained using cross-entropy loss or regression loss. During the inference phase, the network performs forward computation on the input facial image patch, outputting the category probability or numerical estimate of each attribute. Simultaneously, the intermediate layer feature vectors of the backbone (e.g., 256 / 512-dimensional features after global average pooling) are used as auxiliary attribute feature vectors for subsequent multimodal feature fusion and association matching. This invention does not limit the specific network structure or training data, as long as it can stably identify the attributes and output the corresponding feature vectors.

[0090] General appearance feature extraction: To address the issue of inaccurate biometric features when faces are in poor pose or occluded, an additional model pre-trained on general objects (ResNet50) is used to extract general appearance features such as color and texture of the face region.

[0091] S3. Based on the quality assessment results, determine the fusion weight of the biometric features and the general appearance features, and calculate the association similarity between the current frame detection target and each existing trajectory based on the weighted fusion features; update the tracking status of each trajectory based on the association similarity, the tracking status includes confirmed status, unconfirmed status and lost status; wherein, when the association similarity of a trajectory is continuously higher than a first threshold, its tracking status is updated or maintained as confirmed status;

[0092] In this embodiment, based on the quality assessment results, the fusion weight of the biometric feature and the general appearance feature is determined, specifically:

[0093] Based on at least one quantitative indicator from the quality assessment results, the fusion weight coefficients of the biometric feature and the general appearance feature are calculated using a predefined weight mapping function or rule.

[0094] The weight mapping function or rule is configured such that: when the quality assessment result indicates that the face image is more suitable for identity determination, a higher weight coefficient is assigned to the biometric feature than to the general appearance feature; when the quality assessment result indicates that the face image is not suitable for identity determination, a higher weight coefficient is assigned to the general appearance feature than to the biometric feature.

[0095] In this embodiment, S3 further includes: generating a feature cache for each trajectory to store historical multimodal features; when calculating the correlation similarity, comparing the fused features of the current frame with the historical features in the trajectory feature cache.

[0096] In this embodiment, when the correlation similarity of the trajectory continues to be higher than the first threshold, updating or maintaining its tracking status as confirmed further includes:

[0097] For trajectories that are in a confirmed state, their associated identity tags will be used as prior information in the calculation of association similarity for subsequent frames.

[0098] For example, this embodiment designs a dynamic identity association and state management module: it is responsible for associating newly detected faces with existing trajectories, and dynamically managing the state of each trajectory according to the confidence level of the association.

[0099] Implementation method: Maintain a state (Confirmed, Uncertain, Lost) and a short-term memory buffer containing historical multimodal features for each activity trajectory.

[0100] Association Cost Calculation: When a new detection box matches a trajectory in location, the weighted similarity between the multimodal features of the new detection box and the features in the trajectory buffer is calculated. This weight is dynamic and determined by the results of the Face Quality Assessment (FQA) module. For example, when the face pose is poor, the weight of biometric features is reduced, while the weight of general appearance features is increased. For instance, regarding the specific quantification of weight allocation in feature fusion, the core design principle of this scheme is based on dynamic adjustment of quality; that is, the weight allocation between biometric features and general appearance features is not fixed but dynamically calculated based on the output of the Face Quality Assessment (FQA) module. The allocation principle will be explained below, along with specific quantification examples.

[0101] 1. Weighting principle: Dynamic adjustment based on quality.

[0102] The weighting is designed to balance the reliability of the two types of features: under high-quality face conditions, highly discriminative biometric features should dominate; when quality deteriorates, the weight of more stable but less discriminative general appearance features should be increased accordingly to maintain the robustness of trajectory association. This adjustment directly depends on the overall face quality score Q_face output by the FQA module (usually normalized to the [0,1] interval) and can be further subdivided into sub-indicators (such as pose score Q_pose).

[0103] 2. Example of a specific allocation mechanism: The specific allocation of weights can be implemented through predefined function mappings. Here are two example schemes:

[0104] Option 1: Piecewise function allocation (example)

[0105] This scheme divides the quality into different levels based on the numerical range of Q_face, and assigns a fixed weight pair to each level.

[0106] High-quality range (suitable for identity recognition): When Q_face ≥ 0.8 (indicating good clarity, correct posture, normal lighting, and minimal occlusion).

[0107] Biometric feature weight w_id = 0.8

[0108] The weight of the general appearance feature is w_app = 0.2

[0109] Medium quality range (general quality): when 0.5 ≤ Q_face < 0.8.

[0110] Biometric feature weight w_id = 0.5

[0111] The weight of the general appearance feature is w_app = 0.5

[0112] Low quality range (significantly unsuitable for identity recognition): when Q_face < 0.5 or a specific sub-indicator (such as pose score Q_pose < 0.5).

[0113] Biometric feature weight w_id = 0.3

[0114] The weight of the general appearance feature, w_app, is 0.7.

[0115] Option 2: Linear Continuous Assignment (Example)

[0116] To make the weight changes smoother, linear interpolation can be used. For example, we can define w_id as changing linearly with Q_face within a preset interval:

[0117] w_id = 0.3 + 0.5 * Q_face (as Q_face changes from 0 to 1, w_id increases linearly from 0.3 to 0.8)

[0118] w_app = 1 - w_id

[0119] 3. Criteria for determining thresholds and weights

[0120] The specific thresholds (e.g., 0.5, 0.8) and weight values ​​(e.g., 0.3 / 0.7, 0.5 / 0.5, 0.8 / 0.2) in the above examples are illustrative embodiments, and their determination is based on:

[0121] Thresholding should be correlated with the quality level of face images in real-world scenarios, ensuring they are "reliable for identity recognition." For example, Q_face = 0.5 might correspond to an empirical quality threshold; below this value, the recognition error rate increases significantly. In practical application deployment, scenario data with trajectory association ground truth annotations can be used to optimize the overall trajectory maintenance accuracy (e.g., the MOTA metric) by adjusting these thresholds and weights, thereby determining the optimal configuration. For scenarios with clear challenges (e.g., extreme poses), sub-metrics (e.g., Q_pose) can be introduced to directly participate in weight calculations or as segmentation conditions for finer control. State transition: If the weighted similarity is higher than the high confidence threshold, the association is considered successful, the trajectory remains in the Confirmed state, and the cache is updated with new high-quality features. Regarding the specific method for obtaining the high confidence threshold, this solution is based on data-driven statistical analysis and retains adjustable flexibility in actual deployment to ensure its applicability and robustness in different scenarios. The following will elaborate on the definition of this threshold, its specific acquisition method, and the design considerations behind it.

[0122] 1. Threshold definition and function

[0123] In this system, the "high confidence threshold" (denoted as T_high) is a critical value used to evaluate the confidence level of the correlation similarity between trajectories. The correlation similarity is usually obtained by calculating the cosine similarity or normalized Euclidean distance of feature vectors (such as appearance features) and mapping them to the interval [0,1]. The higher the score, the greater the probability of correlation.

[0124] When the association similarity is greater than or equal to T_high, the system considers the confidence level of this association to be high and the result to be reliable, which can be used to directly confirm trajectory matching or update status.

[0125] The system typically also sets a "low confidence threshold" (T_low). When the similarity is ≤ T_low, the association confidence is considered low, and the match should be rejected.

[0126] The interval between T_high and T_low is defined as the "uncertain interval". Matches within this interval need to be further decided by combining other information (such as time series prediction and other modal features).

[0127] 2. Specific acquisition methods and example standards

[0128] The value of T_high is not arbitrarily set, but follows a statistically based example process:

[0129] Offline data preparation and statistics: Collect video datasets with accurate trajectory annotations, containing known "correctly associated sample pairs" (continuous detection boxes belonging to the same target) and "incorrectly associated sample pairs" (detection boxes belonging to different targets). Calculate the association similarity between all these sample pairs to form the similarity distributions for the correct and incorrectly associated groups.

[0130] Determining the threshold based on performance goals: Analyze the similarity distribution of erroneous association samples. Set a target for the highest tolerable erroneous association rate for the system (e.g., requiring the error acceptance rate to be below 1%). Find the similarity value corresponding to the target error rate (e.g., 1%) on the similarity distribution of these erroneous associations. Initially determine this value as the default value for T_high. Empirically, this value usually falls within the range of 0.7 to 0.8.

[0131] Meanwhile, observe the similarity distribution of correctly associated samples, and select a similarity point that is significantly lower than its typical value (e.g., the low tail of the distribution) as T_low, with an empirical value range of, for example, 0.4 to 0.5. The interval between the two is the "uncertain interval" that needs to be handled with caution.

[0132] 3. Design considerations and engineering adjustments

[0133] The consideration for establishing an "uncertainty interval": Directly setting a single threshold may lead to erroneous black-and-white judgments regarding boundary situations. Establishing an uncertain interval between T_low and T_high essentially introduces a buffer zone for the system. For matches falling within this interval, the system will not force a high-confidence or low-confidence conclusion, but will trigger a more complex fusion decision or remain in an undetermined state. This significantly improves the system's robustness in complex scenarios (such as transient occlusion or sudden appearance changes). The offline statistics obtained above are robust initial values ​​for general scenarios. When actually deployed in specific environments (such as shopping malls or transportation hubs), the initial small amount of field data can be used to observe the association effect through trial runs, and T_high and T_low can be fine-tuned semi-automatically or automatically to better adapt the thresholds to the unique factors of the current scene, such as lighting and crowd density.

[0134] Obtaining the high-confidence threshold T_high is an offline analysis process based on labeled data statistics, with the core objective of controlling the error rate. By simultaneously setting T_low to clarify the "uncertain interval," the system enhances its fault tolerance capability in judging fuzzy correlations. Finally, combined with scenario-based fine-tuning during deployment, this threshold mechanism ensures theoretical rigor while also possessing good engineering practicality and environmental adaptability.

[0135] If the similarity is between the high and low thresholds, or if the target is temporarily lost (occluded), the trajectory enters the Uncertain state, and the system will continue to try to re-associate it in the next few frames.

[0136] If an Uncertain trajectory cannot be successfully associated for an extended period, it is marked as Lost. If a bounding box cannot be associated with any existing trajectory, a New trajectory is created.

[0137] S4. For trajectories that are in a confirmed state and whose face image quality is higher than the second threshold, their biometric features are compared with the identity database to perform identity recognition or registration, and the successfully recognized identity tag is bound to the trajectory; when a trajectory is bound to a new identity tag in a subsequent frame, a time-series backtracking correction is triggered: using the new identity tag as the correction benchmark, the identity tags of the trajectory in the historical frames within the backtracking time window are recalculated, and the historical identity tags are replaced with consistent ones.

[0138] In this embodiment, for faces in a confirmed state with image quality exceeding a second threshold—specifically, how to judge "high face quality"—this solution designs a multi-dimensional, quantifiable, and hierarchical evaluation mechanism. This mechanism calculates an overall quality score by integrating multiple key visual indicators and makes a judgment based on a preset threshold to determine whether it is suitable to trigger advanced operations such as identity recognition. The specific mechanism and an example scheme will be described in detail below.

[0139] 1. Specific mechanisms for quality assessment: multi-dimensional weighted synthesis

[0140] The output of the Face Quality Assessment (FQA) module is an overall quality score Q_face (range [0,1]), which is obtained by a weighted sum of multiple sub-indicators. These sub-indicators typically include:

[0141] Clarity (Q_blur): Evaluates the degree of blurriness in an image; the higher the value, the clearer the image.

[0142] Pose (Q_pose): Evaluates the angle of the face relative to the camera (such as yaw and pitch). The higher the value, the closer it is to a frontal view.

[0143] Illumination (Q_illum): Evaluates whether the uniformity and intensity of illumination are appropriate; a higher value indicates better illumination conditions.

[0144] Occlusion (Q_occ): Assess the proportion of the face that is obscured (e.g., by hands, glasses, or masks). A higher value indicates less occlusion.

[0145] Example weighted formula:

[0146] Q_face = w1 * Q_blur + w2 * Q_pose + w3 * Q_illum + w4 * Q_occ

[0147] Among them, the weights w1 + w2 + w3 + w4 = 1. According to experience, sharpness and pose usually have the greatest impact on recognition, so relatively high weights can be assigned. For example:

[0148] Q_face = 0.4 * Q_blur + 0.3 * Q_pose + 0.2 * Q_illum + 0.1 * Q_occ

[0149] 2. Example scheme for "high quality" determination

[0150] The determination of "high quality" not only depends on the overall score, but also needs to consider whether the key sub - indicators meet the standards, forming a two - layer determination rule.

[0151] Set the overall quality threshold (the second threshold T_qual):

[0152] For example, set T_qual = 0.7. This threshold is the first threshold for determining "high quality".

[0153] Set the lower limits of key sub - indicators:

[0154] To prevent the situation of "high total score but extremely low key indicator" (such as extremely blurred but excellent illumination), set the minimum requirements for core sub - indicators. For example:

[0155] Q_blur ≥ 0.6 (not too blurred)

[0156] Q_pose ≥ 0.6 (not deviate too far from the front)

[0157] Final determination rule:

[0158] High - quality frame: When and only when Q_face ≥ T_qual (for example, ≥ 0.7) and all key sub - indicators meet the lower - limit requirements (for example, Q_blur ≥ 0.6 and Q_pose ≥ 0.6), the frame is determined as a "high - quality frame". Such frames are allowed to trigger the comparison with the identity database or new identity registration.

[0159] Non - high - quality frame: If Q_face < T_qual or any key sub - indicator fails to meet the standard, it is determined that the quality is insufficient. Such frames are only used for appearance - based tracking association and will not trigger the formal identity recognition or registration process to avoid introducing unreliable recognition results.

[0160] 3. Determination criteria for thresholds and weights

[0161] Weight allocation is based on the fact that the weights of sub-indicators (e.g., 0.4, 0.3, 0.2, 0.1) are determined based on prior knowledge or experimental analysis of their impact on the performance of the identity recognition algorithm. Clarity and pose are usually the most critical factors.

[0162] Threshold (T_qual) Acquisition: Determining the threshold T_qual is a data-driven process. In the offline phase, an annotated validation set is used to analyze the relationship between the recognition accuracy and trigger frequency of frames triggering recognition under different Q_face thresholds. A balance point that maintains a high recognition accuracy (e.g., >95%) while ensuring a sufficient trigger frequency is selected as the default value for T_qual (e.g., 0.7). After actual deployment, fine-tuning can be performed based on on-site statistics such as false recognition rate and false negative rate. In this embodiment, for trajectories in a confirmed state with face image quality higher than the second threshold, their biometric features are compared with the identity database for identity recognition or registration, and the successfully recognized identity tag is bound to the trajectory. Specifically:

[0163] If the similarity between the biometric feature and any registered feature in the identity database is higher than the third threshold, the identity recognition is deemed successful, and the corresponding registered identity tag is bound to the trajectory.

[0164] If the similarity between the biometric feature and all registered features in the identity database is lower than the third threshold, then after meeting the preset conditions, the biometric feature will be registered as a new identity in the database, and a new identity label will be assigned to the trajectory.

[0165] It is important to note that the definition of the requirement "meeting preset conditions" in the new identity registration rules may be ambiguous—for example, the specific quantitative standards for "multiple consecutive frames" and "high quality" are unclear, which may lead to overly conservative (missed registration) or overly aggressive (false registration) registration behavior. This solution addresses this issue through parametric engineering definition, hierarchical quality assessment, and multi-mechanism collaborative constraints.

[0166] 1. Engineering-based quantitative definition of fuzzy conditions

[0167] Quantization of "continuous multiple frames": This parameter is not a fixed value, but is preset based on the typical frame rate of the application scenario and the expected duration of user dwell time. For example, in engineering configuration, it can be explicitly set to "continuous N frames" (e.g., N=5). This value can be used as an adjustable parameter to adapt to the requirements of registration speed and rigor in different scenarios.

[0168] The refinement and grading of "high quality": A face quality assessment mechanism can be adopted, which can output multi-dimensional quality scores for each frame. For the critical operation of new identity registration, the system does not simply reuse the "second threshold" used for recognition triggering, but defines a more stringent "registration-specific quality threshold". This threshold is usually on top of the basic quality score, with additional stricter constraints on factors such as pose (e.g., requiring a near-frontal view), degree of occlusion, and illumination uniformity, forming a composite quality judgment condition, thereby ensuring that the registration process is only initiated when sufficient information is captured that is suitable for identity modeling.

[0169] 2. Multi-layered mechanisms work together to balance registration risks

[0170] Registration triggering is not solely determined by a single quality or consecutive frame count condition, but rather by a combination of the quality rating result, the consecutive frame count threshold, and the current state of the trajectory state machine (e.g., the target is in a stable tracking state). This joint judgment reduces the probability of systematic misjudgments caused by deviations in a single condition setting. The system allows for management of registered identities in subsequent processes. For example, if post-analysis reveals that two identity tags belong to the same person (possibly due to initial tracking segment splitting), the system can initiate an identity merging operation. This mechanism provides an opportunity to correct redundant or erroneous registrations that may have occurred due to "overly aggressive" condition settings in the early stages, while also mitigating the problem of temporary non-registration due to "overly conservative" settings—as long as the person reappears and meets the conditions, there is still a possibility of correct registration.

[0171] This solution, through the comprehensive application of multiple mechanisms such as quality grading, frame count threshold, state machine collaboration, and post-event backtracking and merging, enables the system to find a robust compromise that adapts to specific scenarios between the two conflicting goals of "avoiding the omission of legitimate personnel registration" and "preventing the registration of interference items by mistake," thereby ensuring the effectiveness and controllability of identity database growth.

[0172] Furthermore, this application provides examples of the aforementioned preset conditions as follows:

[0173] Regarding the specific mechanism for determining whether a biometric feature has a sufficiently high similarity to any registered feature in the identity database, specifically the mechanism for judging "sufficient similarity to a feature of a known identity in the database," the core of this solution lies in defining a quantifiable and configurable similarity metric and decision-making process. The following sections will elaborate on the specific similarity measurement method, the "sufficiently high" judgment rule (example solution), and the method for obtaining the key threshold (example standard).

[0174] 1. Similarity Measurement Methods

[0175] This scheme adopts a metric method based on the distance in the feature vector space. The specific steps are as follows:

[0176] Feature representation: For each registered identity in the identity database, store one or more normalized face depth feature vectors (e.g., 512-dimensional vectors) of it. Similarly, the same normalization process is also performed on the current biometric features of the track to be recognized.

[0177] Metric calculation: Cosine similarity is used as the core metric. Calculate the cosine similarity between the current feature vector and each registered identity feature vector (or the average representative vector of multiple vectors) in the database to obtain a set of similarity scores {S_i}, where each S_i ∈ [0, 1]. The closer the score is to 1, the more consistent the directions of the two features in the vector space and the higher the similarity.

[0178] 2. Decision rule for "high enough" (example scenario)

[0179] Determining whether it is "high enough" is a decision-making process based on a preset threshold, and the specific rules are as follows:

[0180] Find the maximum similarity: From all the calculated similarities {S_i}, find the maximum value, denoted as S_max, and the corresponding identity label is ID*.

[0181] Threshold comparison decision:

[0182] Set a third threshold T_id (identity matching threshold), for example, T_id = 0.80.

[0183] Decision success: If S_max ≥ T_id, it is considered that the similarity between the features of the current track and the features of identity ID* is "high enough", and the identity recognition is successful. The system binds the identity label ID* to the current track.

[0184] Decision failure: If S_max < T_id, it is considered that the similarity between the current features and all known identities in the database does not reach the "high enough" standard, and it is determined as a mismatch. Subsequently, if the target meets the preset new identity registration conditions such as "appearing with high quality in multiple consecutive frames", it may be registered as a new identity.

[0185] 3. Method for obtaining threshold T_id (example standard)

[0186] The setting of the threshold T_id is not an arbitrary value, but is obtained through data-driven statistical analysis to ensure the reliability of the decision:

[0187] Offline statistics and determination: Before the system is deployed, use a large face dataset with accurate identity annotations for offline analysis. The specific steps include:

[0188] Calculate the similarity between all pairs of samples representing the same person in the dataset, forming a positive sample similarity distribution. Calculate the similarity between a large number of pairs of samples representing different people, forming a negative sample similarity distribution. Analyze these two distributions and, based on the system's acceptable false acceptance rate (FAR, the rate at which different people are mistakenly identified as the same person) target (e.g., setting FAR below 1%), determine the corresponding similarity threshold from the negative sample distribution. This threshold can be used as the default T_id value (e.g., falling within the range of 0.75 to 0.85). The T_id obtained by the above method is a robust initial value. In actual deployment, the false recognition rate and false negative rate can be fine-tuned based on on-site data according to the different security and convenience requirements of the specific application scenario, thereby finally determining the optimal T_id for that scenario.

[0189] In summary, a concrete, implementable, and adjustable "sufficiently high similarity" evaluation mechanism is constructed through cosine similarity calculation, a maximum value comparison decision rule based on a fixed threshold (T_id), and a threshold setting method based on offline dataset statistical analysis and target error rate determination. This mechanism balances theoretical rigor with engineering flexibility, providing a clear technical basis for identity recognition functions.

[0190] In this embodiment, using the new identity tag as a correction benchmark, the identity tags of the trajectory in historical frames within the backtracking time window are recalculated. Specifically:

[0191] Using the historical multimodal features stored within the backtracking time window and the new identity label as prior information, feature matching is re-executed.

[0192] For example, this embodiment designs an identity recognition and time-series backtracking correction module responsible for confirming the true identity of the tracked target and binding the identity information with the trajectory. When a past recognition error is detected, a backtracking mechanism is activated for correction.

[0193] Implementation method:

[0194] Identity Recognition and Update: When a trajectory is in the Confirmed state and the facial quality is high, the system will compare its biometric features with the identity database.

[0195] Match successful: If the feature similarity with a known identity in the database is high enough, the trajectory is bound to the known IDb, and the aggregated attribute information of the ID in the database is updated using the newly collected facial attributes (age, gender, etc.).

[0196] Match failure: If the similarity with all identities in the database is low, the system considers this a new person. After certain conditions are met (e.g., high-quality frontal faces appearing in multiple consecutive frames), a new identity registration operation is performed, its high-quality feature vector is stored in the database, and a new unique identity ID is assigned.

[0197] Temporal backtracking correction: This module is triggered when a trajectory previously identified as "unknown" or incorrectly identified as "identity B" is subsequently confirmed with high confidence as "identity A" in a later frame. It backtracks a period of time from the past, using the definitive information of "identity A" as a strong prior, to recalculate and correct the identity label for that time period. This effectively corrects identity jumps caused by brief occlusion or pose changes, generating a clean and continuous identity trajectory.

[0198] This embodiment of the method also includes: persistently storing identity data and trajectory data, and providing query and analysis interfaces. The final output is a structured data stream, with each record containing: timestamp, frame number, unique identity ID, target bounding box of the current frame, current face attributes, etc., for further analysis and display by upper-layer applications.

[0199] It is important to note that, given the exponential increase in complexity resulting from the deeply coupled closed-loop design of the system, and the problems caused by the intricate states and data flows between modules (for example, inaccurate scoring by the face quality assessment module in complex scenarios may lead to errors in subsequent decision chains such as feature fusion and recognition triggering), this solution manages complexity and isolates and mitigates error propagation through clear division of module responsibilities and a state-based conditional triggering mechanism. The core principle is to decouple the two key tasks of temporal continuity and identity recognition, and to transform the high-cost recognition operation into sparse, controlled correction events, with tracking as the main thread.

[0200] 1. System Architecture Design: Tracking is the main focus, with identity recognition serving as a sparse correction node.

[0201] The system does not perform a highly coupled "detection-recognition-fusion" operation in every frame. Its core operating logic is as follows:

[0202] The multi-target tracking module serves as the foundation for continuity: the system maintains the continuity of the face's trajectory across frames by combining motion and appearance features in a multi-target tracking module. This stage involves "identity-free tracking," with the core objective of ensuring the trajectory's spatiotemporal coherence and stability, without relying on identity recognition results.

[0203] Face recognition serves as a condition-triggered correction mechanism: the identity recognition module (face comparison / registration) is not a dependency on every tracked frame, but rather designed as a "correction node" triggered under specific conditions. Its triggering is strictly controlled by both the face quality assessment results and the trajectory state machine. The system will only trigger identity recognition when both "the face quality of the current frame exceeds a preset high threshold" and "the trajectory is in a predefined stable state such as 'confirmed' or 'high confidence'." This design decouples identity recognition from a mandatory task of continuous maintenance, transforming it into a sparse, high-quality-driven confirmation and update event.

[0204] 2. Error isolation and risk control mechanisms

[0205] Suppressing false identification of low-quality frames: By setting a high quality threshold as the identification trigger condition, the system avoids unreliable identification on frames with low quality due to factors such as lighting, pose, and occlusion. Even if the quality assessment module has some deviation, the dual conditions (high quality + stable state) significantly reduce the probability of false triggering.

[0206] Preventing the contamination and propagation of erroneous labels: The results of identity recognition are primarily used to update and solidify the labels of the tracks to which they belong. Since recognition is performed only under high-quality, stable conditions, the results have high confidence. Simultaneously, the track state machine manages the "confirmation" and "locking" logic of identity labels, ensuring that low-confidence or temporary recognition results do not easily overwrite confirmed identity information. This effectively isolates single recognition errors, preventing them from contaminating the identity database or propagating continuously in subsequent frames.

[0207] The system manages system complexity and enhances robustness through two main strategies: architectural decoupling and conditional triggering. It ensures continuity through tracking, achieves correction through sparse and controlled identity recognition, and precisely determines the timing of identity intervention through joint gating of quality assessment and state machines. This design significantly reduces the reliance on the perfection of the outputs of individual modules (especially the quality assessment module, which is susceptible to environmental interference), confining potential single-point assessment errors to local levels and preventing them from triggering global decision chain errors. This achieves controllable risk management in a complexly coupled system.

[0208] Furthermore, addressing the issues arising during the cold start phase when the identity database is empty or very small, and the trajectory status is mostly "unconfirmed"—namely, the lack of prior identity information prevents the system's performance advantages from being immediately apparent, and early erroneous trajectories may not be corrected due to the inability to meet backtracking conditions, thus affecting the initial output quality—this solution comprehensively considers three aspects: system phased behavior design, error correction timing mechanism, and explicit status management. Its core lies in accepting the objective limitations of the cold start phase and ensuring, through architectural design, that system performance gradually improves with runtime and data accumulation.

[0209] 1. Behavioral patterns during the cold start phase: functional degradation and resource optimization

[0210] In the initial stages, when the identity database is scarce, the system cannot immediately leverage the full advantages of the "identity-aware" closed loop. Therefore, the system is designed to degenerate into an enhanced multi-target tracker.

[0211] In the "unconfirmed" state, the correlation and maintenance of the trajectory rely entirely on the motion model and appearance features. Its behavior is equivalent to a high-quality, identity-aware multi-target tracker, thus ensuring the availability of basic tracking functions. Even if the identity module cannot provide effective information temporarily, the system can still optimize the scheduling of computing resources using the face quality assessment module. For example, it can suppress high-cost feature extraction or recognition attempts on low-quality frames, thus demonstrating an advantage over blind processing systems from the outset in terms of efficiency.

[0212] 2. Correctability of early errors: Incremental optimization based on long-term backtracking

[0213] The key design feature of this solution is that trajectory errors arising early due to a lack of prior knowledge are not permanently locked by the system. Its correction mechanism is dynamic and delayed in triggering.

[0214] Condition-triggered backtracking correction: As the system runs for a period of time, the identity database gradually becomes richer with the registration of new identities. Once a target's identity is confirmed with high confidence in a subsequent frame (i.e., the triggering conditions of "high quality + stable state" are met), the system will not only update the current and future trajectory status, but also perform backtracking correction on the historical trajectory window associated with that identity. This means that even erroneous associations or labels generated during the cold start phase have the opportunity to be automatically corrected after obtaining reliable identity evidence, thereby continuously improving the accuracy and consistency of the overall trajectory over a longer time scale.

[0215] 3. Explicit Management and Output Strategies for State Machines

[0216] The system explicitly distinguishes between "unconfirmed" and "confirmed" states using a state machine to manage the output confidence and downstream impact at different stages.

[0217] Conservative output for "unconfirmed" states: During the cold start phase, the system maintains most targets in the "unconfirmed" state. For such trajectories, the system only outputs their spatiotemporal continuity information, while avoiding over-interpretation or high-confidence assertions regarding their identity labels. This reduces the risk of misleading subsequent application modules that rely on identity information.

[0218] "Confirmed" status triggers in-depth processing: Only when a trajectory enters the "confirmed" or "high confidence" status will the system activate the in-depth processing chain associated with that identity, including backtracking corrections and increased resource investment. This ensures that high-cost, high-impact operations are only based on a reliable identity.

[0219] This solution addresses the cold start phase by ensuring basic performance through functional degradation and maintaining efficiency through quality-aware scheduling. More importantly, by managing output confidence through a state machine and relying on a long-term backtracking correction mechanism, the system ensures that early errors are gradually corrected as the database expands and identity verification events accumulate. Therefore, system performance does not peak at startup but gradually approaches and eventually stabilizes at the optimal level of the closed-loop design over time, achieving a smooth transition and continuous optimization from "no prior tracking" to "strong identity awareness."

[0220] Furthermore, this application designs multiple thresholds in the solution (first threshold: high association similarity threshold; second threshold: quality threshold for triggering identification; third threshold: identity matching threshold; backtracking window length, etc.). Addressing the issues that the setting of several key thresholds in the system (such as the high association similarity threshold, the quality threshold for triggering identification, the identity matching threshold, and the backtracking window length) depends on specific application scenarios, making them difficult to apply universally. Moreover, in actual deployment, they require a large amount of data and repeated trials for optimization, leading to high deployment costs and difficulties in scenario migration. This solution considers both the functional positioning and configuration strategy of the thresholds, aiming to reduce the absolute dependence on the accuracy of the thresholds and construct a scalable, layered parameter management framework to improve the system's deployability and scenario adaptability.

[0221] 1. Threshold function positioning: Controlling collaborative logic, rather than the correctness of the core algorithm.

[0222] This scheme clearly distinguishes the different roles of thresholds, with most thresholds primarily used to regulate the collaborative logic and resource scheduling strategies between modules, rather than determining the core correctness of the underlying algorithm. This design reduces the system's sensitivity to the absolute precision of the threshold values.

[0223] Taking the quality threshold for triggering identification (the second threshold) as an example: the main function of this threshold is to determine when to deploy the computationally expensive identity recognition module. If set too loosely, identification triggers will be more frequent, potentially increasing computational overhead and producing unreliable results on some low-quality frames; if set too tightly, identification triggers will be more conservative, potentially slightly delaying identity confirmation, but ensuring a high confidence level for the identification event. Regardless of whether it is loose or tight, its impact is mainly reflected in the trade-off between "identification frequency and computational efficiency," without disrupting tracking continuity or causing functional errors in the identity module itself. The system can still maintain basic availability even when the threshold is not optimal.

[0224] 2. Layered Configurable Strategy: General Benchmarks and Scenario-Based Templates

[0225] To avoid performing time-consuming threshold tuning from scratch for each new scenario, this solution adopts a layered configuration strategy:

[0226] General Layer (Determined through Offline Evaluation): For parameters reflecting the performance boundaries of the algorithm itself, such as the identity matching threshold, a robust initial value that balances false acceptance and false rejection rates in most cases can be determined by evaluating them on large, diverse offline datasets (e.g., plotting ROC curves). This value serves as the basic configuration of the system and has good generalization ability.

[0227] Scene Layer (Pre-defined Profiles): For parameters that heavily rely on environmental factors, such as quality assessment thresholds, correlation similarity thresholds, and backtracking window lengths, a set of parameter profiles can be pre-defined based on typical business scenarios (e.g., "indoor fixed monitoring," "high-density pedestrian flow in outdoor plazas," "low-light entrances and exits," etc.). Each profile contains a set of parameters optimized for the lighting, viewing angle, and pedestrian density characteristics of that scenario. During deployment, engineers can select the closest profile based on the site environment, eliminating the need for parameter tuning from scratch, greatly reducing deployment complexity and cost.

[0228] This solution utilizes a general and scenario-based configuration system to pre-process time-consuming parameter tuning by performing offline analysis and scenario template preparation. In actual deployment, only simple scenario matching or limited calibration is required to achieve satisfactory performance. This design significantly improves the system's engineering convenience and cross-scenario migration capabilities, effectively controlling deployment and maintenance costs.

[0229] Furthermore, addressing potential issues with time-series backtracking correction mechanisms (such as target confusion leading to the inability to obtain high-confidence labels, failure to trigger backtracking for transient targets, and historical contamination by erroneous labels) and their potential problems in real-time performance and resource consumption, this solution comprehensively considers and balances these issues through conservative design principles, strict limitations on triggering conditions, multimodal verification of the correction process, and targeted optimization of computation and storage. Its core idea is that this mechanism aims to significantly improve error accumulation in mainstream scenarios, rather than guaranteeing a perfect solution for all edge cases; simultaneously, its implementation is efficiency-optimized to control the impact on system real-time performance and resources. Specifically:

[0230] 1. Robust design of the backtracking trigger mechanism: handling complex scenarios

[0231] This scheme does not assume that absolutely correct high-confidence identity labels can be obtained under all circumstances. Backtracking correction is considered a reinforcement mechanism to improve long-term consistency when certain conditions are met, and its design includes the following protective mechanisms:

[0232] Strict and complex triggering conditions: Identity verification and subsequent backtracking require not only high confidence in a single identification result but also compatibility with existing multimodal historical features of the trajectory (such as long-term appearance and movement patterns). This dual verification mechanism significantly reduces the probability of erroneous backtracking triggered by transient similarities between targets (such as twins or individuals dressed identically). In scenarios where long-term confusion makes reliable differentiation impossible, the system is more likely to maintain the relevant trajectory in a conservative "unconfirmed" state rather than forcibly assigning a potentially incorrect identity.

[0233] Differentiated target handling strategy: The system clearly distinguishes between "targets with medium to long durations" and "targets with extremely short appearances." The backtracking mechanism primarily serves the former, as it provides a sufficient time window to accumulate information and meet triggering conditions. For targets whose appearance time is shorter than the backtracking window, the system neither relies on nor insists on identity correction through backtracking. Instead, it outputs high-quality tracking trajectories with conservative identity labels (such as "unconfirmed"), which is a reasonable result in such scenarios.

[0234] Multimodal verification of the correction process: Backtracking is not simply about overwriting the historical window with new labels. Its essence is to re-execute a quality-weighted multimodal matching process within the historical time window, using the newly confirmed identity features as a priori. This process searches for the detection target that best matches the identity features and rigorously verifies its compatibility with the general appearance features of the trajectory and motion continuity. This design ensures that even if backtracking based on incorrect new labels is triggered, the verification of historical data is highly likely to be interrupted due to compatibility verification failure, thus preventing the correct historical trajectory from being "contaminated" on a large scale.

[0235] 2. Optimization of computational and storage efficiency of the backtracking mechanism

[0236] To address the potential computational spikes and memory pressures caused by backtracking, this solution primarily involves feature matching operations during the backtracking correction process. The required historical data, such as deep features, is already cached in the trajectory. Feature matching (e.g., calculating vector similarity) is a relatively low-computation operation and does not incur the enormous overhead of re-performing deep learning inference. Therefore, even when triggered centrally in high frame rate or multi-target scenarios, the additional computational load is manageable and will not pose a fundamental challenge to the system's real-time performance.

[0237] The historical feature cache maintained to support backtracking can be addressed through the following design. First, high-dimensional deep features (such as facial recognition features, FR) are represented using compressed representations (e.g., 512-dimensional FP16 format), resulting in a small data volume. Quantitative analysis shows that in a scenario with tens of thousands of users, even if only one representative feature is stored for each person, the required memory is only around 10MB. Second, other features used for matching and verification (such as general appearance features) are typically lower-dimensional vectors or sparse representations. Therefore, in long-term, high-density scenarios, memory usage growth is linear and within the range easily managed by modern hardware.

[0238] This solution employs conservative triggering, multimodal verification, and scenario-differentiated processing to improve long-term tracking consistency in mainstream scenarios while effectively mitigating or avoiding edge risks such as target confusion, transient appearances, and the propagation of incorrect labels. Regarding resource consumption, by relying on lightweight feature matching and optimized storage representation, it ensures that the mechanism will not trigger uncontrollable computational peaks or memory pressure during actual deployment, thus meeting the performance requirements of real-time systems.

[0239] Under the current technological paradigm, the development of real-time face recognition and tracking systems typically revolves around several independent and even inherently conflicting technical indicators. These indicators mainly include: tracking robustness in maintaining trajectory continuity under complex conditions such as occlusion and deformation; recognition accuracy in ensuring correct identity determination in single-frame or short-time comparisons; and real-time system efficiency that meets actual deployment needs and is related to processing speed and resource consumption. Over the long term, technological evolution has led to different development paths with varying focuses: to enhance robustness, trackers based on motion models and appearance features (such as DeepSORT) have emerged, but they lack identity recognition capabilities and are difficult to deeply integrate with recognition modules; to improve recognition rates, high-precision face recognition is often relied upon for each frame or key frame, but this incurs high computational overhead, and isolated recognition makes it difficult to utilize temporal context, leading to a disconnect from the tracking trajectory; to ensure efficiency, lightweight models or frequency reduction processing are often used, but this inevitably results in a loss of accuracy or trajectory continuity.

[0240] Against this technological backdrop, a consensus has emerged in the field: long-term identity consistency, high recognition accuracy, and high real-time efficiency are difficult to achieve simultaneously. Any technological improvement can usually only focus on one or two of these indicators, at the expense of others. Therefore, existing solutions are mostly localized optimizations targeting single performance bottlenecks, lacking a holistic approach that addresses multiple challenges from a system-wide perspective.

[0241] To address the issue that existing technologies often limit themselves to localized optimizations in individual dimensions and fail to establish a collaborative mechanism at the system level to overcome these mutually constraining challenges holistically, this invention achieves a comprehensive leap in technological effectiveness by redefining and collaboratively addressing this system-level problem.

[0242] Traditional methods typically design face tracking and identity recognition as independent modules, either serial or parallel, resulting in fragmented optimization objectives. The tracking module focuses on trajectory continuity but lacks identity awareness, while the recognition module prioritizes single-frame accuracy while ignoring temporal context. Furthermore, indiscriminate computation at the system level often leads to resource waste. This fragmented design paradigm makes it difficult to construct a holistic solution that simultaneously guarantees long-term consistent identity tracking and efficient resource scheduling. To address this, this invention restructures the problem as: how to construct a closed-loop system with state awareness, decision feedback, and self-correction capabilities in real-time streaming processing, thereby collaboratively achieving adaptive instantaneous association, cross-frame tag consistency, and intelligent allocation of computing resources.

[0243] To address the aforementioned comprehensive issues, this invention proposes a closed-loop framework using face quality assessment as the unified perception signal and identity confidence as the core state variable. This framework dynamically adjusts the fusion weights of multimodal features based on real-time quality assessment, enabling scenario-adaptive association decisions. By embedding high-confidence identity information into tracking state management, it achieves continuous feedback from identity recognition to the association process. Furthermore, it designs a temporal backtracking mechanism based on strong evidence triggering to proactively correct historical identity labels. Simultaneously, the quality assessment signal directly drives computational resource scheduling, ensuring that high-cost recognition operations are triggered only for high-quality face images.

[0244] In summary, this invention achieves an overall improvement in technical indicators without significantly increasing computational complexity: the system can maintain long-term stability and consistency of identity trajectories in complex scenarios, while significantly reducing invalid computations, thereby achieving synergistic optimization of robustness, consistency and efficiency.

[0245] Example 2

[0246] Please see Figure 3 A real-time face recognition and tracking system based on multimodal features and temporal backtracking, based on the method described in one embodiment, the system comprising:

[0247] The face module is used to perform frame-by-frame face detection on the input video stream and track the detected face targets in real time, generating a trajectory with temporary trajectory markers for each face target;

[0248] The multimodal feature extraction and quality assessment module is used to extract multimodal features of the face target in the current frame of the trajectory. The multimodal features include at least biometric features extracted based on a face recognition model and general appearance features extracted based on a general visual model. At the same time, the module performs quality assessment on the face image of the current frame to obtain quality assessment results.

[0249] The dynamic identity association and state management module is used to determine the fusion weight of the biometric features and the general appearance features based on the quality assessment results, and calculate the association similarity between the current frame detection target and each existing trajectory based on the weighted fusion features; update the tracking status of each trajectory according to the association similarity, the tracking status includes confirmed status, unconfirmed status and lost status; wherein, when the association similarity of a trajectory is continuously higher than a first threshold, its tracking status is updated or maintained as confirmed status;

[0250] The identity recognition and temporal backtracking correction module is used to compare the biometric features of trajectories that are in a confirmed state and whose face image quality is higher than a second threshold with the identity database for identity recognition or registration, and to bind the successfully recognized identity tag to the trajectory. When a trajectory is bound with a new identity tag in a subsequent frame, temporal backtracking correction is triggered: using the new identity tag as the correction benchmark, the identity tags of the trajectory in the historical frames within the backtracking time window are recalculated, and the historical identity tags are replaced with consistent ones.

[0251] In this embodiment, in order to better utilize the method described in one of the embodiments, this application proposes a real-time face recognition and tracking system based on multimodal features and temporal backtracking. Each module corresponds to each step of the above method, and its specific principle has been described above and will not be repeated here.

[0252] Example 3

[0253] A real-time face recognition and tracking device based on multimodal features and temporal backtracking includes:

[0254] At least one processor and a memory communicatively connected to said at least one processor;

[0255] The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method described in one of the embodiments.

[0256] In this embodiment, to better run and process the method described in one of the embodiments, the above method is stored in a memory, and the stored method is executed using a processor. It should be noted that the principle and effect of each step have been described above and will not be elaborated further here.

[0257] Example 4

[0258] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in one of the embodiments.

[0259] In this embodiment, to better operate and use the method described in one of the embodiments, the above method is stored in a computer-readable storage medium, and the above method is implemented using a processor. It should be noted that the principle and effect of each step have been described above and will not be elaborated further here.

[0260] The above description is only a part of the embodiments of the present invention and does not limit the scope of protection of the present invention. Any equivalent device or equivalent process transformation made based on the content of the present invention specification and drawings, or direct or indirect application in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A real-time face recognition and tracking method based on multimodal features and temporal backtracking, characterized in that, The method includes: S1. Perform face detection frame by frame on the input video stream and track the detected face targets in real time, generating a trajectory with temporary trajectory markers for each face target; S2. Extract the multimodal features of the face target in the current frame from the trajectory. The multimodal features include at least biometric features extracted based on the face recognition model and general appearance features extracted based on the general visual model. At the same time, perform quality assessment on the face image of the current frame to obtain the quality assessment result. S3. Based on the quality assessment results, determine the fusion weight of the biometric features and the general appearance features, and calculate the association similarity between the current frame detection target and each existing trajectory based on the weighted fusion features; update the tracking status of each trajectory based on the association similarity, the tracking status includes confirmed status, unconfirmed status and lost status; wherein, when the association similarity of a trajectory is continuously higher than a first threshold, its tracking status is updated or maintained as confirmed status; S4. For trajectories that are in a confirmed state and whose face image quality is higher than the second threshold, their biometric features are compared with the identity database to perform identity recognition or registration, and the successfully recognized identity tag is bound to the trajectory; when a trajectory is bound to a new identity tag in a subsequent frame, a time-series backtracking correction is triggered: using the new identity tag as the correction benchmark, the identity tags of the trajectory in the historical frames within the backtracking time window are recalculated, and the historical identity tags are replaced with consistent ones.

2. The real-time face recognition and tracking method based on multimodal features and temporal backtracking as described in claim 1, characterized in that, The quality assessment includes quantifying at least one of the following: the sharpness of the face image, pose angle, lighting conditions, and degree of occlusion.

3. The real-time face recognition and tracking method based on multimodal features and temporal backtracking as described in claim 1, characterized in that, Based on the quality assessment results, the fusion weights of the biometric features and the general appearance features are determined, specifically: Based on at least one quantitative indicator from the quality assessment results, the fusion weight coefficients of the biometric feature and the general appearance feature are calculated using a predefined weight mapping function or rule. The weight mapping function or rule is configured such that: when the quality assessment result indicates that the face image is more suitable for identity determination, a higher weight coefficient is assigned to the biometric feature than to the general appearance feature; when the quality assessment result indicates that the face image is not suitable for identity determination, a higher weight coefficient is assigned to the general appearance feature than to the biometric feature.

4. The real-time face recognition and tracking method based on multimodal features and temporal backtracking as described in claim 1, characterized in that, The S3 further includes: generating a feature cache for each trajectory to store historical multimodal features; when calculating the correlation similarity, comparing the fused features of the current frame with the historical features in the trajectory feature cache.

5. The real-time face recognition and tracking method based on multimodal features and temporal backtracking as described in claim 1, characterized in that, When the correlation similarity of trajectories remains above a first threshold, the tracking status is updated or maintained as confirmed. This also includes: For trajectories that are in a confirmed state, their associated identity tags will be used as prior information in the calculation of association similarity for subsequent frames.

6. The real-time face recognition and tracking method based on multimodal features and temporal backtracking as described in claim 1, characterized in that, For trajectories that are in a confirmed state and whose facial image quality is higher than the second threshold, their biometric features are compared with the identity database for identity recognition or registration, and the successfully recognized identity tag is bound to the trajectory. Specifically: If the similarity between the biometric feature and any registered feature in the identity database is higher than the third threshold, the identity recognition is deemed successful, and the corresponding registered identity tag is bound to the trajectory. If the similarity between the biometric feature and all registered features in the identity database is lower than the third threshold, then after meeting the preset conditions, the biometric feature will be registered as a new identity in the database, and a new identity label will be assigned to the trajectory.

7. The real-time face recognition and tracking method based on multimodal features and temporal backtracking as described in claim 1, characterized in that, Using the new identity label as a correction benchmark, the identity labels of the trajectory in historical frames within the backtracking time window are recalculated, specifically: Using the historical multimodal features stored within the backtracking time window and the new identity label as prior information, feature matching is re-executed.

8. A real-time face recognition and tracking system based on multimodal features and temporal backtracking, characterized in that, Based on the method according to any one of claims 1-7, the system comprises: The face module is used to perform frame-by-frame face detection on the input video stream and track the detected face targets in real time, generating a trajectory with temporary trajectory markers for each face target; The multimodal feature extraction and quality assessment module is used to extract multimodal features of the face target in the current frame of the trajectory. The multimodal features include at least biometric features extracted based on a face recognition model and general appearance features extracted based on a general visual model. At the same time, the module performs quality assessment on the face image of the current frame to obtain quality assessment results. The dynamic identity association and state management module is used to determine the fusion weight of the biometric features and the general appearance features based on the quality assessment results, and calculate the association similarity between the current frame detection target and each existing trajectory based on the weighted fusion features; update the tracking status of each trajectory according to the association similarity, the tracking status includes confirmed status, unconfirmed status and lost status; wherein, when the association similarity of a trajectory is continuously higher than a first threshold, its tracking status is updated or maintained as confirmed status; The identity recognition and temporal backtracking correction module is used to compare the biometric features of trajectories that are in a confirmed state and whose face image quality is higher than a second threshold with the identity database for identity recognition or registration, and to bind the successfully recognized identity tag to the trajectory. When a trajectory is bound with a new identity tag in a subsequent frame, temporal backtracking correction is triggered: using the new identity tag as the correction benchmark, the identity tags of the trajectory in the historical frames within the backtracking time window are recalculated, and the historical identity tags are replaced with consistent ones.

9. A real-time face recognition and tracking device based on multimodal features and temporal backtracking, characterized in that, include: At least one processor and a memory communicatively connected to said at least one processor; The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

Images