A face recognition method based on multi-modal feature fusion

Abstract

This invention discloses a face recognition method based on multimodal feature fusion, belonging to the fields of computer vision and artificial intelligence technology. The method includes the following steps: Step S1, Data reception and parsing: Obtaining basic video parameters; merging the three types of metadata of each coding tree unit into pixels to generate a three-channel feature map; Step S2, Fast face detection: Using the three-channel feature map as input, a shallow convolutional neural network is used to determine whether it contains a face; for three-channel feature maps that detect faces, a real-time object detection model is selected to simultaneously complete face localization, key point annotation, and gaze point prediction; Step S3, Dynamic tracking and tracking ID correction; Step S4, Gaze point optimization and output. This solution integrates compressed domain, depth, and spatiotemporal multimodal features, taking into account both illumination and pose robustness and detailed representation, balancing real-time performance and efficiency, and has low deployment costs.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and artificial intelligence, specifically to a face recognition method based on multimodal feature fusion. Background Technology

[0002] With the rapid popularization of facial recognition technology, facial spoofing attacks have become a serious security threat. In daily life, we increasingly rely on facial recognition technology—from unlocking phones to payment verification, from airport security checks to community access control. However, malicious attackers can deceive recognition systems by printing photos, playing videos, or creating 3D masks. Existing anti-spoofing technologies have three main problems:

[0003] 1. Limitations of traditional feature representation capabilities.

[0004] Mainstream methods rely on hand-designed shallow features (such as Haar-like features and HOG features), which are essentially statistical models of local texture gradients. These features have three major drawbacks: First, in scenes with drastic lighting changes (such as backlighting or shadow coverage), the feature response exhibits gradient saturation, leading to a decrease in recognition rate; second, they are sensitive to pose changes, and when the face deviates from the frontal viewpoint by more than 30°, the directional gradient distribution of HOG features becomes distorted; finally, they lack the ability to represent high-frequency details and cannot effectively capture subtle discriminative features such as iris texture and eyelash shadows.

[0005] 2. Defects of the single-modal feature fusion mechanism.

[0006] Existing technical solutions generally rely on single-modal features for decision-making, resulting in significant underutilization of information. For example, systems based on RGB single-modality are vulnerable to two-dimensional forgery attacks (paper photos, electronic displays); systems relying on infrared thermal imaging can capture blood vessel distribution features, but are ineffective against high-precision silicone masks; and systems relying solely on depth information have blind spots when dealing with projection attacks.

[0007] 3. The contradiction between real-time performance and computational complexity.

[0008] High-performance anti-counterfeiting systems generally face computing power bottlenecks: proactive solutions based on 3D reconstruction require dedicated hardware support and have high deployment costs; while lightweight solutions can be deployed on mobile devices, their anti-counterfeiting accuracy is low. Summary of the Invention

[0009] To address the shortcomings of existing technologies, this invention provides a face recognition method based on multimodal feature fusion.

[0010] The present invention employs the following technical means.

[0011] A face recognition method based on multimodal feature fusion includes the following steps:

[0012] Step S1, Data Reception and Parsing: Obtain basic video parameters; using the HEVC compressed stream as input, extract basic video information by parsing the sequence parameter set and image parameter set, and perform entropy decoding; then, perform shallow decoding on the header information of each coding tree unit to extract three types of metadata: intra-frame prediction mode, transform unit size, and coding bit depth, and map them to grayscale values ​​of 0~255; finally, merge the three types of metadata of each coding tree unit into pixels to generate a three-channel feature map, preserving the HEVC structural features;

[0013] Step S2, Fast Face Detection: Using a three-channel feature map as input, a shallow convolutional neural network is used to determine whether it contains a face; for the three-channel feature map that detects a face, a real-time object detection model is selected to simultaneously complete face localization, key point annotation and gaze point prediction.

[0014] Step S3, Dynamic Tracking and Tracking ID Correction: Extract feature vectors that reflect individual uniqueness from the face image and assign a unique tracking ID to the feature vector. Then, complete the face tracking and matching by motion matching, feature matching and intersection-union ratio correction to determine whether it is the same person from different dimensions.

[0015] Step S4, line-of-sight optimization and output: Using the tracking ID output in step S3 and the unsmoothed line-of-sight coordinates output in step S2 as input, Kalman filtering is used to smooth out transient interference and output a JSON structured result.

[0016] Further, step S1 includes:

[0017] Step S101, Data reception and parsing:

[0018] Parse the sequence parameter set to extract basic video information, including: video resolution, coding tree unit size, and chroma sampling format; calculate the feature image size, and determine the width and height of the intra-frame transform block feature image based on the video resolution and coding tree unit size;

[0019] Parse the image parameter set: Extract encoding configuration parameters, including quantization parameters, entropy coding mode, and maximum depth of transform units, to guide the subsequent metadata parsing logic;

[0020] Step S102, metadata extraction, performing shallow decoding on the entropy decoding output of the HEVC compressed stream:

[0021] Shallow decoding in intra-prediction mode:

[0022] 1. Since coding tree units in HEVC can be recursively divided into prediction units, all coding tree units in the current frame are traversed to locate the prediction unit level;

[0023] 2. For intra-coded prediction units, the coding index of the intra-prediction mode is read from its header information to obtain the prediction direction coding value; the prediction direction coding value ranges from 0 to 34, corresponding to 35 modes in different directions.

[0024] 3. Normalization mapping: The predicted direction encoding values ​​of the 35 modes are linearly mapped to a grayscale range of 0~255. ;

[0025] Among them, I IPM It is the grayscale mapping value of the intra-pred mode; intra_pred_mode is the encoding index of the intra-pred mode, with a value range of 0~34; It is a floor function; 35 is the total number of modes; 255 is the maximum value of 8-bit grayscale.

[0026] Shallow decoding with varying unit size:

[0027] 1. Analyze the segmentation structure of the coding tree unit: For each frame of the HEVC image, it is first divided into multiple coding tree units. Each coding tree unit is recursively divided into coding units, and each coding unit is further divided into transform units.

[0028] 2. Read the header information of the transformation unit:

[0029] Each transform unit has its own header information. The focus is on extracting fields related to the exponential encoding and obtaining the size encoding of the transform unit according to the size mapping rules.

[0030] 3. The size of the transform unit is encoded as the transform unit size, and the transform unit size is linearly mapped to grayscale values ​​of 0-255: ;

[0031] Among them, I TUS The grayscale mapping value is the size of the transformation unit; TUS is the size of the transformation unit.

[0032] Shallow decoding of encoded bits:

[0033] Average normalization based on 8×8 blocks is adopted: ;

[0034] Among them, BN avg It is the average number of bits encoded within an 8×8 block; BN original It is the number of bits in the original coding unit; k is the number of 8×8 blocks;

[0035] Then perform grayscale linear mapping: ;

[0036] Among them, I BN It is the grayscale mapping value of the encoded bit depth, BNmin and BN max These are the minimum and maximum values ​​of the intra-frame coding bit length, respectively;

[0037] Step S103, Image Construction and Fusion: Generate a three-channel feature map;

[0038] Metadata of each coding tree unit (I IPM I TUS I BN Mapped to 1 pixel, generating a three-channel feature map I. ITB =(I IPM I TUS I BN ).

[0039] Furthermore, in step S2, the real-time target detection model outputs three-scale feature maps: a 160×160 semantic layer, an 80×80 transition layer, and a 40×40 localization layer. Using a bidirectional multi-scale feature fusion architecture of feature pyramid network and path aggregation network, the hierarchical transmission of semantic and location information is realized, and multi-scale feature maps that fuse semantic and location information are output: a 160×160 large-scale feature map and a 40×40 small-scale feature map.

[0040] Furthermore, in step S2, a decoupled head is used to process five tasks independently: bounding box regression branch, classification branch, confidence branch, key point regression branch, and gaze point regression branch, outputting the face bounding box, five key points, and gaze point coordinates.

[0041] Furthermore, in step S3, the FastReID model is used to transform the input face image into a 128-dimensional feature vector. ;

[0042] Motion matching: By observing the facial movement trajectory, it determines whether they belong to the same person, using Mahalanobis distance d. M It measures the positional similarity between the current frame's detection bounding box and the previous frame's tracking bounding box; the judgment condition is: when d M If the value is less than 2.5, determine that the two boxes correspond to the same person; otherwise, proceed to the next step.

[0043] Feature matching: By comparing the 128-dimensional feature vectors of the face Based on the similarity of facial feature vectors, the cosine distance d is used. C The system measures the consistency of identity between the current frame and historical features to determine whether they belong to the same person; the judgment condition is: when d C If the value is less than 0.3, determine that the two boxes correspond to the same person; otherwise, proceed to the next step.

[0044] Intersection over Union (IoU) to fill gaps: If the first two steps of matching do not find the same person, try matching again using the overlap of the two bounding boxes to avoid missing the real same person: Judgment condition: If IoU > 0.5, it means that the two face bounding boxes to be matched correspond to the same person; otherwise, mark the face corresponding to the box as a new tracking ID.

[0045] Furthermore, in step S4, the JSON structured result includes the tracking ID, line-of-sight coordinates, and tracking status; the tracking status includes:

[0046] Continuous tracking: When the tracking ID of the target object in the current frame is completely consistent with that in the previous frame, and within a time window of 3 consecutive frames, the confirmation logic of dynamically tracking the same person in step S3 is satisfied, and the target is not obscured by other objects or misjudged due to visual confusion, then the tracking status and original tracking ID of the target are maintained.

[0047] Tracking ID switching: If the tracking ID of the target in the current frame is inconsistent with that in the previous frame, but the cosine distance is less than 0.3 through similarity matching of target features, the tracking ID of the current target is determined, and then the tracking ID correction operation is performed to ensure the continuity of the target identity in time.

[0048] New Tracking ID Assignment: When a target object is detected for the first time in a video sequence, a unique tracking ID is assigned to the target as an identification, and the continuous tracking process for the new target is initiated.

[0049] Loss of contact determination: If the target object is not detected by the detection module for 30 consecutive frames, the system determines that the target is in a lost contact state and will trigger the tracking termination logic.

[0050] This solution employs "compressed domain lightweight features + multimodal fusion + lightweight deep learning," fundamentally addressing the three major shortcomings of existing technologies: weak features, single modality, and low efficiency. It achieves integrated face recognition and anti-spoofing with "high precision, low latency, and strong robustness," with the following specific benefits:

[0051] 1. Overcome the limitations of traditional feature representation capabilities: multimodal feature fusion + compressed domain structural features.

[0052] This scheme does not directly rely on manual shallow features, but integrates three types of multimodal features: First, compressed domain features, which extract three types of metadata from the HEVC bitstream: intra-prediction mode, transform unit size, and coding bit depth, and map them to a three-channel feature map. Real faces have concentrated intra-prediction modes (over 60% in the horizontal / vertical direction), small transform units (4×4 / 8×8, mapped to dark colors), and high coding bit depth (mapped to high grayscale) due to symmetry. Forged faces have dispersed modes, large units (16×16 / 32×32, bright colors), and low bit depth, thus distinguishing between real and fake faces and resisting interference from lighting and pose. Second, depth features, which use the FastReID model to extract 128-dimensional feature vectors from the corrected face image (angle between the same person <30°, angle between different people >60°), capturing high-frequency details such as iris texture. Third, spatiotemporal features, which use Kalman filtering to utilize the temporal correlation of viewpoint trajectory (such as human eye rotation inertia) to suppress blinking and head shaking noise, thereby improving detail representation and robustness.

[0053] 2. To address the shortcomings of single-modal feature fusion mechanisms (single modality is vulnerable to specific forgery attacks), this solution employs a multi-modal collaborative anti-counterfeiting approach combining "compressed domain + depth + spatiotemporal": compressed domain features defend against two-dimensional forgery of photos or videos (disordered distribution of three-channel feature maps), depth features distinguish between real faces and 3D masks (differences in vector encoding materials), and spatiotemporal features identify projection attacks (smooth trajectories reveal static features), thus achieving full-type forgery defense.

[0054] 3. To address the conflict between real-time performance and computational complexity, this solution employs direct processing in the compressed domain without full decoding: Step S1 performs only entropy decoding (accounting for 20% of full decoding), skipping inverse quantization / inverse transform, and directly extracts metadata from the bitstream to generate a three-channel feature map; Step S2 uses a shallow CNN (2 convolutional layers + 2 fully connected layers) to filter out faceless regions, and YOLOv5's anchorless box design autonomously learns the bounding box size (adapting to near-large and far-small scenes), combining feature pyramids and path aggregation networks to bidirectionally fuse multi-scale features, achieving real-time detection at 30fps (supporting 8K video); Step S4 uses Kalman filtering to smooth the viewpoint coordinates of 5 consecutive frames, reducing the subsequent computational burden and significantly reducing deployment costs while maintaining high accuracy (>99.5%). Attached Figure Description

[0055] Figure 1 This is a flowchart comparing the traditional solution and the proposed solution in step S1.

[0056] Figure 2 This is a flowchart of the present invention. Detailed Implementation

[0057] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0058] This solution addresses three key technical challenges: First, the low efficiency of high-resolution video processing. With the widespread adoption of 4K and 8K ultra-high-definition video, traditional methods require substantial computational resources to process even a single frame, much like observing the night sky with a regular telescope—the more detail, the slower the processing. Second, the challenge of detecting advanced forgery attacks. Modern forgery techniques, such as deepfakes, can generate extremely realistic fake faces, much like how high-quality counterfeits are difficult to distinguish using simple methods. Third, the disconnect between the identification process and anti-counterfeiting detection. Traditional solutions prioritize identification before anti-counterfeiting, similar to verifying identity before checking authenticity, resulting in low efficiency and a high risk of errors.

[0059] This solution introduces direct compression domain processing technology, much like how X-ray scans can determine the contents of a package without opening it, significantly improving processing efficiency. Simultaneously, a multimodal feature fusion mechanism ensures the system can gather evidence from multiple information sources for comprehensive judgment, much like a judge needs multiple pieces of evidence to support their case.

[0060] Figure 1 This is a flowchart comparing the traditional solution and the proposed solution in step S1. Figure 2 This is a flowchart of the present invention. Figure 1 and Figure 2 As shown. A face recognition method based on multimodal feature fusion includes the following steps:

[0061] Step S1, Data Reception and Parsing: Obtain basic video parameters.

[0062] This step transforms the raw image data from the HEVC compressed stream into a standardized, lightweight feature representation. Through shallow decoding, a three-channel feature map is generated, providing structured input for subsequent analysis. This step not only significantly reduces the computational load but also preserves the video's key feature information, making it particularly suitable for real-time intelligent video analysis scenarios.

[0063] HEVC (High Efficiency Video Coding) is a high-efficiency video coding standard widely used in ultra-high-definition (UHD) video. Its bitstream consists of a multi-layer structure, including the Network Abstraction Layer (NAL) and the Video Coding Layer (VCL).

[0064] Step S101: Data reception and parsing.

[0065] Receive HEVC compressed streams (such as surveillance video), parse the core parameter set, and read only the sequence parameter set (SPS) and image parameter set (PPS).

[0066] Parse the sequence parameter set: Extract basic video information, including video resolution, coding tree unit (CTU) size, and chroma sampling format (e.g., 4:2:0). Calculate the feature image size: Determine the width and height of the intra-frame transform block (ITB) feature image based on the video resolution and CTU size. For example, if the video resolution is 1920×1080 and the CTU size is 64×64, then the horizontal number of CTUs is 1920 / 64=30, and the vertical number is 1080 / 64≈17 (rounded down). The feature image size is 30×17 pixels, with each CTU corresponding to one pixel position.

[0067] Parse the image parameter set: Extract encoding configuration parameters, including quantization parameters (QP), entropy coding mode (CABAC), and maximum depth of transform unit (TU), to guide the subsequent metadata parsing logic.

[0068] The sequence parameter set and image parameter set are parsed, and only key fields in the header of the data packet format (NAL) unit are read. Entropy decoding is then performed, skipping the complete decoding of pixel data. Entropy decoding accounts for 20% of the total decoding time, resulting in a significant efficiency improvement. This process is implemented using HEVC standard reference software (such as HM16.7).

[0069] In the HEVC (High-Efficiency Video Coding) standard's bitstream structure, Network Abstraction Layer (NAL) units use specific start codes (such as 0x00000001) as separators to encapsulate and transmit compressed data. This solution optimizes the bitstream parsing process, prioritizing data extraction from Non-Video Coding Layer (NCL) NAL units. By avoiding complex inverse quantization and inverse transform core video decoding modules, computational overhead is significantly reduced.

[0070] Specifically, the non-video coding layer (NAL) unit contains global configuration information such as the Sequence Parameter Set (SPS) and Picture Parameter Set (PPS). This data is independent of the encoded content of a specific video frame, and the parsing process does not rely on pixel-level reconstruction. For example, this scheme directly extracts the size of the coding tree unit from the sequence parameter set (typically 64×64 pixels, but can also be 32×32 or 16×16 depending on the coding configuration). This parameter serves as the core basis for subsequent metadata grid partitioning—by recursively dividing the video frame according to the CTU size, the basic unit for feature extraction can be determined (e.g., each CTU corresponds to a metadata sampling point), laying a structured foundation for compressed domain feature analysis (such as intra-frame prediction mode and transform unit size).

[0071] This step, which extracts non-video coding layer data (such as SPS), only requires the basic operation of entropy decoding, accounting for less than 20% of the total decoding time. It avoids the redundant link of "full decoding → pixel reconstruction → feature analysis" in traditional solutions, and realizes the direct mapping "from bitstream to features", providing a feasible guarantee for real-time processing of ultra-high-definition video (4K / 8K).

[0072] Step S102: Metadata extraction, performing shallow decoding on the entropy decoding output of the HEVC compressed stream.

[0073] During shallow decoding, the header information of each coding tree unit is shallowly decoded, skipping residual data and decoding tree traversal, and directly extracting the following metadata:

[0074] Shallow decoding in Intra-Predictive Mode (IPM):

[0075] Intra-prediction mode (IPM) reflects the spatial prediction relationships of image content. In HEVC encoding, each image patch selects a prediction mode to reduce redundancy. Real faces have specific symmetry and texture features, and their intra-prediction mode distribution exhibits regularity; while fake faces, due to being captured or re-encoded from a screen, often have a chaotic and disordered intra-prediction mode distribution. This scheme maps 35 intra-prediction modes to a grayscale range of 0-255 to generate an intra-prediction mode (IPM) feature map.

[0076] 1. Since coding tree units in HEVC can be recursively divided into prediction units (PUs), all coding tree units in the current frame are traversed to locate the prediction unit level;

[0077] 2. For intra-coded prediction units, read the encoding index of the intra-pred_mode from its header information (intra_pred_mode field) to obtain the prediction direction encoding value; the prediction direction encoding value ranges from 0 to 34, corresponding to 35 modes in different directions;

[0078] 3. Normalization mapping: The predicted direction encoding values ​​of the 35 modes are linearly mapped to a grayscale range of 0~255. ;

[0079] Among them, I IPM It is the grayscale mapping value of the intra-pred mode; intra_pred_mode is the encoding index of the intra-pred mode, with a value range of 0~34 (the HEVC encoding standard defines 35 intra-prediction direction modes, with the index starting from 0 to correspond to different directions). It is a floor function; 35 is the total number of modes (the index range is 0~34, a total of 35 values); 255 is the maximum value of 8-bit grayscale (corresponding to the grayscale range of a single-channel image).

[0080] This mapping mechanism provides an intuitive quantitative basis for distinguishing between real and fake faces by leveraging the differences in the concentration and dispersion of grayscale distribution.

[0081] I. Real Human Faces: Ordered IPM mapping results in concentrated grayscale values, reflecting the symmetry of biological structures. Due to the symmetry of physiological structures (e.g., horizontal alignment of the eyes and vertical orientation of the nasolabial folds), real human faces exhibit a regular distribution when selecting intra-prediction modes (IPM) in HEVC encoding—horizontal (mode 10), vertical (mode 26), and adjacent directional modes (e.g., modes 9, 11, 25, 27) are used significantly more frequently than other directions. At this point, among the 35 IPM modes (coding indices 0-34), effective modes are concentrated in a narrow range (e.g., the proportion of horizontal / vertical modes in real human faces exceeds 60%). After mapping, the corresponding grayscale values ​​form a single-peaked histogram within the range of 0-255 (e.g., horizontal mode 10 is mapped to I...). IPM ≈73, Vertical mode 26 is mapped to I IPM ≈189, concentrated in the 70~190 range). This "concentrated grayscale" feature directly reflects the natural spatial symmetry of the human face, providing structural evidence for real human faces.

[0082] II. Forged Faces: Disorganized IPM mapping results in scattered grayscale, exposing unnatural distortions. Forged faces (such as screen captures or printed photos) suffer from perspective distortion (side-view shots), moiré interference (electronic screen displays), or material texture distortion (printed paper texture). Consequently, the IPM mode selection becomes irregular: among the 35 modes, the probability of each directional mode (0-34) tends to be random, with no clear dominant direction. After mapping using the same formula, the grayscale values ​​are evenly distributed within the range of 0-255 (the histogram is flat, with no obvious peaks). For example, mode 0 (planar prediction) and mode 34 (diagonal prediction) may alternate, causing the grayscale values ​​to be scattered across the entire range of 0-255. This "scattered grayscale" characteristic exposes the unnatural coding patterns of forged faces, showing a significant difference from the concentrated distribution of real faces.

[0083] Shallow decoding of Transform Unit Size (TUS):

[0084] Transform Unit Size (TUS) is a key syntax element in the HEVC coding standard, representing the size of the transform block (TU). Its values ​​range from {4×4, 8×8, 16×16, 32×32, 64×64} (corresponding to powers of 2). The core value of TUS lies in: representing texture complexity: small TUS sizes (e.g., 4×4) correspond to high-frequency detail regions in the image (e.g., edges, areas with dense texture), while large TUS sizes (e.g., 64×64) correspond to smooth regions (e.g., solid-color backgrounds); and guiding quantization and entropy coding: TUS size directly affects the number of basis functions and energy concentration characteristics of the Discrete Cosine Transform (DCT), thus affecting compression efficiency.

[0085] Real faces contain rich details, and encoders use smaller transform units (such as 4x4 or 8x8) to preserve these details. In contrast, fake faces, due to multiple compressions and transmissions, suffer significant detail loss, and encoders tend to use larger transform units (such as 16x16 or 32x32). The system can distinguish between real and fake faces by analyzing the distribution of transform unit sizes: fake faces or low-quality videos have larger transform unit sizes.

[0086] 1. Analysis of coding tree unit segmentation structure: For each frame of the HEVC image, it is first divided into multiple coding tree units. Each coding tree unit is recursively divided into coding units (CU), and each coding unit (CU) is further divided into transform units (TU).

[0087] 2. Read the header information of the transformation unit:

[0088] Each transform unit has its own header information. The focus is on extracting the fields related to the exponential encoding, namely the tu_size field, and obtaining the size encoding of the transform unit according to the size mapping rules.

[0089] The `tu_size` field uses exponential encoding to store the size code of the transformation unit, where the size code of the transformation unit = 2. tu _size For example, when tu_size=3, the actual size is 2. 3 =8, therefore, the size code of the transformation unit (TU) is 8×8 pixels.

[0090] 3. The size of the transform unit is encoded as the transform unit size, and the transform unit size is linearly mapped to grayscale values ​​of 0-255: ;

[0091] Among them, I TUSThe grayscale mapping value is the transform unit size; TUS is the transform unit size. Small TUS (e.g., 4×4) is mapped to dark colors (low grayscale), emphasizing details; large TUS (e.g., 32×32) is mapped to bright colors (high grayscale), representing smooth areas. Shallow decoding of the transform unit size (TUS) uses grayscale to distinguish texture details (small TUS → high grayscale) from smooth areas (large TUS → low grayscale), aiding in determining the realism of a face.

[0092] This step achieves efficient conversion from HEVC bitstream to transform unit size feature map through "hierarchical structure parsing → tu_size field extraction → segmented grayscale mapping".

[0093] Shallow decoding of encoded bit length (BN):

[0094] The encoding bit depth is the number of bits required to encode each prediction unit. Recording the compressed bit consumption of each encoding unit indirectly indicates the importance of the region. Real faces have rich textures and complex details, requiring more bits for encoding; fake faces have relatively simple textures, requiring fewer bits. By analyzing the encoding bit depth distribution, the complexity and quality of face images can be evaluated.

[0095] To avoid contour breakage caused by excessively large encoding bit values, an average normalization based on 8×8 blocks is adopted:

[0096] ;

[0097] Among them, BN avg It is the average number of bits encoded within an 8×8 block; BN original is the number of bits in the original coding unit; k is the number of 8×8 blocks.

[0098] Then perform grayscale linear mapping:

[0099] ;

[0100] Among them, I BN It is the grayscale mapping value of the encoded bit depth, BN min and BN max These are the minimum and maximum values ​​of the intra-frame coding bit depth, respectively. This method smooths the grayscale distribution and uses grayscale to reflect the complexity of the region (high coding bit depth → high grayscale, complex texture; low coding bit depth → low grayscale, simple region).

[0101] Step S103, Image Construction and Fusion: Generate a three-channel feature map.

[0102] Metadata of each coding tree unit (I IPM I TUS I BN Mapped to 1 pixel, generating a three-channel feature map I. ITB=(I IPM I TUS I BN ).

[0103] Three-channel fusion (I IPM I TUS I BN A pseudo-color image (similar to RGB) carries structural information rather than pixel color.

[0104] This step outputs a three-channel feature map. Although it does not contain pixel information, it retains the structural features in HEVC encoding, solving the high computational problem of traditional full decoding and providing rich clues and lightweight input for subsequent steps.

[0105] Figure 1 This is a flowchart comparing the traditional and proposed solutions in step S1. The traditional solution's flow is: encoding → full decoding → pixel-level analysis, requiring the reconstruction of all pixels before performing complex AI tasks. The proposed solution's flow is: encoding → entropy decoding → feature-level analysis. The H.265 entropy decoder only performs the "entropy decoding" step, directly extracting features from the entropy-decoded data (without reconstructing all pixels), significantly reducing computation (entropy decoding is several times faster than full decoding), while retaining sufficient information to support lightweight AI analysis. For HEVC compressed streams, the system employs a partial decoding strategy. Traditional methods require complete decompression of the entire video stream, like opening a package to check its contents, while this invention only performs "shallow decoding," extracting only metadata information without completely decompressing the image content. Specifically, the system parses the HEVC NAL unit header, extracting three key parameters: Intra-Prediction Mode (IPM), Transform Unit Size (TUS), and Bit Numerical Number (BN), retaining key HEVC encoding features. The computational cost is only 20% of full decoding, providing input for subsequent analysis.

[0106] Step S2, rapid face detection.

[0107] Using the three-channel feature map output from step S1 (mapped from the intra-prediction mode, transform unit size, and encoding bit depth data extracted from the HEVC compressed stream, with a size of 128×128×3) as input, the system simultaneously completes face existence verification, precise localization (face bounding box, marking face position), key point annotation (5 key points: coordinates of eye center, nose tip, and left and right corners of mouth), and gaze point prediction (the point where the gaze lands on the screen) through a two-level process of shallow filtering and real-time multi-task detection, providing structured output for subsequent dynamic tracking (step S3).

[0108] Step S201, shallow filtering: Select a shallow convolutional neural network (CNN) to determine whether there is a face in the three-channel feature map, quickly filter out the "face" region, and reduce subsequent computational overhead.

[0109] The reason for choosing a shallow CNN instead of a deep network is that the three-channel feature map is already highly abstract, and a shallow network is sufficient to capture key differences; shallow networks have a small computational cost, which is suitable for the low latency requirements of real-time monitoring.

[0110] The CNN architecture uses 2 convolutional layers and 2 fully connected layers. Its structure is as follows: Input layer: receives a 128×128×3 three-channel feature map; Convolutional layer 1: 5×5 convolutional kernels (100), stride 1, Same padding, ReLU activation; Convolutional layer 2: 3×3 convolutional kernels (100), stride 1, Same padding, ReLU activation; Fully connected layer 1: 500 neurons, ReLU activation; Fully connected layer 2: 1 neuron, SigmoID activation (outputs face probability; if the face probability is > 0.5, it is determined to be a face).

[0111] Training data for shallow CNNs: "Face" samples are trained using the LFW dataset (Labeled Faces in the Wild, containing diverse lighting, poses, and expressions in natural scenes); "Non-Face" samples are trained using the LSVRC dataset (LargeScale Visual Recognition Challenge, containing thousands of non-face targets); and the model learns to distinguish between "faces" and "background" (such as documents, walls, and clutter) through "positive and negative sample comparison".

[0112] Step S202, Real-time Object Detection and Multi-task Output: For a three-channel feature map with a face, a real-time object detection model (YOLOv5) is selected to simultaneously complete face localization, key point annotation and gaze point prediction, solving the problem of detecting small targets and occluded faces in multi-person scenes.

[0113] 1. Model Selection: YOLOv5 was chosen as the real-time object detection model, with CSPDarknet as its backbone network. It utilizes feature reuse between residual blocks (cross-layer skip connections, reusing low-level features from previous layers such as edges or textures, and combining them with high-level semantic features from the current layer such as facial structure) and multi-scale feature decoupling (separating large-scale semantic features from small-scale positional features) to output a three-scale feature map.

[0114] 160×160 semantic layer: Use global semantic alignment technology for re-identification (contrastive learning to narrow the distance between cross-view features of the same face) to improve the discrimination of "who is the face" (such as distinguishing between a face and a human-shaped doll).

[0115] An 80x80 transition layer connects semantic and positional information;

[0116] A 40×40 positioning layer: using a re-identification local feature matching mechanism (constraining the similarity of eye or mouth features) to enhance the positioning accuracy of "where is the face" (e.g., a small face in the distance, or a side face that is obscured).

[0117] 2. Using a bidirectional multi-scale feature fusion architecture of Feature Pyramid Network (FPN) and Path Aggregation Network (PAN), the hierarchical transfer of semantic and location information is realized, and the multi-scale feature map with fused semantic and location information is output: a large-scale feature map of 160×160 and a small-scale feature map of 40×40.

[0118] Feature Pyramid Network (FPN) achieves multi-scale feature map fusion and optimization through a top-down semantic information transmission mechanism. Its core process involves: using a 160×160 large-scale feature map (semantic layer) as a base, upsampling to 80×80 and 40×40, and then element-wise superimposing it with the original small-to-medium-scale feature maps. This injects semantic discriminative capabilities into the small-scale maps (such as judging the "facial attributes" of an occluded profile). After fusion, the performance of small-scale feature maps (such as the 40×40 localization layer) is significantly improved. Features that originally only reflected positional details such as edges and contours gain the ability to discriminate the essential attributes of the target by combining with the semantic information of the large-scale features. For example, when faced with local details such as a profile obscured by a hand, the fused features can retain the localization information of key areas such as the eyes and mouth, and can also determine whether the area belongs to a real facial structure rather than background clutter or other interference based on semantic capabilities, thus effectively enhancing the detection accuracy of small targets and occluded targets.

[0119] The Path Aggregation Network (PAN) enhances the localization accuracy of multi-scale feature maps through a bottom-up mechanism that transfers positional details. Its core function is to downsample and condense a 40×40 small-scale feature map (localization layer) to 80×80 and 160×160, then overlay it with the original large-scale feature map, injecting detailed localization capabilities (such as the precise coordinates of a distant face) into the large-scale map. Through this fusion, the large-scale feature map (e.g., 160×160) gains the advantage of detailed localization from small-scale features on top of its original overall semantic discrimination capability, forming a dual verification mechanism of semantics and detail. For example, when dealing with targets like humanoid dolls that have a human-like facial outline but lack realistic facial details, the collaborative judgment of semantic attributes and detailed localization can more accurately distinguish them from real faces, significantly improving the reliability of target detection in complex scenes.

[0120] The Feature Pyramid Network (FPN) and Path Aggregation Network (PAN) are bidirectionally fused to form a bidirectional transmission architecture:

[0121] Large-scale feature map of 160×160: details are supplemented through Feature Pyramid Network (FPN) and semantic enhancement is achieved through Path Aggregation Network (PAN), ultimately enhancing the semantic discrimination of human faces and non-human faces (such as distinguishing between "human face" and "human doll").

[0122] 40×40 small-scale feature map: semantic guidance is obtained through path aggregation network (PAN) and overall localization is obtained through feature pyramid network (FPN), which ultimately improves the localization accuracy of small targets and occluded faces (such as small faces in the distance and side faces covered by hands).

[0123] 3. A decoupled head is adopted, with independent branches handling 5 tasks, solving the problem of "multi-task interference": The decoupled head adopts a multi-branch independent convolutional architecture.

[0124] Bounding box regression branch: Optimize face location (considering overlap, center distance, and aspect ratio); use CIoU Loss to consider intersection-over-union ratio, center distance, and aspect ratio, formula: Among them, L IOU ρ(b,b) is the intersection-union ratio loss between the predicted and ground truth bounding boxes, reflecting the degree of overlap between the two boxes; gt () represents the predicted bounding box center point b and the true bounding box center point b. gt The Euclidean distance is used to constrain the center point position deviation; c is the diagonal length of the smallest rectangle that simultaneously encloses the predicted and ground truth bounding boxes, serving as a normalization reference for the distance; α is the center point adjustment parameter; v is the aspect ratio adjustment parameter; α and v, compared to the traditional IOU loss, can more accurately optimize the bounding box position. The bounding box regression branch generates the face bounding box, accurately predicting the coordinates of the top-left and bottom-right corners of the face. The input feature map size of this branch is consistent with other branches, at 160×160, 80×80, or 40×40. Its internal network structure consists of two 3×3 standard convolutional layers connected in series, with the number of channels compressed from 256 to 128, and finally connected to a 1×1 convolutional layer for channel dimensionality reduction. This branch ultimately outputs a 4-dimensional vector, corresponding to the center point coordinates x and y of the bounding box, and the width and height w and h, respectively. In terms of activation functions, this branch does not set any additional non-linear activations, directly outputting the offset used for regression prediction.

[0125] Classification branch: Determine "face / non-face". Focal Loss is used to address imbalanced samples and focus on small, hard-to-classify targets and occluded faces. Formula: Among them, L Cls It is the classification loss, α t For class weights (e.g., assigning different weights to faces and non-faces to balance task importance), p tThis branch represents the model's predicted probability that the current sample belongs to the positive class (face). γ is the focusing parameter, which addresses the imbalanced sample size problem and enhances classification robustness by reducing the weight of easily classified samples and increasing the weight of difficult-to-classify samples (such as small target faces in multi-person scenes or occluded faces). The input feature map size of this branch is consistent with other branches, at 160×160, 80×80, or 40×40. Its internal network structure includes two 3×3 convolutional layers (the number of channels is compressed from 256 to 64) and one 1×1 convolutional layer. This classification branch outputs a 1-dimensional scalar, representing the probability that the current candidate box belongs to "face," and a sigmoid activation function is applied at the end to achieve a normalized probability output for binary classification.

[0126] Confidence branch: Determines whether the bounding box contains a face; the confidence loss is L. Conf The input feature map size for this branch remains consistent with other branches, at 160×160, 80×80, or 40×40. Its internal network structure includes two 3×3 convolutional layers (compressing the number of channels from 256 to 64) and one 1×1 convolutional layer. This classification branch outputs a 1-dimensional scalar representing the confidence score of the target within the bounding box, and a sigmoid activation function is applied at the end to achieve a normalized probability output for binary classification.

[0127] Keypoint Regression Branch: By learning the geometric structure of the face (e.g., eyes horizontally aligned, nose tip centered between the eyes), it simultaneously outputs the coordinates of five keypoints (center of the eyes, nose tip, and left and right corners of the mouth). The keypoint regression loss is L. kpt The formula is defined as:

[0128] ; where N kpt =5, indicating 5 key points, (x kpt,pred i, y kpt,pred (x,i) represents the coordinates of the i-th keypoint predicted by the model; (x,i) represents the coordinates of the i-th keypoint predicted by the model. kpt,target i, y kpt,target Let ,i) represent the true coordinates of the i-th keypoint. The keypoint regression branch labels the keypoints of the five faces. This branch is enhanced in depth, containing three 3×3 convolutional layers. The number of channels is compressed layer by layer, from 256 to 128 and finally to 64, before being output through a 1×1 convolutional layer. This branch ultimately outputs a 10-dimensional vector, representing the normalized x and y coordinates of five predefined facial keypoints (e.g., the center of the eyes, the tip of the nose, and the corners of the mouth). Because its task is to regress continuous coordinate values, this branch does not use a specific activation function, but rather a linear output mode to ensure the unbiasedness of the coordinate prediction.

[0129] The gaze point regression branch employs a lightweight design using 3×3 depthwise separable convolutions ×2 to output 2D gaze point coordinates (x, y), achieving end-to-end regression of the 2D gaze point coordinates. The gaze point regression loss is L.Gaze Its weight is 20, which means that "a small mistake will cause the overall loss to increase sharply", forcing the model to prioritize optimizing this part;

[0130] ; Where, N gaze It is the number of samples involved in the calculation; (x gaze,pred,i y gaze,pred,i (x) represents the coordinates of the line-of-sight point predicted by the model for the i-th sample; gaze,target,i y gaze,target,i `i` represents the true gaze point coordinates of the i-th sample; the gaze point regression branch predicts the gaze point. This branch undergoes extreme lightweight modifications while maintaining accuracy, using only one 3×3 depthwise separable convolution layer, reducing the number of channels from 256 to 128, followed by a 1×1 standard convolution layer. This special structure significantly reduces the number of parameters and computational overhead. The branch ultimately outputs a 2D vector, representing the x and y coordinates of the gaze point, also using a linear output method without activation functions to facilitate subsequent high-precision smoothing processing.

[0131] L Cls It's the classification loss, which measures the accuracy of determining whether "this region is a face or not." Misclassification results in a large loss; the classification loss L... Cls Solve the problem of positive and negative sample imbalance in multi-person scenarios with small targets and occluded faces;

[0132] L Conf This is the confidence loss, which is determined by whether the model's "confidence" in its predictions is reasonable. For example, if the model is overly confident even when it has not captured a face, the loss will be significant.

[0133] L Gaze It is the gaze point regression loss, which specifically looks at the accuracy of the prediction of "gaze direction + 5 key points". Its weight is 20, which means that "if one point is wrong, the overall loss will increase sharply", forcing the model to prioritize the optimization of this part.

[0134] Construct a multi-task joint optimization objective L Total Its formula is:

[0135] ;

[0136] Among them, L IOU It is the bounding box intersection-union loss. Where, IOU is the intersection-union ratio of the predicted bounding box and the ground truth bounding box, reflecting the degree of overlap between the two boxes; ρ(b,b) gt () represents the predicted bounding box center point b and the true bounding box center point b. gtThe Euclidean distance is used to constrain the center point position deviation; c is the diagonal length of the smallest rectangle that simultaneously encloses the predicted bounding box and the ground truth bounding box, serving as a normalization reference for the distance; α is the center point adjustment parameter; v is the aspect ratio adjustment parameter; α and v, compared to the traditional IOU loss, can more accurately optimize the bounding box position; W Gaze The optimal weight (value 20) in the experiment allows the model to focus on gaze point prediction, eliminating interference from blinking and head shaking; W kpt These are the weights for keypoint regression (requires experimental tuning).

[0137] Through the collaborative design and weight optimization of the multi-task loss mentioned above, efficient collaboration of multiple tasks such as bounding box localization, face classification, confidence calibration, and gaze point regression can be achieved, solving the problems of target detection and gaze tracking in multi-person scenarios.

[0138] This step employs a decoupling mechanism. Traditional object detection models often combine tasks such as bounding box prediction, category determination, and keypoint or gaze point regression into a single branch, much like a single person handling multiple tasks. This can easily lead to decreased accuracy due to conflicting task objectives. This solution adopts a "division of labor" approach, breaking down different tasks into independent branches: one branch focuses on drawing face bounding boxes (localization), another branch accurately labels the category (face or non-face), and a third branch calculates gaze point coordinates (regression). Each branch independently optimizes its objective, avoiding interference between tasks. This is akin to a professional team each performing its specific duties, ultimately achieving more accurate detection and prediction results than a single person handling everything.

[0139] This step employs an anchor-free bounding box approach. Existing object detection models (such as Faster R-CNN) require a pre-defined set of fixed-size anchor boxes (e.g., small, medium, and large sizes) when dealing with faces of varying distances and sizes. This is akin to using different-sized molds to fit clothing, attempting to match targets of different body types. While effective when target sizes are relatively fixed, this approach struggles to cover all sizes in multi-person scenarios where faces appear larger in the foreground and smaller in the background (e.g., large faces in the foreground and small faces in the background). This leads to issues like missed detections when using small boxes for large faces and redundant detections when using large boxes for small faces. Furthermore, manually designing anchor boxes requires extensive parameter tuning, resulting in poor flexibility. This solution utilizes an anchor-free bounding box design, allowing the model to autonomously learn the size and position of the target bounding box. The model directly predicts the center point coordinates and aspect ratio of the bounding box from the feature map, without relying on pre-defined anchor boxes. This design is particularly suitable for multi-person scenarios: faces at different distances have significantly different sizes. The model can dynamically adjust the box size based on the semantics of the feature map, using large boxes for large faces in the foreground and small boxes for small faces in the background, avoiding pre-defined biases and improving robustness for detecting targets of different scales.

[0140] This step outputs:

[0141] Face bounding box: Marks the position of the face in the image, providing a "target region" for subsequent dynamic tracking (step S3);

[0142] Five key points: assisting in identity recognition (such as focusing on facial features when extracting features in the FastReID model) and posture analysis (such as determining the head turning angle);

[0143] Eye gaze coordinates: quantify the direction of human eye gaze, providing a calculable basis for "eye tracking" and "attention analysis".

[0144] This solution employs gaze point prediction, essentially transforming the "attention" of the human eye as it gazes at the screen into quantifiable two-dimensional coordinates. Similar to using arrows to indicate the direction each person is looking in a group photo, this solution learns the correlation between facial features (especially the eye area) and gaze direction through a model, directly regressing the coordinates of the gaze's landing point on the screen from the facial image.

[0145] Step S3: Dynamic tracking and tracking ID correction.

[0146] Input: Face bounding box, gaze point coordinates, and face image (112×112 pixels) output from step S2.

[0147] Output: Tracking ID (continuous numbering for the same person), cross-frame identity consistency result (determining whether faces in different frames belong to the same person).

[0148] Step S301: Extract facial features.

[0149] The FastReID model is used to extract feature vectors that reflect individual uniqueness from face images and assign a unique tracking ID to each feature vector.

[0150] Model Architecture: The backbone network of the FastReID model is ResNet50. The output is compressed into a feature vector by global average pooling (GAP), and then normalized by batch normalization (BN) layers to transform the input face image into a 128-dimensional feature vector. This serves as an identity identifier for subsequent matching.

[0151] Training objective: Through comparative learning, the angle between the feature vectors of individuals with the same identity is less than 30 degrees, while the angle between the feature vectors of individuals with different identities is greater than 60 degrees, thereby achieving identity discrimination.

[0152] Step 302: Three-step tracking and matching: Complete face tracking and matching by motion matching, feature matching, and intersection-union comparison to determine whether it is the same person from different dimensions.

[0153] Step 1: Motion Matching: By observing the facial movement trajectory, determine whether it is the same person, using Mahalanobis distance d. M This measures the positional similarity between the current frame's detection bounding box and the previous frame's tracking bounding box. The formula is:

[0154] Where z is the location of the detected face bounding box in the current frame; x is the location of the face bounding box predicted from the previous frame; H is a matrix used to convert the predicted location into a form comparable to the current detection location; P is the covariance matrix; and T is the matrix transpose operator. M The smaller the value, the more similar the movement trajectory of the current face and the predicted face are, and the more likely they are the same person.

[0155] Judgment condition: when d M If the value is less than 2.5, determine that the two boxes correspond to the same person; otherwise, proceed to the next step.

[0156] Step 2: Feature matching: By comparing the 128-dimensional feature vectors of the face Based on the similarity of facial feature vectors, the cosine distance d is used. C The system measures the consistency of identity features between the current frame and historical features to determine if they belong to the same person. The formula is:

[0157] Among them, r current It is the feature vector corresponding to the face in the current frame; r prev It is the feature vector of the face at the corresponding location predicted in the previous frame; " represents the dot product of two vectors; It is the module length.

[0158] Judgment condition: when d C If the value is less than 0.3, determine that the two boxes correspond to the same person; otherwise, proceed to the next step.

[0159] Step 3: Intersection over Union (IOU) for omissions: If the first two steps fail to find the same person, try matching again using the overlap of the two bounding boxes to avoid missing the same person. The formula for calculating the Intersection over Union (IOU) is:

[0160] The intersection area is the area of ​​the overlapping portion of the two bounding boxes of the faces to be matched; the area of ​​box A and the area of ​​box B are the areas of the two bounding boxes themselves; min is the smaller of the two areas.

[0161] Judgment condition: If the Intersection over Union (IOU) ratio is greater than 0.5, it means that the two face bounding boxes to be matched correspond to the same person; otherwise, the face corresponding to the bounding box is marked as a new tracking ID (marked as a new object that has not appeared before).

[0162] In this step, the 128-dimensional feature vector extracted by FastReID provides an identity for feature matching. This, along with motion matching (trajectory continuity) and intersection-union (IU) error correction (spatial overlap), forms a three-level verification to ensure the stability of the tracking ID in occlusion and deformation scenarios. For example, when the target reappears after a brief occlusion, feature matching recovers the tracking ID through vector similarity, motion matching assists in localization through trajectory prediction, and IU error correction handles extreme overlap situations.

[0163] Step S4: Optimize and output the line of sight.

[0164] Using the tracking ID output in step S3 and the coordinates of the unsmoothed line-of-sight point output in step S2 as input, Kalman filtering is used to smooth out transient interference and output a JSON structured result (including fields such as tracking ID, optimized line-of-sight point, and tracking status).

[0165] Step S401: Smooth the line of sight.

[0166] The viewpoint coordinates (x, y) output in step S2 are based on the YOLOv5 single model (with W). Gaze Although the end-to-end prediction results with a high-weight loss of 20 have improved accuracy through multi-task joint optimization (boundary box localization + gaze point regression) (the error in the face region has been reduced from 26.28cm to 15.77cm), they are still affected by transient disturbances (such as blinking or slight head shaking), resulting in high-frequency small fluctuations in coordinates.

[0167] To address this issue, this step employs a Kalman filter to perform temporal smoothing on the gaze point coordinates across five consecutive frames. The Kalman filter is a recursive optimal estimation algorithm that uses a "prediction-update" loop to fuse current measurements with historical states, outputting a more stable estimate. Its principle is similar to "predicting the current position using past trends, and then correcting the prediction using current measurements," making it suitable for handling noisy dynamic sequences.

[0168] 1. State Definition: Treating the coordinates (x, y) of the line-of-sight point as a two-dimensional dynamic system, the state vector is defined as follows: ; where x k y k Let K be the coordinates of the viewpoint in the k-th frame. , The rate of change (velocity) of the coordinates.

[0169] 2. Prediction steps.

[0170] Based on the state x of the (k-1)th frame k-1 Predict the state of the k-th frame. and state covariance matrix :

[0171] ; ;in,

[0172] F is the state transition matrix (if the coordinates change at a constant velocity, then...). );

[0173] : The state covariance matrix of the (k-1)th frame, representing the uncertainty of the state estimation;

[0174] Q: The process noise covariance matrix represents the error in the model prediction, such as... ).

[0175] 3. Update steps.

[0176] Using the raw measurement values ​​of the k-th frame Update the predicted state and calculate the Kalman gain K. k Updated status x k With covariance matrix P k :

[0177] Kalman gain formula: Among them, K k It is the Kalman gain, which balances the weights of prediction and measurement; The predicted state covariance matrix describes the uncertainty of the "predicted state"; H is the observation matrix, which maps the state vector to the measurement space, such as... ); H T R is the transpose of the observation matrix H; R is the measurement noise covariance matrix, representing the error of the original predicted values, such as... .

[0178] State update formula: ; where x k It is the updated state vector at time k; It is the predicted state vector at time k; z k It is the measurement vector of the k-th frame, such as ; It is the measurement residual, which reflects the difference between the "actual measured value" and the "predicted measured value".

[0179] Covariance update formula: Among them, P k I is the updated state covariance matrix at time k; I is the identity matrix.

[0180] 4. Output the smoothed coordinates.

[0181] After five consecutive frames of prediction and update, the smoothed back view point coordinates for the fifth frame are output. This will be the output of this step.

[0182] In this step, Kalman filtering is used to denoise the coordinates of the gaze point in 5 consecutive frames, which increases the denoising capability: eliminating the effects of blinking (instantaneous coordinate shift) and slight head shaking (high-frequency small fluctuation), making the gaze point trajectory smoother.

[0183] The line-of-sight smoothing process employs a Kalman filter algorithm to eliminate transient interference. Its core principle is to treat the line-of-sight coordinates as a two-dimensional dynamic system containing position and velocity, optimizing the results through prediction and updates. In the prediction phase, the current position is estimated based on the previous frame's state. In the update phase, the prediction deviation is corrected using the current measurement value. The Kalman gain automatically balances the weights of prediction and measurement, similar to combining historical routes with real-time positioning correction in navigation. Specifically, a state vector is defined containing the line-of-sight coordinates and movement velocity. During prediction, uniform motion is assumed to calculate the new position. During updates, the difference between the measured and predicted coordinates is compared, and the fusion ratio is adjusted using the gain. After five consecutive frame cycles, smoothed coordinates are output, effectively eliminating high-frequency fluctuations caused by blinking and head shaking.

[0184] Step S402: JSON output.

[0185] It contains the tracking ID, line-of-sight coordinates (x, y), and tracking status (such as "continuous tracking" or "tracking ID switching"), which can be called or read.

[0186] Output field definitions: The JSON result contains the following fields:

[0187] IDentity_ID: The type is string, which is the tracking ID output in step 3. It serves as a unique identifier for the tracked object. The example value is ID_001.

[0188] gaze_coordinate: of type array, is the smoothed gaze point coordinate, containing x and y, in pixels, with an example value of [320, 240];

[0189] tracking_state: of type string, used to describe the relationship between the current tracking ID and the previous frame. Possible states include continuous tracking, tracking ID switching, new tracking ID, and lost contact. Example values ​​are continuous tracking / tracking ID switching / new tracking ID / lost contact.

[0190] timestamp: of type string, is the timestamp of the current frame, in the format YYYY-MM-DD HHMMSSSSS, example value is 2026-04-01 090005123;

[0191] frame_number: of type int, is the sequence number of the current frame, starting from 1 and incrementing, with an example value of 150;

[0192] confidence: type float, is the confidence level of the line-of-sight prediction, and the value ranges from 0 to 1. The larger the value, the higher the confidence level. The example value is 0.92.

[0193] The JSON structured output encapsulates the optimized results in an easily readable format. Output fields include: Identity Tracking ID (the continuous tracking number assigned in step S3, uniquely identifying the target object); gaze point coordinates (smoothed horizontal and vertical pixel values, quantifying the point of human eye gaze); tracking status categorized into four types: continuous tracking (tracking ID matches the previous frame and matches for three consecutive frames); tracking ID switching (tracking ID changes but features are similar, indicating the same person); new tracking ID (first target detection); and lost contact (no detection for thirty consecutive frames, releasing resources); a timestamp recording the current frame time accurate to milliseconds; an incrementally increasing frame number; and confidence level indicating the reliability of the gaze point prediction. These fields collaboratively present comprehensive information on identity, gaze, and status, directly supporting downstream applications such as attention analysis.

[0194] Detailed tracking status description:

[0195] Continuous tracking: When the tracking ID of the target object in the current frame is completely consistent with that in the previous frame, and within a time window of 3 consecutive frames, the confirmation logic of dynamically tracking the same person in step S3 is satisfied, and the target is not obscured by other objects or misjudged due to visual confusion, then the tracking status and original tracking ID of the target are maintained.

[0196] Tracking ID switching: If the tracking ID of the target in the current frame is inconsistent with that in the previous frame, but the cosine distance is less than 0.3 through similarity matching of target features, the tracking ID of the current target is determined, and then the tracking ID correction operation is performed to ensure the continuity of the target identity in time.

[0197] New Tracking ID Assignment: When a target object is detected for the first time in a video sequence, a unique tracking ID is assigned to the target as an identification, and the continuous tracking process for the new target is initiated.

[0198] Loss of contact determination: If the target object is not detected by the detection module for 30 consecutive frames (corresponding to the parameter configuration of max_age=30 in process step S3, i.e. the maximum number of undetected frames allowed by the tracker), the system determines that the target is in a lost contact state and will trigger the tracking termination logic (such as releasing the resource occupation of the target tracking ID, stopping the tracking calculation of the target, etc.).

[0199] This solution overcomes the shortcomings of existing technologies through an integrated design of compressed domain lightweight features, multimodal fusion, and lightweight deep learning. It integrates compressed domain features (HEVC metadata mapping to a three-channel image to distinguish the symmetry and texture complexity of real and fake faces), depth features (FastReID extracting a 128-dimensional vector to capture iris texture), and spatiotemporal features (Kalman filtering to suppress noise), balancing robustness to lighting and pose with detailed representation, achieving a 4K video detection accuracy >99.5%. The collaborative anti-spoofing approach using compressed domain, depth, and spatiotemporal multimodal features resists 2D, 3D, and projection attacks, achieving a liveness detection accuracy >98.7%. It employs compressed domain processing without full decoding (only entropy decoding accounts for 20%), shallow CNN filtering + YOLOv5 anchor-box detection (30fps@RTX 3090, supports 8K), and Kalman filtering to smooth the viewpoint (reducing error by 40.1%), balancing real-time performance and efficiency with low deployment costs.

[0200] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A face recognition method based on multimodal feature fusion, characterized in that, Includes the following steps: Step S1, Data Reception and Parsing: Obtain basic video parameters; using the HEVC compressed stream as input, extract basic video information by parsing the sequence parameter set and image parameter set, and perform entropy decoding; then, perform shallow decoding on the header information of each coding tree unit to extract three types of metadata: intra-frame prediction mode, transform unit size, and coding bit depth, and map them to grayscale values ​​of 0~255; finally, merge the three types of metadata of each coding tree unit into pixels to generate a three-channel feature map, preserving the HEVC structural features; Step S2, Fast Face Detection: Using a three-channel feature map as input, a shallow convolutional neural network is used to determine whether it contains a face; for the three-channel feature map that detects a face, a real-time object detection model is selected to simultaneously complete face localization, key point annotation and gaze point prediction. Step S3, Dynamic Tracking and Tracking ID Correction: Extract feature vectors that reflect individual uniqueness from the face image and assign a unique tracking ID to the feature vector. Then, complete the face tracking and matching by motion matching, feature matching and intersection-union ratio correction to determine whether it is the same person from different dimensions. Step S4, line-of-sight optimization and output: Using the tracking ID output in step S3 and the unsmoothed line-of-sight coordinates output in step S2 as input, Kalman filtering is used to smooth out transient interference and output a JSON structured result.

2. The face recognition method based on multimodal feature fusion according to claim 1, characterized in that, Step S1 includes: Step S101, Data reception and parsing: Parse the sequence parameter set to extract basic video information, including: video resolution, coding tree unit size, and chroma sampling format; calculate the feature image size, and determine the width and height of the intra-frame transform block feature image based on the video resolution and coding tree unit size; Parse the image parameter set: Extract encoding configuration parameters, including quantization parameters, entropy coding mode, and maximum depth of transform units, to guide the subsequent metadata parsing logic; Step S102, metadata extraction, performing shallow decoding on the entropy decoding output of the HEVC compressed stream: Shallow decoding in intra-prediction mode:

1. Since coding tree units in HEVC can be recursively divided into prediction units, all coding tree units in the current frame are traversed to locate the prediction unit level; 2. For intra-coded prediction units, the coding index of the intra-prediction mode is read from its header information to obtain the prediction direction coding value; the prediction direction coding value ranges from 0 to 34, corresponding to 35 modes in different directions.

3. Normalization mapping: The predicted direction encoding values ​​of the 35 modes are linearly mapped to a grayscale range of 0~255. ; Among them, I IPM It is the grayscale mapping value of the intra-pred mode; intra_pred_mode is the encoding index of the intra-pred mode, with a value range of 0~34; It is a floor function; 35 is the total number of modes; 255 is the maximum value of 8-bit grayscale. Shallow decoding with varying unit size:

1. Analyze the segmentation structure of the coding tree unit: For each frame of the HEVC image, it is first divided into multiple coding tree units. Each coding tree unit is recursively divided into coding units, and each coding unit is further divided into transform units.

2. Read the header information of the transformation unit: Each transform unit has its own header information. The focus is on extracting fields related to the exponential encoding and obtaining the size encoding of the transform unit according to the size mapping rules.

3. The size of the transform unit is encoded as the transform unit size, and the transform unit size is linearly mapped to grayscale values ​​of 0-255: ； Among them, I TUS The grayscale mapping value is the size of the transformation unit; TUS is the size of the transformation unit. Shallow decoding of encoded bits: Average normalization based on 8×8 blocks is adopted: ; Among them, BN avg It is the average number of bits encoded within an 8×8 block; BN original It is the number of bits in the original coding unit; k is the number of 8×8 blocks; Then perform grayscale linear mapping: ; Among them, I BN It is the grayscale mapping value of the encoded bit depth, BN min and BN max These are the minimum and maximum values ​​of the intra-frame coding bit length, respectively; Step S103, Image Construction and Fusion: Generate a three-channel feature map; Metadata of each coding tree unit (I IPM I TUS I BN Mapped to 1 pixel, generating a three-channel feature map I. ITB =(I IPM I TUS I BN ).

3. The face recognition method based on multimodal feature fusion according to claim 2, characterized in that, In step S2, the real-time target detection model outputs three-scale feature maps: a 160×160 semantic layer, an 80×80 transition layer, and a 40×40 localization layer. A bidirectional multi-scale feature fusion architecture using a feature pyramid network and a path aggregation network is used to achieve hierarchical transmission of semantic and location information, outputting multi-scale feature maps that fuse semantic and location information: a 160×160 large-scale feature map and a 40×40 small-scale feature map.

4. The face recognition method based on multimodal feature fusion according to claim 3, characterized in that, In step S2, a decoupled head is used to process five tasks independently: bounding box regression branch, classification branch, confidence branch, key point regression branch, and gaze point regression branch, outputting the face bounding box, five key points, and gaze point coordinates.

5. A face recognition method based on multimodal feature fusion according to claim 1 or 4, characterized in that, In step S3, the FastReID model is used to transform the input face image into a 128-dimensional feature vector. ; Motion matching: By observing the facial movement trajectory, it determines whether they belong to the same person, using Mahalanobis distance d. M It measures the positional similarity between the current frame's detection bounding box and the previous frame's tracking bounding box; the judgment condition is: when d M If the value is less than 2.5, determine that the two boxes correspond to the same person; otherwise, proceed to the next step. Feature matching: By comparing the 128-dimensional feature vectors of the face Based on the similarity of facial feature vectors, the cosine distance d is used. C The system measures the consistency of identity between the current frame and historical features to determine whether they belong to the same person; the judgment condition is: when d C If the value is less than 0.3, determine that the two boxes correspond to the same person; otherwise, proceed to the next step. Intersection over Union (IoU) to fill gaps: If the first two steps of matching do not find the same person, try matching again using the overlap of the two bounding boxes to avoid missing the real same person: Judgment condition: If IoU > 0.5, it means that the two face bounding boxes to be matched correspond to the same person; otherwise, mark the face corresponding to the box as a new tracking ID.

6. The face recognition method based on multimodal feature fusion according to claim 5, characterized in that, In step S4, the JSON structured result includes the tracking ID, line-of-sight coordinates, and tracking status. Tracking status includes: Continuous tracking: When the tracking ID of the target object in the current frame is completely consistent with that in the previous frame, and within a time window of 3 consecutive frames, the confirmation logic of dynamically tracking the same person in step S3 is satisfied, and the target is not obscured by other objects or misjudged due to visual confusion, then the tracking status and original tracking ID of the target are maintained. Tracking ID switching: If the tracking ID of the target in the current frame is inconsistent with that in the previous frame, but the cosine distance is less than 0.3 through similarity matching of target features, the tracking ID of the current target is determined, and then the tracking ID correction operation is performed to ensure the continuity of the target identity in time. New Tracking ID Assignment: When a target object is detected for the first time in a video sequence, a unique tracking ID is assigned to the target as an identification, and the continuous tracking process for the new target is initiated. Loss of contact determination: If the target object is not detected by the detection module for 30 consecutive frames, the system determines that the target is in a lost contact state and will trigger the tracking termination logic.

Images