Face recognition method, system, medium and product based on rk3576

Abstract

A face recognition method, system, medium and product based on RK3576, relating to the field of data recognition, the method comprising: acquiring a current frame image in a video stream, reading an accumulated confidence value; substituting the quality score and the accumulated confidence value into a resource demand calculation formula to calculate a resource scheduling score; when the resource scheduling score is less than a preset computing power activation threshold, calling a CPU unit to calculate geometric displacement data to generate a lightweight feature vector; when the resource scheduling score is greater than or equal to the computing power activation threshold, calling an NPU unit and a CPU unit to extract apparent texture features, geometric structure features and optical flow motion features in parallel to generate a multi-modal feature set; calculating the semantic distance between each modal feature vector to obtain a weighted feature vector; performing vector retrieval comparison based on the lightweight feature vector or the weighted feature vector to obtain a face recognition result at the current time. The application can improve the utilization rate of computing power resources for face recognition in non-cooperative scenarios.

Description

Technical Field

[0001] This application relates to the field of data recognition, and in particular to a face recognition method, system, medium and product based on RK3576. Background Technology

[0002] In edge computing and IoT applications, embedded facial recognition devices need to perform real-time authentication of continuous video streams locally. These devices are typically equipped with heterogeneous processors such as the RK3576, which have specific NPU (Neural Processing Unit) computing power. They need to process non-cooperative facial images that include changes in lighting, pose shifts, and motion blur, under limited power consumption and computing resource constraints.

[0003] In related technologies, edge-based face recognition methods typically employ a linear hierarchical processing approach based on the quality of a single frame of image. This approach calculates an image quality score for each frame of the video stream, considering its sharpness, brightness, and facial angle. When the image quality score is high, the system determines that the frame contains rich details and uses a computationally intensive deep convolutional neural network for full feature extraction. Conversely, when the image quality score is low, the system determines that the frame lacks sufficient information and switches to a less computationally intensive lightweight network for processing or skips the frame altogether. This statistically controls the average load on the device.

[0004] However, for non-cooperative individuals, when they are facing the camera and being continuously tracked by the system, the captured subsequent frames, although of high image quality, contain redundant information, and full feature extraction would waste computing power. When the target is in a side profile, in motion, or occluded state, the image quality score is low, and related technologies will reduce computing power input or discard images due to the low score, resulting in interruption of recognition continuity or missed identity detection, forcing subsequent frames to require several times the computing power for image recapture, causing a waste of overall computing power resources. Summary of the Invention

[0005] This application provides a face recognition method, system, medium, and product based on RK3576, which can improve the utilization rate of computing resources for face recognition in non-cooperative scenarios.

[0006] In a first aspect, this application provides a face recognition method based on RK3576, which includes a CPU unit and an NPU unit. This face recognition method based on RK3576 is applied to a face recognition system. The method includes: acquiring the current frame image from a video stream and reading the cumulative confidence score of the current target object in the current frame image from memory in the previous frame image; substituting the quality score and cumulative confidence score of the current frame image into a resource requirement calculation formula to calculate a resource scheduling score characterizing the necessity of feature extraction for the current frame; and when the resource scheduling score is less than a preset computing power activation threshold, calling the CPU unit to calculate the current frame... The geometric displacement data of the image relative to the previous frame is used to generate a lightweight feature vector. When the resource scheduling score is greater than or equal to the computing power activation threshold, the NPU unit and CPU unit are invoked to extract the appearance texture features, geometric structure features, and optical flow motion features of the current frame image in parallel, generating a multimodal feature set. The semantic distance between the feature vectors of each modality in the multimodal feature set is calculated, and weighting coefficients are generated based on the semantic distance and the gating network. Weighted fusion and dimensionality reduction are performed on each modality feature vector to obtain a weighted feature vector. Vector retrieval and comparison are performed based on the lightweight feature vector or the weighted feature vector to obtain the face recognition result at the current moment.

[0007] In the above embodiments, the face recognition system adopts the above technical solution and dynamically schedules CPU / NPU computing power by combining image quality and historical recognition confidence. This avoids over-computation of high-quality redundant frames and loss processing of low-quality key frames, realizes on-demand allocation of computing resources, and improves the utilization rate of computing resources for face recognition in non-cooperative scenarios.

[0008] In conjunction with some embodiments of the first aspect, in some embodiments, after the step of performing vector retrieval and comparison based on lightweight feature vectors or weighted feature vectors to obtain the face recognition result at the current moment, the method further includes: determining the comparison similarity value corresponding to the face recognition result; updating the cumulative confidence value in memory according to the comparison similarity value as a feedback basis for resource demand assessment of the next frame image.

[0009] In the above embodiments, the face recognition system adopts the above technical solution to update the cumulative confidence score by feeding back the similarity of the recognition results. This enables the system to dynamically adjust the resource investment of subsequent frames according to the success or failure of recognition, thereby enhancing the adaptive ability to changes in the target state and improving tracking stability.

[0010] In conjunction with some embodiments of the first aspect, in some embodiments, before substituting the quality score and cumulative confidence value of the current frame image into the resource requirement calculation formula to calculate the resource scheduling score characterizing the necessity of feature extraction of the current frame, the method further includes: extracting the current identity hash fingerprint from the facial region of the current target object in the current frame image; reading the previous frame identity hash fingerprint associated with the cumulative confidence value from memory; calculating the difference between the current identity hash fingerprint and the previous frame identity hash fingerprint to obtain an identity continuity score; and resetting the cumulative confidence value to the initial confidence value when the identity continuity score is lower than a preset continuity score threshold.

[0011] In the above embodiments, the face recognition system adopts the above technical solution to quickly verify the continuity of identity using hash fingerprints and resets the confidence level when switching targets. This avoids the incorrect application of the confidence level of the previous target to the new target, ensures the accuracy of resource scheduling scoring, and prevents recognition errors caused by identity confusion.

[0012] In conjunction with some embodiments of the first aspect, in some embodiments, after substituting the quality score and cumulative confidence value of the current frame image into the resource requirement calculation formula to calculate the resource scheduling score characterizing the necessity of extracting the features of the current frame, the method further includes: when the resource scheduling score is greater than or equal to the computing power activation threshold, calling the NPU unit to extract the deep liveness features of the current frame image; when the resource scheduling score is less than the computing power activation threshold, calling the CPU unit to extract the shallow liveness features of the current frame image; and generating a verification result of the face recognition result based on the deep liveness features or the shallow liveness features.

[0013] In the above embodiments, the face recognition system adopts the above technical solution, dynamically selects deep or shallow liveness detection based on resource scheduling score, uses NPU for high-precision detection when it is highly necessary, and uses CPU for low-power detection when it is not necessary, thus achieving a balance between security and resource consumption.

[0014] In conjunction with some embodiments of the first aspect, in some embodiments, before substituting the quality score and cumulative confidence value of the current frame image into the resource requirement calculation formula to calculate the resource scheduling score characterizing the necessity of feature extraction of the current frame, the method further includes: detecting the number of face regions in the current frame image to obtain a face count value; when the face count value is greater than a single face threshold, extracting the independent cumulative confidence value corresponding to each face region to form a confidence array; calculating the product of the reciprocal of each value in the confidence array and the corresponding face region area to obtain a processing urgency score; sorting the processing urgency scores in descending order to obtain a processing priority sequence; and selecting the top N face regions from the processing priority sequence as priority processing targets for recognition processing according to the upper limit of the concurrent processing capability of the NPU unit.

[0015] In the above embodiments, the face recognition system, by adopting the above technical solution, prioritizes faces in multi-target scenarios by comprehensively considering the recognition confidence and the urgency of face area calculation, ensuring that limited NPU resources are allocated to the targets that need the most processing first, thus optimizing concurrent processing efficiency.

[0016] In conjunction with some embodiments of the first aspect, in some embodiments, after the step of selecting the top N face regions as priority processing targets from the processing priority sequence according to the upper limit of the concurrent processing capability of the NPU unit for recognition processing, the method further includes: extracting the row exposure timestamp of the original sensor data corresponding to the current frame image from the ISP cache; calculating the temporal offset of the face region of each priority processing target during the sensor exposure process according to the row exposure timestamp; converting the temporal offset into pixel coordinate correction values, performing distortion correction on the boundary coordinates of each face region to obtain corrected face regions; calculating the spatial overlap between multiple corrected face regions, determining face image pairs with occlusion relationships, and determining the visible area ratio of each face image in the face image pair; reallocating the NPU processing time according to the visible area ratio, so that face images with a larger visible area ratio receive priority processing rights; and generating an NPU task scheduling table containing processing allocation time and spatial coordinate correction information.

[0017] In the above embodiments, the face recognition system adopts the above technical solution, uses ISP timestamps to correct rolling shutter distortion, and dynamically allocates NPU processing time according to the occlusion situation, thereby avoiding the decrease in recognition accuracy caused by sensor effects and mutual occlusion in multi-target scenarios.

[0018] In conjunction with some embodiments of the first aspect, in some embodiments, after generating an NPU task scheduler containing processing allocation time and spatial coordinate correction information, the method further includes: calculating the expected completion time of each face feature extraction based on the processing allocation time in the NPU task scheduler; calculating the time difference between the expected completion time and the current frame acquisition time to obtain a temporal delay prediction value; determining a target face image whose temporal delay prediction value is greater than a preset synchronization threshold, and extracting motion trajectory parameters of the target face image in a historical frame sequence; calculating the predicted center coordinates of the target face image at the expected completion time based on the motion trajectory parameters; and associating the extracted feature vectors with the predicted center coordinates after the NPU unit completes feature extraction to compensate for spatial position deviations caused by computational delay.

[0019] In the above embodiments, the face recognition system, by adopting the above technical solution, predicts the NPU processing delay and combines it with the spatial position of the motion trajectory compensation feature vector, thereby avoiding positional deviation caused by the asynchrony between the calculation delay and the target movement, and improving the recognition accuracy in dynamic scenes.

[0020] In a second aspect, embodiments of this application provide a face recognition system, which includes: one or more processors and a memory; the memory is coupled to the one or more processors, and the memory is used to store computer program code, which includes computer instructions, and the one or more processors call the computer instructions to cause the face recognition system to perform the method described in the first aspect and any possible implementation thereof.

[0021] Thirdly, embodiments of this application provide a computer program product containing instructions that, when the computer program product is run on a face recognition system, cause the face recognition system to perform the method described in the first aspect and any possible implementation thereof.

[0022] Fourthly, embodiments of this application provide a computer-readable storage medium including instructions that, when executed on a face recognition system, cause the face recognition system to perform the method described in the first aspect and any possible implementation thereof.

[0023] Understandably, the face recognition system provided in the second aspect, the computer program product provided in the third aspect, and the computer storage medium provided in the fourth aspect are all used to execute the methods provided in the embodiments of this application. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods, and will not be repeated here.

[0024] One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages:

[0025] 1. By employing a resource scheduling method that combines the current frame image quality score and historical cumulative confidence to calculate the resource scheduling score, and dynamically calling the CPU for lightweight feature extraction or parallel calling of the CPU and NPU for multimodal feature extraction based on this score, when the system processes a highly confirmed target, only the CPU is used to calculate the geometric displacement, saving computing power. However, when a new target appears or the state of an old target changes drastically, the cumulative confidence is low, which leads to an increase in the resource scheduling score. This activates the NPU and CPU to extract appearance, geometry, optical flow, and other multimodal features in parallel. This solves the drawbacks of related technologies that only perform linear resource allocation based on the quality of a single frame, wasting computing power to process high-quality redundant frames, and abandoning the processing of key turning point frames due to low quality scores. It realizes on-demand allocation of computing resources on heterogeneous processors such as the RK3576, improving the utilization rate of computing resources.

[0026] 2. Because the comparison similarity value of the face recognition results is used to update the cumulative confidence, after the vector retrieval comparison is completed, not only the recognition result is obtained, but also the comparison similarity that quantifies its reliability is obtained. The cumulative confidence of the target is increased or decreased based on the similarity. This solves the rigidity problem of resource scheduling strategy lacking feedback and difficulty in self-correction based on recognition effect in related technologies, and enhances the long-term stable operation capability of the system in real complex environments.

[0027] 3. By employing a strategy that calculates the processing urgency score based on the face region area and the individual cumulative confidence scores in multi-target scenarios, and prioritizing multiple targets accordingly, the system selects the priority target based on the upper limit of NPU concurrent processing capacity. Therefore, when multiple faces appear in the field of view, the system can calculate the processing urgency for each face, rank the urgency scores in descending order, and avoid the average distribution of computing power or its occupation by unimportant targets through a resource allocation priority mechanism. This achieves intelligent and differentiated scheduling of NPU core computing resources in crowded or dynamic multi-person scenarios, improving the system's key target capture rate and recognition efficiency in complex environments. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of the topology of a face recognition system in an embodiment of this application;

[0029] Figure 2 This is a flowchart illustrating a face recognition method based on RK3576 in an embodiment of this application.

[0030] Figure 3 This is another flowchart illustrating the face recognition method based on RK3576 in the embodiments of this application;

[0031] Figure 4This is a schematic diagram of the physical device structure of a face recognition system in an embodiment of this application. Detailed Implementation

[0032] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification of this application, the singular expressions “a,” “an,” “the,” “the,” and “this” are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this application refers to any or all possible combinations including one or more of the listed items.

[0033] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature, and in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0034] Please see Figure 1 This is a schematic diagram of the topology of a face recognition system in an embodiment of this application. Figure 1 The diagram illustrates the overall topology of a multimodal face feature extraction system built on the RK3576 chip in an embodiment of this application. This system embodies a complete processing pipeline from data input to feature output, its core being the utilization of the RK3576's heterogeneous computing units (CPU, GPU, and NPU) to achieve efficient and robust face feature extraction.

[0035] Specifically, the system's processing logic is as follows:

[0036] First, the input layer acquires video streams from multiple cameras. Then, the image preprocessing module uses GPU-accelerated OpenCP to perform operations such as color space conversion and histogram equalization to enhance image quality and normalize the face region size to 112x112, preparing it for subsequent neural network processing.

[0037] The core processing layer extracts features from three different modalities in parallel:

[0038] First, face detection is performed using NPU-accelerated RetinaFace, and 5-point key point alignment is performed, which is the basis for appearance feature extraction.

[0039] Secondly, the CPU-run Dlib library is used to detect 68 facial key points and calculate their relative distances to form geometric features;

[0040] Thirdly, the pixel motion between consecutive frames is calculated on the CPU using optical flow to capture dynamic changes such as facial expressions and form temporal features.

[0041] It should be noted that the three features mentioned here—appearance features, geometric features, and temporal features—are features in three essential dimensions for face recognition. In this application, they are specifically referred to as appearance texture features, geometric structure features, and optical flow motion features. These three features respectively characterize the data of the face at the texture, structure, and image motion levels. Optical flow motion features refer to the motion features of a spatially moving object on the observation imaging plane. For ease of description, they can also be simplified to optical flow features or described as temporal features in the following text.

[0042] These three features are then fed into different processing pathways. The appearance features are processed by a lightweight neural network called LightFaceNet (running on an NPU, containing structures such as convolutions, depthwise separating convolutions, attention modules, and dilated convolutions, and accelerated using INT8 quantization) to extract deep texture information, resulting in a 512-dimensional appearance feature vector (F_app). The geometric and temporal features are then formed into 128-dimensional (F_geo) and 256-dimensional (F_temp) vectors, respectively.

[0043] In the feature fusion and output layer, the system adopts a gated attention fusion mechanism to perform a weighted summation of the feature vectors of the three modalities (here it is assumed that F_geo and F_temp are mapped to 512 dimensions) (F_fused=α·F_app+β·F_geo+γ·F_temp). The weights α, β, and γ are dynamically generated by a small network (Concat→SoftMax) based on the concatenated features of the three modalities to adaptively adjust the contribution of each modality.

[0044] Finally, in the feature compression output stage, the fused 512-dimensional features are processed through a fully connected layer (which can be selectively compressed to 256 dimensions) and L2 normalized to finally output a standardized 512-dimensional feature vector for subsequent face comparison and recognition tasks.

[0045] The rationale behind the architecture lies in its allocation of tasks with different computational characteristics to the most suitable hardware units (such as NPU for parallel computation-intensive CNNs, CPU for logic control and traditional algorithms, and GPU for image batch processing), and its use of multimodal fusion and attention mechanisms to effectively address changes in pose, lighting, and expression in non-cooperative scenarios.

[0046] The following describes the process of the method provided in this implementation. Please refer to [link / reference]. Figure 2 This is a flowchart illustrating a face recognition method based on RK3576 in an embodiment of this application.

[0047] S201. Obtain the current frame image in the video stream, and read the cumulative confidence value of the current target object in the previous frame image from memory.

[0048] Here, the current frame image refers to a single frame captured in real time from the video stream by the face recognition system. The current target object refers to the face in this frame image that needs to be identified. The cumulative confidence score is a variable stored in memory that represents the system's continued trust in the identification result of the target object; this score is updated and retained after processing the previous frame.

[0049] Specifically, in a continuous video processing flow, the face recognition system first captures a new image frame through the camera interface. Simultaneously, to achieve temporal association decisions, the system can access RAM or cache to read the cumulative confidence score associated with the currently tracked target ID, calculated and stored after processing the previous frame. For example, if the similarity score of the previous frame was 0.9 and the initial confidence score was 0.5, the updated cumulative confidence score might be 0.7.

[0050] In some embodiments, this step can be implemented in several ways: Optionally, the system maintains a tracking list containing the ID, bounding box, and confidence score of each target, and associates the face detected in the current frame with the targets in the list through IOU matching, then reads the confidence score; Optionally, the system employs a more complex tracking algorithm such as SORT or DeepSORT, whose tracker state itself contains historical information, from which the cumulative confidence score can be directly extracted or calculated. It is understood that other methods can also be used to achieve inter-frame association of targets and information reading, which are not limited here.

[0051] In some embodiments, when a target briefly leaves the screen and then returns, the system may treat it as a new target, resulting in a loss of accumulated confidence. To address this, a temporary storage state is set for the briefly lost target. If a face with similar features reappears within a certain time (e.g., 2 seconds), its tracking ID and accumulated confidence are restored, thereby ensuring the continuity of recognition.

[0052] S202. Substitute the quality score and cumulative confidence value of the current frame image into the resource requirement calculation formula to calculate the resource scheduling score that represents the necessity of feature extraction of the current frame.

[0053] The quality score is a quantitative assessment of attributes such as sharpness, illumination, and pose of the face region in the current frame image. The resource requirement calculation formula is a pre-defined mathematical formula used to combine the quality score and cumulative confidence. The resource scheduling score is the output of the calculation formula, which represents the urgency and necessity of performing high-computational feature extraction on the current frame.

[0054] Specifically, the face recognition system first performs a quality assessment on the face region of the current frame. For example, it uses the Laplacian operator to calculate a sharpness score, assesses the head pose angle, and analyzes the brightness distribution to obtain a quality score Q. Then, it substitutes this quality score Q and the cumulative confidence score C read from step S201 into a formula, for example, Score = w1 * Q - w2 * C. Here, w1 and w2 are preset weighting coefficients. A higher Q value will increase the Score, while a higher C value will decrease the Score. The Score is a resource scheduling score.

[0055] In some embodiments, this step can be implemented in several ways: Optionally, a simple linear weighted formula Score=a*Q+b*(1-C) can be used, where a and b are hyperparameters; alternatively, a more complex nonlinear function can be used, such as Score=(1+tanh(Q-Q_threshold))*exp(-C), making the score more sensitive to changes in quality and confidence, where Q_threshold is a preset quality score threshold. It is understood that lookup tables or small neural networks can also be used to fit this calculation process, and this is not limited here.

[0056] It should be noted that the core idea of ​​this resource requirement calculation formula is to establish an evaluation model: high-quality images (high Q) provide high potential for returns, but if the target has been highly confirmed (high C), the marginal returns of repeated extraction are diminishing; conversely, low confidence (low C) means that the current state is highly uncertain, and in this case, even if the image quality is average, the necessary cost of performing deep feature extraction is worthwhile.

[0057] A more specific nonlinear calculation formula can be designed as: Score = (α*f(Q)) / (β+g(C)).

[0058] Here, f(Q) is a sigmoid activation function (such as sigmoid), which maps the quality score from 0 to 100 to the range of 0 to 1, making the score more sensitive to changes in the middle region and saturating at the very good or very poor levels, thus avoiding excessive influence from extreme quality values. g(C) can be an exponential function, such as exp(γ*C), which makes the denominator grow faster as the cumulative confidence C increases, thereby exponentially enhancing the inhibitory effect on the total score. α, β, and γ are hyperparameters that need to be calibrated by performing a grid search on a validation dataset containing various scenarios (such as the appearance of new targets, the target remaining stable, the target's attitude changing drastically, and changes in illumination).

[0059] The goal of calibration is to optimize the final resource scheduling score distribution by separating different scenarios, ensuring that the score consistently exceeds the computing power activation threshold in key frames that require the most NPU computing power (such as identity switching or the instant when the pose changes from side to front), while the score consistently remains below this threshold in redundant frames (such as multiple consecutive high-quality frontal faces). This achieves a balance between minimizing average computing power consumption and minimizing recognition accuracy loss across the entire test set. For example, after calibration, the final formula can be:

[0060] Score = (100 * sigmoid((Q - 50) / 10)) / (0.1 + exp(3.0 * C)), where Q is the quality score from 0 to 100 and C is the confidence level from 0 to 1.

[0061] S203. When the resource scheduling score is less than the preset computing power activation threshold, the CPU unit is called to calculate the geometric displacement data of the current frame image relative to the previous frame image and generate a lightweight feature vector.

[0062] The computing power activation threshold is a preset value used to determine which processing path to adopt. The CPU unit is a general-purpose processor. Geometric displacement data refers to the displacement vector calculated by comparing the coordinate changes of facial key points between the current frame and the previous frame. Lightweight feature vectors are feature representations with low dimensionality and low computational cost.

[0063] Specifically, the face recognition system compares the resource scheduling score calculated in step S202 with a preset computing power activation threshold (e.g., 50). If the score is less than 50, it indicates that the system considers the current frame information redundant or of low value, and there is no need to use the NPU. At this time, the system can call the CPU to track 5 or 68 key points of the face in the previous and next frames, calculate its average displacement, rotation, scaling, and other geometric transformation parameters, and combine these parameters into a low-dimensional lightweight feature vector.

[0064] In some embodiments, this step can be implemented in several ways: optionally, the average displacement vector of only 5 core key points (such as the corner of the eye and the tip of the nose) can be calculated as a feature; alternatively, based on 68 key points, Procrustes analysis can be used to align the point clouds of two frames and extract a more accurate affine transformation matrix as a feature vector. It is understood that methods such as optical flow can also be used to assist in calculating displacement to generate more robust and lightweight features; this is not limited here.

[0065] In some embodiments, when the target moves rapidly or is occluded, keypoint detection may be inaccurate, leading to incorrect calculated geometric displacement data. To address this, a keypoint confidence check is added before displacement calculation to eliminate unreliable points with low confidence, using only stable high-confidence points to calculate geometric displacement. Alternatively, when the displacement vector amplitude is abnormally large, lightweight feature generation is abandoned, and resource scheduling scores are temporarily increased, forcing entry into a high-computing-power mode.

[0066] It should be noted that constructing a discriminative and stable feature vector from raw displacement data (such as translation, rotation, and scaling parameters) involves four steps: keypoint extraction, rigid transformation alignment, displacement parameter calculation, and feature vectorization. First, the system uses a lightweight keypoint detector (such as a 68-point model from the previous frame already in memory) on the CPU to extract the coordinates of 68 keypoints from the face in the current and previous frames. Second, to eliminate the influence of overall head movement and focus on non-rigid changes such as facial expressions, the system employs Procrustes Analysis to calculate an optimal affine transformation matrix (including rotation, translation, and scaling) to align the keypoint cloud from the previous frame to the point cloud of the current frame. The parameters of this transformation matrix itself (such as rotation angle and scaling factor) can be used as part of the feature vector. Then, the Euclidean distance between corresponding keypoints in the two frames after alignment is calculated, forming a 68-dimensional residual vector. This vector primarily reflects non-rigid deformations caused by changes in facial expressions. Finally, the parameters of the affine transformation matrix (e.g., the four degrees of freedom parameters of the transformation matrix), the statistics of the residual vector (e.g., mean, variance, maximum value), and the translation vector (x, y displacement) of the face bounding box center point are concatenated and L2 normalized to form a lightweight feature vector with low dimensionality (e.g., 16 dimensions) but containing information on subtle changes in position, pose, and expression. For example, a vector can be composed of [dx, dy, scale, rotation_angle, residual_mean, residual_std, ..., residual_max], which includes dx (horizontal displacement), dy (vertical displacement), scale (scaling), rotation_angle (rotation angle), ..., residual_mean (residual mean), residual_std (residual standard deviation), and residual_max (residual maximum value). Although this vector cannot be used for precise identity recognition, it is sufficient to track minor changes in the same individual in the short term and compare it with the previous deep features of that ID in the database to confirm the continuity of identity.

[0067] S204. When the resource scheduling score is greater than or equal to the computing power activation threshold, the NPU unit and CPU unit are invoked to extract the appearance texture features, geometric structure features and optical flow motion features of the current frame image in parallel to generate a multimodal feature set.

[0068] The NPU (Neural Processing Unit) is a neural network processing unit specifically designed to accelerate AI models. Parallel extraction refers to the simultaneous operation of the CPU and NPU. Appearance and texture features refer to features extracted through deep learning networks that reflect facial skin texture and lighting details. Geometric structure features refer to the positional relationships of key points. Optical flow motion features refer to pixel-level motion patterns. The multimodal feature set is a combination of these three types of features.

[0069] Specifically, when the resource scheduling score is greater than or equal to the computing power activation threshold, the system determines that a comprehensive analysis is needed and initiates a parallel processing flow: On one hand, a 112x112 face image is fed into the NPU to run a deep network such as LightFaceNet to extract high-dimensional appearance texture features; on the other hand, algorithms such as Dlib are run on the CPU simultaneously to detect 68 key points to generate geometric structure features and to calculate the optical flow field to generate optical flow motion features. After these three steps are completed, they together constitute a multimodal feature set.

[0070] In some embodiments, this step can be implemented in several ways: Optionally, the NPU extracts apparent features, while the CPU extracts only geometric features, ignoring optical flow to save computational power; alternatively, if the CPU performance is strong, some lightweight network layers (such as the initial few layers of the network) can also be run on the CPU, forming a deeper pipeline parallelism with the NPU. It is understood that, depending on the specific hardware characteristics of the RK3576, different tasks can be more finely allocated to different CPU cores or NPU subunits, which is not limited here. In some embodiments, the asynchronous execution times of CPU and NPU tasks can lead to waiting bottlenecks. For example, the NPU completes the computation quickly, but the CPU's optical flow computation is slow. To address this, an asynchronous task management and callback mechanism can be established. After the NPU and CPU tasks are completed, a callback function is triggered, storing the results in a shared buffer. Subsequent fusion steps are only triggered when features from all modalities are ready, thereby decoupling the execution processes of different units and improving parallel efficiency.

[0071] S205. Calculate the semantic distance between each modal feature vector in the multimodal feature set, and generate weighting coefficients based on the semantic distance and the gating network. Perform weighted fusion and dimensionality reduction on each modal feature vector to obtain a weighted feature vector.

[0072] Semantic distance measures the similarity of different feature vectors in the feature space. A gating network is a small neural network used to generate gate signals that control the flow of information. Weighting coefficients are the weights of the gating network's output. Weighted fusion refers to adding the feature vectors according to their weights. Dimensionality reduction refers to reducing the dimensionality of the feature vectors.

[0073] Specifically, the face recognition system inputs the multimodal feature set (appearance, geometry, optical flow) generated in step S204 into the fusion module. First, these features may be concatenated and fed into a gated network consisting of a fully connected layer and a Softmax activation function. This network outputs three weighting coefficients α, β, and γ, which sum to 1. These coefficients reflect the importance of each modal feature in the current scene. Then, the system uses these coefficients to perform a weighted summation of the original modal feature vectors, obtaining a fused feature vector. Finally, dimensionality reduction may be performed using a fully connected layer.

[0074] In some embodiments, when the input quality of a modality is extremely poor (e.g., a pure black image renders appearance features invalid), the gating network may still assign a non-zero weight, introducing noise. To address this, a modality validity prediction is added before fusion. For example, if the image brightness is too low, the appearance feature weight α can be directly set to 0, and β and γ can be renormalized. This hard gating prevents invalid features from contaminating the final fusion result.

[0075] It should be noted that the gating network itself mainly relies on the concatenated multimodal feature vector to dynamically perceive the overall information quality of the current frame. For example, 512-dimensional appearance features, 128-dimensional geometric features, and 256-dimensional optical flow features are concatenated into an 896-dimensional vector, which is then input into a small MLP (Multilayer Perceptron). This MLP outputs three weights (α, β, γ). Simultaneously, the system calculates the cosine similarity between each pair of modal features in parallel, obtaining three semantic distance values: d_ag (appearance-geometric), d_ao (appearance-optical flow), and d_go (geometric-optical flow).

[0076] These distance values ​​reflect the consistency of descriptions across different modalities. For example, when a face is in good pose and expressionless, appearance and geometric features should be highly correlated, resulting in high d_ag. When illumination changes drastically, appearance features may differ significantly from the other two, leading to low d_ag and d_ao.

[0077] These semantic distance values ​​can be used to post-process and correct the original weights output by the gating network. For example, if the gating network gives a weight set [0.6, 0.2, 0.2], but the calculation finds that the semantic distance between the appearance feature and other features is extremely low (indicating that the appearance feature may have become invalid), the system can attenuate the weights of the appearance feature according to a preset rule α'=α*clamp(d_ag, 0, 1)*clamp(d_ao, 0, 1), and renormalize the weights of β and γ accordingly.

[0078] This mechanism combines data-driven dynamic weighting (gated network) and rule correction based on feature consistency testing (semantic distance), making the fusion process more robust and effectively suppressing interference caused by a sudden deterioration in the quality of a certain modality of data.

[0079] S206. Perform vector retrieval and comparison based on lightweight feature vectors or weighted feature vectors to obtain the face recognition result at the current moment.

[0080] The lightweight feature vector comes from step S203, and the weighted feature vector comes from step S205. Vector retrieval and comparison involves searching a pre-registered facial feature database for the vector most similar to the current input feature vector. The face recognition result is typically the identity ID with the highest similarity and its similarity score in the database.

[0081] Specifically, based on the execution path of the previous steps, the facial recognition system obtains a feature vector (either lightweight or multimodal fusion). The system then calculates the cosine similarity or Euclidean distance between this vector and the feature vectors of thousands of registered individuals pre-stored in the database. Finally, it finds the registration vector with the highest similarity (or closest distance), and its corresponding identity ID is used as the current facial recognition result. Simultaneously, this highest similarity value is also output.

[0082] In some embodiments, this step can be implemented in several ways: Optionally, to accelerate retrieval, instead of brute-force comparison, an index can be built using an Approximate Nearest Neighbor (ANN) search library such as Faiss or Annoy, achieving millisecond-level retrieval of massive databases; alternatively, different databases or different similarity thresholds can be used for lightweight features and weighted features, as their representational capabilities and stability differ. It is understood that a multi-level comparison strategy can also be adopted, first using lightweight features for coarse screening, and then using weighted features for fine ranking; this is not limited here.

[0083] In some embodiments, when the highest similarity value of the identification result is also low (e.g., below 0.5), the result is unreliable. Direct output may lead to false identification. To address this, an identification threshold can be set. Only when the highest similarity exceeds this threshold is the identification result accepted and an identity ID output; otherwise, an unknown person is output, and this low similarity can be used as feedback to reduce the cumulative confidence of the target in the next step.

[0084] The following provides a more detailed description of the process of the method provided in this implementation. Please refer to [link / reference]. Figure 3 This is another flowchart illustrating the face recognition method based on RK3576 in this application embodiment.

[0085] S301. Obtain the current frame image in the video stream, and read the cumulative confidence value of the current target object in the previous frame image from memory.

[0086] Refer to step S201, which will not be repeated here.

[0087] S302. Detect the number of face regions in the current frame image and obtain the face count value.

[0088] The number of face regions refers to the number of faces that the face recognition system successfully locates and outlines in the current frame image using a face detection algorithm (such as RetinaFace). The face count value is the specific numerical value of this number.

[0089] Specifically, after acquiring the current frame image, but before performing in-depth feature extraction, the face recognition system can first perform a rapid face detection scan of the entire image. This detection process is typically accelerated by an NPU to ensure real-time performance. The detection algorithm returns one or more bounding boxes, each defining a region that is considered a face. The system then counts the total number of returned bounding boxes; this total is the face count. For example, if three bounding boxes are detected, the face count is 3. This step is a prerequisite for triggering subsequent multi-object processing strategies.

[0090] In some embodiments, this step can be implemented in several ways to adapt to different needs: Optionally, a lightweight face detector such as Ultra-Light-Fast-Generic-Face-Detector-1MB (a lightweight general-purpose face detection model) can be used for fast initial detection on the CPU to save NPU resources for subsequent more important tasks; Optionally, in addition to returning the number of faces, the detector can also provide a detection confidence score for each face bounding box. The system can set a confidence threshold and only count the number of faces that are higher than the threshold to filter out false positives that are of poor quality or very blurry. It is understood that other more advanced detection algorithms can also be used to achieve face localization and counting, which is not limited here.

[0091] In some embodiments, in crowded scenes, face detectors may miss or incorrectly merge multiple overlapping faces into a single detection box, resulting in inaccurate face counts and affecting subsequent priority ranking. To address this, a model supporting dense face detection can be employed, combined with a variant of non-maximum suppression (NMS), such as Soft-NMS. This approach does not drastically remove overlapping boxes but rather reduces their confidence, thus effectively distinguishing more real faces while preserving more of them, leading to more accurate face counts.

[0092] S303. When the face count value is greater than the single face threshold, extract the independent cumulative confidence value corresponding to each face region to form a confidence array.

[0093] The single-face threshold is a preset integer, usually 1, used to determine whether the scene is a multi-object scene. The independent cumulative confidence score refers to the individual confidence score maintained for each individual object tracked in the video. The confidence array is a data structure used to store the cumulative confidence scores corresponding to all detected faces in the current frame.

[0094] Specifically, the face recognition system compares the face count obtained in step S302 with a single face threshold (e.g., set to 1). If the count is greater than 1, the system determines that it has entered the multi-target processing mode. Next, the system can traverse each face region detected in the current frame and associate it with a list of tracking targets maintained in memory using a tracking algorithm (such as IOU matching) to find the historical tracking ID corresponding to each face region. Then, based on this ID, the system extracts the individual, independent cumulative confidence value of each target from memory and collects these values ​​into an array to form a confidence array. For example, if three faces are detected, corresponding to tracking IDs A, B, and C, with cumulative confidence values ​​of 0.9, 0.2, and 0.8 respectively, the confidence array would be [0.9, 0.2, 0.8].

[0095] In some embodiments, this step can be implemented in several ways to enhance robustness: Optionally, for newly appearing faces in the frame that do not yet have an associated tracking ID, a new ID is assigned and an initial confidence value (e.g., 0.1) is given, then added to the confidence array; alternatively, if the tracking algorithm itself can output the tracking confidence or age (number of tracked frames) for each target, this information can be combined with the accumulated identification confidence to generate a more comprehensive confidence index, which is then stored in the array. It is understood that more complex data structures, such as hash tables, can also be used to store and retrieve the confidence of each target to improve efficiency; this is not limited here.

[0096] In some embodiments, when targets cross paths, the tracking algorithm may experience ID switching, where A's ID is incorrectly assigned to B, causing B to inherit A's accumulated confidence, and vice versa. This can severely disrupt subsequent priority calculations. To address this, short-term feature comparisons can be introduced as auxiliary verification for tracking. During ID association, in addition to position overlap (IOU), a very shallow feature (such as a color histogram or LBP feature) is quickly extracted, and the feature similarity of targets in preceding and following frames is compared. ID inheritance is only confirmed when both position and features are sufficiently matched, which effectively reduces the occurrence of ID switching.

[0097] S304. Calculate the product of the reciprocal of each value in the confidence array and the corresponding face region area to obtain the processing urgency score.

[0098] The face region area refers to the number of pixels occupied by the face detection bounding box in the image. The processing urgency score is a comprehensive metric used to quantify the priority of processing a particular face.

[0099] Specifically, after entering the multi-target processing mode, the face recognition system needs to calculate a priority score for each face. The system can iterate through the confidence array and the detection box information of each face formed in step S303. For the i-th face, its cumulative confidence is Ci, and its detection box area is Ai. The system can calculate its processing urgency score Pi, which can be calculated as Pi = (1 / Ci) * Ai. This formula means that the lower the confidence (the larger 1 / Ci), the more uncertain the identity, and the more priority it needs to be processed; at the same time, the larger the face area (the larger Ai), the closer the target is, the clearer the information, and the more important it is, so it also needs to be processed first. Multiplying these two factors yields a comprehensive urgency score.

[0100] In some embodiments, this step can be implemented in several ways to flexibly adjust the strategy: Optionally, to prevent the confidence level from being too low (close to 0) and causing the score to be infinitely large, the confidence level can be smoothed, such as Pi = (1 / (Ci+ε))*Ai, where ε is a very small normal number; Optionally, different weights can be introduced for the area and confidence level, such as Pi = w_c*(1 / Ci) + w_a*Ai, and the weights w_c and w_a can be adjusted to control which factor is more important. It is understood that other factors, such as the speed of facial movement and the angle of pose deviation, can also be introduced to calculate a more complex urgency score, which is not limited here.

[0101] In some embodiments, the calculation of the face region area may be affected by perspective distortion, causing distant faces, even if important, to have a small area, resulting in a lower urgency score. To address this, if the system can obtain depth information (e.g., through a binocular camera or ToF sensor), the area Ai can be replaced with the true size corrected for depth information, or the inverse of the distance can be directly multiplied into the formula. If depth information is unavailable, the relative size can be estimated based on the distance between facial key points instead of directly using pixel area, to partially compensate for perspective effects.

[0102] S305. Sort the urgency scores in descending order to obtain the processing priority sequence.

[0103] In this context, descending order refers to sorting a set of values ​​from largest to smallest. The processing priority sequence is an ordered list containing the face IDs or indices to be processed, with their order representing the order in which they are processed.

[0104] Specifically, after calculating the processing urgency score for each face in the current frame, the face recognition system obtains a score list that corresponds one-to-one with the face list. For example, the urgency scores for three faces A, B, and C are [50, 200, 80], respectively. The system then performs a standard sorting algorithm (such as quicksort or mergesort) on this score list to sort it in descending order. The sorted scores are [200, 80, 50], corresponding to the original face order [B, C, A]. This [B, C, A] is the final generated processing priority sequence, indicating that the system should process face B first, then C, and finally A.

[0105] In some embodiments, this step can be implemented in several ways to optimize performance or add logic: Optionally, during sorting, if two targets have the same urgency score, a second sorting rule can be introduced, for example, prioritizing targets that are closer to the center of the image; alternatively, the sorting process can not just generate a static sequence, but can be dynamically adjusted. For example, if too much time is spent processing high-priority targets, the processing level of subsequent targets can be dynamically reduced or they can be skipped directly. Understandably, a more efficient data structure, such as a priority queue (heap), can also be used to maintain this priority relationship, where the highest priority element can be retrieved directly each time; this is not limited here.

[0106] S306. Based on the upper limit of the concurrent processing capability of the NPU unit, select the top N face regions from the processing priority sequence as priority processing targets for recognition processing.

[0107] The concurrent processing capacity limit of an NPU unit refers to the maximum number of tasks that the hardware unit can process in parallel per unit of time. For example, an NPU may be able to handle the inference tasks of 2 or 4 face recognition models simultaneously. The top N face regions refer to the N targets that are ranked first in the processing priority sequence. Priority targets are the faces selected for high-computing-power recognition.

[0108] Specifically, the face recognition system first needs to know the hardware specifications of the NPU on its RK3576 chip, especially its supported concurrent batch size or multi-task parallel capability, which is set to N. Then, the system can select the first N elements from the processing priority sequence generated in step S305. For example, if the priority sequence is [B, C, A, D, E], and the NPU's concurrency limit N=2, then the system will select B and C as the priority processing targets for this frame. These two targets will then be sent to the subsequent high-performance, multimodal feature extraction process (i.e., step S309 and beyond).

[0109] In some embodiments, this step can be implemented in several ways to achieve finer-grained resource control: Optionally, for other unselected targets (such as A, D, E), the system will not completely ignore them, but can perform a very low-power operation for them, such as simply updating their tracking position, or performing lightweight feature extraction in step S203 to maintain basic tracking and state awareness; Optionally, the value of N can be dynamic, and the system can dynamically adjust the size of N according to the current total power consumption or temperature, actively reducing the number of concurrent connections for protection when the device overheats. It is understood that more complex scheduling strategies, such as round-robin scheduling, can also be used, allowing all targets to have a chance to be processed, with higher-priority targets receiving more time slices; this is not limited here.

[0110] In some embodiments, after determining the priority of multi-target processing, the face recognition system may perform sensor-level distortion correction and resource fine-tuning to further improve processing accuracy. Specifically, the face recognition system can extract the row exposure timestamp of the original sensor data corresponding to the current frame image from the ISP cache; calculate the temporal offset of the face region of each priority processing target during the sensor exposure process based on the row exposure timestamp; convert the temporal offset into pixel coordinate correction values, perform distortion correction on the boundary coordinates of each face region to obtain the corrected face region; calculate the spatial overlap between multiple corrected face regions, identify face image pairs with occlusion relationships, and determine the visible area ratio of each face image in the face image pair; reallocate NPU processing time according to the visible area ratio, so that face images with a larger visible area ratio receive priority processing; and generate an NPU task scheduling table containing processing allocation time and spatial coordinate correction information.

[0111] The ISP cache refers to the storage area within the image signal processor used to temporarily store raw sensor data. The row exposure timestamp is the precise exposure start time recorded by the sensor for each row or group of rows in the image. The timing offset represents the time difference between the exposure of a row in the image and the exposure of the first row. The pixel coordinate correction value is the amount of adjustment needed to the horizontal position of pixels to compensate for the movement of objects during rolling shutter scanning. The corrected face region is the face bounding box after rolling shutter distortion correction. Spatial overlap is usually measured by the intersection-over-union (IoU) ratio, which measures the degree of overlap between two regions. The visible area percentage refers to the ratio of the area of ​​the face region not occluded by other objects to its total area. The NPU task scheduler is a structured instruction set that defines in detail the tasks that the NPU needs to process and their related parameters.

[0112] Specifically, after the face recognition system selects the top N face regions as priority processing targets based on the Top-N algorithm, a refined preprocessing and scheduling step is performed to handle the complexities of rapid movement and mutual occlusion. First, for each priority processing target, the system accesses the ISP driver interface to obtain the row exposure timestamp associated with the current frame image. Combining this with the target's horizontal motion velocity estimated by optical flow or a tracker, the system can calculate the temporal offset of each row of pixels within the face bounding box due to the rolling shutter effect and the corresponding pixel coordinate correction value, thereby correcting the original rectangular face bounding box into a more accurate tilted quadrilateral region. Next, the system calculates the spatial overlap between each pair of these N corrected face regions. If the overlap of a pair of face regions is greater than zero, they are determined to have an occlusion relationship. For occluded faces, the system can estimate the proportion of their visible area (e.g., by calculating the area ratio of the non-overlapping parts). Finally, instead of simply allocating NPU resources equally among the N targets, the system performs a weighted allocation based on the proportion of the visible area. A target with 90% of its visible area will receive more processing priority or time slices than a target with only 30% visible area. All this information—the corrected coordinates and processing time allocation—is integrated into an NPU task scheduling table and sent to the NPU for execution.

[0113] In some embodiments, this fine-grained scheduling can be implemented in several ways: Optionally, a simplified rolling shutter correction can be used. Specific steps include: First, assuming the target moves at a constant velocity in a straight line within the frame, the horizontal velocity v is calculated by comparing the target center points of two consecutive frames; second, the total readout time of the sensor and the number of image rows are obtained, and the readout time t_row for each row is calculated; then, for any row y in the face bounding box, its horizontal coordinate correction value is v*y*t_row; finally, this correction value is applied to the left and right boundaries of the face bounding box to obtain the corrected region. Optionally, a segmentation-based visible region calculation can be used. Specific steps include: First, for a pair of overlapping faces A and B, their respective contours are roughly outlined using a lightweight face segmentation model; second, the intersection region of the contours of A and B is calculated; then, the area of ​​the intersection region is subtracted from the total contour area of ​​A to obtain the visible area of ​​A; finally, the visible area is divided by the total contour area of ​​A to obtain its visible region percentage. Understandably, hardware-accelerated graphics APIs can also be used to efficiently calculate the overlap and area between regions, which is not limited here.

[0114] In some embodiments, direct access to the ISP cache and acquisition of precise line exposure timestamps may be difficult to achieve on many embedded systems due to driver or permission restrictions, making physical model-based rolling shutter correction infeasible. To address this, a data-driven distortion correction method can be employed. Specifically, a large amount of data can be pre-collected offline, including image pairs taken by the specific camera at different motion speeds (distortion-free images at high shutter speeds and distorted images at rolling shutter speeds). Using this data, a miniaturized CNN model is trained. This model takes a distorted face image patch as input and outputs a distortion field to correct the distortion or directly outputs a corrected image patch. In actual operation, the system only needs to run this small CNN on the CPU to effectively correct distortion in the face region, thus bypassing the dependence on the underlying hardware interface.

[0115] In addition, in some embodiments, after generating a refined NPU task schedule, the face recognition system can perform temporal delay prediction and compensation to address the problem of processing latency. Specifically, the face recognition system can calculate the expected completion time of each face feature extraction based on the processing allocation time in the NPU task schedule; calculate the time difference between the expected completion time and the current frame acquisition time to obtain the temporal delay prediction value; identify target face images whose temporal delay prediction value is greater than a preset synchronization threshold, and extract the motion trajectory parameters of the target face image in the historical frame sequence; based on the motion trajectory parameters, calculate the predicted center coordinates of the target face image at the expected completion time; after the NPU unit completes feature extraction, associate the extracted feature vector with the predicted center coordinates to compensate for the spatial position deviation caused by the calculation delay.

[0116] The expected completion time refers to the time point predicted by the system based on task complexity and NPU performance benchmarks when the NPU will complete feature extraction. The predicted temporal delay is the time span between the occurrence of an event (image acquisition) and the output of its result (feature extraction completion). The synchronization threshold is a preset time length used to determine if the delay is large enough to require compensation. Motion trajectory parameters are a set of data describing the target's motion state, such as position, velocity, and acceleration, typically maintained by a tracking algorithm (e.g., a Kalman filter). The predicted center coordinates are the spatial position of the target at a future time (i.e., the expected completion time) calculated based on the motion trajectory parameters. The association operation refers to binding the calculated feature vector with the predicted future coordinates, rather than binding it with outdated original coordinates.

[0117] Specifically, before the face recognition system distributes tasks to the NPU, it performs future simulations to compensate for the system latency between image acquisition and feature output. First, the system consults the NPU task scheduler, calculating the expected time for each task based on the processing task assigned to each face (e.g., which model to run, batch size, etc.) and known NPU performance data, thus estimating the expected completion time. Then, it subtracts the current frame's acquisition timestamp from the expected completion time to obtain the predicted temporal latency. If this latency exceeds a synchronization threshold (e.g., 150 milliseconds), the system initiates a compensation procedure for that target. It extracts the target's motion trajectory parameters (e.g., a state vector maintained by a Kalman filter, containing current position and velocity) from the target's tracker. Using these parameters and the predicted temporal latency, the system predicts the target's center coordinates at the expected completion time using kinematic formulas. Finally, when the NPU actually completes the computation and returns the feature vector, the system no longer associates it with the original position detected in the current frame, but rather with the newly predicted future position coordinates. In this way, even if the target moves during the calculation, its identity features can be accurately attached to its latest location.

[0118] In some embodiments, this delay compensation can be implemented in several ways: Optionally, a Kalman filter-based prediction method can be used. The specific steps include: first, maintaining a Kalman filter for each tracked target, whose state includes position and velocity; second, updating the filter state in each frame using the detected position; then, when prediction is needed, calling the filter's prediction function, inputting the time delay prediction value as the time interval dt, to directly obtain the predicted position state; finally, associating the features output by the NPU with this predicted position. Optionally, a simpler linear extrapolation method can be used. The specific steps include: first, calculating the target's average velocity vector using its position in the most recent two or three frames; second, adding "velocity vector × time delay prediction value" to the current position to obtain the predicted center coordinates; finally, associating the features with the coordinates. While this method is less accurate than the Kalman filter, it has lower computational overhead. It is understood that more complex motion models, such as a second-order model considering acceleration, can also be used for more accurate trajectory prediction; this is not limited here.

[0119] S307. Substitute the quality score and cumulative confidence value of the current frame image into the resource requirement calculation formula to calculate the resource scheduling score that represents the necessity of feature extraction of the current frame.

[0120] Refer to step S202, which will not be repeated here.

[0121] In some embodiments, before conducting resource requirement assessment, the face recognition system may perform an identity continuity check to ensure the validity of the accumulated confidence score. Specifically, the face recognition system may extract the current identity hash fingerprint from the facial region of the current target object in the current frame image; read the previous frame identity hash fingerprint associated with the accumulated confidence score value from memory; calculate the difference between the current identity hash fingerprint and the previous frame identity hash fingerprint to obtain an identity continuity score; and reset the accumulated confidence score value to the initial confidence score value when the identity continuity score is lower than a preset continuity score threshold.

[0122] The current identity hash fingerprint refers to a compact binary sequence generated from the current face image using a very lightweight algorithm. It's used to quickly characterize the uniqueness of a face, similar to a "perceptual summary" of a face. The previous frame identity hash fingerprint is the hash fingerprint calculated and stored for the same tracking target in the previous frame. Dissimilarity typically represents the number of different bits between two hash fingerprints, i.e., the Hamming distance. The identity continuity score is a normalized score derived from the dissimilarity score, used to quantify the probability that the faces in the two frames belong to the same person. The preset continuity score threshold is a critical value used to determine whether an identity has changed. The initial confidence value is a preset, low default value used to indicate that the confidence in recognizing a new target or a target whose identity has changed is in its initial state.

[0123] Specifically, before the face recognition system uses historical cumulative confidence scores to evaluate the processing strategy for the current frame, it performs a quick identity continuity check to prevent the confidence scores of a previous target from being incorrectly applied to a newly appearing target. This scenario is common when target A leaves the frame, and target B immediately enters and is mistakenly identified by the tracking algorithm as a continuation of A. The system first calculates a short binary hash fingerprint (e.g., 64 bits) for the facial region of the target object in the current frame (e.g., a 64x64 grayscale image). Then, it reads the hash fingerprint associated with the target's tracking ID stored in the previous frame from memory. By calculating the Hamming distance between these two 64-bit fingerprints (e.g., a distance of 5), the system obtains a dissimilarity score. This dissimilarity score is converted into an identity continuity score (e.g., a score of (64-5) / 64 = 0.92). The system compares this score to a preset continuity score threshold (e.g., 0.85). If the score 0.92 is higher than the threshold 0.85, the face is considered continuous, and the cumulative confidence score is valid. Conversely, if the score is 0.7, which is below the threshold, the system determines that an identity switch has occurred. In this case, the system will forcibly reset the cumulative confidence score of the target to the initial confidence score value (e.g., 0.1), thereby ensuring that subsequent resource scheduling decisions are based on a zeroed-out, unbiased basis and avoiding incorrect confidence score inheritance.

[0124] In some embodiments, this identity continuity verification can be implemented in several ways: Optionally, the classic perceptual hashing algorithm (pHash) can be used. Specific steps include: first, scaling the face region to a very small size, such as 8x8 pixels; second, converting the image to grayscale; then, calculating the average grayscale value of these 64 pixels; finally, iterating through all pixels, recording a value greater than the average as 1, otherwise as 0, thus generating a 64-bit hash fingerprint. Optionally, a learning-based method can be used. Specific steps include: first, training a very small convolutional neural network (CNN), whose final output is a low-dimensional vector (e.g., 32-dimensional); second, passing this vector through a tanh activation function, making its values ​​range from -1 to 1; finally, taking the sign of each element in the vector (positive numbers as 1, negative numbers or zero as 0), generating a 32-bit hash fingerprint. This method is more robust to changes in lighting and pose compared to traditional hashing. It is understandable that other hashing techniques, such as average hashing (aHash) or differential hashing (dHash), can also be used to achieve this function, and no limitation is made here.

[0125] In some embodiments, for the same target, a significant change in hash fingerprint may occur due to drastic pose shifts or illumination variations, causing the identity continuity score to fall below a threshold and erroneously resetting the accumulated confidence level, which contradicts the initial intention of maintaining confidence continuity. To address this, a multi-region hash verification mechanism can be employed. Specifically, instead of calculating a single hash fingerprint for the entire face, the system divides the face region into multiple stable sub-regions, such as the left eye region, right eye region, and mouth region. The system independently calculates and compares the hash fingerprints of each sub-region, obtaining multiple identity continuity scores. The final identity continuity score can be a weighted average or the maximum value of these sub-region scores. In this way, even if a part of the face undergoes a drastic change due to pose or occlusion, the hash fingerprints of other stable regions remain consistent, providing a more reliable overall continuity judgment and reducing the misjudgment rate caused by changes in external conditions.

[0126] In addition, in some embodiments, the face recognition system can perform corresponding levels of liveness detection in parallel based on the resource scheduling score to ensure security. That is, when the resource scheduling score is greater than or equal to the computing power activation threshold, the face recognition system can call the NPU unit to extract the deep liveness features of the current frame image; when the resource scheduling score is less than the computing power activation threshold, it can call the CPU unit to extract the shallow liveness features of the current frame image; and generate a verification result of the face recognition result based on the deep liveness features or the shallow liveness features.

[0127] Deep liveness detection features are extracted using a complex neural network model running on an NPU. These features can identify subtle forgery clues and are used to defend against advanced spoofing attacks such as highly realistic 3D masks and high-definition video playback. Shallow liveness detection features are extracted using traditional computer vision algorithms or simple models running on a CPU. These are primarily used to detect low-cost planar attacks such as printed photos and screen captures. The verification result is the final output of the liveness detection module, typically a binary judgment of "real" or "forged," used to confirm the validity of the face recognition result.

[0128] Specifically, after determining the resource scheduling level of the current frame, the face recognition system will simultaneously initiate a matching security verification process. This process is executed in parallel with the main recognition process to avoid adding additional latency. When the resource scheduling score calculated in step S307 is high, triggering the NPU to perform high-computation recognition, the system considers the recognition task of the current frame to be crucial and therefore requires the highest level of security. At this time, the system can utilize part of the NPU's computing power to load and run a dedicated deep liveness detection model (such as a lightweight CNN). This model analyzes the subtle textures, reflections, and distortions of the face image, extracts deep liveness features, and provides a high-precision liveness determination. Conversely, when the resource scheduling score is low and the system only uses the CPU for lightweight processing, in order to maintain low power consumption, the system can call the CPU to execute shallow liveness detection algorithms, such as tracking key points of the eyes in several consecutive frames, calculating the eye aspect ratio (EAR) to determine whether there is a natural blinking action, or analyzing the frequency domain features of the image to detect moiré patterns on the screen. Ultimately, regardless of the method used to generate the liveness feature, it will be fed into a classifier to generate the final verification result. This result can serve as a gatekeeper; if it is determined to be "forged," the facial recognition result will be rejected.

[0129] In some embodiments, this hierarchical liveness detection can be implemented in several ways: Optionally, a shallow liveness detection based on blink detection can be implemented. Specific steps include: first, locating the eyes using 68 extracted facial key points; second, calculating the eye aspect ratio (EAR) of each frame based on the coordinates of the eye key points; then, storing the EAR values ​​of consecutive frames into a time window queue; finally, analyzing whether the EAR sequence in the queue exhibits a typical blink waveform of rapid decline followed by rapid recovery; if so, it is determined to be a real person. Optionally, a deep liveness detection based on deep learning can be implemented. Specific steps include: first, obtaining a liveness detection CNN model pre-trained on a large number of real people and various attack samples; second, using the face image of the current frame as input, performing the forward propagation of the model on the NPU; finally, the model's output layer typically directly provides a score between 0 and 1, representing the probability of being a real person, and the system compares this score with a threshold to draw a conclusion. It is understandable that other techniques, such as shallow methods based on color space analysis or deep methods based on remote photoplethysmography (rPPG), can also be used to perform liveness detection, and no limitation is made here.

[0130] It's important to note that the deep liveness detection model is a convolutional neural network specifically designed to distinguish real faces from various deceptive attacks (such as printed photos, screen playback, and 3D masks). Training the model requires a dedicated dataset containing a large number of positive and negative samples. Positive samples are real live face videos or images captured from different individuals under varying lighting and environmental conditions, while negative samples encompass as many attack types and methods as possible, such as photo attacks using different printers and paper, video attacks played on different mobile phones and tablets, and 3D mask attacks using different materials.

[0131] The training objective is to learn a binary classifier, and its loss function is typically the binary cross-entropy loss. To improve the model's generalization ability, a variety of data augmentation techniques are employed during training, such as random cropping, color dithering, Gaussian blur, and specific enhancement methods that simulate moiré patterns and screen reflections.

[0132] This model architecture is typically lighter than face recognition models because it focuses more on detecting subtle forgery clues rather than identity verification. A typical deep liveness detection model might be based on MobileNetV3-Small or a custom-designed multi-layer CNN, taking a cropped and normalized face image (e.g., 128x128 or 256x256) as input. Its design emphasizes capturing the image's microscopic texture, color distortion, reflectivity, and temporal information in motion (for sequence models). For example, the model might include a module analyzing the image's frequency domain to detect moiré patterns caused by screen refresh rates, or a module analyzing the color space (e.g., YCrCb) to detect color deviations in printed photos.

[0133] When the resource scheduling score is high, requiring deep liveness detection, the face image of the current frame is fed into the model running on the NPU. The model's forward propagation process extracts liveness-related deep features and ultimately outputs a score between 0 and 1 through a sigmoid activation function. This score represents the probability that the input image is of a real person. The system compares this probability to a preset threshold (e.g., 0.95) determined on the validation set based on the equal error rate (EER). If the score is higher than the threshold, a "real person" verification result is generated; otherwise, a "fake" verification result is generated, thus ensuring the security of the face recognition results.

[0134] S308. When the resource scheduling score is less than the preset computing power activation threshold, the CPU unit is called to calculate the geometric displacement data of the current frame image relative to the previous frame image and generate a lightweight feature vector.

[0135] Refer to step S203, which will not be repeated here.

[0136] S309. When the resource scheduling score is greater than or equal to the computing power activation threshold, the NPU unit and CPU unit are invoked to extract the appearance texture features, geometric structure features and optical flow motion features of the current frame image in parallel to generate a multimodal feature set.

[0137] Refer to step S204, which will not be repeated here.

[0138] S310. Calculate the semantic distance between each modal feature vector in the multimodal feature set, and generate weighting coefficients based on the semantic distance and the gating network. Perform weighted fusion and dimensionality reduction on each modal feature vector to obtain a weighted feature vector.

[0139] Refer to step S205, which will not be repeated here.

[0140] S311. Perform vector retrieval and comparison based on lightweight feature vectors or weighted feature vectors to obtain the face recognition result at the current moment.

[0141] Refer to step S206, which will not be repeated here.

[0142] S312. Determine the similarity value corresponding to the face recognition result.

[0143] The face recognition result refers to the most similar registered identity ID found through vector comparison in step S311. The comparison similarity value is a quantified score calculated for this comparison, such as the cosine similarity value, which usually ranges from -1 to 1 or from 0 to 1. The higher the value, the more similar the two individuals are.

[0144] Specifically, after the facial recognition system performs vector retrieval and comparison, it obtains a "best match" identity and a corresponding similarity score. For example, after comparing the feature vector extracted from the current frame with the database, the system finds that it has the highest similarity (0.85) with the registered vector of the ID "Zhang San". Therefore, the facial recognition system will identify "Zhang San" as the result and specifically extract the value 0.85 as the similarity score for subsequent steps. This value is the direct basis for measuring the reliability of the recognition.

[0145] In some embodiments, this step can be implemented in several ways to obtain a more reliable similarity metric: Optionally, if an approximate nearest neighbor search is used, multiple candidate results and their similarities may be returned. The system can select the one with the highest score, or comprehensively evaluate the similarity of the top candidate results (e.g., check the difference between the highest and second-highest scores) to generate a more robust confidence metric; Optionally, the similarity value can be calibrated before outputting it. For example, a pre-trained calibration model can be used to map the original similarity score to a probability value, making it more consistent with the true confidence probability. It is understood that other distance or similarity metrics, such as the reciprocal of Euclidean distance, can also be used as comparison similarity values, which is not limited here.

[0146] In some embodiments, different individuals or different quality of registered photos may result in different baseline levels of similarity comparison. For example, A's photos may generally score higher than B's. This would make a uniform similarity threshold unfair to different people. To address this, an independent similarity distribution statistic (mean and variance) can be maintained for each registered identity in the database. When determining the similarity comparison value, the raw score is not used directly, but rather a Z-score normalized to the identity-specific distribution is used. This allows the similarity value to reflect the degree of matching relative to the individual's own baseline level, making it more fair and accurate.

[0147] S313. Update the cumulative confidence value in memory based on the comparison similarity value, and use it as feedback for resource demand assessment of the next frame image.

[0148] The cumulative confidence score in memory is the historical value read in step S301. Feedback refers to the fact that the updated confidence score will be used as one of the inputs for resource requirement assessment of the next frame image (i.e., step S307), thereby influencing future decisions.

[0149] Specifically, the face recognition system combines the similarity value S determined in step S312 with the cumulative confidence C_old of the target currently in memory to calculate a new cumulative confidence C_new. This update process typically employs a smoothing or filtering method, such as exponential moving average (EMA): C_new = α * S + (1 - α) * C_old, where α is a smoothing factor that controls the influence of the new observation (similarity S) on the cumulative value. After calculating C_new, the system can use this new value to overwrite the old value of the corresponding target in memory. Thus, when the next frame of the image arrives, the system reads this updated C_new, thereby forming a closed-loop feedback control.

[0150] In some embodiments, this step can be implemented in several ways to achieve a more intelligent update: Optionally, a simpler update rule can be used, such as slightly increasing the confidence score when the similarity S is above a certain threshold and significantly decreasing it when it is below. Optionally, the update rule can be non-linear; for example, when C_old is already high, even if S is also high, the increase in C_new should be smaller, and conversely, when C_old is low, a high S should allow C_new to have a large jump, which can be achieved using functions such as Sigmoid. It is understandable that other information, such as the number of consecutive successful recognitions, can also be combined to jointly determine how to update the cumulative confidence score; this is not limited here.

[0151] In some embodiments, if the smoothing factor α is set improperly, the cumulative confidence may become overly sensitive to noise (such as misidentification in a single frame) or unresponsive to real changes in state (such as character switching). To address this, the smoothing factor α itself can be dynamic. For example, when the similarity S differs significantly from C_old, it indicates a possible abrupt change; in this case, α should be increased to allow C_new to approach S more quickly. Conversely, when S and C_old are very close, it indicates a stable state; in this case, α should be decreased to enhance the smoothing effect and suppress noise. This adaptive smoothing factor allows confidence updates to be both stable and sensitive.

[0152] In this embodiment, by employing a resource scheduling scoring mechanism based on both image quality and historical cumulative confidence, as well as a priority ranking strategy for multi-target scenarios, the system can intelligently switch between low-power CPU lightweight processing and high-performance NPU multimodal processing according to the real-time value and necessity of each recognition task. It also prioritizes the allocation of limited NPU concurrent resources to the most critical targets, effectively solving the problems of wasted computing power (processing redundant information) and missed opportunities (abandoning key information) caused by relying solely on single-frame image quality in related technologies. Furthermore, it addresses the drawbacks of blind and inefficient resource allocation in multi-target scenarios. This enables refined, adaptive, and on-demand scheduling of heterogeneous computing resources on edge computing platforms such as RK3576, improving the system's energy efficiency, recognition robustness, and multi-task processing capabilities in non-cooperative face recognition scenarios.

[0153] The face recognition system in the embodiments of this invention is described below from the perspective of hardware processing. Please refer to [link / reference]. Figure 4 This is a schematic diagram of the physical device structure of a face recognition system in an embodiment of this application.

[0154] It should be noted that, Figure 4 The structure of the face recognition system shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0155] like Figure 4 As shown, the face recognition system includes a CPU 401, which can perform various appropriate actions and processes according to a program stored in ROM 402 or a program loaded into RAM 403 from storage section 408, such as executing the methods described in the above embodiments. RAM 403 also stores various programs and data required for system operation. The CPU 401, ROM 402, and RAM 403 are interconnected via bus 404. I / O interface 405 is also connected to bus 404.

[0156] The following components are connected to I / O interface 405: input section 406 including audio input devices, push-button switches, etc.; output section 407 including liquid crystal display (LCD) and audio output devices, indicator lights, etc.; storage section 408 including hard disks, etc.; and communication section 409 including network interface cards such as LAN (Local Area Network) cards, modems, etc. Communication section 409 performs communication processing via a network such as the Internet. Drive 410 is also connected to I / O interface 405 as needed. Removable media 411, such as disks, optical disks, magneto-optical disks, semiconductor memories, etc., are installed on drive 410 as needed so that computer programs read from them can be installed into storage section 408 as needed.

[0157] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing computer programs for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 409, and / or installed from removable medium 411. When the computer program is executed by CPU 401, it performs the various functions defined in the present invention.

[0158] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. Each block in a flowchart or block diagram may represent a module, program segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those shown in the drawings.

[0159] Specifically, the face recognition system in this embodiment includes a processor and a memory. The memory stores a computer program. When the computer program is executed by the processor, it implements the face recognition method based on RK3576 provided in the above embodiment.

[0160] In another aspect, the present invention also provides a computer-readable storage medium, which may be included in the face recognition system described in the above embodiments; or it may exist independently and not assembled into the face recognition system. The storage medium carries one or more computer programs, which, when executed by a processor of the face recognition system, cause the face recognition system to implement the RK3576-based face recognition method provided in the above embodiments.

[0161] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0162] As used in the above embodiments, depending on the context, the term "when..." can be interpreted as meaning "if...", "after...", "in response to determining...", or "in response to detecting...". Similarly, depending on the context, the phrase "when determining..." or "if (the stated condition or event) is interpreted as meaning "if determining...", "in response to determining...", "when (the stated condition or event) is detected", or "in response to detecting (the stated condition or event)".

Claims

1. A face recognition method based on RK3576, characterized in that, The RK3576 includes a CPU unit and an NPU unit. The face recognition method based on the RK3576 is applied to a face recognition system. The method includes: The current frame image in the video stream is obtained, and the cumulative confidence value of the current target object in the previous frame image is read from memory. Substitute the quality score of the current frame image and the cumulative confidence value into the resource requirement calculation formula to calculate the resource scheduling score that represents the necessity of feature extraction of the current frame. When the resource scheduling score is less than the preset computing power activation threshold, the CPU unit is invoked to calculate the geometric displacement data of the current frame image relative to the previous frame image, and a lightweight feature vector is generated. When the resource scheduling score is greater than or equal to the computing power activation threshold, the NPU unit and CPU unit are invoked to extract the appearance texture features, geometric structure features and optical flow motion features of the current frame image in parallel to generate a multimodal feature set. Calculate the semantic distance between each modal feature vector in the multimodal feature set, and generate weighting coefficients based on the semantic distance and the gating network. Perform weighted fusion and dimensionality reduction on each modal feature vector to obtain a weighted feature vector. Based on the lightweight feature vector or the weighted feature vector, a vector retrieval comparison is performed to obtain the face recognition result at the current moment.

2. The method according to claim 1, characterized in that, After the step of performing vector retrieval and comparison based on the lightweight feature vector or the weighted feature vector to obtain the face recognition result at the current time, the method further includes: Determine the similarity value corresponding to the face recognition result; The cumulative confidence score in memory is updated based on the comparison similarity score, serving as feedback for resource requirement assessment of the next frame image.

3. The method according to claim 1, characterized in that, Before the step of substituting the quality score of the current frame image and the cumulative confidence value into the resource requirement calculation formula to calculate the resource scheduling score characterizing the necessity of feature extraction for the current frame, the method further includes: Extract the current identity hash fingerprint from the facial region of the current target object in the current frame image; Read the identity hash fingerprint of the previous frame associated with the accumulated confidence value from memory; Calculate the difference between the current identity hash fingerprint and the previous frame identity hash fingerprint to obtain an identity continuity score; When the identity continuity score is lower than a preset continuity score threshold, the cumulative confidence score is reset to the initial confidence score.

4. The method according to claim 1, characterized in that, After the step of substituting the quality score of the current frame image and the cumulative confidence value into the resource requirement calculation formula to calculate the resource scheduling score characterizing the necessity of extracting the features of the current frame, the method further includes: When the resource scheduling score is greater than or equal to the computing power activation threshold, the NPU unit is invoked to extract the deep liveness features of the current frame image; When the resource scheduling score is less than the computing power activation threshold, the CPU unit is invoked to extract the shallow liveness features of the current frame image; A verification result for the face recognition result is generated based on the deep liveness feature or the shallow liveness feature.

5. The method according to claim 1, characterized in that, Before the step of substituting the quality score of the current frame image and the cumulative confidence value into the resource requirement calculation formula to calculate the resource scheduling score characterizing the necessity of feature extraction for the current frame, the method further includes: The number of face regions in the current frame image is detected to obtain a face count value; When the face count value is greater than the single face threshold, the independent cumulative confidence value corresponding to each face region is extracted to form a confidence array; The processing urgency score is obtained by multiplying the reciprocal of each value in the confidence array with the corresponding face region area. The urgency scores are sorted in descending order to obtain a processing priority sequence; Based on the upper limit of the concurrent processing capability of the NPU unit, the top N face regions are selected from the processing priority sequence as priority processing targets for recognition processing.

6. The method according to claim 5, characterized in that, After the step of selecting the top N face regions from the processing priority sequence as priority processing targets for recognition processing based on the upper limit of the concurrent processing capability of the NPU unit, the method further includes: Extract the row exposure timestamp of the original sensor data corresponding to the current frame image from the ISP cache; The temporal offset of the face region of each priority processing target during the sensor exposure process is calculated based on the row exposure timestamp; The temporal offset is converted into pixel coordinate correction values, and distortion correction is performed on the boundary coordinates of each face region to obtain the corrected face region. Calculate the spatial overlap between multiple corrected face regions, identify face image pairs with occlusion relationships, and determine the proportion of the visible area of ​​each face image in the face image pair; NPU processing time is redistributed according to the proportion of the visible area, so that face images with a larger visible area are given priority in processing; Generate an NPU task scheduler table containing processing allocation time and spatial coordinate correction information.

7. The method according to claim 6, characterized in that, After the step of generating the NPU task scheduler table containing processing allocation time and spatial coordinate correction information, the method further includes: Based on the processing allocation time in the NPU task scheduling table, calculate the expected completion time for each face feature extraction. Calculate the time difference between the expected completion time and the current frame acquisition time to obtain the predicted timing delay value; A target face image with a predicted time delay value greater than a preset synchronization threshold is identified, and the motion trajectory parameters of the target face image in the historical frame sequence are extracted. Based on the motion trajectory parameters, the predicted center coordinates of the target face image at the expected completion time are calculated; After the NPU unit completes feature extraction, the extracted feature vector is associated with the predicted center coordinates to compensate for the spatial position deviation caused by the computation delay.

8. A face recognition system, characterized in that, The face recognition system includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code including computer instructions, and the one or more processors call the computer instructions to cause the face recognition system to perform the method as described in any one of claims 1-7.

9. A computer-readable storage medium comprising instructions, characterized in that, When the instructions are executed on the face recognition system, the face recognition system performs the method as described in any one of claims 1-7.

10. A computer program product, characterized in that, When the computer program product is run on the face recognition system, it causes the face recognition system to perform the method as described in any one of claims 1-7.

Images