Face multi-modal high-precision identification and payment level two-dimensional code adaptive decoding and self-supervised evolution system and method for restricted computing power access terminal
Patent Information
- Application Number
- CN202611441215.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-16
- Publication Date
- 2026-10-30
AI Technical Summary
传统方案依赖死板的固定坐标硬映射，一旦硬件存在装配公差或在户外夏冬极端温差下产生热胀冷缩形变，算法截取的红外区域就会发生像素错位，导致频繁拒真与活体误判，工厂废品率居高不下；
1. 软件算法硬核自愈机械公差：通过面部关键点动态单应性矩阵逆变换，无需高成本的高精机械模具，由软件在 1 毫秒内自动抹平双目镜头 0.5°~2° 的装配公差与温漂形变，彻底根治量产装配偏差导致的活体拒真问题；
Smart Images
Figure CN123016503A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the fields of IoT smart hardware, edge computing, computer vision, biometric recognition and barcode image processing technology, and in particular relates to a high-precision multimodal face recognition and payment-grade QR code adaptive decoding and self-supervised evolution system and method for access control terminals with limited computing power. Background Technology
[0002] With the deep application of artificial intelligence in access control, facial recognition and QR code scanning have become the core functions of modern smart access control terminals. However, in current engineering mass production and actual community deployment, the following serious technical bottlenecks generally hinder industrial delivery and user experience: First, hardware assembly tolerances are difficult to eliminate in mass production: Access control terminals typically use binocular cameras (visible light RGB lens and near-infrared NIR lens) for liveness detection and recognition. In the large-scale mass production of cost-effective or micro-miniature modules, injection mold tolerances, SMT placement deviations, and differences in manual glue application inevitably result in a mechanical yaw angle of 0.5°~2° or a center displacement of a few pixels between the physical optical axes of the two lenses. Traditional solutions rely on rigid fixed coordinate hard mapping. Once there are assembly tolerances in the hardware or thermal expansion and contraction deformation under extreme temperature differences in summer and winter, pixel misalignment will occur in the infrared area captured by the algorithm, leading to frequent false rejections and false liveness detections, resulting in a high scrap rate in the factory. Second, the complex outdoor environment and the deterioration of the experience of reflective barcodes: Outdoor access control often faces strong backlighting, dark nights and high temperature drift. The existing model parameters are fixed and cannot be adaptively adjusted. At the same time, the mirror reflection of mobile phone screens is serious. When facing complex dynamic payment codes with high information density such as WeChat and Alipay, the existing general scanning libraries (such as ZBar) are prone to recognition lag or inability to decode due to the white spots caused by reflection, which seriously affects the efficiency of passage. Third, multi-person recognition is limited and computational resources are excessively high: Current mainstream embedded solutions (such as general-purpose visual AI chip solutions like Yizhi) blindly adopt 1024-dimensional high-dimensional floating-point or fixed-point vectors and strongly couple face detection with feature extraction, lacking lightweight trajectory tracking and lifecycle caching mechanisms. In practical applications, the appearance of a face in each frame forces full feature extraction, with single face feature extraction taking as long as 60-80 milliseconds. Once multiple people (such as 3-4 people) pass through the entrance / exit simultaneously, the total computational cost will instantly surge to nearly 300 milliseconds, causing the video frame rate to plummet to below 3 frames, resulting in screen stuttering, memory avalanche, and even watchdog restarts. Therefore, existing traditional SDKs can generally only degenerate into a single-person recognition mechanism that forcibly "captures only the largest face in the image," which is extremely limiting and easily causes congestion and queuing during peak travel periods. Fourth, the model is static and cannot adapt to evolve in complex physical environments: Traditional embedded models are completely static black boxes after being burned at the factory, and cannot be dynamically and adaptively optimized for specific entrances and exits as seasons, lighting, and user accessories change; the collection of difficult examples and model updates rely heavily on after-sales manual collection and offline manual annotation, with a long update cycle and extremely high human maintenance costs. Summary of the Invention
[0003] To overcome the shortcomings of existing technologies, this invention provides a system and method for high-precision multimodal facial recognition and payment-grade QR code adaptive decoding and self-supervised evolution for access control terminals with limited computing power. Through software algorithm self-healing of hardware assembly mechanical tolerances, shared lightweight backbone network, multi-target trajectory tracking and computing power segmentation scheduling, reflective filtering and cloud-edge collaborative seamless evolution, it breaks through the single-person recognition bottleneck of existing solutions and achieves an industrial-grade closed loop with default multi-person concurrent high-precision recognition, ultra-low resource consumption and ultra-fast passage under limited computing power.
[0004] The technical solution adopted in this invention is as follows: A high-precision multimodal face recognition and payment-grade QR code adaptive decoding and self-supervised evolution method for access control terminals with limited computing power includes the following steps: Step S201: The system supports dynamic configuration of single-person recognition working mode and multi-person simultaneous recognition working mode, and the default preset is multi-person simultaneous recognition mode; the binocular camera module collects environmental images of the entrance and exit channels. Due to hardware production assembly tolerances or environmental temperature deformation, there is a physical optical axis deviation between the visible light RGB lens and the near-infrared NIR lens of the binocular camera module. Step S202: The end-side lightweight multi-target feature detector locates multiple candidate face targets and key points of each target's face within the field of view in the RGB image and NIR image respectively, and solves the dynamic homography matrix between the two sets of key points in real time using the least squares method; Step S203: Perform subpixel-level perspective inverse transformation mapping on the NIR image based on the dynamic homography matrix to achieve real-time adaptive alignment of the RGB image and the NIR image at the spatial feature scale, thereby eliminating assembly tolerances; Step S204: Input the aligned multimodal image into a shared lightweight backbone network. The backbone network is simultaneously decoupled and split into a face feature extraction branch and a barcode detection branch. When the barcode detection branch detects a high-density QR code candidate region, Sauvola local gradient variance dynamic binarization and four-point sub-pixel perspective distortion correction are performed to filter out white spots reflecting the mirror surface of the mobile phone screen. High-density Reed-Solomon error correction decoding is performed in parallel to output voucher data. Step S205: In the multi-person concurrent recognition mode, the edge multi-target tracker assigns a unique trajectory tracking identifier to each candidate face target in the field of view, and constructs a feature extraction priority queue based on the face clarity quality score and spatial geometric distance; only for newly entered targets or targets with a jump in quality score is the feature extraction branch triggered to output a compact fixed-point quantized feature vector, and the comparison results are cached for a long time during the target's life cycle; at the same time, the onboard temperature sensor reads the hardware ambient temperature, dynamically loads the feature comparison confidence bias threshold of the corresponding temperature range, and performs 1:N ultra-fast feature retrieval in the local static memory base library, realizing millisecond-level seamless concurrent access for multiple people under limited computing power; Step S206: When successful passage is detected and the face comparison confidence is within the preset boundary range, the edge marks the frame image as a difficult sample and fine-tunes the local feature template using momentum decay weighting. The feature fingerprint is then desensitized, encrypted, and asynchronously uploaded to the cloud self-evolution engine. The cloud-based large model generates incremental differential weight patches through unsupervised pseudo-label knowledge distillation and silently hot-loads them back to the edge via OTA, achieving self-evolution with zero human intervention.
[0005] The beneficial effects of this invention include: 1. Software algorithm with hard-core self-healing mechanical tolerance: Through the inverse transformation of the dynamic homography matrix of facial key points, without the need for high-cost high-precision mechanical molds, the software automatically smooths out the assembly tolerance and temperature drift deformation of the binocular lens within 1 millisecond, completely eliminating the problem of liveness rejection caused by mass production assembly deviation. 2. Instant reading of complex and high-density QR codes at payment level: Through Sauvola local variance binarization and sub-pixel perspective correction, the reflective white spots on the mobile phone screen are completely eliminated. Even when faced with ultra-high density and complex payment codes such as WeChat and Alipay, the decoding time can be stably reduced to less than 25 milliseconds. 3. Approximates financial-grade accuracy with extremely low resources: Employing 256 / 512 / 1024-dimensional compact fixed-point quantization vectors, combined with a Large-Margin loss function, requiring only ≤15MB of resident memory, eliminating parasitic dependence on expensive dedicated NPUs, achieving a recognition rate >99.9% and a false recognition rate <0.0001%; 4.0 Human Intervention Cloud-Edge Self-Evolution: Combining strong fact-based self-supervised learning for access control, the equipment becomes more accurate with use, completely eliminating manual labeling and after-sales maintenance costs; 5. Breakthrough achievement of high-speed concurrent passage for multiple users under limited computing power: Through a lightweight multi-target trajectory tracker and lifecycle caching mechanism, "one-time extraction, long-term reuse throughout the entire lifecycle" is achieved. The peak computing power is evenly distributed across each clock cycle, and it can stably support 3 to 5 people to pass through simultaneously without being noticed on a micro embedded terminal with less than 0.5 TOPS computing power and ≤15MB of resident memory, completely breaking the industry's bottleneck of single-person capture. At the same time, it supports dynamic switching to single-person anti-tailgating mode in seconds according to management requirements, balancing extreme passage efficiency and high-level security control. Attached Figure Description
[0006] Figure 1 This is a diagram illustrating the full-link architecture of the multimodal adaptive recognition and self-supervised evolution system for access control terminals with limited computing power, as described in this invention. Figure 2 This is a flowchart of the binocular lens mechanical tolerance dynamic homography matrix straightening and registration algorithm of the present invention; Figure 3 This is a timing diagram of the parallel decoding of face and high-density QR codes via decoupling and diversion in the shared backbone network of this invention; Figure 4 This is a schematic diagram of the closed loop of end-side difficult momentum micro-update and cloud-based unsupervised distillation in this invention; Figure 5 This is a timing diagram for the concurrent identification of multiple targets and trajectory tracking and computing power partitioning scheduling of this invention. Detailed Implementation
[0007] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.
[0008] Example 1: Dynamic Straightening and Registration of Binocular Assembly Tolerances For a certain batch of access control terminals, due to injection molding and curing shrinkage, the NIR lens exhibits a 1.2° pitch tilt and an 8-pixel lateral offset relative to the RGB lens. When a user enters the recognition area, a lightweight SCRFD detector detects five key facial features (left eye, right eye, nose tip, left corner of mouth, and right corner of mouth) in both the RGB and NIR images. The algorithm solves an overdetermined linear equation system using two sets of corresponding points to calculate the homography transformation matrix H in real time. Subsequently, a hardware-accelerated inverse homography affine transformation instantaneously maps the NIR feature map to the RGB geometric coordinate system, with a coordinate alignment error of less than 0.3 pixels, restoring the liveness detection accuracy to 99.99%.
[0009] Example 2: High-density payment code with strong reflectivity and ultra-fast decoding When a user displays their Alipay payment code (containing an encrypted string of over 100 bytes with an extremely dense dot matrix) on their phone under the midday sun, the intense sunlight causes a large, speckled white spot to appear in the center of the phone's high-transparency glass screen. After the system's barcode branch locates the vertices of the QR code quadrilateral, it first applies a four-point perspective transformation to straighten the 45-degree tilted barcode into a square. Then, it calculates the grayscale variance within a 15x15 local sliding window and uses the Sauvola dynamic thresholding algorithm to identify the glaring white spot as a high-frequency DC component, which is then stripped away, accurately restoring the black positioning corner blocks and data code blocks obscured by the white spot. The entire process takes only 18ms on an ARM Cortex-A7 single-core processor, achieving "instant opening."
[0010] Example 3: Extreme High Temperature Drift Compensation and Cloud Edge Self-Evolution On a hot summer afternoon, the internal temperature of the device climbed to 68°C, significantly increasing the dark current thermal noise of the CMOS sensor. The onboard digital temperature sensor (I2C) read the temperature value, and the comparison engine automatically and dynamically shifted the cosine similarity threshold from 0.72 to 0.69 to compensate for the high thermal noise-induced high-frequency texture attenuation. Simultaneously, when a resident who frequently wears different sun hats passed by, the comparison confidence was 0.70 (falling into the 0.68-0.73 hard case range). Combining this with the subsequent physical fact of pressing the door to open, the device fused the new features currently obscured by the hat into the resident's local database template with 8% momentum. The next day, when the resident wore the same type of hat, the comparison confidence directly increased to 0.86, achieving completely imperceptible edge-side self-evolution.
[0011] Example 4: Simultaneous, high-speed concurrent identification of multiple users and computational power sharding scheduling During the morning rush hour, four residents (targets A, B, C, and D) entered the access control channel simultaneously. The access control terminal operated in the default multi-person concurrent recognition mode: the multi-target detector output four candidate bounding boxes simultaneously during a single forward inference, and the multi-target tracker assigned Tracker-IDs 101-104 in real time based on Kalman filtering and Hungarian matching. The priority scheduler calculated the target sharpness and area, prioritizing targets A and B for feature extraction in the first frame, each taking 16ms (using 256-dimensional fixed-point quantized vectors), and completing the comparison in the local database of 50,000 people within 3ms. Their authentication pass status was immediately bound to the lifecycle cache of Tracker-IDs 101 and 102. Subsequently, in the second frame, the scheduler scheduled the feature extraction and comparison of targets C and D in segments. During the subsequent 60 consecutive frames (approximately 2.5 seconds) of passage, for targets 101-104 already in the "authentication passed" cache state, the computationally intensive feature extraction branch was skipped, retaining only the extremely low-cost geometric trajectory tracking (<2ms per frame). The single-core CPU load remained stable below 28%, the video frame rate remained stable at 24 FPS, and the total gate opening delay was reduced to 110ms. Four people could pass side by side without having to stop and queue. When the property management system issued a switching command due to counter-terrorism or special anti-tailgating security requirements, the system instantly switched to single-person anti-tailgating mode in milliseconds, forcibly locking only the single face closest to the camera and with the largest area. Other targets triggered warnings and temporarily suspended gate opening, balancing ultra-high throughput with high-level security flexibility.
Claims
1. A method for high-precision multimodal face recognition and payment-grade QR code adaptive decoding and self-supervised evolution for access control terminals with limited computing power, characterized in that, Includes the following steps: Step S201: The system supports dynamic configuration of single-person recognition working mode and multi-person simultaneous recognition working mode, and the default preset is multi-person simultaneous recognition mode; the binocular camera module collects environmental images of the entrance and exit channels. Due to hardware production assembly tolerances or environmental temperature deformation, there is a physical optical axis deviation between the visible light RGB lens and the near-infrared NIR lens of the binocular camera module. Step S202: The end-side lightweight multi-target feature detector locates multiple candidate face targets and key points of each target's face within the field of view in the RGB image and NIR image respectively, and solves the dynamic homography matrix between the two sets of key points in real time using the least squares method; Step S203: Perform subpixel-level perspective inverse transformation mapping on the NIR image based on the dynamic homography matrix to achieve real-time adaptive alignment of the RGB image and the NIR image at the spatial feature scale, thereby eliminating assembly tolerances; Step S204: Input the aligned multimodal image into a shared lightweight backbone network. The backbone network is simultaneously decoupled and split into a face feature extraction branch and a barcode detection branch. When the barcode detection branch detects a high-density QR code candidate region, Sauvola local gradient variance dynamic binarization and four-point sub-pixel perspective distortion correction are performed to filter out white spots reflecting the mirror surface of the mobile phone screen. High-density Reed-Solomon error correction decoding is performed in parallel to output voucher data. Step S205: In the multi-person concurrent recognition mode, the edge multi-target tracker assigns a unique trajectory tracking identifier to each candidate face target in the field of view, and constructs a feature extraction priority queue based on the face clarity quality score and spatial geometric distance. The feature extraction branch is triggered only for newly entered targets or targets with a leap in quality score to output compact fixed-point quantized feature vectors, and the comparison results are cached for a long time during the target's life cycle; at the same time, the onboard temperature sensor reads the hardware ambient temperature, dynamically loads the feature comparison confidence bias threshold of the corresponding temperature range, and performs 1:N ultra-fast feature retrieval in the local static memory base library, enabling multiple people to pass through in milliseconds without any noticeable concurrency under limited computing power. Step S206: When successful passage is detected and the face comparison confidence is within the preset boundary range, the edge marks the frame image as a difficult sample and fine-tunes the local feature template using momentum decay weighting. The feature fingerprint is then desensitized, encrypted, and asynchronously uploaded to the cloud self-evolution engine. The cloud-based large model generates incremental differential weight patches through unsupervised pseudo-label knowledge distillation and silently hot-loads them back to the edge via OTA, achieving self-evolution with zero human intervention.
2. The face multimodal recognition and QR code adaptive decoding and self-supervised evolution method according to claim 1, characterized in that, The dynamic homography matrix calculation in steps S202 and S203 also incorporates factory-cured prior tolerances: In the equipment production line testing station, the factory baseline mechanical tolerance matrix is calculated by photographing the preset calibration target and then solidified and written into the onboard non-volatile memory area. During operation, the dynamic homography matrix and the factory reference mechanical tolerance matrix are fused using Kalman filtering on the end side to eliminate transient interference from human motion jitter on matrix estimation.
3. The face multimodal recognition and QR code adaptive decoding and self-supervised evolution method according to claim 1, characterized in that, The barcode detection branch in step S204 includes a super-resolution reconstruction mechanism: When the detected QR code is a WeChat or Alipay high-density dynamic payment code and the pixel size is smaller than the preset resolution threshold, the sub-pixel super-resolution reconstruction convolution operator is applied to the QR code candidate region for edge sharpening. The Sauvola Local Gradient Variance Dynamic Binarization dynamically determines the segmentation threshold based on the local grayscale mean and standard deviation within the sliding window, physically separating the strongly reflective area in the center of the mobile phone screen from the black and white dot matrix, with a decoding time of less than 25 milliseconds.
4. The face multimodal recognition and QR code adaptive decoding and self-supervised evolution method according to claim 1, characterized in that, In step S205, the compact fixed-point quantization feature vector is represented using a 256-dimensional, 512-dimensional, or 1024-dimensional fixed-point quantization vector. During the model training phase, a large-margin cosine loss function is introduced to enhance intra-class extreme compactness and inter-class strong exclusivity. The resident memory on the access control terminal is controlled within 15MB. Under single-core CPU operation, the retrieval latency for a 1:N database of 50,000 people is less than 5 milliseconds, the recognition accuracy is greater than 99.9%, and the false recognition rate is less than 0.0001%.
5. The face multimodal recognition and QR code adaptive decoding and self-supervised evolution method according to claim 1, characterized in that, The scheduling mechanism for the concurrent identification mode of multiple users in step S205 includes: The multi-target tracker is based on target trajectory association and lifecycle management between consecutive video frames; For targets that have completed 1:N retrieval and passed authentication, skip the extraction of duplicate features within the preset lifecycle time window and directly inherit the pass authentication status of the target; When multiple targets exceeding the preset concurrency threshold N (N≥3) simultaneously appear within the channel's field of view, the priority scheduler distributes the feature extraction computing power evenly across continuous clock cycle slices based on the face's geometric area and pose flatness. This limits the peak transient computing power of the edge-side single-core CPU and micro NPU to a safe level, supporting simultaneous, seamless concurrent verification for 3-5 people. Furthermore, in response to external security control commands, it supports dynamic downgrading from the multi-person concurrent recognition mode to a forced single-person anti-tailgating verification mode.
6. The face multimodal recognition and QR code adaptive decoding and self-supervised evolution method according to claim 1, characterized in that, The unsupervised pseudo-label knowledge distillation of the cloud-based large model in step S206 includes: The cloud-based self-evolution engine aggregates difficult case feature data packets transmitted back from tens of thousands of edge access control terminals across the network, and uses an unsupervised deep density clustering algorithm to automatically classify pseudo-label categories. The micro student model is guided by a large-parameter teacher model to perform knowledge distillation fine-tuning, generating differential incremental patches that only contain layers with changed convolutional kernels. These patches are then distributed to the edge access control terminal via OTA differential protocol for hot replacement of shared memory.
7. A high-precision multimodal face recognition and payment-grade QR code adaptive decoding and self-supervised evolution system for access control terminals with limited computing power, characterized in that... include: A binocular camera module, comprising a visible light RGB lens and a near-infrared (NIR) lens in a non-coaxial physical assembly state; Onboard hardware sensors, including ambient light sensors and temperature sensors; The edge computing main control chip integrates a shared lightweight backbone network, a multi-target trajectory tracker, a dynamic homography matrix straightening unit, a Sauvola local binarization scanning accelerator, a temperature adaptive threshold regulator, and a momentum self-supervised update module. The cloud-based self-evolving server is network-connected to the edge computing main control chip and includes a difficult example feature aggregation queue, an unsupervised clustering distillation hub, and an incremental differential patch generation service. The systems jointly and collaboratively execute the method described in any one of claims 1 to 6.
8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method as described in any one of claims 1 to 6.
9. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 6.