Automatic fare collection method and system based on multi-mode identity recognition

Through multimodal identity recognition technology, integrating palm vein, palm print, face, QR code and IC card, it solves the problems of poor user experience, insufficient security and limited expansion capabilities of traditional automatic ticket vending and checking systems, and achieves high accuracy, stability and scalability.

CN120726707APending Publication Date: 2025-09-30NINGBO YIKATONG TECHNOLOGY CO LTD +2
View PDF 0 Cites 5 Cited by

Patent Information

Application Number
CN202510830276.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-06-20
Publication Date
2025-09-30

AI Technical Summary

Technical Problem

Traditional rail transit automatic ticket vending and checking systems have poor user experience, insufficient security, and limited scalability. Biometric recognition technology lacks system-level integration deployment with IC cards, QR codes, and facial recognition in AFC scenarios.

Method used

It adopts a multimodal identity recognition method, integrating palm vein, palm print, face, QR code and IC card recognition methods, and realizes unified management and cloud-based collaborative management through multimodal data collection, feature extraction, environmental data processing, machine learning adaptive weight adjustment and DS evidence fusion.

Benefits of technology

It improves recognition accuracy and system stability, optimizes user experience and traffic efficiency, and has good scalability and adaptability to adapt to different user habits and environmental changes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN120726707A_ABST
    Figure CN120726707A_ABST
Patent Text Reader

Abstract

The invention discloses an automatic fare collection method and system based on multi-modal identity recognition, and relates to the technical field of rail transit automatic fare collection systems and multi-modal biological recognition fusion, and the method comprises the following steps: 1, enabling a user to approach a recognition area, carrying out the multi-modal data collection, and carrying out the quality detection and preprocessing; 2, feature extraction and credibility preliminary calculation are carried out; 3, environment data are collected and subjected to standardization processing; step 4, adaptively predicting a weight through machine learning, and constructing weight smoothing and time domain constraints; step 5, carrying out D-S evidence step-by-step fusion, and calculating comprehensive credibility; and step 6, comparing the comprehensive credibility with a preset threshold value, if the comprehensive credibility is greater than or equal to the preset threshold value, determining that the identification is successful and the gate is opened for passing, otherwise, determining that the credibility is insufficient, and performing alternative / manual processing. The system comprises a front-end identification terminal, a read-write industrial control all-in-one machine and a cloud management platform.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of integration of rail transit automatic ticket vending and checking systems and multimodal biometric identification, and in particular to an automatic ticket vending and checking method and system based on multimodal identity recognition. Background Art

[0002] With the rapid development of urban rail transit, the automatic fare collection (AFC) system has become a key link in passenger travel. However, traditional identity authentication methods (IC cards, QR codes, ID cards, etc.) still have the following shortcomings:

[0003] 1. User experience: The operation process is cumbersome, and different groups of people have different familiarity with scanning codes / swiping cards;

[0004] 2. Security and stability: A single identification method is susceptible to environmental interference or counterfeit attacks;

[0005] 3. In terms of expansion capabilities - equipment is usually developed in a split manner, with inconsistent interfaces, making it difficult to quickly upgrade and maintain.

[0006] Biometric recognition (especially contactless methods such as palm vein and palm print) has significant advantages in anti-counterfeiting, hygiene, and recognition accuracy. However, there is currently a lack of system-level solutions for integrating it with IC cards, QR codes, and facial recognition in AFC scenarios.

[0007] Therefore, technical personnel in this field are committed to developing a new AFC identification method and system that can be uniformly managed, multimodal fusion, support dynamic supplementary identification and cloud-based collaborative management to solve the above-mentioned shortcomings in traditional authentication methods. Summary of the Invention

[0008] In view of the above-mentioned defects of the prior art, the technical problem to be solved by the present invention is how to realize an automatic ticket vending and checking method and system with unified management, multimodal fusion, support for dynamic supplementary recognition and cloud-based collaborative management.

[0009] To achieve the above-mentioned object, the present invention provides an automatic ticket vending and checking method based on multimodal identity recognition, characterized in that the method comprises the following steps:

[0010] Step 1: The user approaches the recognition area, collects multimodal data, and performs quality inspection and preprocessing;

[0011] Step 2: Perform feature extraction and preliminary credibility calculation;

[0012] Step 3: Collect environmental data and perform standardization;

[0013] Step 4: Adaptively predict weights through machine learning and construct weight smoothing and time domain constraints;

[0014] Step 5: Perform level-by-level fusion of DS evidence and calculate the comprehensive credibility;

[0015] Step 6: Compare the comprehensive credibility with a preset threshold. If it is greater than or equal to the preset threshold, the identification is successful and the gate is opened for passage. Otherwise, the credibility is considered insufficient and alternative / manual processing is performed.

[0016] Furthermore, the step 1 includes the following sub-steps:

[0017] Step 1.1: When a user approaches the recognition area, the recognition trigger condition is determined based on the sensing signal, and the multimodal data collection path is determined based on the ticket type attributes and the modal scheduling strategy;

[0018] Step 1.2: The palm vein and face cameras enter the pre-acquisition monitoring state, start the image stream, and complete the preliminary image quality assessment;

[0019] Step 1.3: Determine whether to include the image in the recognition process based on pre-set factors. If the recognition process includes the image, perform image quality assessment and preprocessing. Otherwise, skip the biometric path and use only the primary modality recognition result. The pre-set factors include, but are not limited to: whether the current ticket type requires biometric recognition, the working status of the camera and module, whether the preliminary image quality score exceeds the threshold, and whether there is a corresponding historical biometric information binding record.

[0020] Furthermore, step 2 includes the following sub-steps:

[0021] Step 2.1: Extract palm print features and palm vein features, perform weighted fusion, and obtain the credibility score m of the fusion features related to the palm. palm (H):

[0022] m palm (H)=α p ·m vein (H)+β p ·m print (H), α p +β p =1

[0023] Among them, m vein (H) represents the credibility score of the palm vein feature, m print (H) represents the credibility score of the palmprint feature, α p , β p are the fusion weight coefficients corresponding to the palm vein feature and the palm print feature respectively;

[0024] Step 2.2: Extract the two-dimensional features and three-dimensional features of the face, and perform weighted fusion of the face features to obtain the credibility score m of the fusion features related to the face. face (H):

[0025] m face (H)=α f ·m 2d (H)+β f ·m 3d (H), α f +β f =1

[0026] Among them, m 2d (H) represents the basic credibility of the two-dimensional features of the face, m 3d (H) is the basic credibility of the three-dimensional features of the face, α f and β f is the fusion weight coefficient corresponding to the two-dimensional facial feature and the three-dimensional facial feature;

[0027] Step 2.3: Calculate the credibility m of the QR code feature based on the barcode contrast, scanning distance, and image clarity. QR (H), calculated as:

[0028] m QR (H)=γ1·Contrast+γ2·Distance+γ3·Clarity, γ1+γ2+γ3=1

[0029] Wherein, Contrast is the barcode contrast, Distance is the scanning distance, Clarity is the image clarity, γ1, γ2, and γ3 are the fusion weight coefficients corresponding to the barcode contrast, the scanning distance, and the image clarity;

[0030] Step 2.4: Calculate the credibility m of the IC card feature based on signal strength, reading speed, and error rate. IC (H), calculated as:

[0031] m IC (H)=δ1·Signal Strength+δ2·Read Speed+δ3·Error Rate,

[0032] δ1+δ2+δ3=1

[0033] Among them, Signal Strength is the signal strength, Read Speed ​​is the reading speed, ErrorRate is the error rate, and δ1, δ2 and δ3 are the fusion weight coefficients corresponding to the signal strength, the reading speed and the error rate.

[0034] Furthermore, step 3 includes the following sub-steps:

[0035] Step 3.1: Collect the environmental data, including the light sensor continuously measuring the ambient light intensity I L The temperature and humidity sensor measures the ambient temperature T and humidity RH in real time. The infrared humidity sensor focuses on monitoring the dryness or wetness D of the palm skin and normalizes it to the range of 0-1, where 0 represents completely dry and 1 represents completely wet. The historical confidence sensor records the recent historical success rate R of palm print and palm vein recognition. V and R P ;

[0036] Step 3.2: Upload the collected environmental data in real time to establish a first environmental state feature vector:

[0037] E=[I L ,T,RH,D,R V , R P ]

[0038] The first environmental state feature vector is normalized, and the processed first environmental state feature vector is expressed as:

[0039] E′=[I L ′, T′, RH′, D′, R V ′,R P ′]

[0040] Step 3.3: Collect indicators and environmental data related to face recognition, including using a light sensor to monitor the ambient light intensity and calculate the facial light uniformity to obtain the light condition index L; using a face pose estimation algorithm to calculate the face frontal pose confidence, that is, the face angle and pose index P; using a face occlusion detection model to evaluate the face occlusion ratio to form a face occlusion degree index O; using the Laplace operator to measure the clarity and contrast of the currently captured image to obtain the image clarity index C; and statistically analyzing the recent historical accuracy of face 2D and 3D recognition to obtain R. 2D and R 3D ; The collected indicators and environmental data related to face recognition together constitute a second environmental state feature vector:

[0041] F=[L,P,O,C,R 2D , R 3D ]

[0042] The second environmental state feature vector is normalized, and the processed second environmental state feature vector is expressed as:

[0043] F′=[L′,P′,O′,C′,R 2D ′,R 3D ′].

[0044] Furthermore, step 4 includes the following sub-steps:

[0045] Step 4.1: For the palm print feature and the palm vein feature, introduce a pre-trained deep neural network machine learning model, perform deep mapping processing on the first environmental state feature vector E′, and predict the optimal weight coefficient α in the current scene respectively. p , β p ;

[0046] Step 4.2: For the two-dimensional face features and the three-dimensional face features, introduce a pre-trained deep neural network machine learning model, perform deep mapping processing on the second environmental state feature vector F′, and predict the optimal weight coefficient α in the current scene. f , β f ;

[0047] Step 4.3: In terms of weight coefficient update constraints, α p , β p and α f , β f It is limited to the interval [0.3, 0.7]. At the same time, a time domain smoothing constraint mechanism, namely the sliding window mechanism, is introduced.

[0048] Furthermore, the sliding window smoothing method corresponding to the sliding window mechanism in step 4.3 is:

[0049] α t =λ·α model +(1-λ)·α t-1 , 0<λ<1

[0050] Among them, α model is the output of the model at the current moment, α t-1 is the weight value at the previous moment, and λ is the smoothing factor, which indicates the degree of memory of historical weights.

[0051] Furthermore, the step 5 includes the following sub-steps:

[0052] Step 5.1, preliminarily fusing the palm print feature and the palm vein feature to obtain fused palm features;

[0053] Step 5.2, fusing the fused palm features with the facial features;

[0054] Step 5.3: Integrate the QR code features and the IC card features to obtain a final identity recognition result, namely the comprehensive credibility.

[0055] Furthermore, in the process of stepwise fusion of the DS evidence in step 5, it is assumed that m i (H) represents the support of the i-th modal feature for the assumed identity match H, and the multimodal synthesis credibility is:

[0056]

[0057] The confidence m1(H) of the current main modality recognition result is fused with the current valid biometric recognition result or the biometric confidence m2(H) based on historical binding. If both modalities meet the synthesis rules of DS evidence, the final comprehensive confidence m(H) is obtained; if one of the modalities is unavailable or fails, it is automatically adjusted to simplify it to a single-modality decision.

[0058] The present invention also provides an automatic ticket vending and checking system based on multimodal identity recognition, which includes a front-end identification terminal, a reader-writer industrial control integrated machine and a cloud management platform, wherein the front-end identification terminal includes a palm print / palm vein module, a face recognition module, a QR code reading module, an IC card reading module, and an environmental sensor group; the reader-writer industrial control integrated machine is connected to the front-end identification terminal, and includes a feature cache, a DS evidence fusion decision module, and an exception handling module; the cloud management platform is connected to the reader-writer industrial control integrated machine, and includes a microservice architecture and an OTA upgrade channel.

[0059] Furthermore, the system also includes the automatic ticket vending and checking method based on multimodal identity recognition as described in any one of claims 1 to 8.

[0060] The present invention provides an automatic ticket vending and checking method and system based on multimodal identity recognition, which has at least the following technical effects:

[0061] 1. The technical solution provided by this invention can improve recognition accuracy. By combining the advantages of multiple features through multimodal fusion technology, it can reduce the impact of environmental interference on a single modality, such as maintaining accurate recognition under varying lighting conditions or hand stains. Furthermore, through comprehensive judgment based on DS evidence theory, the accuracy of identity authentication is improved, significantly reducing the false positive rate compared to traditional single-mode recognition methods.

[0062] 2. The technical solution provided by this invention enhances system reliability and stability. Through the close collaboration of various components within the integrated system architecture, front-end acquisition, industrial computer processing, and cloud management collaborate to ensure stable system operation. The high-reliability design of the all-in-one read / write industrial computer, its hardware watchdog and anti-collision mechanisms, and the cloud platform's device status monitoring and remote maintenance reduce the probability of system failures and ensure proper operation even in high-concurrency scenarios.

[0063] 3. The technical solution provided by this invention optimizes user experience and transit efficiency. By supporting multiple recognition methods, it adapts to different user habits and simplifies the operational process. Feature caching and a fast comparison mechanism, along with parallel task scheduling, accelerate recognition and reduce passenger wait times. Exception handling mechanisms and dynamic adjustment strategies ensure a smooth recognition process, improving overall transit efficiency and user satisfaction.

[0064] 4. The technical solution provided by this invention boasts excellent scalability and adaptability. Through its modular hardware and software architecture and cloud-based microservices design, it facilitates system expansion and upgrades. The cloud platform can flexibly issue identification strategies and perform software updates based on the needs of different scenarios, enabling the system to quickly adapt to changes in rail transit operations, such as changes in passenger flow and increased safety requirements.

[0065] The concept, specific structure and technical effects of the present invention will be further described below in conjunction with the accompanying drawings to fully understand the purpose, characteristics and effects of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 This is a flowchart of multimodal fusion of DS evidence theory in a preferred embodiment of the present invention;

[0067] Figure 2 This is a structural diagram of a weight coefficient adaptive adjustment model of a preferred embodiment of the present invention;

[0068] Figure 3 1 is a schematic diagram of the overall system architecture of a preferred embodiment of the present invention;

[0069] Figure 4 It is a timing diagram of the identification process and exception handling process of a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0070] The following describes several preferred embodiments of the present invention with reference to the accompanying drawings to make its technical content clearer and easier to understand. The present invention can be embodied in many different forms of embodiments, and the scope of protection of the present invention is not limited to the embodiments mentioned herein.

[0071] The embodiment of the present invention proposes an automatic ticket vending and checking method and system with multimodal identity fusion based on a unified industrial control platform, which integrates the deep fusion and integrated deployment of multiple identification methods such as palm vein, palm print, face, QR code, IC card, etc., and is suitable for rail transit entrance and exit identification control scenarios, improving the security and identification efficiency of rail transit entry and exit. Through a modular software and hardware architecture, it realizes image acquisition control, multimodal feature extraction and fusion, local rapid comparison and access control, cloud-based unified management and policy scheduling and other functions, and has the characteristics of strong robustness and good scalability.

[0072] Example 1

[0073] like Figure 1 As shown, an automatic ticket vending and checking method based on multimodal identity recognition provided by an embodiment of the present invention includes the following steps:

[0074] Step 1: The user approaches the recognition area, collects multimodal data, and performs quality inspection and preprocessing;

[0075] Step 2: Perform feature extraction and preliminary credibility calculation;

[0076] Step 3: Collect environmental data and perform standardization;

[0077] Step 4: Adaptively predict weights through machine learning and construct weight smoothing and time domain constraints;

[0078] Step 5: Perform level-by-level fusion of DS evidence and calculate the comprehensive credibility;

[0079] Step 6: Compare the comprehensive credibility with the pre-set threshold. If it is greater than or equal to the pre-set threshold, the recognition is successful and the gate is opened for passage. Otherwise, it is considered that the credibility is insufficient and alternative / manual processing is performed.

[0080] In particular, step 1 includes the following sub-steps:

[0081] Step 1.1: When the user approaches the recognition area, the recognition trigger condition is determined based on the sensing signal, and the multimodal data collection path is determined based on the ticket type attributes and modality scheduling strategy;

[0082] Step 1.2: The palm vein and face cameras enter the pre-acquisition monitoring state, start the image stream, and complete the preliminary image quality assessment;

[0083] Step 1.3: Determine whether to include the image in the recognition process based on pre-set factors. If the recognition process includes the image, perform image quality assessment and preprocessing. Otherwise, skip the biometric path and use only the primary modality recognition result. Pre-set factors include, but are not limited to: whether the current ticket type requires biometric recognition, the working status of the camera and module, whether the preliminary image quality score exceeds the threshold, and whether there is a corresponding historical biometric information binding record.

[0084] Specifically, when a user approaches the recognition area, the front-end terminal detects a passenger within a 50cm range using a time-of-flight (TOF) sensor or infrared human body sensor, automatically activating the camera and RFID reader, and entering recognition readiness. If the image quality assessment determines the image is valid, image preprocessing operations such as illumination balancing, denoising, and edge enhancement are performed. Otherwise, the biometric path is skipped, and only the primary modality recognition result is used.

[0085] In particular, a multi-factor fusion dynamic decision-making mechanism is used in step 1.1. After the user approaches the recognition area and triggers the recognition process, a comprehensive judgment is made based on the ticket type attributes and modality availability whether to activate high-security recognition paths such as palm vein and face, and whether to include the corresponding recognition results in the fusion process. This mechanism has dynamic perception and fault-tolerant judgment capabilities, and specifically includes the following three types of control logic:

[0086] 1) Ticket type configuration decision logic

[0087] After reading the IC card, QR code, or ID card information, the terminal extracts the corresponding identification strategy configuration items for that ticket type. For example, student tickets use a QR code and palm print and vein recognition as the identification path; senior citizen tickets require an IC card and facial recognition; and ordinary passengers can choose either mode. When the terminal reads the ticket type information (such as after successfully swiping an IC card or QR code), it immediately activates the required recognition mode according to the preset strategy and marks it as "Required / Optional / Not Participated."

[0088] 2) Modal usability evaluation mechanism

[0089] The hardware connection status and imaging quality of each recognition modality are detected in real time, and whether to include it in the recognition is determined as follows: If the device module is abnormal (such as the camera does not respond, the fill light fails, or the module is offline), the modality is directly marked as unavailable; if the device is normal but the image score does not reach the set threshold (such as the face score <0.7), it is considered "currently unavailable" and you can choose to request re-capture or switch to an alternative modality. If the modality score passes and the module status is normal, it is marked as available and enters the recognition process.

[0090] 3) Failure identification degradation mechanism

[0091] In actual applications, recognition failure may be caused by a variety of reasons. It is necessary to design a "degradation response mechanism" for different failure situations to ensure that recognition is not interrupted and users can pass normally. Typical scenarios include:

[0092] a) If modality recognition fails (e.g. palm vein matching is unsuccessful), the system will automatically switch to an alternative modality (e.g. QR code) or attempt facial recognition for auxiliary verification;

[0093] b) If the image cannot be acquired or is of low quality, the failure reason is recorded as "Invalid Image", the mode is skipped, and another path is tried;

[0094] c) If the recognition result has low confidence (does not exceed the threshold), if it is an optional modality, the system will continue to fusion judgement; if it is a required modality, the user will be prompted to try again;

[0095] d) If all modes fail, if the ticket type supports single-mode authentication (such as IC card), passage will be allowed but a risk warning will be recorded; if it is a real-name strong identity binding ticket type, an abnormal channel will be triggered (such as prompting manual verification or uploading an image to the platform background for asynchronous verification); if the recognition failure rate continuously exceeds the threshold, the system will enter "quick pass mode" or "alarm recording mode", and the platform will remotely intervene in configuration optimization.

[0096] In each failure tolerance chain, the identification path, failure mode, switching trajectory and final judgment result will be recorded as a reference for log uploading or training fusion model strategy.

[0097] Specifically, the system searches pre-stored information based on the collected multimodal data, matches the corresponding user ID based on the read IC card UID or QR code content, and retrieves the user's registered or long-term bound biometric template (such as palm print, palm vein, facial vector, etc.). If the current modality image is available, real-time feature extraction is initiated and participates in recognition; if the current modality image is invalid but historical features exist, their statistical features can be referenced in subsequent fusion for confidence inference compensation.

[0098] In particular, when two features conflict significantly (such as inconsistent palm print and facial recognition), the modality with higher confidence is prioritized for judgment. At the same time, the pre-stored information association is queried to obtain more relevant identity information for auxiliary judgment. For example, the associated IC card or QR code usage record is checked to determine whether the current recognition situation is consistent with the user's historical behavior pattern, further improving the accuracy of the judgment. If the identity still cannot be determined, the multimodal joint supplementary judgment mechanism is triggered, and the features of other modalities are combined for re-judgment.

[0099] In particular, the modal fusion weights are dynamically adjusted according to environmental changes (such as light, humidity, and hand status) to improve system robustness. The system monitors environmental changes in real time through sensors, combines pre-stored environment and modality association information, and uses an adaptive algorithm to dynamically adjust the modal fusion weights. For example, in a dimly lit environment, if the pre-stored information shows that the user's face and palm vein recognition effects are better in this environment, the system will appropriately increase the fusion weights of these two modalities to ensure recognition accuracy and stability. At the same time, the fusion weights are personalized according to the user's usage habits and historical recognition data to improve the system's adaptability to different users.

[0100] Example 2

[0101] In a multimodal identity recognition system, palm features (palmprint + palm vein), facial features, QR code features and IC card features exist simultaneously, and each feature has different credibility and discriminability.

[0102] Based on Example 1, step 2 includes the following sub-steps:

[0103] Step 2.1: Extract palm print features and palm vein features, perform weighted fusion, and obtain the credibility score m of the fusion features related to the palm. palm (H):

[0104] m palm (H)=α p ·m vein (H)+β p ·m print (H), α p +β p =1

[0105] Among them, m vein (H) represents the reliability score of palm vein feature, m print (H) represents the credibility score of the palmprint feature, α p , β p are the fusion weight coefficients corresponding to palm vein features and palm print features respectively;

[0106] Step 2.2: Extract the two-dimensional features and three-dimensional features of the face, and perform weighted fusion of the face features to obtain the credibility score m of the fusion features related to the face. face (H):

[0107] m face (H)=α f ·m 2d (H)+β f ·m 3d (H), α f +β f =1

[0108] Among them, m 2d (H) represents the basic credibility of the two-dimensional features of the face, m 3d (H) is the basic credibility of the three-dimensional features of the face, α f and β f is the fusion weight coefficient corresponding to the two-dimensional face features and the three-dimensional face features;

[0109] Step 2.3: Calculate the credibility m of the QR code feature based on the barcode contrast, scanning distance, and image clarity. QR (H), calculated as:

[0110] m QR (H)=γ1·Contrast+γ2·Distance+γ3·Clarity, γ1+γ2+γ3=1

[0111] Among them, Contrast is the barcode contrast, Distance is the scanning distance, Clarity is the image clarity, γ1, γ2 and γ3 are the fusion weight coefficients corresponding to the barcode contrast, scanning distance and image clarity;

[0112] Barcode contrast (Contrast) refers to the grayscale contrast of the barcode image (0-1 range); scanning distance (Distance) is the deviation between the normalized current scanning distance and the optimal scanning distance; image clarity (Clarity) is the clarity of the QR code assessed by the edge sharpness algorithm (0-1), and the weight of each factor is determined by a deep neural model and sliding window method;

[0113] Step 2.4: Calculate the credibility m of the IC card feature based on signal strength, reading speed, and error rate. IC (H), calculated as:

[0114] m IC (H)=δ1·Signal Strength+δ2·Read Speed+δ3·Error Rate,

[0115] δ1+δ2+δ3=1

[0116] Among them, SignalStrength is the signal strength, ReadSpeed ​​is the reading speed, ErrorRate is the error rate, δ1, δ2 and δ3 are the fusion weight coefficients corresponding to the signal strength, reading speed and error rate respectively;

[0117] By monitoring the signal and processing the data of the RFID module, parameters such as signal strength, reading speed and error rate are obtained in real time, and the corresponding weights are obtained according to the algorithm to calculate the credibility of the IC card features.

[0118] In particular, based on information association and pre-storage, feature fusion can rapidly acquire feature vectors and credibility information from different modalities. For example, when calculating the credibility of multimodal synthesis, the system accurately determines the support level of each modal feature for the identity matching hypothesis based on pre-stored association information and performs the fusion calculation according to the evidence synthesis rules of DS evidence theory. The feature caching and fast comparison mechanism involves pre-setting a user feature cache on the local embedded platform and using a local sensitive hashing (LSH) with a directional index structure to accelerate comparison. After a user successfully registers for the first time, the user's multimodal features (palmprint, palm vein, face, QR code, and IC card) are fused to generate a unified feature vector and stored in the local embedded platform's feature cache. Then, using a LSH directional index structure, the user's features are quickly located and retrieved, improving comparison efficiency. When user identity information is frequently used, that user's template is automatically cached first, achieving "quasi-whitelist" recognition optimization. By recording information such as user usage frequency and recognition success rate, the caching strategy is dynamically adjusted. For example, if a user frequently passes through a gate over a period of time and has a high recognition success rate, the system moves the user's feature template to a priority position in the cache. The next time a user is recognized, the user's features will be read from the cache for comparison, reducing recognition time and improving traffic efficiency. For high-traffic and high-concurrency scenarios, parallel comparison task scheduling and NPU local accelerated inference are supported.

[0119] Example 3

[0120] like Figure 2 As shown, based on Example 2, step 3 includes the following sub-steps:

[0121] Step 3.1: Collect environmental data, including the light sensor that continuously measures the ambient light intensity I L , whose measurement unit is Lux, accurately captures the changes in light intensity; the temperature and humidity sensor simultaneously measures the ambient temperature T (measured in degrees Celsius) and humidity RH (presented in percentage) in real time, providing accurate information about the ambient temperature and humidity; the infrared humidity sensor focuses on monitoring the dryness or wetness D of the palm skin contact and normalizes it to the range of 0-1, where 0 represents completely dry and 1 represents completely wet; the historical confidence sensor records the recent historical success rate R of palm print and palm vein recognition V and R P ;

[0122] Step 3.2: Upload the collected environmental data in real time and establish the first environmental state feature vector:

[0123] E=[I L ,T,RH,D,R V , R P ]

[0124] The first environmental state feature vector is normalized, and the processed first environmental state feature vector is expressed as:

[0125] E′=[I L ′, T′, RH′, D′, R V ′,R P ′]

[0126] Step 3.3: To achieve adaptive automatic adjustment of the facial feature fusion weight coefficient, a weight adaptive adjustment model based on facial image quality indicators and environmental information is used to obtain a series of indicators and environmental data related to face recognition in real time, including using a light sensor to monitor the ambient light intensity and calculate the facial light uniformity to obtain the light condition index L; using a face pose estimation algorithm to calculate the face frontal pose confidence, that is, the facial angle and pose index P; using a face occlusion detection model to evaluate the facial occlusion ratio to form a facial occlusion degree index O; using the Laplace operator to measure the clarity and contrast of the currently captured image to obtain the image clarity index C; and statistically analyzing the recent historical accuracy rates of face 2D and 3D recognition to obtain R. 2D and R 3D ; The collected indicators related to face recognition and environmental data together constitute the second environmental state feature vector:

[0127] F=[L,P,O,C,R 2D , R 3D ]

[0128] The second environmental state feature vector is normalized, and the processed second environmental state feature vector is expressed as:

[0129] F′=[L′,P′,O′,C′,R 2D ′,R 3D ′].

[0130] Step 4 includes the following sub-steps:

[0131] Step 4.1: For the palm print features and palm vein features, introduce the pre-trained deep neural network machine learning model, perform deep mapping processing on the first environmental state feature vector E′, and predict the optimal weight coefficient α in the current scene respectively. p , β p ;

[0132] The pre-trained deep neural network (DNN) machine learning model, a type of machine learning model with powerful nonlinear mapping capabilities, consists of multiple hidden layers, each containing a large number of neurons. Environmental information, such as normalized light intensity, temperature, humidity, dryness, and historical confidence levels for palm print and vein recognition, is fed into the DNN model's input layer as input data. Once the data enters the input layer, it is passed through the model layer by layer. Within each layer of neurons, the data undergoes complex mathematical operations. Specifically, the neurons perform a weighted summation of the input signals, with the weights continuously adjusted and optimized during the model training process.

[0133] For example, for the i-th neuron in a hidden layer, it receives the output signal x from the neurons in the previous layer. j , (j represents the number of the neuron in the previous layer), and the corresponding weight w ij , then the sum of the input signals received by the neuron is ∑ j w ij x j , and then, this sum will be processed by a ReLU (Rectified Linear Unit) activation function, whose expression is f(x) = max(0,x). The role of the activation function is to introduce nonlinear factors into the model. After being processed by the activation function, the neuron generates an output signal and continues to pass it to the next layer of neurons. As the data is continuously transmitted and processed in multiple hidden layers, the model gradually conducts in-depth mining and learning of the complex relationships in the environmental feature vectors. After deep processing of multiple hidden layers, the data finally reaches the output layer. The number of neurons in the output layer corresponds to the number of weight coefficients that need to be predicted. In the embodiment of the present invention, there are two neurons, corresponding to the weight coefficients α and β respectively. The training process of the entire model is based on supervised learning. During the training stage, a large amount of data on the correspondence between multiple sets of environmental factor features and palm print and palm vein recognition accuracy collected from laboratories or actual scenarios are input into the model. These data contain different environmental feature vectors E ′ and the corresponding known optimal weight coefficients α and β. The DNN model continuously adjusts the weights of the connections between internal neurons to minimize the error between the model's predicted weight coefficients and the actual known weight coefficients. The error is measured using the mean squared error loss function L(θ):

[0134]

[0135] Among them, y i is the actual weight coefficient value, is the weight coefficient value predicted by the model, θ represents all parameters (i.e., weights) that need to be adjusted in the DNN model, and n is the number of training data samples. Through the backpropagation algorithm, the DNN model can propagate the error of the loss function from the output layer to each hidden layer and input layer, thereby updating and optimizing the weights between neurons in each layer. After multiple iterative training, the model gradually learns how to accurately predict the optimal weight coefficients α and β based on the input environmental feature vector, thereby achieving adaptive adjustment to environmental changes and providing key support for the precise operation of subsequent systems.

[0136] Step 4.2: For the two-dimensional and three-dimensional features of the face, introduce the pre-trained deep neural network machine learning model to calculate the second environmental state feature vector F ′ Perform depth mapping processing and predict the optimal weight coefficient α for the current scene f , β f ;

[0137] Similarly, the deep neural network DNN model is applied to this link, and F ′ As input, the DNN model outputs the optimal weight coefficient α in real time f and β f The DNN model training data comes from the accuracy of 2D / 3D feature performance under different lighting conditions, facial postures, and occlusion levels. In terms of smoothing and constraints, a strategy similar to palm print and palm vein fusion is used to perform sliding window smoothing using historical weights and real-time weights to ensure stable system operation and avoid frequent and large fluctuations in weight coefficients.

[0138] Step 4.3: In terms of weight coefficient update constraints, α p , β p and α f , β f The system is constrained to the interval [0.3, 0.7]. A temporal smoothing constraint mechanism, also known as a sliding window mechanism, is also introduced. Although machine learning can generate adaptive weight coefficients, in practice, to ensure absolute system stability, these coefficients must adhere to a series of strict constraints. Regarding weight update constraints, α and β are each constrained to the interval [0.3, 0.7] during real-time adjustments. This interval constraint acts as a safety margin for weight adjustments, effectively preventing a single feature from dominating the weight calculation and avoiding imbalanced decisions caused by excessive weight bias. Furthermore, the system incorporates a temporal smoothing constraint mechanism, also known as a sliding window mechanism. This mechanism cleverly takes a weighted average of the model's real-time output weight coefficients and historical weight coefficients over a period of time. This sliding window smoothing method allows the weight coefficients to adapt to environmental changes while fully accounting for the impact of historical data, thus preventing system instability caused by sudden and drastic environmental changes.

[0139] In particular, the sliding window mechanism in step 4.3 corresponds to the sliding window smoothing method:

[0140] α t =λ·α model +(1-λ)·α t-1 ,0<λ<1

[0141] Among them, α model is the output of the model at the current moment, α t-1 is the weight value at the previous moment, and λ is a smoothing factor, indicating the degree of memory of historical weights. This rigorous set of constraints and smoothing mechanisms ensures high operational stability while ensuring adaptive adjustment, effectively avoiding frequent and drastic fluctuations in the weight coefficients and providing a solid foundation for the reliable operation of the entire system.

[0142] Example 4

[0143] Based on Example 3, step 5 includes the following sub-steps:

[0144] Step 5.1: Perform preliminary fusion of palm print features and palm vein features to obtain fused palm features;

[0145] Step 5.2, fuse the fused palm features with the facial features;

[0146] Step 5.3: Integrate the QR code features and IC card features to obtain the final identity recognition result, namely the comprehensive credibility.

[0147] By synthesizing the credibility of different modal features layer by layer, a comprehensive decision result is ultimately achieved. During this synthesis process, precise calculations are performed based on the credibility and weight of each modal feature, as well as the correlations between them. For example, when the credibility of palm print and palm vein features is high and they are closely correlated with other modal features, their weight in the comprehensive decision result is increased accordingly, thereby improving the accuracy of the final identity recognition.

[0148] In the process of step-by-step fusion of DS evidence in step 5, assuming that m i (H) represents the support of the i-th modal feature for the assumed identity match H, and the multimodal synthesis credibility is:

[0149]

[0150] The confidence m1(H) of the current main modality recognition result (such as the confidence of IC card or QR code recognition) is fused with the current valid biometric recognition result or the biometric confidence m2(H) based on historical binding. If both modalities meet the synthesis rules of DS evidence, the final comprehensive confidence m(H) is obtained; if one of the modalities is unavailable or fails (such as poor image quality or unbound features), it is automatically adjusted and simplified to a single-modality decision.

[0151] Among them, the Dempster-Shafer Theory (DST) can make weighted judgments on the fusion results of multiple independent features by reasoning about uncertainty and multi-source evidence. It has the following characteristics:

[0152] 1) Evidence Combination: The credibility of different feature sources is combined to generate a joint credibility. The system assigns different weights to the credibility of each modal feature based on its stability, uniqueness, and performance in different environments. For example, in an environment with stable lighting, the credibility weight of facial recognition will be relatively higher; while when hands are clean and the posture is stable, the credibility weight of palm print and palm vein features will be increased. In this way, the credibility of multiple feature sources is organically combined to form a more convincing joint credibility.

[0153] 2) Uncertainty Handling: A conflicting evidence suppression strategy is employed to address potential errors caused by a single modality. When conflicting recognition results from different modalities arise, the system does not simply deny a single result, but instead analyzes the cause of the conflict. For example, if the facial and palmprint recognition results are inconsistent, the system examines current environmental factors (such as whether lighting affects facial recognition or whether hand stains affect palmprint recognition). Combining historical recognition data with pre-stored information, the system determines which modality's result is more reliable, thereby minimizing the potential for conflicting evidence to interfere with the final decision.

[0154] 3) Step-by-step synthesis: Palm print, palm vein, face, QR code, and IC card features are gradually weighted and integrated to form the final judgment. During the fusion process, the system processes each modality feature according to a specific order and weight.

[0155] In step 6, if the comprehensive confidence level m(H) is ≥ 0.8, the recognition is successful, the gate is opened, and the recognition record is uploaded to the cloud. If the comprehensive confidence level m(H) is < 0.8, the user is prompted to switch to other modes for recognition or seek manual processing. The threshold is set based on a large amount of experimental data and actual application scenario tests, taking into account the recognition accuracy and error rate. For example, in scenarios with high security requirements, the threshold is appropriately increased to ensure the accuracy of identity authentication; while in scenarios with high traffic efficiency requirements, the threshold T is appropriately lowered to increase the recognition speed while ensuring a certain level of security.

[0156] Example 5

[0157] like Figure 3 As shown, an embodiment of the present invention provides an automatic ticket vending and checking system based on multimodal identity recognition, which integrates multiple identity authentication methods such as palm vein, palm print, face, QR code, and ID card, and constructs a complete link structure of "multimodal image perception → feature recognition fusion → local decision control → cloud-based collaborative management", including a front-end recognition terminal, a read-write industrial control all-in-one machine and a cloud-based management platform, which is widely applicable to rail transit entrance and exit control scenarios, wherein the front-end recognition terminal includes a palm print / palm vein module, a face recognition module, a QR code reading module, an IC card reading module, and an environmental sensor group; the read-write industrial control all-in-one machine is connected to the front-end recognition terminal, including a feature cache, a DS evidence fusion decision module, and an exception handling module; the cloud-based management platform is connected to the read-write industrial control all-in-one machine, including a microservice architecture and an OTA upgrade channel.

[0158] In particular, the automatic ticket vending and checking system based on multimodal identity recognition also includes the automatic ticket vending and checking method based on multimodal identity recognition included in any of the foregoing embodiments.

[0159] Specifically, the front-end identification terminal integrates a palm print and palm vein acquisition module (using NIR imaging technology), a facial recognition module (equipped with RGB and NIR cameras), a QR code scanner, a dedicated ID card recognition module, an IC card reader / writer, and environmental sensors. It supports contactless ISO / IEC14443 Type A and B cards. This terminal collects multimodal passenger information and, based on pre-stored association information, quickly identifies user data, providing foundational data support for subsequent identification processes.

[0160] In particular, the integrated reader / writer industrial control machine utilizes a domestically produced 64-bit ARM multi-core processor with a built-in NPU acceleration unit, offering powerful computing capabilities and compatibility with multi-protocol IC cards. Its modular design facilitates device maintenance and supports hardware watchdog, anti-collision mechanisms, and fast local comparison caching, providing stable computing power for multimodal recognition in high-concurrency scenarios.

[0161] Specifically, the cloud-based management platform utilizes a microservices architecture to enable unified management of front-end devices and user identity data. The platform captures identification logs and transaction records in real time for big data analysis and visual O&M, providing managers with a comprehensive understanding of system operations. Furthermore, the platform can issue identification policies (such as DS synthesis thresholds and environmental adaptation parameters) and perform over-the-air (OTA) software updates, ensuring the system adapts to diverse scenarios and maintains optimized performance.

[0162] In particular, Figure 4As shown in the figure, it is a timing diagram of the recognition process and exception handling process. When a pedestrian (user) approaches the terminal, the front-end recognition terminal starts multimodal data collection and uploads the acquired features to the reader-writer industrial control integrated machine. The reader-writer industrial control integrated machine requests the fusion strategy from the cloud management platform. The fusion strategy is returned from the cloud management platform to the reader-writer industrial control integrated machine. The reader-writer industrial control integrated machine performs DS fusion calculation and judgment to obtain the first recognition result and returns the first recognition result to the front-end recognition terminal. If the confidence level is greater than or equal to the threshold, the recognition is successful and the gate is opened for passage. The front-end recognition terminal prompts the pedestrian that the passage is successful. Otherwise, it is considered that the confidence level is less than the threshold, and the reader-writer industrial control integrated machine starts supplementary recognition. The front-end The identification terminal uploads supplementary data to the reader-writer industrial control integrated computer for secondary DS fusion calculation and judgment. The reader-writer industrial control integrated computer returns the supplementary identification result to the front-end identification terminal. If the secondary confidence level is ≥ the threshold, the identification is successful, the gate is opened for passage, and the pedestrian is prompted to pass successfully. Otherwise, it is considered that the secondary confidence level is < the threshold. The reader-writer industrial control integrated computer initiates cloud-based manual verification support to the cloud-based management platform. The cloud-based management platform performs manual verification and returns the manual verification results to the reader-writer industrial control integrated computer and the front-end identification terminal in turn. If the verification is passed, the front-end identification terminal prompts the pedestrian to pass successfully. Otherwise, the verification is rejected, and the front-end identification terminal prompts the pedestrian to pass rejected.

[0163] The preferred embodiments of the present invention have been described in detail above. It should be understood that numerous modifications and variations based on the concepts of the present invention are possible without inventive effort by those skilled in the art. Therefore, any technical solution that can be derived by one skilled in the art through logical analysis, reasoning, or limited experimentation based on the concepts of the present invention and the prior art should be within the scope of protection defined by the claims.

Claims

1. An automatic ticket vending and checking method based on multimodal identity recognition, characterized in that: The method comprises the following steps: Step 1: The user approaches the recognition area, collects multimodal data, and performs quality inspection and preprocessing; Step 2: Perform feature extraction and preliminary credibility calculation; Step 3: Collect environmental data and perform standardization; Step 4: Adaptively predict weights through machine learning and construct weight smoothing and time domain constraints; Step 5: Perform level-by-level fusion of DS evidence and calculate the comprehensive credibility; Step 6: Compare the comprehensive credibility with a preset threshold. If it is greater than or equal to the preset threshold, the identification is successful and the gate is opened for passage. Otherwise, the credibility is considered insufficient and alternative / manual processing is performed.

2. The automatic ticket vending and checking method based on multimodal identity recognition according to claim 1, characterized in that: The step 1 includes the following sub-steps: Step 1.1: When a user approaches the recognition area, the recognition trigger condition is determined based on the sensing signal, and the multimodal data collection path is determined based on the ticket type attributes and the modal scheduling strategy; Step 1.2: The palm vein and face cameras enter the pre-acquisition monitoring state, start the image stream, and complete the preliminary image quality assessment; Step 1.3: Determine whether to include the image in the recognition process based on pre-set factors. If the recognition process includes the image, perform image quality assessment and preprocessing. Otherwise, skip the biometric path and use only the primary modality recognition result. The pre-set factors include, but are not limited to: whether the current ticket type requires biometric recognition, the working status of the camera and module, whether the preliminary image quality score exceeds the threshold, and whether there is a corresponding historical biometric information binding record.

3. The automatic ticket vending and checking method based on multimodal identity recognition according to claim 1, characterized in that: The step 2 includes the following sub-steps: Step 2.1: Extract palm print features and palm vein features, perform weighted fusion, and obtain the credibility score m of the fusion features related to the palm. palm (H): m palm (H)=a p ·m vein (H)+β p ·m print (H),a p +b p =1 Among them, m vein (H) represents the credibility score of the palm vein feature, m print (H) represents the credibility score of the palmprint feature, α p , β p are the fusion weight coefficients corresponding to the palm vein feature and the palm print feature respectively; Step 2.2: Extract the two-dimensional features and three-dimensional features of the face, and perform weighted fusion of the face features to obtain the credibility score m of the fusion features related to the face. face (H): m face (H)=a f ·m 2d (H)+β f ·m 3d (H),a f +b f =1 Among them, m 2d (H) represents the basic credibility of the two-dimensional features of the face, m 3d (H) is the basic credibility of the three-dimensional features of the face, α f and β f is the fusion weight coefficient corresponding to the two-dimensional facial feature and the three-dimensional facial feature; Step 2.3: Calculate the credibility m of the QR code feature based on the barcode contrast, scanning distance, and image clarity. QR (H), calculated as: m QR (H)=γ1·Contrast+γ2·Distance+γ3·Clarity,γ1+γ2+γ3=1 Wherein, Contrast is the barcode contrast, Distance is the scanning distance, Clarity is the image clarity, γ1, γ2, and γ3 are the fusion weight coefficients corresponding to the barcode contrast, the scanning distance, and the image clarity; Step 2.4: Calculate the credibility m of the IC card feature based on signal strength, reading speed, and error rate. IC (H), calculated as: m IC (H)=δ1·Signal Strength+δ2·Rread SPeed+δ3·Error Rate, δ1+δ2+δ3=1 Among them, Signal Strength is the signal strength, Read Speed ​​is the reading speed, Error Rate is the error rate, and δ1, δ2 and δ3 are the fusion weight coefficients corresponding to the signal strength, the reading speed and the error rate.

4. The automatic ticket vending and checking method based on multimodal identity recognition according to claim 3, characterized in that: Step 3 includes the following sub-steps: Step 3.1: Collect the environmental data, including the light sensor continuously measuring the ambient light intensity I L The temperature and humidity sensor measures the ambient temperature T and humidity RH in real time. The infrared humidity sensor focuses on monitoring the dryness or wetness D of the palm skin and normalizes it to the range of 0-1, where 0 represents completely dry and 1 represents completely wet. The historical confidence sensor records the recent historical success rate R of palm print and palm vein recognition. V and R P ; Step 3.2: Upload the collected environmental data in real time to establish a first environmental state feature vector: E=[I L ,T,RH,D,R V ,R P ] The first environmental state feature vector is normalized, and the processed first environmental state feature vector is expressed as: E′=[I L ′,T′,RH′,D′,R V ′,R P ′] Step 3.3: Collect indicators and environmental data related to face recognition, including using a light sensor to monitor the ambient light intensity and calculate the facial light uniformity to obtain the light condition index L; using a face pose estimation algorithm to calculate the face frontal pose confidence, that is, the face angle and pose index P; using a face occlusion detection model to evaluate the face occlusion ratio to form a face occlusion degree index O; using the Laplace operator to measure the clarity and contrast of the currently captured image to obtain the image clarity index C; and statistically analyzing the recent historical accuracy of face 2D and 3D recognition to obtain R. 2D and R 3D ; The collected indicators and environmental data related to face recognition together constitute a second environmental state feature vector: F=[L,P,O,C,R 2D ,R 3D ] The second environmental state feature vector is normalized, and the processed second environmental state feature vector is expressed as: F′=[L′,P′,O′,C′,R 2D ′,T 3D ′]。 5. The automatic ticket vending and checking method based on multimodal identity recognition according to claim 4, characterized in that: The step 4 includes the following sub-steps: Step 4.1: For the palm print feature and the palm vein feature, introduce a pre-trained deep neural network machine learning model, perform deep mapping processing on the first environmental state feature vector E′, and predict the optimal weight coefficient α in the current scene respectively. p , β p ; Step 4.2: For the two-dimensional face features and the three-dimensional face features, introduce a pre-trained deep neural network machine learning model, perform deep mapping processing on the second environmental state feature vector F′, and predict the optimal weight coefficient α in the current scene. f , β f ; Step 4.3: In terms of weight coefficient update constraints, α p , β p and α f , β f It is limited to the interval [0.3, 0.7]. At the same time, a time domain smoothing constraint mechanism, namely the sliding window mechanism, is introduced.

6. The automatic ticket vending and checking method based on multimodal identity recognition according to claim 5, characterized in that: The sliding window smoothing method corresponding to the sliding window mechanism in step 4.3 is: a t =l·a model +(1-λ)·a t-1 ,0<λ<1 Among them, α model is the output of the model at the current moment, α t-1 is the weight value at the previous moment, and λ is the smoothing factor, which indicates the degree of memory of historical weights.

7. The automatic ticket vending and checking method based on multimodal identity recognition according to claim 5, characterized in that: The step 5 includes the following sub-steps: Step 5.1, preliminarily fusing the palm print feature and the palm vein feature to obtain fused palm features; Step 5.2, fusing the fused palm features with the facial features; Step 5.3: Integrate the QR code features and the IC card features to obtain a final identity recognition result, namely the comprehensive credibility.

8. The automatic ticket vending and checking method based on multimodal identity recognition according to claim 5, characterized in that: In the process of step 5 of DS evidence fusion, assuming that m i (H) represents the support of the i-th modal feature for the assumed identity match H, and the multimodal synthesis credibility is: The confidence m1(H) of the current main modality recognition result is fused with the current valid biometric recognition result or the biometric confidence m2(H) based on historical binding. If both modalities meet the synthesis rules of DS evidence, the final comprehensive confidence m(H) is obtained; if one of the modalities is unavailable or fails, it is automatically adjusted to simplify it to a single-modality decision.

9. An automatic ticket vending and checking system based on multimodal identity recognition, characterized in that: The system includes a front-end identification terminal, a reader-writer industrial control integrated machine and a cloud management platform, wherein the front-end identification terminal includes a palm print / palm vein module, a face recognition module, a QR code reading module, an IC card reading module, and an environmental sensor group; the reader-writer industrial control integrated machine is connected to the front-end identification terminal and includes a feature cache, a DS evidence fusion decision module, and an exception handling module; the cloud management platform is connected to the reader-writer industrial control integrated machine and includes a microservice architecture and an OTA upgrade channel.

10. The automatic ticket vending and checking system based on multimodal identity recognition according to claim 9, characterized in that: The system also includes the automatic ticket vending and checking method based on multimodal identity recognition as described in any one of claims 1 to 8.

Citation Information

Cited By

  • Subway BOM ticket business processing method and system based on artificial intelligence

    CN121767154A

  • Rail transit passage management and control method and read-write industrial control all-in-one machine adopting same

    CN121786430A

  • Rail transit access control method and read-write industrial control all-in-one machine adopting the same

    CN121786430B

  • An adaptive verification method for target identity in a dynamic environment

    CN122365464A

  • A target identity self-adaptive verification method in a dynamic environment

    CN122365464B