Gait and facial feature fusion recognition method and system of deep learning architecture

Abstract

The invention is suitable for the technical field of biological recognition, and provides a gait and facial feature fusion recognition method and system of a deep learning architecture, and the method comprises the following steps: monitoring a card swiping event of a subway gate, analyzing the ID information of a card, and generating a dynamic time window signal; collecting plantar pressure distribution and a face video of a card swiping person, and processing the plantar pressure distribution and the face video to obtain space-time aligned plantar pressure time sequence data and an anti-occlusion face image; reconstructing continuous gait phase features according to the foot pressure time sequence data, reversely restoring three-dimensional facial biological features in an extrusion state by using an anti-shielding facial image, and integrating to obtain a human body biological feature vector; and carrying out confidence evaluation on the human body biological feature vector and carrying out dynamic weighted fusion to obtain a multi-modal feature, and generating a final identity authentication result through a multi-round voting mechanism in a time window. According to the method, high-precision and low-false-recognition-rate fusion recognition is realized in a complex scene, and a stable and reliable solution is provided for intelligent management of the subway turnstile.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of biometric identification, in particular to a gait and facial feature fusion recognition method and system based on deep learning architecture. BACKGROUND

[0002] Gait and facial feature fusion recognition is a method that combines two biometric identification technologies. The fusion of this technology takes advantage of the characteristics of gait recognition that can still work effectively under long-distance and low-resolution conditions, and the advantages of facial recognition that provides high-precision recognition under close distances.

[0003] The current subway gate biometric identification system has certain technical defects. In the physical layer, the gait stagnation phenomenon that inevitably occurs when passengers swipe their cards causes the continuous gait sequence to break, and the feature extraction error rate of the existing gait recognition model based on time convolution network increases sharply, while the conventional method of using interpolation compensation will cause distortion of key motion features. At the same time, the complex mechanical action of the crowd in the gate channel during peak hours causes the face geometry to be distorted instantaneously beyond the processing capacity of traditional deformation models, which greatly reduces the practical value of face recognition in real-world scenarios.

[0004] More importantly, the mainstream fusion architecture usually assumes that gait and facial features remain continuous and synchronous in space and time, but in the actual gate scene, there is a significant time offset between the gait stagnation stage and the maximum face compression time, which can easily cause inter-modal conflicts. The average time from sensor acquisition to completion of recognition decision exceeds the acceptable transit time window of passengers, mainly because the hard real-time correlation mechanism between card swiping behavior, mechanical pressure changes and biometric feature acquisition has not been established, therefore, a gait and facial feature fusion recognition method and system based on deep learning architecture is proposed to solve the above problems. SUMMARY

[0005] In view of the deficiencies in the prior art, the purpose of the present application is to provide a gait and facial feature fusion recognition method and system based on deep learning architecture to solve the problems in the background art.

[0006] The present application is implemented as follows: a gait and facial feature fusion recognition method based on deep learning architecture, the method comprising the following steps:

[0007] Monitoring the card swiping event of the subway gate, analyzing the card ID information and generating a dynamic time window signal, the card swiping event being obtained by reading the traffic card signal through the card reader;

[0008] When the dynamic time window signal is acquired, the foot pressure distribution of the card swiping person and the face video are collected and processed to obtain the spatio-temporal aligned foot pressure time series data and the anti-occlusion face image, the foot pressure distribution is collected by the high-density array pressure sensor distributed in the gate passage, and the face video is collected by the multi-view camera device;

[0009] According to the foot pressure time series data, the continuous gait phase characteristics are reconstructed, the three-dimensional face biometric features in the extrusion state are inversely restored by using the anti-occlusion face image, and the standardized human biometric feature vector is obtained by integrated processing;

[0010] The human biometric feature vector is subjected to confidence evaluation and dynamic weighted fusion to obtain a multi-modal feature, and the final identity authentication result is generated through a multi-round voting mechanism within a time window and the gate passage is controlled.

[0011] As a further scheme of the application: the step of monitoring the card swiping event of the subway gate, analyzing the card ID information and generating a dynamic time window signal, specifically comprises:

[0012] The high-frequency carrier and the modulated data in the card swiping signal are separated by the card reader, the card ID information is obtained by decoding the modulated data by using an envelope detector;

[0013] Based on the card ID information, a real-time queuing density image is obtained by combining a convolutional neural network to predict the optimal collection time, and a dynamic time window instruction for variable biometric feature capture is generated, the real-time queuing density image is obtained by a camera at the top of the gate;

[0014] The dynamic time window instruction is converted into a nanosecond-level photoelectric synchronous signal, and is transmitted to the array pressure sensor and the multi-view camera device for controlling the accurate start-stop time sequence of the two;

[0015] The time overlap of the card swiping events of multiple channels is monitored, and the collection priority is dynamically allocated to ensure that the card swiping signal processing is conflict-free under high-density passenger flow.

[0016] As a further scheme of the application: the step of collecting the foot pressure distribution of the card swiping person and the face video and processing to obtain the spatio-temporal aligned foot pressure time series data and the anti-occlusion face image, specifically comprises:

[0017] The foot pressure distribution is sensed by the array pressure sensor in the gate passage, which is converted into a charge distribution map, and the foot pressure time series data is obtained after processing by a charge amplifier;

[0018] The anti-occlusion face image is collected by a multi-view camera device, and the multi-spectral image is output by fusing the face texture features and the subcutaneous blood vessel distribution through the multi-view camera device;

[0019] The foot pressure time sequence data and the anti-occlusion face image are compensated by time domain interpolation based on the vibration sensor data of the gate, to obtain synchronized bimodal data.

[0020] The output voltage of the array pressure sensor is monitored in real time, so that when the output voltage is saturated, the charge shunt protection circuit is triggered.

[0021] As a further scheme of the present application: the step of reconstructing continuous gait phase features according to foot pressure time sequence data, and inversely restoring three-dimensional face biometric features in the squeezed state by using anti-occlusion face images, and integrating processing to obtain a standardized human biometric feature vector, specifically includes:

[0022] Based on the foot pressure time sequence data, the gait cycle feature vector is obtained by using the second-order differential equation of the foot pressure trajectory to calculate the interrupted motion phase during gait arrest;

[0023] Based on the anti-occlusion face image, the corrected three-dimensional face topology is obtained by inversely calculating the face squeezing deformation by using the biomechanical elastic model;

[0024] Fusion of gait cycle feature vector and three-dimensional face topology, noise reduction processing by using conditional adversarial network;

[0025] The standardized human biometric feature vector is obtained by using the spatio-temporal pyramid network to extract features from the noise-reduced data.

[0026] As a further scheme of the present application: the step of obtaining the corrected three-dimensional face topology by inversely calculating the face squeezing deformation by using the biomechanical elastic model based on the anti-occlusion face image, specifically includes:

[0027] Combined with clinical data, the Young's modulus reference value is obtained by extracting the local elastic modulus based on the anti-occlusion face image, and the Young's modulus reference value is associated with age and gender;

[0028] The stress state of deep tissue is obtained by inversely calculating the squeezing force distribution by the displacement of the marker points of the anti-occlusion face image and the optical flow method, to obtain the spatio-temporal continuous mechanical sensing signal;

[0029] Based on the Young's modulus reference value and the mechanical sensing signal, the dynamic displacement field is obtained by using three-dimensional nonlinear wave equation and display integral algorithm;

[0030] The corrected three-dimensional face topology is generated by combining the dynamic displacement field with the initial model by using the adversarial generation network, and the initial model is the initial state of the three-dimensional face topology.

[0031] As a further scheme of the present application: the step of performing confidence evaluation on the human biological feature vector and dynamically weighting and fusing to obtain multi-modal features, and then generating a final identity authentication result through a multi-round voting mechanism within a time window, specifically comprises:

[0032] The real-time confidence weights of the gait and the face are calculated by analyzing the gait stagnation index and the face deformation entropy in the human biological feature vector;

[0033] The features of the gait and the face are dynamically aggregated using a gated attention mechanism to obtain fused multi-modal features;

[0034] Based on the weighted majority voting of the fusion decision of the multi-modal features within the time window, a preliminary identity authentication result is obtained;

[0035] When a decision conflict is detected, a final identity authentication result is output using a Bayesian inference engine.

[0036] Another object of the present application is to provide a gait and face feature fusion recognition system of a deep learning architecture, which comprises:

[0037] A signal response module is configured to monitor the card swiping event of the subway gate, analyze the card ID information and generate a dynamic time window signal, wherein the card swiping event is obtained by reading the traffic card through a card reader;

[0038] A micro-motion capture module is configured to, when the dynamic time window signal is acquired, collect the plantar pressure distribution and the face video of the card swiping person and process them to obtain spatiotemporally aligned foot pressure time series data and anti-occlusion face images, wherein the plantar pressure distribution is collected by a high-density array-type pressure sensor distributed in the passageway of the gate, and the face video is collected by a multi-view camera device;

[0039] A feature reconstruction module is configured to reconstruct continuous gait phase features according to the foot pressure time series data, and inversely restore three-dimensional face biological features in a squeezed state using the anti-occlusion face images, and integrate and process them to obtain a standardized human biological feature vector;

[0040] A dynamic decision module is configured to perform confidence evaluation on the human biological feature vector and dynamically weight and fuse to obtain multi-modal features, and then generate a final identity authentication result through a multi-round voting mechanism within a time window and control the gate passage.

[0041] As a further scheme of the present application: the signal response module comprises:

[0042] A signal processing unit is configured to separate the high-frequency carrier and the modulated data in the card swiping signal through the card reader, and decode the modulated data using an envelope detector to obtain the card ID information;

[0043] A dynamic window generation unit is configured to combine a real-time queuing density image obtained by a camera on top of a gate with card ID information, predict an optimal collection time length through a convolutional neural network, and generate a dynamic time window instruction for variable biometric feature capture;

[0044] A hardware trigger unit is configured to convert the dynamic time window instruction into a nanosecond-level photoelectric synchronization signal and transmit the signal to an array pressure sensor and a multi-view camera device to control the accurate start-stop timing of the two devices;

[0045] A conflict arbitration unit is configured to monitor the time overlap of multiple channel card swiping events and dynamically allocate collection priorities to ensure conflict-free card swiping signal processing under high-density passenger flow.

[0046] As a further scheme of the present application, the micro-motion capture module comprises:

[0047] A foot pressure sensing unit is configured to sense foot pressure distribution through an array pressure sensor in a gate passage, convert the foot pressure distribution into an electric charge distribution map, and obtain foot pressure timing data after processing by a charge amplifier;

[0048] A multi-spectral imaging unit is configured to collect an anti-shielding facial image through a multi-view camera device, and the multi-view camera device can output a fused multi-spectral image by collecting facial texture features and subcutaneous blood vessel distribution;

[0049] A data alignment unit is configured to perform time domain interpolation compensation on the foot pressure timing data and the anti-shielding facial image based on gate vibration sensor data to obtain synchronized bimodal data;

[0050] An overload protection unit is configured to monitor the output voltage of the array pressure sensor in real time, and trigger a charge shunt protection circuit when the output voltage is saturated.

[0051] As a further scheme of the present application, the feature reconstruction module comprises:

[0052] A gait phase field reconstruction unit is configured to calculate a continuous gait cycle feature vector by using a foot pressure trajectory second-order differential equation to calculate the interrupted motion phase during gait stagnation based on the foot pressure timing data;

[0053] A deformation mechanics solving unit is configured to calculate a corrected three-dimensional facial topological structure by using a biomechanical elastic model to reversely calculate facial extrusion deformation based on the anti-shielding facial image;

[0054] A multi-modal noise reduction unit is configured to fuse the gait cycle feature vector and the three-dimensional facial topological structure and perform noise reduction processing by using a conditional adversarial network;

[0055] The feature extraction unit is configured to extract a standardized human biological feature vector from the denoised data by using a spatio-temporal pyramid network.

[0056] Compared with the prior art, the application has the following beneficial effects:

[0057] The application can accurately capture the behavior timing of passengers in the gate channel by monitoring the card swiping event and generating a dynamic time window signal, solves the problem of continuous sequence breakage caused by gait stagnation in the traditional method, and avoids the distortion of motion features; after collecting the plantar pressure distribution of the high-density array pressure sensor and the anti-occlusion face image obtained by the multi-view camera device, the three-dimensional face biometric feature in the extrusion state is reversely restored, effectively overcoming the problem of instantaneous distortion of face geometry caused by crowd extrusion during peak hours. In addition, the application dynamically fuses multi-modal features and introduces a multi-round voting mechanism within the time window, not only alleviating the modal conflict caused by the spatio-temporal offset of the gait stagnation stage and the face extrusion moment, but also greatly shortening the time consumption from sensor collection to recognition decision through the linkage mechanism of hard real-time association of card swiping behavior, mechanical pressure change and biometric feature collection, ensuring that identity authentication is completed within the acceptable travel time window of passengers. In summary, the application realizes high-precision and low-misrecognition-rate fusion recognition in complex scenarios, and provides a stable and reliable solution for intelligent management of subway gates. BRIEF DESCRIPTION OF DRAWINGS

[0058] Figure 1 A flowchart of a gait and face feature fusion recognition method of a deep learning architecture.

[0059] Figure 2 A flowchart of generating a dynamic time window signal in a gait and face feature fusion recognition method of a deep learning architecture.

[0060] Figure 3 A flowchart of obtaining spatio-temporally aligned foot pressure timing data and anti-occlusion face images in a gait and face feature fusion recognition method of a deep learning architecture.

[0061] Figure 4 A flowchart of obtaining a standardized human biological feature vector in a gait and face feature fusion recognition method of a deep learning architecture.

[0062] Figure 5 A flowchart of obtaining a corrected three-dimensional face topology in a gait and face feature fusion recognition method of a deep learning architecture.

[0063] Figure 6 A flowchart of generating a final identity authentication result in a gait and face feature fusion recognition method of a deep learning architecture.

[0064] Figure 7 This is a schematic diagram of a gait and facial feature fusion recognition system based on a deep learning architecture.

[0065] Figure 8 This is a schematic diagram of the signal response module of a gait and facial feature fusion recognition system based on a deep learning architecture.

[0066] Figure 9 This is a schematic diagram of the micro-motion capture module of a gait and facial feature fusion recognition system based on a deep learning architecture.

[0067] Figure 10 This is a schematic diagram of the feature reconstruction module of a gait and facial feature fusion recognition system based on a deep learning architecture. Detailed Implementation

[0068] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0069] The specific implementation of the present invention will be described in detail below with reference to specific embodiments.

[0070] like Figure 1 As shown, this embodiment of the invention provides a gait and facial feature fusion recognition method based on a deep learning architecture, the method comprising the following steps:

[0071] S100 monitors card swiping events at subway turnstiles, parses card ID information, and generates dynamic time window signals. The card swiping events are obtained by reading the signal from the transportation card using a card reader.

[0072] S200, when the dynamic time window signal is acquired, the foot pressure distribution and facial video of the card swiper are collected and processed to obtain spatiotemporally aligned foot pressure time series data and anti-occlusion facial image. The foot pressure distribution is collected by a high-density array of pressure sensors distributed in the gate passage, and the facial video is collected by a multi-view camera device.

[0073] S300 reconstructs continuous gait phase features based on foot pressure time-series data, and uses anti-occlusion facial images to inversely restore three-dimensional facial biometrics under compression conditions, integrating and processing them to obtain standardized human biometric vectors.

[0074] The S400 performs confidence assessment on human biometric vectors and performs dynamic weighted fusion to obtain multimodal features. Then, it generates the final identity authentication result through a multi-round voting mechanism within a time window and controls the gate passage.

[0075] It should be noted that the dynamic time window is a time period adjusted in real time according to the card swiping event, which is used for synchronizing data acquisition and processing. For example, when the passenger swipes the card, the system triggers a time window (such as 10 seconds) during which the plantar pressure and facial video data are collected, ensuring that the data is strictly associated with the card swiping behavior. The array type pressure sensor is composed of an array of multiple high-density pressure sensors covering the gate passage area. Compared with the single-point sensor, the array type sensor can accurately capture the spatial distribution of the plantar pressure, supporting the reconstruction of the continuous gait phase characteristics.

[0076] In the embodiment of the present application, by monitoring the card swiping event and generating a dynamic time window signal, the behavior timing of the passenger in the gate passage can be accurately captured, the problem of continuous sequence breakage caused by gait stagnation in the traditional method is solved, and the distortion of motion features is avoided; after the plantar pressure distribution collected by the high-density array type pressure sensor and the anti-occlusion facial image obtained by the multi-view camera device, the three-dimensional facial biometric features in the extrusion state are reversely restored, effectively overcoming the problem of instantaneous distortion of facial geometry caused by crowd extrusion during peak hours. In addition, by dynamically weighting and fusing multi-modal features and introducing a multi-round voting mechanism within the time window, not only is the modal conflict caused by the spatio-temporal shift between the gait stagnation stage and the facial extrusion moment alleviated, but also the time-consuming from sensor collection to recognition decision is greatly shortened through the linkage mechanism of hard real-time association of card swiping behavior, mechanical pressure change and biometric feature collection, ensuring that identity authentication is completed within the passenger's acceptable passage time window. In summary, the present application realizes high-precision and low-misrecognition-rate fusion recognition in complex scenarios, providing a stable and reliable solution for the intelligent management of subway gates.

[0077] As shown in Figure 2 As a preferred embodiment of the present application, the step of monitoring the card swiping event of the subway gate, analyzing the card ID information and generating a dynamic time window signal specifically includes:

[0078] S101, separate the high-frequency carrier and the modulated data in the card swiping signal through the card reader, and decode the modulated data to obtain the card ID information using an envelope detector;

[0079] S102, based on the card ID information, combine the real-time queuing density image obtained by the camera at the top of the gate to predict the optimal collection time through a convolutional neural network, and generate a dynamic time window instruction for variable biometric feature capture;

[0080] S103, convert the dynamic time window instruction into a nanosecond-level photoelectric synchronous signal, and transmit it to the array type pressure sensor and the multi-view camera device for controlling the accurate start and stop timing of the two;

[0081] S104, monitor the time overlap of multiple channel card swiping events, and dynamically allocate collection priorities to ensure conflict-free card swiping signal processing under high-density passenger flow.

[0082] In the embodiment of the present application, the card reader extracts card ID information by separating high-frequency carrier and modulated data using an envelope detector. This process ensures fast analysis of card swiping signals. Combined with real-time queuing density images captured by the top camera of the gate, the optimal collection time is predicted by a convolutional neural network (CNN) (for example, the CNN may predict that the time window needs to be shortened to 3 seconds when the queue is dense during peak hours, and extended to 5 seconds during off-peak hours), and a variable biometric capture time window instruction is dynamically generated to ensure that the collection time and passenger flow state are adaptively matched. After the dynamic time window instruction is converted into a nanosecond-level photoelectric synchronization signal, it accurately controls the start and stop timing of the array pressure sensor and multi-view camera equipment. The system monitors the time overlap of multiple channel card swiping events, and uses a dynamic priority allocation algorithm (such as timestamp sorting or queue scheduling strategy) to handle conflicts under high-density passenger flow (for example, when two passengers swipe their cards at the same time, the system prioritizes processing the signal of the passenger closer to the gate to avoid data overlap or processing delay. This method not only solves the biometric capture distortion problem caused by gait stagnation or crowd compression in traditional gates, but also ensures efficient and conflict-free passage during peak hours through nanosecond-level synchronization signals and dynamic priority allocation, thereby significantly improving the real-time performance and reliability of the gate system.

[0083] As shown in Figure 3 As a preferred embodiment of the present application, the step of collecting the foot pressure distribution of the card swiping person and the face video and processing to obtain spatiotemporally aligned foot pressure time series data and anti-occlusion face images, specifically includes:

[0084] S201, sensing the foot pressure distribution through the array pressure sensor in the gate passage, converting it into a charge distribution map, and obtaining foot pressure time series data after processing by a charge amplifier;

[0085] S202, collecting anti-occlusion face images through multi-view camera equipment, which can output fused multi-spectral images by collecting facial texture features and subcutaneous blood vessel distribution;

[0086] S203, performing time domain interpolation compensation on the foot pressure time series data and anti-occlusion face images based on gate vibration sensor data to obtain synchronized bimodal data;

[0087] S204, monitoring the output voltage of the array pressure sensor in real time, so that when the output voltage is saturated, a charge shunt protection circuit is triggered.

[0088] In the embodiment of the present application, the array pressure sensor in the passageway of the gate machine converts the passenger foot pressure distribution into an electric charge distribution map through piezoelectric effect or piezoresistive effect. For example, when the passenger walks, the pressure changes in different areas of the foot will cause the deformation of nanowires or piezoresistive resistors in the sensor, generating an electric charge signal proportional to the pressure. These electric charge signals are then processed by a charge amplifier, the cable capacitance interference is reduced through the Miller effect, and low-impedance foot pressure time sequence data is output, so as to accurately capture the key phase characteristics such as touch and swing in the gait cycle. At the same time, the multi-view camera device generates an anti-shielding multi-spectral face image through the cooperative collection of visible light and near-infrared light spectrum. When the passenger may be partially face-shielded due to hair, mask or side face posture, the multi-view camera can jointly capture the face texture (such as skin texture under visible light) and subcutaneous blood vessel distribution (such as blood oxygen signal under infrared light) under different wavelengths, and generate a high-confidence face feature vector by using the multi-spectral image fusion algorithm mentioned in document [9] to complete the shielding area. In order to realize the space-time alignment of the foot pressure time sequence data and the face image, a gate vibration sensor (such as an accelerometer) is needed to monitor the mechanical vibration signal in real time, but it needs to be combined with the frequency domain zero padding time domain interpolation method to compensate for the time sequence deviation caused by the difference in sensor sampling rate or data loss. For example, if the pressure sensor samples at 100 Hz and the camera samples at 30 Hz, the vibration sensor data can be used as a reference signal to generate foot pressure data for the intermediate frame through the vibration amplitude interpolation of adjacent time points, so that the two data are strictly synchronized on the time axis. In addition, by monitoring the output voltage of the array pressure sensor in real time, the charge shunt protection circuit is triggered when the voltage is close to saturation. For example, when the passenger's body weight is too large to cause the output voltage of the sensor to approach the maximum dynamic range of the charge amplifier, the MOSFET in the shunt protection circuit will automatically conduct, releasing the excess charge through a low-impedance path to prevent the sensor from being damaged due to overload. This mechanism ensures the long-term stable operation of the sensor under high-density passenger flow, avoiding the recognition interruption of the whole system caused by single-point failure. In summary, through the cooperative action of charge signal processing, multi-spectral image fusion, time domain interpolation compensation and charge protection circuit, the technical bottleneck of the traditional method in the problems of shielding, space-time asynchronization and hardware overload is solved, and a high-accuracy solution is provided for the intelligent feature collection of the subway gate.

[0089] As Figure 4 shown, as a preferred embodiment of the present application, the step of reconstructing continuous gait phase characteristics according to foot pressure time sequence data, using anti-shielding face image to inversely restore three-dimensional face biometric features under extrusion state, and integrating processing to obtain standardized human biometric feature vector, specifically includes:

[0090] S301, based on the foot pressure time sequence data, using the second-order differential equation of the foot pressure trajectory to calculate the continuous gait cycle feature vector of the interrupted motion phase during gait stagnation;

[0091] S302, based on the anti-occlusion face image, the biomechanical elastic model is used to reversely calculate face extrusion deformation to obtain a corrected three-dimensional face topology structure;

[0092] S303, the gait cycle feature vector and the three-dimensional face topology structure are fused, and a conditional adversarial network is used for noise reduction processing;

[0093] S304, a spatio-temporal pyramid network is used for feature extraction on the noise-reduced data to obtain a standardized human biological feature vector.

[0094] In the embodiment of the application, based on the foot pressure time sequence data, the system uses the second-order differential equation of the foot pressure trajectory (uses the second-order differential equation to describe and predict the movement trajectory of the foot pressure center in the walking process) to calculate the interrupted movement phase during gait stagnation. For example, when the passenger temporarily stops due to the congestion of the gate, the second-order differential equation can predict the phase continuity of the interruption stage through the dynamic relationship of foot pressure acceleration and speed (such as formula wherein represents the second-order derivative of position with respect to time, F(t) is the foot pressure distribution function, and k is the proportional constant) to reconstruct the discrete foot pressure signal into a continuous gait cycle feature vector, which contains parameters such as step frequency and step length. The anti-occlusion face image (such as the multi-view modeling technology of Wild2Avatar mentioned in document [2]) reversely calculates the face extrusion deformation through the biomechanical elastic model (such as the nonlinear elastic equation σ=E·ε, wherein σ is stress, E is Young's modulus, and ε is strain). For example, due to the face geometric distortion caused by the passenger flow extrusion in the peak period, the system inputs the multi-view multispectral image (such as visible light texture and near-infrared blood vessel distribution) into the elastic model to simulate the reverse deformation of the deep tissue under stress, and finally corrects the standard three-dimensional face topology structure. Then, the fusion of the gait cycle feature vector and the three-dimensional face topology structure can use the conditional adversarial network for noise reduction processing. For example, the original data may be distorted due to sensor noise or light interference, and the conditional adversarial network evaluates the data authenticity through the discriminator, and the generator generates more robust feature representation using the complementary information of gait and face. The spatio-temporal pyramid network performs multi-granularity analysis on the noise-reduced data: in the time dimension, the time sequence dependence (such as the step frequency change rule) of the gait cycle is extracted through the sliding window; in the spatial dimension, the spatial correlation of the face key points (such as the nose tip and eye distance) and the foot pressure hot area is captured through the pyramid convolution kernel, and finally the standardized human biological feature vector is output. Therefore, in this process, not only the limitations of traditional methods in occlusion and movement interruption are solved, but also the physical modeling and deep learning are optimized in cooperation to achieve high-confidence identity authentication in complex scenarios.

[0095] For example, Figure 5As shown, as a preferred embodiment of the present application, the step of obtaining the corrected three-dimensional facial topology based on the anti-occlusion facial image using the biomechanical elastic model to reversely calculate the facial extrusion deformation specifically comprises:

[0096] S312, in combination with clinical data, a Young's modulus reference value is obtained based on the anti-occlusion facial image extraction local elastic modulus, the Young's modulus reference value is associated with age and gender;

[0097] S322, the displacement of the marker point of the anti-occlusion facial image is calculated by the optical flow method to reversely deduce the stress state of the deep tissue to obtain a time-space continuous mechanical sensing signal;

[0098] S332, based on the Young's modulus reference value and the mechanical sensing signal, a dynamic displacement field is obtained by a three-dimensional nonlinear wave equation and a display integral algorithm;

[0099] S342, the dynamic displacement field is combined with the initial model by using the generative adversarial network to generate the corrected three-dimensional facial topology, and the initial model is the initial state of the three-dimensional facial topology.

[0100] In the embodiment of the present application, in order to solve the problem of facial deformation distortion caused by crowd pushing and shoving in high-density scenes such as subway gates. Combined with clinical data (such as skin and subcutaneous tissue hardness measurement values of different age groups and gender groups in medical image research), the system extracts the local deformation features of the key areas (such as the cheekbone, the nose wing, and the lower jaw) from the anti-occlusion facial image, and estimates the Young's modulus reference value associated with the individual's age and gender. For example, the facial tissue of young men is usually more compact, and its Young's modulus (a physical quantity that measures the stiffness of a material) may be set to a higher value, while older women may have a lower value due to collagen loss. This reference value provides personalized material parameters for subsequent mechanical modeling. Using marker point displacement analysis and optical flow tracking of the small movement trajectory of the facial surface during the extrusion process, by analyzing the gradient and direction of these displacements, the extrusion pressure distribution acting on the facial surface is reversely deduced, and the stress state of the deep tissue such as subcutaneous muscle and fat is further deduced by combining the principle of elastic mechanics, forming a time-space continuous mechanical sensing signal evolving with time, based on the known Young's modulus reference value and the calculated mechanical sensing signal, a three-dimensional nonlinear wave equation (partial differential equation describing the propagation of stress waves in elastic medium, such as ρ represents the density of the human facial tissue; is the second-order partial derivative of the displacement field with respect to time, representing the acceleration of a point in the medium; is the divergence of stress tensor s; f is the external force density per unit volume, in the face extrusion scene, f is provided by the extrusion force distribution backstepping by the optical flow method, namely the time and space continuous mechanical sensing signal) to simulate the complex deformation behavior of soft tissue under non-uniform extrusion force, the equation is solved by numerical method through explicit integration algorithm (such as central difference method), and the dynamic displacement field of each voxel (3D pixel) of the face in the extrusion process is calculated step by step, that is, the moving path of each point in three-dimensional space. In order to perfectly fuse the displacement field of this physical simulation with the real face appearance, and handle the possible slight errors or unnaturalness in the simulation, at this time, the generative adversarial network (GAN) is introduced, the calculated dynamic displacement field is input as a condition, and an initial three-dimensional face model is acted on. The generator of GAN draws a corrected three-dimensional structure which conforms to the physical law and is highly realistic, and the discriminator constantly evaluates the authenticity of the generated result, and finally outputs a corrected three-dimensional face topology. For example, when the face of the passenger is locally depressed due to the extrusion of the backpack of the passenger beside, the system can backstep the internal force according to the surface deformation, simulate the tissue rebound by using the personalized Young's modulus, and generate a high-fidelity three-dimensional model which removes the extrusion artifacts and restores the real face contour through GAN, so as to provide a reliable basis for subsequent identity recognition.

[0101] As shown in Figure 6 , as a preferred embodiment of the present application, the step of performing confidence evaluation on the human biological feature vector and dynamically weighting fusion to obtain multi-modal features, and then generating the final identity authentication result through the multi-round voting mechanism in the time window, specifically includes:

[0102] S401, calculating the real-time confidence weight of gait and face by analyzing the gait stagnation index and face deformation entropy in the human biological feature vector;

[0103] S402, dynamically aggregating the features of gait and face by using the gated attention mechanism to obtain the fused multi-modal features;

[0104] S403, obtaining the preliminary identity authentication result based on the weighted majority voting of the fusion decision of multi-modal features in the time window;

[0105] S404, when detecting decision conflict, outputting the final identity authentication result by using the Bayesian inference engine.

[0106] In the embodiment of the present application, the real-time confidence weight is calculated by analyzing the gait stagnation index and the face deformation entropy. For example, when the passenger suddenly stops due to the congestion of the gate, the gait signal may be abnormal, and the reliability thereof is quantified by calculating the gait stagnation index; if the face is locally depressed or the texture is distorted due to the pressing of the adjacent passenger, the system evaluates the distortion degree thereof by the face deformation entropy. The real-time confidence weight of the gait and the face modal is determined by the two indexes. The features of the gait and the face are dynamically aggregated by using the gated attention mechanism. The gating mechanism (such as the gating unit in LSTM or GRU) allocates the attention degree of different modal according to the current confidence weight. For example, when the face weight is high, the system enhances the extraction accuracy of the face features, and at the same time, suppresses the part of the gait features disturbed by the stagnation. The preliminary identity authentication result is generated based on the weighted majority voting of multiple rounds in the time window. The system divides the multi-modal features in the continuous several seconds into a time window (such as 100 frames in 5 seconds), and performs identity matching (such as similarity calculation with the database template) on the weighted fusion features of each frame. For example, in the time window, the gait modal may match to user A with a probability of 70% (weight 20%), and the face modal matches to user A with a probability of 90% (weight 80%), and the weighted total probability is 0.7*0.2+0.9*0.8=0.86. The system will count the weighted probability of all frames in the time window, and if it exceeds the threshold (such as 85%) and most frames point to the same user, the preliminary authentication result is output. For the decision conflict problem, for example, the gait matches to user A, but the face matches to user B, at this time, the Bayesian inference engine is started, and the final decision is corrected combined with the user historical behavior prior probability (such as the frequency of user A using the gate in the time period is greater than user B) and the current modal weight. It should be noted that the Bayesian inference engine is a conventional technical means for solving uncertainty problems, and is excellent in the scene of sparse data, dynamic updating or risk quantification, and therefore will not be described in detail.

[0107] As shown in Figure 7 The embodiment of the present application also provides a gait and face feature fusion recognition system of deep learning architecture, which comprises:

[0108] The signal response module 100 is used for monitoring the card swiping event of the subway gate, analyzing the card ID information and generating a dynamic time window signal, wherein the card swiping event is obtained by reading the signal of the traffic card through the card reader;

[0109] The micro-motion capture module 200 is used for collecting the plantar pressure distribution and the face video of the card swipers and processing to obtain the spatio-temporal aligned foot pressure time sequence data and the anti-occlusion face image when the dynamic time window signal is obtained, wherein the plantar pressure distribution is collected by the high-density array type pressure sensor distributed in the gate passage, and the face video is collected by the multi-view camera device;

[0110] The feature reconstruction module 300 is configured to reconstruct continuous gait phase features according to foot pressure timing data, reversely restore three-dimensional facial biometric features in a squeezed state by using anti-occlusion facial images, and integrate and process to obtain a standardized human biometric feature vector.

[0111] The dynamic decision module 400 is configured to perform confidence evaluation on the human biometric feature vector, dynamically weighted fusion to obtain multi-modal features, and generate a final identity authentication result through a multi-round voting mechanism in a time window and control the gate passage.

[0112] In the embodiment of the present application, by monitoring the card swiping event and generating a dynamic time window signal, the behavior timing of passengers in the gate passage can be accurately captured, and the problem of continuous sequence breakage caused by gait stagnation in the traditional method is solved, and the motion feature distortion is avoided. After the plantar pressure distribution collected by the high-density array pressure sensor and the anti-occlusion facial image obtained by the multi-view camera device, the three-dimensional facial biometric features in the squeezed state are reversely restored, and the problem of instantaneous distortion of facial geometry caused by crowd squeezing during peak hours is effectively overcome. In addition, by dynamically weighted fusion of multi-modal features and introduction of a multi-round voting mechanism in a time window, not only the modal conflict caused by the space-time offset of the gait stagnation stage and the facial squeezing moment is alleviated, but also the linkage mechanism of hard real-time association of card swiping behavior, mechanical pressure change and biometric feature collection is introduced, which greatly shortens the time consumption from sensor collection to recognition decision, and ensures that the identity authentication is completed within the acceptable passage time window of passengers.

[0113] As shown in Figure 8 , as a preferred embodiment of the present application, the signal response module 100 comprises:

[0114] The signal processing unit 101 is configured to separate the high-frequency carrier and the modulated data in the card swiping signal through the card reader, and decode the modulated data by using an envelope detector to obtain card ID information;

[0115] The dynamic window generation unit 102 is configured to predict the optimal collection time based on the card ID information in combination with a real-time queuing density image by using a convolutional neural network, and generate a dynamic time window instruction for variable biometric feature capture, wherein the real-time queuing density image is obtained by a camera at the top of the gate;

[0116] The hardware triggering unit 103 is configured to convert the dynamic time window instruction into a nanosecond-level photoelectric synchronous signal, and transmit it to the array pressure sensor and the multi-view camera device for controlling the accurate start-stop timing of the two;

[0117] The conflict arbitration unit 104 is configured to monitor the time overlap of card swiping events in multiple channels, and dynamically allocate collection priorities to ensure that the card swiping signal processing is conflict-free under high-density passenger flow.

[0118] As shown in Figure 9 As a preferred embodiment of the present application, the micro-motion capture module 200 comprises:

[0119] The foot pressure sensing unit 201 is configured to sense the foot pressure distribution through the arrayed pressure sensors in the gate passage, convert the foot pressure distribution into an electric charge distribution map, and obtain the foot pressure time series data after processing by the electric charge amplifier.

[0120] The multi-spectral imaging unit 202 is configured to collect the anti-occlusion facial image through the multi-view camera device, which can output the fused multi-spectral image by collecting the facial texture features and subcutaneous blood vessel distribution.

[0121] The data alignment unit 203 is configured to perform time domain interpolation compensation on the foot pressure time series data and the anti-occlusion facial image based on the gate vibration sensor data to obtain the synchronized bimodal data.

[0122] The overload protection unit 204 is configured to monitor the output voltage of the arrayed pressure sensor in real time, so that when the output voltage is saturated, the charge shunt protection circuit is triggered.

[0123] As shown in Figure 10 As a preferred embodiment of the present application, the feature reconstruction module 300 comprises:

[0124] The gait phase field reconstruction unit 301 is configured to calculate the continuous gait cycle feature vector by using the foot pressure trajectory second-order differential equation to calculate the interrupted motion phase during gait stagnation based on the foot pressure time series data.

[0125] The deformation mechanics solving unit 302 is configured to calculate the corrected three-dimensional facial topological structure by using the biomechanical elastic model to reversely calculate the facial extrusion deformation based on the anti-occlusion facial image.

[0126] The multi-modal noise reduction unit 303 is configured to fuse the gait cycle feature vector and the three-dimensional facial topological structure, and perform noise reduction processing by using the conditional adversarial network.

[0127] The feature extraction unit 304 is configured to perform feature extraction on the noise-reduced data by using the spatio-temporal pyramid network to obtain the standardized human biological feature vector.

[0128] The above only describes the preferred embodiments of the present application in detail, and does not limit the present application. Any modification, equivalent replacement and improvement made within the spirit and principles of the present application shall be included in the protection scope of the present application.

[0129] It should be understood that, even though a series of steps are shown in the flowcharts of the embodiments of the present application, the steps are not necessarily performed in the order shown by the arrows. Unless otherwise specified herein, the steps are not necessarily performed in the order shown by the arrows. The steps can be performed in other orders. Moreover, at least some of the steps in the embodiments can include a plurality of sub-steps or a plurality of stages, which are not necessarily performed at the same time, but can be performed at different times, and the order of the sub-steps or stages is not necessarily sequential, but can be performed in rotation or alternation with at least some of the other steps or sub-steps or stages of other steps.

[0130] It is understood by those skilled in the art that all or part of the processes in the above-mentioned embodiments can be implemented by a computer program instructing the relevant hardware, and the program can be stored in a non-volatile computer readable storage medium. When the program is executed, it can include the processes of the above-mentioned embodiments. Any reference to memory, storage, database or other medium used in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0131] Other embodiments of the present disclosure will be apparent to those skilled in the art with the disclosure herein. The present application is intended to cover any variations, uses or adaptive changes of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or conventional technical means in the art not disclosed by the present disclosure. The specification and embodiments are only considered as exemplary, and the true scope and spirit of the present disclosure are indicated by the claims.

Claims

1. A gait and facial feature fusion recognition method of a deep learning architecture, characterized in that, The method comprises the following steps: Monitoring the card swiping event of the subway gate, analyzing the card ID information and generating a dynamic time window signal, the card swiping event being obtained by reading the signal of the traffic card by the card reader; When the dynamic time window signal is obtained, the foot pressure distribution and the face video of the card swiping person are collected and processed to obtain the spatio-temporal aligned foot pressure time series data and the anti-occlusion face image, the foot pressure distribution being collected by the high-density array pressure sensor distributed in the gate passage, and the face video being collected by the multi-view camera device; According to the foot pressure time series data, the continuous gait phase characteristics are reconstructed, the three-dimensional face biometric features in the squeezed state are inversely restored by using the anti-occlusion face image, and the standardized human biometric feature vector is obtained by integration processing; The human biometric feature vector is confidence evaluated and dynamically weighted and fused to obtain a multi-modal feature, and then the final identity authentication result is generated through a multi-round voting mechanism within the time window, and the gate passage is controlled. 2.The gait and facial feature fusion recognition method of deep learning architecture according to claim 1, characterized in that, The step of monitoring the card swiping event of the subway gate, analyzing the card ID information and generating a dynamic time window signal comprises: Separating the high-frequency carrier from the modulated data in the card swiping signal by the card reader, and decoding the modulated data by the envelope detector to obtain the card ID information; Based on the card ID information, combining the real-time queuing density image, predicting the optimal collection time by the convolutional neural network, and generating a dynamic time window instruction for variable biometric feature capture, the real-time queuing density image being obtained by the camera at the top of the gate; The dynamic time window instruction is converted into a nanosecond-level photoelectric synchronous signal, which is transmitted to the array pressure sensor and the multi-view camera device for controlling the accurate start-stop timing of the two; The time overlap of the card swiping events of multiple channels is monitored, and the collection priority is dynamically allocated to ensure that the card swiping signal processing is conflict-free under high-density passenger flow. 3.The gait and facial feature fusion recognition method of deep learning architecture according to claim 1, characterized in that, The step of collecting the foot pressure distribution and the face video of the card swiping person and processing to obtain the spatio-temporal aligned foot pressure time series data and the anti-occlusion face image comprises: The foot pressure distribution is sensed by the array pressure sensor in the gate passage, and is converted into an electric charge distribution map, and the foot pressure time series data is obtained after processing by the charge amplifier; The anti-occlusion face image is collected by the multi-view camera device, which can output a fused multi-spectral image by collecting the face texture features and subcutaneous blood vessel distribution; Based on the gate vibration sensor data, the foot pressure time series data and the anti-occlusion face image are time domain interpolated and compensated to obtain synchronized bimodal data; The output voltage of the array pressure sensor is monitored in real time, so that when the output voltage is saturated, the charge shunt protection circuit is triggered. 4.The gait and facial feature fusion recognition method of deep learning architecture according to claim 1, characterized in that, The step of reconstructing the continuous gait phase characteristics according to the foot pressure time series data, inversely restoring the three-dimensional face biometric features in the squeezed state by using the anti-occlusion face image, and integrating processing to obtain the standardized human biometric feature vector comprises: Based on the foot pressure time series data, the continuous gait cycle characteristic vector is obtained by using the foot pressure trajectory second-order differential equation to calculate the interrupted motion phase during gait arrest; Based on the anti-occlusion face image, the corrected three-dimensional face topology is obtained by inversely calculating the face squeezing deformation by using the biomechanical elastic model. The gait cycle feature vector is fused with the three-dimensional face topology, and a conditional adversarial network is used for noise reduction processing; The spatial-temporal pyramid network is used for feature extraction on the noise-reduced data to obtain a standardized human biological feature vector. 5.The gait and facial feature fusion recognition method of deep learning architecture according to claim 4, characterized in that, The step of obtaining the corrected three-dimensional face topology based on the anti-occlusion face image and using a biomechanical elastic model to reversely calculate the face extrusion deformation, specifically includes: Combining clinical data, a Young's modulus reference value is obtained based on the anti-occlusion face image and the local elastic modulus, and the Young's modulus reference value is associated with age and gender; The displacement of the marker points of the anti-occlusion face image and the optical flow method are used to calculate the extrusion force distribution to inversely deduce the stress state of the deep tissue to obtain a time-space continuous mechanical sensing signal; Based on the Young's modulus reference value and the mechanical sensing signal, a dynamic displacement field is obtained through a three-dimensional nonlinear wave equation and a display integral algorithm; The dynamic displacement field is combined with an initial model by using an adversarial generative network to generate a corrected three-dimensional face topology, and the initial model is an initial state of the three-dimensional face topology. 6.The gait and facial feature fusion recognition method of deep learning architecture according to claim 1, characterized in that, The step of performing confidence evaluation on the human biological feature vector and dynamically weighting fusion to obtain multi-modal features, and then generating a final identity authentication result through a multi-round voting mechanism within a time window, specifically includes: The real-time confidence weights of the gait and the face are calculated by analyzing the gait stagnation index and the face deformation entropy in the human biological feature vector; The features of the gait and the face are dynamically aggregated by using a gated attention mechanism to obtain fused multi-modal features; Based on the fusion decision of the multi-modal features within the time window, a weighted majority voting is performed to obtain a preliminary identity authentication result; When a decision conflict is detected, a final identity authentication result is output by using a Bayesian inference engine.

7. A gait and facial feature fusion recognition system of deep learning architecture, characterized in that, The system includes: A signal response module for monitoring a card swiping event of a subway gate, analyzing card ID information, and generating a dynamic time window signal, wherein the card swiping event is obtained by a card reader reading a traffic card signal; A micro-motion capture module for, when the dynamic time window signal is acquired, collecting and processing the plantar pressure distribution and the face video of the card swiping person to obtain spatio-temporally aligned foot pressure time series data and anti-occlusion face images, wherein the plantar pressure distribution is collected by a high-density array pressure sensor distributed in the gate passage, and the face video is collected by a multi-view camera device; A feature reconstruction module for reconstructing continuous gait phase features according to the foot pressure time series data, reversely restoring three-dimensional face biological features in the extrusion state by using the anti-occlusion face images, and integrating processing to obtain a standardized human biological feature vector; A dynamic decision module for performing confidence evaluation on the human biological feature vector and dynamically weighting fusion to obtain multi-modal features, and then generating a final identity authentication result through a multi-round voting mechanism within a time window and controlling the gate passage. 8.The gait and facial feature fusion recognition system of deep learning architecture according to claim 7, wherein, The signal response module includes: A signal processing unit for separating the high-frequency carrier and the modulated data in the card swiping signal by a card reader, and decoding the modulated data by an envelope detector to obtain the card ID information; A dynamic window generation unit is configured to combine a real-time queuing density image obtained by a camera on top of a gate with card ID information, predict an optimal collection time length through a convolutional neural network, and generate a dynamic time window instruction for variable biometric feature capture; A hardware trigger unit is configured to convert the dynamic time window instruction into a nanosecond-level photoelectric synchronization signal and transmit the signal to an array pressure sensor and a multi-view camera device for controlling the accurate start-stop timing of the two devices; A conflict arbitration unit is configured to monitor the time overlap of multiple channel card swiping events and dynamically allocate collection priorities to ensure conflict-free card swiping signal processing under high-density passenger flow. 9.The gait and facial feature fusion recognition system of deep learning architecture according to claim 7, wherein, The micro-motion capture module includes: A foot pressure sensing unit is configured to sense foot pressure distribution through an array pressure sensor in a gate passage, convert the foot pressure distribution into an electric charge distribution map, and obtain foot pressure timing data after processing by a charge amplifier; A multi-spectral imaging unit is configured to capture an anti-shielding facial image through a multi-view camera device, which can output a fused multi-spectral image by capturing facial texture features and subcutaneous blood vessel distribution; A data alignment unit is configured to perform time domain interpolation compensation on foot pressure timing data and anti-shielding facial images based on gate vibration sensor data to obtain synchronized bimodal data; An overload protection unit is configured to monitor the output voltage of the array pressure sensor in real time, so that when the output voltage is saturated, a charge shunt protection circuit is triggered. 10.The gait and facial feature fusion recognition system of deep learning architecture of claim 7, wherein, The feature reconstruction module includes: A gait phase field reconstruction unit is configured to calculate the continuous gait cycle feature vector by using a foot pressure trajectory second-order differential equation to calculate the interrupted motion phase during gait stagnation based on foot pressure timing data; A deformation mechanics solving unit is configured to calculate the corrected three-dimensional facial topology by using a biomechanical elastic model to inversely calculate facial extrusion deformation based on the anti-shielding facial image; A multi-modal noise reduction unit is configured to fuse the gait cycle feature vector and the three-dimensional facial topology and perform noise reduction processing using a conditional adversarial network; A feature extraction unit is configured to extract standardized human biometric feature vectors from the noise-reduced data using a spatio-temporal pyramid network.

Images