A zero trust multi-modal biometric continuous authentication method against deepfake

Abstract

The present application relates to multi-modal biometric continuous authentication technology in a zero-trust access scenario, aiming at the technical problem that existing multi-modal authentication only performs independent matching or simple fusion, and is difficult to identify deep forgery, remote control and identity theft. The present application collects face video sequences, voiceprint audio sequences, behavior operation sequences and device fingerprint sequences, maps them to a unified time axis, extracts cross-modal relationship features, calculates the time sequence consistency violation degree based on a coupling constraint model, determines the authentication strength in combination with the device fingerprint dynamic evolution state and the access resource risk level, determines the real user continuous interaction process through an artificial intelligence model, and performs non-susceptible release, access restriction, secondary authentication or session termination. The present application can be used in remote medical, financial digital wallet and government secret system access control scenarios.

Description

Technical Field

[0001] This invention relates to the field of computer information security technology, and in particular to a zero-trust multimodal biometric continuous authentication method resistant to deepfakes. Background Technology

[0002] In identity authentication technologies, existing multimodal biometric authentication typically collects facial images, fingerprints, voiceprints, gait, touch behavior, device status, or other contextual data, and determines the authenticity of the user's identity through feature extraction, template matching, score fusion, behavioral template comparison, or liveness detection. In continuous authentication scenarios, the system typically continuously acquires behavioral or device data after the user logs in, matching the current features with preset templates to determine whether the current operator is still an authorized user. In anti-counterfeiting authentication scenarios, the system also uses infrared spectroscopy, visible light images, action commands, liveness detection, or facial comparison to determine the authenticity of biometric features, thereby improving the ability to identify forged fingerprints, forged faces, or abnormal behavior.

[0003] The problem with the existing technologies is that current multimodal authentication mainly relies on independent matching results of features from each modality, liveness detection results, or simple fusion results for identity determination. Its focus is on determining whether the face, fingerprint, behavior, or device is genuine, without adequately characterizing the temporal correlation between face video sequences, voiceprint audio sequences, behavioral operation sequences, and device fingerprint sequences. When attackers employ combined attack methods such as deepfake videos, speech synthesis, remote control, device hijacking, or switching users after login, a single modality may still show high matching or low anomalies, but the overall multimodal data does not conform to the continuous interaction process of a real user. Existing solutions struggle to identify situations where audio and video are partially consistent but the actual operator is inconsistent, device state changes are accompanied by abrupt behavioral changes, or some modalities are forged while others remain normal. This allows authentication to still pass under deepfake attacks, remote control, or identity theft scenarios.

[0004] Publication No. CN118627049A discloses a multimodal biometric comparison method and system. This method acquires identity information, the fingerprint features of a person's four consecutive fingers, and a facial image. After initial consistency between the fingerprint features and the facial image, it controls a red light emitting device and infrared light to collect red light images of the fingers and the infrared reflection spectrum of a designated area of ​​the face. Based on the red light images and infrared reflection spectra, it determines whether the four consecutive fingers and the face are genuine, thus addressing the problem of difficulty in identifying high-quality forged fingerprints and faces. This solution is similar to the aforementioned problem in that it focuses on multimodal identity verification and forgery detection. However, it primarily targets optical liveness comparison of four consecutive fingers and the face, failing to address the temporal correlation modeling problem of facial video sequences, voiceprint audio sequences, behavioral operation sequences, and device fingerprint sequences in identity authentication sessions. It also fails to identify remote control, proxy operation, device hijacking, or local deep forgery through cross-modal coupling terms and differences.

[0005] Publication number CN111353139A discloses a method, apparatus, electronic device, and storage medium for continuous authentication. It acquires a user's current gait and touch data, extracts gait and touch features respectively, and obtains behavioral features through feature fusion. These behavioral features are then matched against behavioral feature templates in a behavioral template library to determine the user's identity, thereby addressing the problems of traditional authentication being easily cracked, having high security risks, and degrading the continuous authentication experience. This solution is similar to the aforementioned problem in that it involves continuous identity authentication and behavioral feature fusion, but its main processing objects are gait and touch data, and it determines user identity through behavioral feature template matching. It does not involve cross-modal temporal consistency analysis between facial video, voiceprint audio, device fingerprint, and behavioral operations, nor does it disclose the identification of deepfake attacks, remote control, or identity theft based on multimodal collaborative anomalies and local modal anomalies. Summary of the Invention

[0006] The technical problem to be solved by this invention is that the existing multimodal authentication is based only on independent matching or simple weighted fusion of features of each modality, which cannot characterize the temporal correlation between different modalities. As a result, authentication may still pass under deepfake attacks, remote control or identity theft scenarios. To this end, a zero-trust multimodal biometric continuous authentication method that is resistant to deepfakes is proposed.

[0007] To achieve the above objectives, this application adopts the following technical solution: a zero-trust multimodal biometric continuous authentication method resistant to deepfakes. This method does not simply weight the identity matching scores of multiple modalities, but extracts cross-modal relationship features between different modalities under a unified time axis, and constructs a temporal consistency violation degree based on the coupling and difference relationships between the cross-modal relationship features, thereby determining whether the multimodal data deviates from the real user's continuous interaction process; the real user's continuous interaction process refers to the interaction process in which the face video, voiceprint audio, behavioral operation and device status naturally generated by the same target user in the same identity authentication session satisfy the relationships of synchronicity, responsiveness and continuity.

[0008] This invention provides a zero-trust, multimodal biometric continuous authentication method resistant to deepfakes, comprising the following steps:

[0009] S1, In the identity authentication session, the target user's facial video sequence, voiceprint audio sequence, behavioral operation sequence, and terminal device fingerprint sequence are collected. The identity authentication session includes a continuous access session after login authentication, a remote business session, a transaction confirmation session, a device unlocking session, a government identity verification session, and a cloud desktop access session. The facial video sequence is continuous video frame data used to extract lip movements, head posture, and gaze changes. The voiceprint audio sequence is the user's voice input signal used to extract speech rhythm, phoneme boundaries, and acoustic features. The behavioral operation sequence is the target user's operation record on the terminal device, including keyboard input, mouse movement, touch operation, page stay, and command confirmation. The device fingerprint sequence is a set of information representing changes in the terminal device's state, including hardware information, system environment information, network environment information, input / output device status, and security module status.

[0010] S2, the face video sequence, the voiceprint audio sequence, the behavior operation sequence, and the device fingerprint sequence are time-synchronized and mapped to a unified time axis to construct multimodal time-series data; the time synchronization includes timestamp alignment, time window division, resampling, and event aggregation processing. Through the unified time axis, face videos, voiceprint audio, behavior operations, and device states with different sampling frequencies can establish corresponding relationships within the same time window, so as to facilitate subsequent cross-modal relationship analysis.

[0011] S3. Based on the multimodal temporal data, extract cross-modal relationship features. These cross-modal relationship features include audio-video temporal correlation features, speech-behavior coupling features based on time delay attenuation, video-behavior response features, and device state and behavior continuity features used to determine whether device changes and behavior changes are continuous and consistent. Specifically, the audio-video temporal correlation features describe the synchronization relationship between lip movements and speech; the speech-behavior coupling features describe the delay response relationship between speech rhythm and operation rhythm; the video-behavior response features describe the time response relationship between changes in user visual attention and operational behavior; and the device state and behavior continuity features describe whether changes in device state and changes in user operation remain continuous and consistent.

[0012] Preferably, the voice and behavior coupling feature is obtained by calculating the correlation between the voice rhythm sequence and the behavior operation rhythm sequence under different time delay conditions and introducing a delay attenuation mechanism. The time delay is the time difference between voice input and behavior response. The delay attenuation mechanism is used to reduce the false correlation generated under unnatural delay conditions, thereby distinguishing between real user natural interaction and remote control, proxy operation or automatic script control.

[0013] Preferably, the device status and behavior continuity features are used to determine whether changes in the device fingerprint sequence are consistent with changes in the behavior operation sequence. Device changes are not used as a sole basis for denying access, but are used together with behavioral mutations to determine the risk of device hijacking or identity theft, thereby reducing misjudgments caused by normal network switching, system updates or peripheral changes.

[0014] S4. Construct a temporal consistency violation degree based on the cross-modal relationship features. The temporal consistency violation degree includes a coupling term for characterizing multimodal collaborative anomalies and a difference term for characterizing local modal anomalies. The temporal consistency violation degree is used to characterize the degree to which multimodal data deviates from the real user's continuous interaction process. The coupling term is used to characterize the degree of collaborative anomalies when multiple modalities deviate from the real interaction process at the same time. The difference term is used to characterize a local forgery state where some modalities are abnormal while other modalities remain normal.

[0015] Preferably, the temporal consistency violation is constructed based on audio-video consistency violations, voice and behavior consistency violations, video and behavior consistency violations, device and behavior consistency violations, and session continuity violations. Each violation is normalized to the same value range. The normalization process is used to ensure that different violations can participate in the coupling constraint calculation at the same scale.

[0016] Furthermore, the timing consistency violation is calculated using the following formula: ,in, Indicates the degree of timing consistency violation; This indicates a violation of audio-visual consistency. This indicates a violation of the consistency between speech and behavior; This indicates a violation of consistency between the video and the behavior; This indicates a violation of the consistency between equipment and behavior; to These are the weighting coefficients for each violation. These are the cross-modal coupling coefficients; is the modal difference sensitivity coefficient. Wherein, This indicates the degree of coupling between multiple modes that are simultaneously abnormal; This indicates the difference in the degree of anomalousness between different modes.

[0017] S5, determine the authentication strength threshold based on the dynamic evolution state of the device fingerprint sequence and the risk level of the accessed object, and compare the temporal consistency violation with the authentication strength threshold to determine the authentication strength; the dynamic evolution state of the device fingerprint sequence is the changing trend of the device fingerprint within a continuous time window, and the risk level of the accessed object is the security level determined based on the sensitivity and operation type of the accessed object. By first determining the threshold based on the device status and the risk level of the accessed object, and then comparing the temporal consistency violation with the threshold, a circular judgment between the authentication strength and the threshold can be avoided.

[0018] S6, perform zero-trust access control operations based on the authentication strength, and identify deepfake attacks or identity theft based on the temporal consistency violation; the zero-trust access control operations include seamless access, restricted access, secondary authentication, or session termination, wherein seamless access means not requiring the target user to actively cooperate with additional verification in a low-risk state, restricted access means restricting operations such as downloading, copying, transferring funds, sending externally, or modifying permissions, secondary authentication means re-collecting biometrics, verifying device certificates, or performing other enhanced authentication, and session termination means blocking the current identity authentication session and generating audit records.

[0019] Preferably, in step S2, the face video sequence, the voiceprint audio sequence, the behavior operation sequence, and the device fingerprint sequence are segmented according to a unified time window, and the correspondence between each modality data within the same time window is established under a unified time axis. The unified time window can be a fixed-length window or a window that is dynamically adjusted according to the risk level of the access object or the operation type.

[0020] Preferably, in step S3, the speech-behavior coupling feature is determined by the speech-behavior coupling degree. Characterization, the degree of coupling between speech and behavior Obtain it as follows: ,in, Indicates the degree of coupling between speech and behavior; Indicates the time delay between the speech rhythm sequence and the behavioral operation rhythm sequence; This represents the deferred set that allows for searching; Indicates the first Speech rhythm features at each time point; Indicates delay The corresponding behavioral rhythm characteristics; Indicates the mean of speech rhythm features; This represents the mean of the behavioral rhythm characteristics; This indicates the number of sampling points within the same time window; This represents a stable term to prevent the denominator from being zero. This represents the delay decay coefficient; the formula is formed by combining the correlation coefficient and the exponential decay function. The correlation coefficient is used to characterize the degree of rhythmic coordination, and the exponential decay function is used to penalize delays that do not conform to natural interaction. If speech and behavior are completely unrelated, the correlation coefficient is close to zero; if speech and behavior are highly correlated but the delay is too large, the exponential decay term will reduce the coupling degree. This can avoid misjudging remote control behavior as real interaction based solely on statistical correlation.

[0021] Preferably, in step S1, the device fingerprint sequence is divided into a stable layer, a semi-stable layer, a dynamic layer, and a behavior coupling layer. The stable layer is used to characterize the device hardware or security module characteristics, the semi-stable layer is used to characterize the system and software environment characteristics, the dynamic layer is used to characterize the network and peripheral device status, and the behavior coupling layer is used to characterize the correlation between device changes and user operations.

[0022] Preferably, in step S3, when a change in the device fingerprint sequence is detected, the degree of change in the behavior operation sequence is combined to determine whether the device change is consistent with the continuity of user behavior. When the device change and the sudden change in behavior occur simultaneously, it is determined to be a risk of device hijacking or identity theft.

[0023] Furthermore, in step S4, the session continuity is detected by comparing the distribution of behavioral operation features in the current time window with the distribution of behavioral features in the historical time windows. When a sudden change in the distribution of behavioral features is detected, it is determined that the session is broken, thereby identifying the behavior of switching users or remote takeover after logging in.

[0024] Furthermore, in step S5, the authentication strength threshold includes a first threshold, a second threshold, and a third threshold. The first threshold, the second threshold, and the third threshold are determined based on the dynamic evolution of the device fingerprint sequence and the risk level of the access object. When the time sequence consistency violation is less than the first threshold, seamless continuous authentication is performed. When the time sequence consistency violation is greater than or equal to the first threshold and less than the second threshold, access restriction or silent verification is performed. When the time sequence consistency violation is greater than or equal to the second threshold and less than the third threshold, high-strength authentication is performed, which includes deepfake detection or two-factor authentication. When the time sequence consistency violation is greater than or equal to the third threshold, the current session is terminated or the access request is blocked.

[0025] Preferably, in step S6, a relationship feature vector is constructed based on the cross-modal relationship features and the temporal consistency violation degree, and the relationship feature vector after irreversible transformation or encryption is input into the identity authenticity determination model to output a deep forgery risk value, an identity theft risk value, or a risk type label.

[0026] The technical effects and advantages of this invention are as follows: The cross-modal relationship feature extraction module solves the problem of existing multimodal authentication's difficulty in characterizing cross-modal temporal relationships by constructing audio-video temporal correlation, voice-behavior coupling, and device-behavior continuity features, thus improving the identification stability of deepfake attacks, remote control, and identity theft. The temporal consistency violation calculation module solves the problem of simple weighted fusion's difficulty in identifying collaborative anomalies and local forgeries by jointly modeling coupling and difference relationships, reducing the risk of misjudgment in multimodal forgery scenarios. The authentication strength decision module solves the problem of fixed authentication strategies being difficult to adapt to different risk states by combining temporal consistency violation, device fingerprint dynamic evolution status, and access resource risk level, reducing invalid calls to high-load models. The system as a whole implements a continuous authentication mechanism based on real user continuous interaction processes, improving the security, stability, and robustness of zero-trust access control, and enhancing cross-modal temporal consistency authentication capabilities. Attached Figure Description

[0027] The disclosure of this invention is illustrated with reference to the accompanying drawings. It should be understood that the drawings are for illustrative purposes only and are not intended to limit the scope of protection of this invention. In the drawings, the same reference numerals are used to refer to the same parts:

[0028] Figure 1 This is an overall flowchart of the zero-trust multimodal biometric continuous authentication method against deepfakes according to the present invention; Figure 2 This is a flowchart of the cross-modal relationship feature extraction process of the present invention; Figure 3 A flowchart for constructing the timing consistency violation of this invention; Figure 4This is a flowchart illustrating the authentication strength determination and authentication triggering process of the present invention. Figure 5 This is a flowchart of the device fingerprint dynamic evolution and device-behavior consistency judgment of the present invention; Figure 6 This is a schematic diagram illustrating how the timing consistency violation degree of the present invention changes with a uniform time window; Figure 7 This is a schematic diagram illustrating the impact of coupling and difference relationships on timing consistency violation in this invention. Detailed Implementation

[0029] It is readily understood that, based on the technical solution of this invention, those skilled in the art can propose various interchangeable structural methods and implementations without altering the essential spirit of the invention. Therefore, the following detailed embodiments and accompanying drawings are merely illustrative examples of the technical solution of this invention and should not be considered as the entirety of the invention or as limitations or restrictions on the technical solution of this invention.

[0030] Example 1: This example provides a zero-trust, multimodal biometric continuous authentication method resistant to deepfakes. For example... Figure 1 As shown, the method includes collecting multimodal sequences, synchronizing time and constructing multimodal time-series data, extracting cross-modal relationship features, constructing time consistency violation, determining authentication strength threshold, judging whether the time consistency violation exceeds the authentication strength threshold, determining authentication strength, identifying deepfake attacks or identity theft, and performing zero-trust access control operations.

[0031] The method is applied to identity authentication sessions. These sessions include remote medical diagnosis and treatment sessions, financial digital wallet transaction confirmation sessions, government classified system access sessions, cloud desktop usage sessions, remote operation and maintenance sessions, and continuous terminal device usage sessions. The method is executed collaboratively by a terminal acquisition program, an identity authentication server, and an access control gateway. The terminal acquisition program is used to collect facial video sequences, voiceprint audio sequences, behavioral operation sequences, and device fingerprint sequences. The identity authentication server is used to perform time synchronization, cross-modal relationship feature extraction, temporal consistency violation calculation, authentication strength threshold determination, and identity authenticity verification. The access control gateway is used to perform seamless access control, access restriction, secondary authentication, or session termination based on authentication strength.

[0032] The terminal acquisition program includes a video acquisition submodule, an audio acquisition submodule, a behavior acquisition submodule, a device fingerprint acquisition submodule, and a data verification submodule. The video acquisition submodule acquires continuous video frames from the terminal's camera or an authorized video input component, and extracts facial key points, lip key points, head posture angles, and gaze direction. The audio acquisition submodule acquires continuous audio segments from the terminal's microphone or an authorized audio input component, and extracts speech energy, fundamental frequency, Mel-frequency characteristics, phoneme boundary characteristics, and speech rhythm characteristics. The behavior acquisition submodule acquires keyboard input, mouse movement, touch operations, page dwell, button clicks, and sensitive command trigger information through the operating system event listening interface or business application event interface. The device fingerprint acquisition submodule acquires hardware status, system environment, network environment, browser environment, camera status, microphone status, peripheral device status, and security module status. The data verification submodule adds timestamps, acquisition source identifiers, and integrity verification values ​​to the above-mentioned data.

[0033] In step S1, the terminal acquisition program collects multimodal data of the target user after the identity authentication session is initiated. The face video sequence includes consecutive video frames, video frame timestamps, camera source identifiers, and video integrity verification values. The video acquisition submodule extracts facial key points, lip key points, head pose angles, and gaze direction within each time window. The facial key points are used to determine the stability of the face region; the lip key points are used to calculate lip opening and closing degree and lip shape change; the head pose angles represent the pose change of the target user's face relative to the terminal camera; and the gaze direction represents the trend of the target user's attention area changing over time.

[0034] The voiceprint audio sequence includes continuous audio segments, audio segment timestamps, microphone source identifiers, and audio integrity verification values. The audio acquisition submodule performs noise reduction, endpoint detection, and speech activity detection on the audio signal, and extracts speech energy, fundamental frequency, Mel-spectral features, phoneme boundary features, and speech rhythm features. To reduce the risk of biometric data leakage, the terminal can delete the original audio segments after completing relational feature extraction, retaining only speech rhythm, phoneme boundaries, and acoustic statistical features.

[0035] The behavioral operation sequence includes event type, event value, event timestamp, and event source identifier. The behavior collection submodule records different operation events according to the business scenario. In a remote medical diagnosis and treatment session, the behavioral operation sequence includes opening medical records, browsing medical records, zooming in on images, filling in diagnostic templates, and confirming medical orders. In a financial digital wallet transaction confirmation session, the behavioral operation sequence includes unlocking, entering amount, receiving party confirmation, transaction confirmation, and triggering secondary verification. In a government classified system access session, the behavioral operation sequence includes opening documents, scrolling through documents, copying requests, downloading requests, sending requests, and permission operations.

[0036] The device fingerprint sequence includes a stable layer, a semi-stable layer, a dynamic layer, and a behavior coupling layer. For example... Figure 5 As shown, the stable layer includes device hardware identification, security chip status, secure boot status, and device certificate status. The semi-stable layer includes operating system version, browser version, driver information, and security plugin status. The dynamic layer includes network type, IP address changes, camera status, microphone status, peripheral connection status, and virtual device status. The behavioral coupling layer includes changes in the operational rhythm before and after a device status change. The device fingerprint sequence is not only used to determine whether devices are the same, but also to determine whether device status changes match the continuity of the target user's behavior.

[0037] To ensure the reliability of the collected data, the terminal acquisition program generates integrity verification values ​​for video, audio, behavior, and device fingerprints. These integrity verification values ​​can be calculated using a hash algorithm. The timestamps can be generated from the terminal's trusted clock, server time calibration information, or session synchronization clock. The acquisition source identifier is used to mark the source of the camera, microphone, input device, browser interface, and operating system interface. When any modality experiences a short-term absence, the system records the missing interval, duration, and source, and reduces the credibility of that modality within the corresponding time window, avoiding directly classifying short-term absences as attack behavior.

[0038] In step S2, the identity authentication server or the terminal acquisition program synchronizes the face video sequence, voiceprint audio sequence, behavioral operation sequence, and device fingerprint sequence in time and maps them to a unified time axis to construct multimodal time-series data. Since video frames, audio samples, behavioral events, and device fingerprint updates have different sampling frequencies, the system divides the identity authentication session into continuous time windows. Each time window can have a fixed length or be dynamically adjusted according to the risk level of the accessed object and the type of operation. A relatively long time window can be used for ordinary browsing operations; a relatively short time window can be used for high-risk operations such as transaction confirmation, prescription issuance, and downloading confidential documents.

[0039] Within each time window, video frames are mapped to a unified timeline using video frame timestamps, audio segments are mapped to a unified timeline using voice activity timestamps, behavioral events are mapped to a unified timeline using operation event timestamps, and device fingerprint changes are mapped to a unified timeline using state change timestamps. For continuous signals such as video and audio, the system employs resampling, interpolation, or window aggregation. For discrete events such as keystrokes, clicks, and device changes, the system uses event counts, event intervals, change intensity, and event duration for statistical analysis, thereby forming comparable multimodal time-series data within the same time window.

[0040] In step S3, the system extracts cross-modal relationship features based on the multimodal time-series data. For example... Figure 2 As shown, the multimodal temporal data is used to generate audio-video temporal correlation features, speech-behavior coupling features, video-behavior response features, and device state-behavior continuity features, which together form cross-modal relationship features. Unlike directly comparing a single modality with a registered template, these cross-modal relationship features describe whether there is synchronicity, responsiveness, and continuity between different modalities within the same time window.

[0041] The audio-visual temporal correlation features are obtained from lip movement sequences and speech sequences. The lip movement sequence includes lip opening and closing, lip key point displacement, and mouth shape changes. The speech sequence includes speech energy, phoneme boundaries, and spectral variations. In real user speech, there is usually a stable temporal relationship between lip opening and closing changes and speech energy changes. However, in cases of face-swapped videos, pre-recorded videos, voice cloning, or audio injection, the relationship between lip movement and speech may become premature, delayed, or decrease in correlation.

[0042] The voice-behavior coupling features are obtained from voice rhythm sequences and behavioral operation rhythm sequences. The voice rhythm sequence includes speech rate, pauses, volume variations, and voice activity segments. The behavioral operation rhythm sequence includes keystroke intervals, mouse movement speed, touch swipe speed, and operation pauses. In real users' speaking, reading, confirming, and input processes, there is usually an interpretable temporal relationship between voice rhythm and operation rhythm. For example, a doctor states a diagnosis before filling out a medical record; a user confirms a transfer amount before confirming a transaction; and a user in a confidential system maintains a continuous relationship between eye movement changes and scrolling operations when reading documents. In remote control, proxy operation, or automated script control scenarios, the voice rhythm of the person in front of the screen usually does not match the behavioral rhythm of the actual operator.

[0043] To ensure that the voice-behavior coupling features reflect natural interaction processes, the system calculates the correlation between voice rhythm sequences and behavioral operation rhythm sequences under different time delay conditions and introduces a delay attenuation mechanism. The time delay represents the time difference between voice input and behavioral response. For natural users, the correlation is high within a reasonable delay range. For remote control, proxy operation, or mechanical scripting, the delay may be too long, too short, or lack a stable pattern. The delay attenuation mechanism is used to reduce spurious correlations caused by unnatural delays.

[0044] The video and behavioral response features are obtained from changes in head posture, gaze direction, and behavioral operation. The system extracts head posture angle, gaze direction, facial orientation, and screen operation changes. In real-world scenarios, changes in visual attention and operational behavior are typically temporally correlated when users are reading documents, confirming transactions, or filling out diagnostic reports. If the user's gaze in the video does not correspond to a change in operation, or if operations are frequent while the video attention state remains stable, the system can determine that the video and behavioral response are abnormal.

[0045] The continuity of device state and behavior is obtained from changes in device fingerprints and behavioral operations. Normal device changes typically have a continuous transition; for example, user behavior should not immediately abruptly change after a network switch. If the device's camera source, microphone source, browser fingerprint, network environment, or security module state changes, and the behavioral operation rhythm also abruptly changes, the risk of device hijacking, virtual device injection, or identity theft increases. Figure 5 As shown, the system inputs the dynamic evolution status of the device fingerprint and the degree of change in the behavior operation sequence into the device and behavior consistency judgment process. When the dynamic evolution status of the device fingerprint is abnormal and the behavior operation sequence changes synchronously, it is determined that the risk of device hijacking or identity theft has increased.

[0046] In step S4, the system analyzes the cross-modal relationship features based on a multimodal temporal consistency model and calculates the temporal consistency violation degree between multimodal data. The temporal consistency violation degree characterizes the degree to which multimodal data deviates from the actual continuous user interaction process. The actual continuous user interaction process refers to the situation where, during a natural operation by the same user in the same authentication session, facial video, voiceprint audio, behavioral operations, and device status satisfy a temporal continuity relationship, a physical synchronization relationship, and a behavioral response relationship.

[0047] like Figure 3As shown, the system further transforms the cross-modal relationship features into audio-video consistency violations, speech-behavior consistency violations, video-behavior consistency violations, device-behavior consistency violations, and session continuity violations. The audio-video consistency violation indicates anomalies in synchronization between lip movements and speech. The speech-behavior consistency violation indicates anomalies in coupling between speech rhythm and behavioral rhythm. The video-behavior consistency violation indicates anomalies in the response between changes in visual attention and changes in operation. The device-behavior consistency violation indicates inconsistencies between changes in device state and behavioral continuity. The session continuity violation indicates abrupt changes in behavioral distribution between the current time window and historical time windows.

[0048] To avoid overlooking the collaborative relationships between different anomalies due to simple weighted fusion, the system employs a coupling constraint model to calculate the temporal consistency violation degree. This model not only considers the anomaly degree of each violation itself, but also characterizes the collaborative anomalies when speech and behavior violations and video and behavior violations occur simultaneously through coupling terms, and characterizes the inconsistency between the degree of audio-video violations and the degree of speech-behavior violations through difference terms.

[0049] Each violation term is normalized before participating in the coupling constraint model. Normalization can employ max-min normalization, quantile normalization, or normalization based on training sample statistics to ensure that audio / video consistency violations, speech / behavior consistency violations, video / behavior consistency violations, and device / behavior consistency violations are on the same value scale. The temporal consistency violation degree, after exponential mapping, remains between zero and one. When all violations are close to zero, the temporal consistency violation degree is close to zero. As any violation term, coupling term, or difference term increases, the temporal consistency violation degree also increases.

[0050] like Figure 6 As shown, within a continuous, unified time window, audio / video consistency violations, voice and behavior consistency violations, video and behavior consistency violations, device and behavior consistency violations, and session continuity violations can all change over time. When all violations are at low levels, the temporal consistency violation is below the low authentication strength threshold, and the system performs seamless continuous authentication. As voice and behavior consistency violations, video and behavior consistency violations, or device and behavior consistency violations continuously increase, the temporal consistency violation also increases, entering a range that deviates from the actual continuous user interaction process. If the temporal consistency violation exceeds the low authentication strength threshold, the system triggers high-strength authentication. If it further exceeds the high authentication strength threshold, the system performs identity verification and identifies deepfake attacks or identity theft.

[0051] like Figure 7As shown, when both the degree of voice-behavior consistency violation and the degree of video-behavior consistency violation are low, the normalized temporal consistency violation is in a low-risk region, and the system can perform seamless continuous authentication. When the degree of one type of violation increases while the degree of the other type remains at a low level, the system enters a local forgery risk region, which can correspond to cross-modal injection, partial modality forgery, or local modality anomaly. When both the degree of voice-behavior consistency violation and the degree of video-behavior consistency violation increase simultaneously, the system enters a collaborative anomaly region, which can correspond to risks of remote control, proxy operation, or identity theft. Therefore, the coupling term is used to enhance the risk response when multiple modalities are simultaneously anomalous, and the difference term is used to identify local forgery states where some modalities are anomalous while other modalities remain normal.

[0052] Session continuity is obtained by comparing the behavioral feature distribution of the current time window with that of historical time windows. Optionally, the system uses Jensen-Shannon divergence, cosine distance, or other distribution distances to measure the difference. If the behavioral distribution of the current window differs significantly from that of historical windows, it is determined that session continuity has decreased. This process can be used to identify user switching after login, remote takeover, and automated script takeover.

[0053] In step S5, the system determines the authentication strength threshold based on the dynamic evolution of the device fingerprint sequence and the risk level of the accessed object, and compares the temporal consistency violation with the authentication strength threshold. Figure 4 As shown, the system uses the timing consistency violation, the dynamic evolution status of the device fingerprint, and the risk level of the access object as the basis for determining the authentication strength, and uses a threshold judgment to determine whether to enter a higher-strength authentication process.

[0054] The dynamic evolution of the device fingerprint sequence is jointly determined by the stable layer, semi-stable layer, dynamic layer, and behavior coupling layer. Changes in the stable layer typically carry a high risk. Changes in the semi-stable layer may be caused by system updates or software upgrades, requiring judgment based on both time and behavior. Changes in the dynamic layer may be caused by network switching or peripheral device changes and should not be used as a sole indicator of high risk. The behavior coupling layer is used to determine whether user operations are continuous before and after device changes.

[0055] The risk level of the accessed object is determined by the identity authentication system based on the accessed object and the type of operation. In remote medical diagnosis scenarios, patient medical records, diagnostic conclusions, and prescription issuance are considered high-risk objects. In financial digital wallet scenarios, transfers, withdrawals, card binding, and key modification are considered high-risk objects. In government classified system scenarios, downloading, copying, distributing, and modifying permissions of classified documents are considered high-risk objects. The risk level is used to trigger different authentication strengths for the same degree of violation in different business scenarios, avoiding frequent disturbances to users in low-risk scenarios, while ensuring timely strengthening of authentication in high-risk scenarios.

[0056] The authentication strength thresholds include a first threshold, a second threshold, and a third threshold. The first threshold, the second threshold, and the third threshold are determined based on the dynamic evolution of the device fingerprint sequence and the risk level of the accessed object. When the time consistency violation is less than the first threshold, seamless continuous authentication is performed. When the time consistency violation is greater than or equal to the first threshold and less than the second threshold, access restriction or silent authentication is performed. When the time consistency violation is greater than or equal to the second threshold and less than the third threshold, high-strength authentication is performed. When the time consistency violation is greater than or equal to the third threshold, the current session is terminated or the access request is blocked.

[0057] like Figure 6 As shown, the authentication strength threshold may include a low authentication strength threshold and a high authentication strength threshold. In a specific implementation, the low authentication strength threshold may correspond to a first threshold, and the high authentication strength threshold may correspond to a second or third threshold. The system determines the corresponding threshold based on the dynamic evolution of the device fingerprint sequence and the risk level of the accessed object, and compares the temporal consistency violation with the threshold. In a low-risk state, the system maintains seamless continuous authentication. In a medium-to-high-risk state, the system sequentially performs access restriction, silent verification, high-strength authentication, or session termination.

[0058] In step S6, the system performs zero-trust access control operations based on authentication strength and identifies deepfake attacks or identity theft based on temporal consistency violations. If the temporal consistency violation is low and the risk level of the accessed object is low, the access control gateway maintains the authentication session and performs seamless access. If a medium risk occurs, the access control gateway restricts some operations, such as prohibiting downloads, limiting transfer amounts, restricting copying, or requiring silent verification. If a high risk occurs, the system triggers secondary authentication, which can involve re-collecting a live face, re-collecting a voiceprint, verifying device certificates, or requiring administrator confirmation. If an extremely high risk occurs, the access control gateway terminates the session and generates an audit log.

[0059] In one specific implementation, the system constructs a relationship feature vector based on cross-modal relationship characteristics and temporal consistency violation. This relationship feature vector, after irreversible transformation or encryption, is then input into an identity authenticity determination model to output a deepfake risk value, an identity theft risk value, or a risk type label. The identity authenticity determination model can be implemented using a rule-based model, a statistical anomaly detection model, a temporal neural network, or a temporal Transformer model. Its inputs include audio / video synchronization offset, audio / video correlation, voice and behavior coupling degree, video and behavior response difference, device and behavior consistency characteristics, session continuity characteristics, and temporal consistency violation. Its output is used by the access control gateway to execute corresponding access control operations.

[0060] When the identity authenticity determination model is implemented using an artificial intelligence model, the artificial intelligence model includes an input normalization layer, a modal relation encoding layer, a temporal encoding layer, a coupled attention layer, and a risk output layer. The input normalization layer is connected to the modal relation encoding layer, used to map relation features of different dimensions to a unified scale and input them into the corresponding relation encoding channels. The modal relation encoding layer is connected to the temporal encoding layer, used to encode audio-video relations, voice behavior relations, video behavior relations, and device behavior relations into window-level relation vectors. The temporal encoding layer is connected to the coupled attention layer, used to learn the changing patterns between continuous time windows. The coupled attention layer is connected to the risk output layer, used to learn the mutual influence between different relation features, especially the coupling relationship when voice and behavior anomalies and video and behavior anomalies occur simultaneously. The risk output layer outputs the confidence score of genuine continuous interaction, the deepfake risk value, the identity theft risk value, and a risk type label.

[0061] In a reproducible model structure, the input normalization layer normalizes the mean and variance or quantiles of each relation feature dimension. The modal relation encoding layer includes fully connected encoding channels corresponding to audio-video relations, voice behavior relations, video behavior relations, and device behavior relations, respectively, with each channel including at least one layer of linear mapping and nonlinear activation. The temporal encoding layer can employ a gated recurrent unit, a long short-term memory network, or a temporal Transformer structure, with its input being a sequence of relation vectors within a continuous time window. The coupled attention layer receives the multi-window hidden states output by the temporal encoding layer and calculates the attention weights between different relation channels. The risk output layer includes a risk scoring output unit and a risk type classification unit. The risk scoring output unit outputs deepfake risk values ​​and identity theft risk values, while the risk type classification unit outputs at least one label selected from real continuous interaction, deepfake, remote control, device hijacking, and session takeover.

[0062] During AI model training, training samples consist of legitimate user real-world conversation samples, deepfake samples, remote control samples, device anomaly samples, and session takeover samples. Legitimate user real-world conversation samples are used to learn normal cross-modal temporal relationships. Deepfake samples include face-swapping videos, speech synthesis, pre-recorded videos, and audio / video injection samples. Remote control samples include screen sharing control, remote desktop control, and proxy operation samples. Device anomaly samples include virtual cameras, virtual microphones, browser fingerprint mutations, and peripheral switching samples. Session takeover samples include user switching after login and mid-session takeover operation samples.

[0063] The training process includes sample collection, time alignment, relation feature extraction, sample labeling, sample partitioning, model training, and threshold calibration. The sample collection phase collects multimodal data according to step S1. The time alignment phase forms a unified time window according to step S2. The relation feature extraction phase extracts cross-modal relation features according to step S3. The sample labeling phase labels at least real continuous interactions, deepfakes, remote control, device hijacking, and session takeover. The sample partitioning phase divides the samples into training samples, validation samples, and test samples. The model training phase uses relation feature vectors as input and outputs risk type and risk value as targets. The threshold calibration phase determines a first threshold, a second threshold, and a third threshold on the validation samples to match the risk levels of access objects in different scenarios.

[0064] The loss mechanism used in model training can include risk type classification loss, true continuous interaction confidence loss, and temporal consistency constraint loss. The risk type classification loss enables the model to distinguish between true continuous interaction, deepfakes, remote control, device hijacking, and session takeover. The true continuous interaction confidence loss ensures that the model output aligns with the credibility of manually or rule-based annotations. The temporal consistency constraint loss enhances the model's sensitivity to cross-modal temporal anomalies, particularly improving risk response to samples exhibiting both speech and behavioral anomalies, as well as video and behavioral anomalies simultaneously. During training, batch gradient descent or adaptive gradient optimization algorithms can be employed. The learning rate, batch size, number of iterations, and early stopping conditions are determined based on the risk classification performance and false trigger rate of the validation samples. After model training is complete, the model parameters are fixed, and the model is deployed to an identity authentication server or edge authentication node.

[0065] To ensure the model's usability in specific domains, training data for telemedicine scenarios includes processes such as doctor video consultations, medical record reading, voice diagnosis, and doctor's order confirmation. Training data for financial digital wallet scenarios includes processes such as login, amount input, transfer confirmation, and secondary verification. Training data for government classified systems scenarios includes processes such as login, document browsing, permission operations, and download control. Different scenarios can share a base model or be retrained or fine-tuned using scenario data. The inherent relationship between model input and output lies in the fact that input features describe whether multimodal data conforms to real continuous interaction patterns, while output results indicate the trustworthiness of the current session and potential attack types.

[0066] In this embodiment, the relationship features can be processed in the form of relationship feature vectors. These relationship feature vectors do not contain data that can directly reconstruct the original facial image or original voiceprint audio. The terminal can perform irreversible transformations or encryption on the relationship feature vectors before uploading them to the identity authentication server. Irreversible transformations can employ hash encoding, random projection, or secure embedding methods. Encryption can employ symmetric encryption or homomorphic computation. This design enables the identity authentication server to determine identity credibility based on relationship features, while simultaneously reducing the risk of leakage of original biometric data.

[0067] The following combination Figures 1 to 7 This document describes the operational processes in scenarios involving remote medical diagnosis, financial digital wallets, and access to classified government systems. All of the above scenarios are implemented in accordance with... Figure 1 The overall process shown is executed, and according to... Figures 2 to 7 The process of cross-modal relationship feature extraction, violation degree construction, authentication strength determination, dynamic evolution judgment of device fingerprint, and temporal consistency violation degree change process shown in the figure completes continuous identity authentication.

[0068] Example 2: In a telemedicine diagnosis scenario, a doctor uses a terminal to enter a remote diagnosis and treatment authentication session and access patient medical records and the video consultation system. The system initiates continuous authentication after the doctor logs in. The terminal collects the doctor's facial video sequence, voiceprint audio sequence, medical record browsing behavior, diagnosis entry behavior, confirmation button click behavior, as well as camera source, microphone source, operating system status, and network status.

[0069] During consultations, doctors describe the patient's symptoms and simultaneously input their diagnostic opinions into the system. The system performs a delay coupling analysis on the rhythm of the spoken words and the rhythm of the diagnostic input. If the doctor's input behavior occurs within a reasonable time after speaking, the coupling between speech and behavior is considered high. The system also analyzes the synchronization between the doctor's lip movements and speech; if there is a significant deviation between lip movements and speech, the audio-video consistency violation is increased. The system further analyzes whether the doctor's gaze direction is consistent with their medical record browsing behavior; if the video shows the doctor not paying attention to the screen but the system shows continuous medical record modification behavior, the video-behavior consistency violation is increased.

[0070] During normal consultations, the doctor's facial video, voiceprint audio, medical record entry behavior, and device fingerprint remain continuous, resulting in a low temporal consistency violation. The system then performs seamless, continuous authentication. However, if an attacker participates in the consultation using face-swapped video and voice cloning, but the actual medical record entry is performed remotely, the system will detect abnormal coupling between voice and behavior, as well as abnormal video and behavior responses. Due to the increased coupling, the temporal consistency violation significantly increases, triggering high-strength authentication and suspending medical order submission or requiring a re-authentication of liveness.

[0071] Example 3: In a financial digital wallet scenario, when a user opens the wallet application and prepares to transfer funds, the system first collects the user's facial video sequence, voiceprint audio sequence, touch behavior sequence, and device fingerprint sequence during the unlocking process. When the user enters an amount and clicks confirm, the risk level of the access target increases, and the system tightens the authentication strength threshold based on the dynamic evolution of the device fingerprint and the risk level of the access target.

[0072] If the user operates on their familiar device, and the touch trajectory, input rhythm, camera status, and network status remain consistent with low temporal consistency violations, the system allows the transaction to continue. However, if an attacker remotely controls the user's phone to complete the transfer, but the video still displays the user's normal face, the system will detect inconsistencies between the touch behavior rhythm and the user's historical behavior distribution, as well as changes in device status or control chain. Increased session continuity violations and device / behavior consistency violations will trigger secondary authentication or limit the transaction amount.

[0073] If an attacker uses synthesized speech commands and forged videos to induce payment, audio-video consistency may remain normal in certain areas, but the coupling between speech and behavior will be abnormal. The discrepancy is used to capture this localized forgery state, allowing the system to improve overall risk and trigger deepfake detection or terminate the transaction.

[0074] Example 4: In a government classified system access scenario, users enter a classified system authentication session via a government cloud desktop. The system collects the user's facial video sequence, behavioral operation sequence, and device fingerprint sequence. If the user continuously browses a document, the direction of their gaze, mouse scrolling, and page dwell time should show a continuous relationship. If the system detects a change in camera source, remote desktop control traces, or a sudden change in browser fingerprint, and the user's behavior rhythm changes significantly, the device and behavior consistency violation increases. If the current behavior distribution differs significantly from the historical behavior distribution, the session continuity violation increases.

[0075] When a user downloads or sends classified documents, the risk level of the accessed object increases. Even if the temporal consistency violation is at a moderate level, the system can still trigger secondary authentication or approval confirmation according to the zero-trust policy. If the identity authenticity judgment model outputs an identity theft risk, the access control gateway terminates the session and generates an audit log. The audit log includes the time window number, risk type, temporal consistency violation, device change summary, and access control result, but does not save the original facial video and original voice content.

[0076] The technical scope of this invention is not limited to the content described above. Those skilled in the art can make various modifications and variations to the above embodiments without departing from the technical concept of this invention, and all such modifications and variations should fall within the protection scope of this invention.

Claims

1. A zero-trust multimodal biometric continuous authentication method resistant to deepfakes, characterized in that, Includes the following steps: S1. In the identity authentication session, collect the target user's face video sequence, voiceprint audio sequence, behavioral operation sequence, and terminal device fingerprint sequence; S2. Synchronize the face video sequence, voiceprint audio sequence, behavioral operation sequence, and device fingerprint sequence in time and map them to a unified time axis to construct multimodal time-series data; S3. Extract cross-modal relationship features based on the multimodal time-series data. The cross-modal relationship features include audio-video time-series correlation features, speech-behavior coupling features based on time delay attenuation, video-behavior response features, and device state and behavior continuity features used to determine whether device changes and behavior changes are continuous and consistent; S4. Construct a time-series consistency violation degree based on the cross-modal relationship features. The time-series consistency violation degree includes coupling terms used to characterize multimodal collaboration anomalies and difference terms used to characterize local modal anomalies. The time-series consistency violation degree is calculated by constructing a coupling constraint model of cross-modal relationships. Represented as: ,in, For timing consistency violation, This is an audio / video consistency violation. This is a violation of the consistency between speech and behavior. This is a violation of consistency between video and behavior. This is a violation of the consistency between equipment and behavior. , , , Here are the weighting coefficients for each violation. For cross-modal coupling coefficients, The modal difference sensitivity coefficient is the term. This indicates the coupling relationship between voice and behavioral violations, and between video and behavioral violations, used to characterize the degree of collaborative anomaly when multiple modalities simultaneously deviate from the real interaction process; the term It represents the inconsistency between the degree of violation of different modalities and is used to characterize the abnormal state in which some modalities are forged while other modalities remain normal. S5. Determine the authentication strength threshold based on the dynamic evolution state of the device fingerprint sequence and the risk level of the access object, and compare the temporal consistency violation with the authentication strength threshold to determine the authentication strength; S6. Perform zero-trust access control operation based on the authentication strength, and identify deepfake attacks or identity theft based on the temporal consistency violation.

2. The zero-trust multimodal biometric continuous authentication method against deepfakes according to claim 1, characterized in that, In step S2, the face video sequence, the voiceprint audio sequence, the behavior operation sequence, and the device fingerprint sequence are segmented according to a unified time window, and the correspondence between each modal data within the same time window is established under the unified time axis.

3. The zero-trust multimodal biometric continuous authentication method against deepfakes according to claim 2, characterized in that, In step S3, the voice and behavior coupling feature is obtained by calculating the correlation between the voice rhythm sequence and the behavior operation rhythm sequence under different time delay conditions, and an attenuation mechanism is introduced based on the time delay to characterize the response characteristics of human operation behavior to voice input, thereby distinguishing between real user interaction and asynchronous control behavior.

4. The zero-trust multimodal biometric continuous authentication method against deepfakes according to claim 3, characterized in that, In step S1, the device fingerprint sequence is divided into a stable layer, a semi-stable layer, a dynamic layer, and a behavior coupling layer. The stable layer is used to characterize the device hardware or security module characteristics, the semi-stable layer is used to characterize the system and software environment characteristics, the dynamic layer is used to characterize the network and peripheral device status, and the behavior coupling layer is used to characterize the correlation between device changes and user operations.

5. The zero-trust multimodal biometric continuous authentication method against deepfakes according to claim 4, characterized in that, In step S3, when a change in the device fingerprint sequence is detected, the degree of change in the behavior operation sequence is combined to determine whether the device change is consistent with the continuity of user behavior. When the device change and the sudden change in behavior occur at the same time, it is determined to be a risk of device hijacking or identity theft.

6. The zero-trust multimodal biometric continuous authentication method against deepfakes according to claim 5, characterized in that, In step S4, the session continuity is detected by comparing the distribution of behavioral operation features in the current time window with the distribution of behavioral features in the historical time window. When a sudden change in the distribution of behavioral features is detected, it is determined that the session is broken, thereby identifying the behavior of switching users or remote takeover after logging in.

7. The zero-trust multimodal biometric continuous authentication method against deepfakes according to claim 6, characterized in that, In step S4, the temporal consistency violation is constructed based on audio-video consistency violations, voice and behavior consistency violations, video and behavior consistency violations, device and behavior consistency violations, and session continuity violations. Each violation is normalized to the same value range.

8. The zero-trust multimodal biometric continuous authentication method against deepfakes according to claim 7, characterized in that, In step S5, the authentication strength threshold includes a first threshold, a second threshold, and a third threshold, and the first threshold, the second threshold, and the third threshold are determined based on the dynamic evolution state of the device fingerprint sequence and the risk level of the access object; when the temporal consistency violation is less than the first threshold, seamless continuous authentication is performed; When the temporal consistency violation is greater than or equal to the first threshold and less than the second threshold, access restriction or silent verification is performed; when the temporal consistency violation is greater than or equal to the second threshold and less than the third threshold, high-strength authentication is performed, which includes deepfake detection or two-factor authentication; when the temporal consistency violation is greater than or equal to the third threshold, the current session is terminated or access requests are blocked.

9. A zero-trust multimodal biometric continuous authentication method resistant to deepfakes according to claim 8, characterized in that, In step S6, a relationship feature vector is constructed based on the cross-modal relationship features and the temporal consistency violation degree. The relationship feature vector after irreversible transformation or encryption is then input into the identity authenticity determination model to output a deep forgery risk value, an identity theft risk value, or a risk type label.

Images