Identity verification method and apparatus, electronic device, and program product

Abstract

The application discloses an identity verification method and device, electronic equipment and program product, and relates to the field of biological identification. The method comprises the following steps: receiving a face video returned by a user according to a random instruction, extracting a response video frame corresponding to the random instruction from the face video, and obtaining an interaction response sequence; intercepting local images of multiple face components, establishing a voting relationship between the local images and face response states, and obtaining a component voting sequence; determining a living body prediction state range based on the component voting states at different time points, updating the voting relationship, and generating a living body state sequence; determining a state change position according to the living body state sequence, extracting an abnormal segment, and determining an identity verification result of the face video. Through the application, the problem that the structuralized corresponding relationship between the face local position and the overall living body state is insufficiently expressed in the remote identity verification process in the related art, it is difficult to reflect the real cooperative change of each face component, and the detection result is inaccurate is solved.

Description

Technical Field

[0001] This application relates to the field of biometrics, and more specifically, to an authentication method, device, electronic device, and program product. Background Technology

[0002] With the development of fintech, remote identity verification services by banks, such as online account opening, remote face-to-face verification, and authorization for large-amount transfers, are becoming increasingly common. These services typically require users to perform actions such as blinking, opening their mouth, and turning their head via random commands on a mobile or computer terminal to confirm the user's identity and verify their consistency with identification information. However, increasingly sophisticated deepfake attacks, such as screen capture, face-swapping, facial retargeting, and generative video forgery, pose serious security challenges to remote identity verification by banks.

[0003] Existing technologies for remote face liveness detection and identity presentation attack detection mainly include methods based on static image texture analysis, action response detection, and video temporal modeling. Static image texture analysis methods typically extract skin texture, edge, or frequency domain features from single frames or keyframes, combining them with convolutional neural networks to distinguish between real and fake faces. Action response detection methods detect facial key points, measure single action indicators such as blink amplitude and mouth opening amplitude, and make a judgment based on command matching results. Video temporal modeling methods utilize 3D convolution, recurrent neural networks, or attention mechanisms to learn representations of action evolution in continuous videos and output classification results. Some solutions also combine multimodal data or first segment the video according to instructions, then evaluate the segments, and finally synthesize the results.

[0004] While the aforementioned solutions have improved verification capabilities to some extent, they still have shortcomings in adapting to remote identity verification scenarios in financial institutions. On the one hand, many solutions simply interpret random command responses as the completion of a single action, or directly process continuous video using temporal classification networks, lacking in-depth modeling of the structured correspondence between facial features and the overall liveness state. This results in insufficient expression of the structured correspondence between facial features and the overall liveness state, making it difficult to reflect the true coordinated changes of various facial components. On the other hand, existing temporal models often lack explicit constraints on whether state changes between consecutive moments fall within the naturally reachable range of a truly live person. When faced with deeply fake identities that can perform surface actions but exhibit unnatural transitions, the judgment criteria are not stable enough. These deficiencies lead to inaccurate detection results, making it difficult to meet the stringent requirements of banks for the authenticity of interactions and the continuity of processes in remote identity verification.

[0005] There is currently no effective solution to the problem that the structured correspondence between local facial features and the overall liveness status is insufficient in the remote identity verification process of related technologies, making it difficult to reflect the real coordinated changes of various facial components and resulting in inaccurate detection results. Summary of the Invention

[0006] The main purpose of this application is to provide an identity verification method, device, electronic device, and program product to solve the problem in the related technology that the structured correspondence between local facial locations and the overall liveness state is insufficient in the remote identity verification process, making it difficult to reflect the real coordinated changes of various facial components and resulting in inaccurate detection results.

[0007] To achieve the above objectives, according to one aspect of this application, an authentication method is provided, comprising: receiving a facial video returned by the user according to a random instruction when user authorization to collect biometric information is detected; extracting response video frames corresponding to the random instruction from the facial video to obtain an interaction response sequence; extracting local images of multiple facial components from the interaction response sequence; determining facial response states based on the interaction response sequence and the random instruction; establishing a voting relationship between the local images and the facial response states to obtain a component voting sequence; determining a liveness prediction state range based on the component voting states at different times in the component voting sequence; updating the voting relationship between the local images and the facial response states based on the liveness prediction state range to generate a liveness state sequence; determining the positions of state changes exceeding the predicted state range in the interaction response sequence based on the liveness state sequence; extracting abnormal segments; and determining the authentication result of the facial video based on the abnormal segments.

[0008] Further, the process involves extracting response video frames corresponding to random commands from the facial video to obtain an interactive response sequence. This includes: determining the display order of the random commands and their timestamp information, where the timestamp information includes the start and end times of the display; extracting consecutive facial video frames corresponding to the random commands from the facial video based on the timestamp information and sorting them chronologically to obtain a candidate response frame sequence; identifying the facial regions and facial components in each frame of the candidate response frame sequence using an object detection algorithm; selecting facial video frames with continuous facial movement and complete facial components from the candidate response frame sequence based on the facial regions and facial components to obtain facial response frames; and sorting the facial response frames according to the display order of the random commands and the time information of the facial response frames to obtain the interactive response sequence.

[0009] Further, local images of multiple facial components are extracted from the interaction response sequence. Facial response states are determined based on the interaction response sequence and random commands. A voting relationship is established between the local images and the facial response states to obtain a component voting sequence. This includes: performing frame alignment processing on facial response frames in the interaction response sequence based on the positional information of multiple facial components; extracting local images of multiple facial components from each aligned frame to obtain multiple first feature capsules corresponding to each facial response frame; determining the facial response state corresponding to the random command based on the correspondence between the random command and the facial response frames in the interaction response sequence and the multiple first feature capsules corresponding to each facial response frame; constructing a second feature capsule based on the facial response state; establishing transformation weights between the multiple first feature capsules and the second feature capsules corresponding to each facial response frame; constructing the component voting state of each facial response frame based on the transformation weights of the multiple first feature capsules corresponding to each facial response frame; and sorting the component voting states of each facial response frame according to the chronological order of the interaction response sequence to obtain a component voting sequence.

[0010] Furthermore, the liveness prediction state range is determined based on the component voting states at different times in the component voting sequence. The voting relationship between the local image and the facial response state is updated based on the liveness prediction state range to generate a liveness state sequence. This includes: extracting the first component voting state corresponding to the first time step and the second component voting state corresponding to the second time step from the component voting sequence in chronological order, where the first time step and the second time step are adjacent time steps; inputting the first component voting state into a pre-trained Kopman operator model to obtain the liveness prediction state range at the second time step; comparing the liveness prediction state range with the second component voting state to obtain a comparison result; performing a weighted summation of the transformation weights of multiple first feature capsules corresponding to the second time step in the component voting sequence based on the comparison result to obtain the liveness feature capsule response state at the second time step; and generating a liveness state sequence based on the liveness feature capsule response state corresponding to each time step in the component voting sequence.

[0011] Further, based on the liveness state sequence, the location of state change exceeding the predicted state range in the interaction response sequence is determined, and abnormal segments are extracted, including: aligning the facial response frames in the liveness state sequence and the interaction response sequence in chronological order to obtain a state correspondence sequence; extracting the liveness feature capsule response states at adjacent time points from the state correspondence sequence, and determining candidate positions of state changes based on the change relationship between the liveness feature capsule response states at adjacent time points; comparing the liveness predicted state range corresponding to the candidate positions of state changes with the liveness change results at the candidate positions of state changes to obtain position deviation information; if the position deviation information meets the preset deviation conditions, the candidate positions of state changes are determined as state change positions, resulting in a set of state change positions; based on the position distribution information of the set of state change positions in the interaction response sequence, the temporally continuous state change positions in the set of state change positions are merged and connected to obtain a set of candidate abnormal segments; based on the interaction response sequence segments corresponding to each state change position in the set of candidate abnormal segments, candidate abnormal segments that overlap with the range of random command are selected, and abnormal segments are determined based on the selected candidate abnormal segments.

[0012] Further, determining the identity verification result of the facial video based on the abnormal segments includes: identifying the start and end state change positions corresponding to the abnormal segments in the interaction response sequence, and counting the number of state change positions in the abnormal segments to obtain the number of abnormalities; determining the percentage of abnormalities based on the number of abnormalities and the total number of state change positions in the interaction response sequence; if the percentage of abnormalities is greater than a preset abnormal threshold, determining the criteria for deep fake identity detection based on the time distribution information of the abnormal segments in the interaction response sequence, the number of abnormalities, and the percentage of abnormalities; if the percentage of abnormalities is less than or equal to the preset abnormal threshold, determining the criteria for real liveness detection based on the time distribution information of the abnormal segments in the interaction response sequence, the number of abnormalities, and the percentage of abnormalities; determining the target detection criteria based on the deep fake identity detection criteria or the real liveness detection criteria, and generating an identity verification result for the user based on the target detection criteria.

[0013] Furthermore, the target determination criterion is determined based on either the deepfake determination criterion or the real liveness determination criterion, and an identity verification result for the user is generated based on the target determination criterion, including: if the target determination criterion is a deepfake determination criterion, determining the deepfake identity determination result based on the determination rules corresponding to the deepfake determination criterion and the deepfake determination criterion; if the target determination criterion is a real liveness determination criterion, determining the real liveness determination result based on the determination rules corresponding to the real liveness determination criterion and the real liveness determination criterion; and determining the identity verification result for the user based on either the deepfake identity determination result or the real liveness determination result.

[0014] To achieve the above objectives, according to another aspect of this application, an authentication device is provided, comprising: an extraction unit, configured to receive a facial video returned by a user according to a random instruction when user authorization to collect biometric information is detected, extract response video frames corresponding to the random instruction from the facial video, and obtain an interactive response sequence; a voting unit, configured to extract local images of multiple facial components from the interactive response sequence, determine facial response states based on the interactive response sequence and the random instruction, and establish a voting relationship between the local images and the facial response states, to obtain a component voting sequence; a prediction unit, configured to determine a liveness prediction state range based on the component voting states at different times in the component voting sequence, update the voting relationship between the local images and the facial response states based on the liveness prediction state range, and generate a liveness state sequence; and a verification unit, configured to determine the positions of state changes exceeding the predicted state range in the interactive response sequence based on the liveness state sequence, extract abnormal segments, and determine the authentication result of the facial video based on the abnormal segments.

[0015] Further, the extraction unit includes: a first determining subunit, used to determine the display order of random instructions and the timestamp information of random instructions, wherein the timestamp information includes: start display time and end display time; a first extraction subunit, used to extract continuous facial video frames corresponding to random instructions from the facial video based on the timestamp information, and sort them according to time order to obtain a candidate response frame sequence; a first filtering subunit, used to identify the facial region and facial parts of each frame in the candidate response frame sequence based on the object detection algorithm, and filter facial video frames with continuous facial movement and complete facial parts in the candidate response frame sequence based on the facial region and facial parts to obtain facial response frames; and a first sorting subunit, used to sort the facial response frames according to the display order of random instructions and the time information of facial response frames to obtain an interactive response sequence.

[0016] Further, the voting unit includes: a first alignment subunit, used to perform frame alignment processing on facial response frames in the interaction response sequence based on the position information of multiple facial components, and extract local images of multiple facial components in each aligned frame to obtain multiple first feature capsules corresponding to each facial response frame; a second determination subunit, used to determine the facial response state corresponding to the random instruction based on the correspondence between the random instruction and the facial response frames in the interaction response sequence and the multiple first feature capsules corresponding to each facial response frame, and construct a second feature capsule based on the facial response state; a construction subunit, used to establish the transformation weights between the multiple first feature capsules and the second feature capsules corresponding to each facial response frame, and construct the component voting state of each facial response frame based on the transformation weights of the multiple first feature capsules corresponding to each facial response frame; and a second sorting subunit, used to sort the component voting states of each facial response frame according to the time order of the interaction response sequence to obtain a component voting sequence.

[0017] Further, the prediction unit includes: a second extraction subunit, used to extract the first component voting state corresponding to the first time step and the second component voting state corresponding to the second time step from the component voting sequence in chronological order, wherein the first time step and the second time step are adjacent time steps; a first comparison subunit, used to input the first component voting state into a pre-trained Kopman operator model to obtain the liveness prediction state range at the second time step, and compare the liveness prediction state range with the second component voting state to obtain a comparison result; a first processing subunit, used to perform a weighted summation of the transformation weights of multiple first feature capsules corresponding to the second time step in the component voting sequence based on the comparison result, to obtain the liveness feature capsule response state at the second time step; and a generation subunit, used to generate a liveness state sequence based on the liveness feature capsule response state corresponding to each time step in the component voting sequence.

[0018] Further, the verification unit includes: a second alignment subunit, used to align the facial response frames in the live state sequence and the interaction response sequence in chronological order to obtain a state correspondence sequence; a second comparison subunit, used to extract the live feature capsule response states at adjacent time points from the state correspondence sequence, and determine the candidate positions of state changes based on the change relationship between the live feature capsule response states at adjacent time points, and compare the live predicted state range corresponding to the candidate positions of state changes with the live change results of the candidate positions of state changes to obtain position deviation information; a third determination subunit, used to determine the candidate positions of state changes as state change positions when the position deviation information meets the preset deviation conditions, to obtain a set of state change positions; a second processing subunit, used to merge and connect the temporally continuous state change positions in the set of state change positions based on the position distribution information of the set of state change positions in the interaction response sequence to obtain a set of candidate abnormal segments; and a second filtering subunit, used to filter candidate abnormal segments that overlap with the range of random command based on the interaction response sequence segments corresponding to each state change position in the set of candidate abnormal segments, and determine abnormal segments based on the filtered candidate abnormal segments.

[0019] Furthermore, the verification unit includes: a statistics subunit, used to determine the start and end state change positions corresponding to abnormal segments in the interaction response sequence, and to count the number of state change positions in the abnormal segments to obtain the number of abnormalities; a fourth determination subunit, used to determine the percentage of abnormalities based on the number of abnormalities and the total number of state change positions in the interaction response sequence; a fifth determination subunit, used to determine the criteria for deep fake identity detection based on the time distribution information of abnormal segments in the interaction response sequence, the number of abnormalities, and the percentage of abnormalities when the percentage of abnormalities is greater than a preset abnormality threshold; a sixth determination subunit, used to determine the criteria for real liveness detection based on the time distribution information of abnormal segments in the interaction response sequence, the number of abnormalities, and the percentage of abnormalities when the percentage of abnormalities is less than or equal to a preset abnormality threshold; and a seventh determination subunit, used to determine the target detection criteria based on the deep fake identity detection criteria or the real liveness detection criteria, and to generate an authentication result for the user based on the target detection criteria.

[0020] Furthermore, the seventh determining subunit includes: a first determining module, used to determine the deepfake identity determination result based on the determination rules corresponding to the deepfake determination basis and the deepfake determination basis when the target determination basis belongs to the deepfake determination basis; a second determining module, used to determine the real liveness determination result based on the determination rules corresponding to the real liveness determination basis and the real liveness determination basis when the target determination basis belongs to the real liveness determination basis; and a third determining module, used to determine the identity verification result for the user based on the deepfake identity determination result or the real liveness determination result.

[0021] To achieve the above objectives, according to one aspect of this application, a computer program product is provided, including a computer program that, when executed by a processor, implements any of the above-described authentication methods, and when executed by a processor, implements the steps of the authentication methods in various embodiments of this application.

[0022] To achieve the above objectives, according to one aspect of this application, a computer-readable storage medium is provided, the computer-readable storage medium including stored computer instructions, wherein any of the above authentication methods are implemented when the computer instructions are executed by a processor.

[0023] To achieve the above objectives, according to one aspect of this application, an electronic device is provided, including one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement any of the above authentication methods.

[0024] In this embodiment, the pixel features of local facial components are transformed into capsule sequences that perform structured voting on the overall liveness response state. The voting relationship is dynamically constrained and updated using the liveness prediction state range. This explicitly models the co-evolutionary relationship between local components such as eyelids, corners of the mouth, lip edges, nasal wings, jaw, and cheek boundaries and the overall liveness state during the detection process. Even if a deepfake attack can simulate a single command action, anomalies can be identified by detecting unnatural deviations in the voting process of local components on the overall state. This avoids misjudgments caused by the lack of a structured correspondence between local and overall components, achieving the technical effect of accurately distinguishing between real liveness and deepfake identity presentation. Therefore, it can solve the technical problem in related technologies where the structured correspondence between local facial locations and the overall liveness state is insufficiently expressed, making it difficult to reflect the real co-changes of various facial components and leading to inaccurate detection results during remote identity verification. This achieves the technical effect of improving the security and accuracy of remote identity verification. Attached Figure Description

[0025] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0026] Figure 1 This is a hardware structure block diagram of a computer terminal (or mobile device) for implementing an authentication method according to Embodiment 1 of this application;

[0027] Figure 2 This is a flowchart of an optional authentication method provided according to Embodiment 1 of this application;

[0028] Figure 3 This is a flowchart illustrating an optional deep pseudo-identity determination method based on capsule network and Koopman operator learning, provided in Embodiment 1 of this application.

[0029] Figure 4 This is a schematic diagram of the network structure of an optional deep pseudo-identity presentation determination method based on capsule network and Koopman operator learning provided in Embodiment 1 of this application;

[0030] Figure 5 This is a schematic diagram of the deviation of optional time and facial components in two-dimensional space according to Embodiment 1 of this application;

[0031] Figure 6 This is a schematic diagram of the optional range of achievable variation in living organisms provided in Embodiment 1 of this application;

[0032] Figure 7This is a schematic diagram of an optional three-dimensional deviation intensity surface map constructed based on time frames and component indexes according to Embodiment 1 of this application;

[0033] Figure 8 This is a schematic diagram of the authentication device provided according to Embodiment 2 of this application;

[0034] Figure 9 This is a schematic diagram of an authentication electronic device provided according to Embodiment 3 of this application. Detailed Implementation

[0035] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0036] It should be noted that the processing method, apparatus, storage medium, and electronic device specified in this application can be used in the field of fintech for liveness authentication, improving the accuracy of verification results. They can also be used in any field other than fintech. The application fields of the processing method, apparatus, storage medium, and electronic device specified in this application are not limited.

[0037] It should be noted that the information collected in this application (including but not limited to user device information, user personal information, collected data, used data, generated data, processed data, etc.) and the data (including but not limited to data used for analysis, stored data, displayed data, collected information, used information, generated information, processed information, etc.) are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of the relevant data all comply with the relevant laws and standards of the relevant regions, do not violate public order and good morals, and provide corresponding operation entry points for users to choose to authorize or refuse. For example, this system has interfaces with relevant users or organizations, providing users with corresponding operation entry points for users to choose to agree to or refuse automated decision results; if the user chooses to refuse, the process proceeds to the expert decision-making process.

[0038] First, some nouns or terms that appear in the description of the embodiments of this application shall be interpreted as follows:

[0039] Capsule Network (CapsNet) is a deep learning architecture that introduces the concept of "capsules," each of which is a group of neurons used to encode the attributes of a specific entity (such as position, pose, scale, etc.). It also preserves the spatial hierarchical relationships between entities through a dynamic routing mechanism, thereby solving the problem of lost spatial information in traditional Convolutional Neural Networks (CNNs).

[0040] The Koopman operator is a linear operator designed to map nonlinear dynamical systems to an infinite-dimensional linear function space, allowing the evolution of nonlinear systems to be described by the action of linear operators, thereby enabling the analysis of complex nonlinear dynamic behavior using linear system theory.

[0041] Deepfake Identity Determination refers to the process of using technical means to identify and distinguish between real human identities and fake identities synthesized through artificial intelligence technologies such as deep learning (e.g., face swapping, facial retargeting). It aims to prevent identity aberration attacks using fake videos or images.

[0042] Example 1

[0043] According to an embodiment of this application, an authentication method embodiment is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0044] The method embodiment provided in Embodiment 1 of this application can be executed in a mobile terminal, computer terminal or similar computing device. Figure 1 This is a hardware structure block diagram of a computer terminal (or mobile device) for implementing an authentication method, according to Embodiment 1 of this application. Figure 1 As shown, the computer terminal 10 (or mobile device) may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) 102 (processor 102 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0045] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0046] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the authentication method in this embodiment. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby implementing the aforementioned authentication method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0047] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0048] The display may be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0049] Under the aforementioned operating environment, this application provides the following: Figure 2The authentication method shown. Figure 2 This is a flowchart of an optional authentication method provided according to Embodiment 1 of this application.

[0050] Step S201: When the user authorizes the collection of biometric information, receive the facial video returned by the user according to the random command, extract the response video frame corresponding to the random command from the facial video, and obtain the interaction response sequence.

[0051] The executing entity in this embodiment 1 can be an identity verification system used for determining the presentation of deep fake identities in banks based on capsule networks and Kopman operator learning. This system extracts interactive response sequences from continuous facial videos generated by users responding to random commands. It constructs low-level capsules representing the local states of multiple facial components such as eyelids, corners of the mouth, lip edges, nose wings, jaw, and cheek boundaries, and these capsules vote on each other to form a component voting sequence in a high-level capsule representing the overall liveness response state. It employs a Koopman operator learning architecture, consisting of a mapping network, a linear state update layer, and a decoding network, to calculate the continuous change from the previous component voting state to the current real liveness component voting, determining the predicted liveness state range (or the liveness reachable change range). Under this range constraint, it updates dynamic routes to aggregate and form a liveness state capsule sequence (i.e., a liveness state sequence). It then locates and merges state changes that deviate from the predicted liveness state range, forming abnormal segments. Based on the abnormality percentage, it outputs a judgment criterion, achieving a continuous evolution relationship between local facial components and the overall liveness response state at the same time. Furthermore, it identifies abnormal response segments based on the real liveness reachable change range, effectively distinguishing between the presentation of a real liveness identity and a deeply fake identity.

[0052] In this embodiment 1, in order to construct structured temporal input data for subsequent deepfake identity determination, facial videos of users responding to random commands can be collected after obtaining user authorization, and valid video frames corresponding to the command time window can be selected and arranged in chronological order to form an interactive response sequence. This allows the acquisition of continuous facial data containing the complete command response process, i.e., the aforementioned interactive response sequence.

[0053] The specific methods for user authorization to collect biometric information are as follows: The system obtains the user's consent to the collection of facial biometric information through interface prompts or pop-ups. When the system detects the user's confirmation action (such as clicking the "agree" button or checking the "authorize" box), the subsequent collection process is triggered. Random instructions refer to the random instructions (such as "please blink," "please open your mouth," "please turn your head to the left," etc.) issued by the server and recorded by the financial institution's remote identity verification terminal, along with the times these instructions are issued and terminated. Facial video refers to the continuous facial video stream captured by the camera during the time period when the random instructions are issued.

[0054] By converting continuous video into a sequence of instruction-based interactive responses, the continuous change relationship between the lower-level components driven by random instructions and the overall liveness response state is preserved, providing a clear structured temporal input for voting from lower-level capsules to higher-level capsules in the subsequent capsule network.

[0055] Step S202: Extract local images of multiple facial components from the interaction response sequence, determine the facial response state based on the interaction response sequence and random instructions, and establish a voting relationship between the local images and the facial response state to obtain the component voting sequence.

[0056] In this embodiment 1, in order to transform the pixel information of the facial image into structured feature data that represents the relationship between facial parts and the overall liveness response state, a low-level capsule can be constructed by locally extracting facial response frames in the interaction response sequence, and a high-level capsule representing the overall liveness response state can be constructed by combining random instruction semantics. A dynamic routing mechanism is then used to make the low-level capsule vote on the high-level capsule, thereby forming a part voting sequence that includes the correspondence between parts and the overall state.

[0057] In this context, a local image refers to determining the spatial positions of multiple facial features (e.g., eyelids, corners of the mouth, lip edges, nose wings, jawline, and cheek boundaries) within each facial response frame in the interactive response sequence, and then cropping local regions from that frame according to fixed cropping rules (e.g., uniformly adjusted to 32×32×3 pixels). A facial response state refers to defining the semantic range of the overall liveness response state at the current moment using the random command (e.g., blinking, opening the mouth) corresponding to the current facial response frame.

[0058] By establishing a clear correspondence between changes in local facial structures and the overall response phase, the random command response process in remote bank identity verification no longer stops at the level of whether the action occurs, but can reflect the collaborative evolution relationship of multiple key parts in continuous moments. This makes it more suitable for identifying deep fake identity presentations where surface actions exist but local linkages are unnatural.

[0059] Step S203: Determine the range of liveness prediction state based on the component voting state at different times in the component voting sequence, update the voting relationship between the local image and the facial response state based on the range of liveness prediction state, and generate a liveness state sequence.

[0060] In this embodiment 1, in order to constrain and correct the voting behavior of facial components driven by random instructions by utilizing the continuous law of the evolution of real living facial states, the voting states of adjacent moments in the component voting sequence can be extracted, and the Koopman operator can be used to learn and predict the achievable change range of real living component voting at the current moment, i.e. the above-mentioned living predicted state range. This range is then used as a constraint condition for dynamic routing to update the voting aggregation relationship of local image to facial response state, thereby forming a living state sequence that conforms to the natural evolution law of living organisms.

[0061] By introducing the liveness prediction state range (i.e., the liveness reachable change range) at the dynamic routing level, the liveness state capsule not only retains the capsule network's ability to express the relationship between components and the whole, but also includes the constraint on the natural continuity of time steps. This enables a more effective distinction between the differences between real liveness responses and deep pseudo-synthetic responses in terms of transition process, part coupling relationship, and state arrival mode, thereby enhancing the robustness of identification and improving the accuracy of deep pseudo-detection.

[0062] Step S204: Based on the liveness state sequence, determine the location of state change that exceeds the predicted state range in the interaction response sequence, extract abnormal segments, and determine the identity verification result of the facial video based on the abnormal segments.

[0063] In this embodiment 1, in order to transform the criteria for deep fake detection from implicit sequence features to explicit, traceable spatiotemporal anomaly distribution, the live state sequence and the interaction response sequence can be time-aligned, the change in the live state capsule at adjacent time points can be calculated and compared with the range of possible changes in the live state to locate the deviation point, and the abnormal segments covering the random instruction triggering process can be merged to form anomaly segments, and the coverage ratio of the abnormal segments in the interaction response sequence can be counted, thereby generating deep fake or real live determination results based on the anomaly distribution.

[0064] Through the above processing, a quantitative judgment mechanism based on the spatiotemporal distribution of abnormal fragments can be established, enabling verifiable identity verification. This feature indirectly solves the problem in related technologies where the structured correspondence between local facial locations and the overall liveness state is insufficient during remote identity verification, making it difficult to reflect the true coordinated changes of various facial components and leading to inaccurate detection results. The rationale for this solution is that existing technologies mostly rely on single-frame anomalies, single-segment action scores, or overall sequence classification results, lacking quantification of the specific distribution and coverage of anomalies during the interaction process; while this feature...

[0065] Optionally, in this embodiment 1, Figure 3 This is a flowchart illustrating an optional deep fake identity determination method based on capsule networks and Koopman operator learning, provided in Embodiment 1 of this application. Figure 3 As shown, in step S301, a continuous facial video generated by the user's response to random commands during remote identity verification at the bank is acquired. Facial response frames corresponding to the random commands are extracted from the continuous facial video to form an interactive response sequence. In step S302, based on the interactive response sequence, the local states of the eyelids, corners of the mouth, lip edges, nose wings, jaw, and cheek boundaries are extracted to construct corresponding low-level capsules. The low-level capsules then vote towards the high-level capsules representing the overall liveness response state, forming a component voting sequence. The component voting sequence is used to represent the continuous change relationship between the lower facial components driven by the random commands and the overall liveness response state. In step S303, based on the component voting sequence, the Koopman operator is used to learn the component voting state from the previous time step. Step S304: Calculate the reachable change range of the real liveness component vote at the current moment, and update the dynamic route at the current moment under the constraint of the reachable change range of the liveness. Aggregate the component votes at the current moment to form a liveness state capsule sequence; Step S305: Calculate the position of the state change in the interaction response sequence that deviates from the reachable change range of the liveness based on the liveness state capsule sequence, forming anomaly segments; Step S306: Calculate the anomalous proportion based on the anomalous segments, and form a deep fake identity judgment basis when the anomalous proportion meets the anomalous threshold, and form a real liveness judgment basis when the anomalous proportion does not meet the anomalous threshold; Step S307: Output the deep fake identity presentation judgment result or the real liveness judgment result based on the deep fake identity judgment basis or the real liveness judgment basis.

[0066] Optionally, in the authentication method provided in Embodiment 1 of this application, extracting response video frames corresponding to random instructions from facial videos to obtain an interactive response sequence includes: determining the display order of random instructions and the timestamp information of random instructions, wherein the timestamp information includes: start display time and end display time; extracting continuous facial video frames corresponding to random instructions from facial videos based on the timestamp information, and sorting them according to time order to obtain a candidate response frame sequence; identifying the facial region and facial parts of each frame in the candidate response frame sequence based on a target detection algorithm, and filtering facial video frames with continuous facial movement and complete facial parts from the candidate response frame sequence based on the facial region and facial parts to obtain facial response frames; and sorting the facial response frames according to the display order of random instructions and the time information of the facial response frames to obtain an interactive response sequence.

[0067] In this embodiment 1, in order to select effective and structurally complete facial response data from the original facial video to construct an ordered interaction sequence, candidate frames can be extracted based on timestamps and response frames with continuous facial movement and complete parts can be selected, thereby constructing an interactive response sequence that contains a complete instruction response process and is in the correct timing.

[0068] Specifically, the acquired continuous facial videos can be represented as: ,in, Indicates the first The acquisition time of a series of facial images Indicates the first A series of consecutive facial images. A random instruction record can be represented as: , where index Arranged in the order of issuance. Indicates the first The start time of displaying a random instruction. Indicates the first The end display time of each random instruction. Construct an instruction time period sequence based on the random instruction record. When the capture time of continuous facial video does not coincide with the start or end time of display, it will satisfy... Furthermore, the consecutive facial images with the smallest time difference are determined as the frames corresponding to the start of display, which will satisfy... Furthermore, the consecutive facial images with the smallest time difference are determined as the frames corresponding to the end of the display. This indicates the preset time matching window. After time matching is completed, consecutive facial images are captured in ascending order of acquisition time within each instruction time period to form a sequence of candidate response frames corresponding to the face. .

[0069] Based on object detection algorithm, candidate response frame sequences are identified. Facial region localization is performed on consecutive facial images. A face position record is generated for each consecutive facial image. ,in, and Indicates the center position of the face area. Represents the area of ​​the face region; simultaneously generates a record of the visible parts of the face. The six components correspond to the visibility status of the eyelids, corners of the mouth, lip edges, nasal alae, jawline, and cheek boundaries, respectively. A value of 1 is assigned when visible and a value of 0 when invisible. The filtering rule is executed in two steps: first, retaining those that meet the criteria... to A series of consecutive facial images, all of which are 1, i.e., complete facial video frames containing the aforementioned facial components; then, consecutive facial images with temporally adjacent faces are compared and their positions are recorded. When the center distance between adjacent facial regions is not greater than a preset displacement threshold... Furthermore, the area difference between adjacent face regions is not greater than a preset area change threshold. At that time, adjacent consecutive facial images are deemed to satisfy the facial motion continuity relationship. After filtering based on facial component integrity and facial motion continuity relationship, a set of facial response frames is obtained. Facial response frame set Each facial response frame retains the complete boundaries of the eyelids, corners of the mouth, lips, nose, jaw, and cheeks, and maintains the continuous change of temporally adjacent facial positions.

[0070] After completing the selection of facial response frames corresponding to a single random command, the entire set of facial response frames is then processed. Sequential reassembly is performed. During reassembly, facial response frames corresponding to the same random command are first arranged in ascending order of acquisition time. Then, facial response frames corresponding to different random commands are connected according to the order in which the random commands were issued. During the reassembly process, each facial response frame is bound to its corresponding random command, forming an interactive response sequence. ,in, , Indicates the first One facial response frame. Indicates the first Random command identifier corresponding to each facial response frame. Index Within the same random command, the increment is based on the acquisition time; between different random commands, the increment is based on the order of issuance. For cases where adjacent indices correspond to different random commands, the random command switching position is preserved in the interactive response sequence, ensuring that the interactive response sequence simultaneously includes the continuous facial response process within a single random command and the switching boundary between different random commands.

[0071] The interactive response sequence obtained through the above steps is not a simple cropping of continuous facial video, but rather an ordered input object that unifies the effective response process, random command boundaries, and facial integrity in continuous facial video. This allows for the direct extraction of local states of facial components such as eyelids, corners of the mouth, lip edges, nose wings, jawline, and cheek boundaries from the facial response frames in the interactive response sequence, and the determination of the overall liveness response state based on the random commands in the interactive response sequence. Furthermore, because the interactive response sequence preserves the temporal continuity within the same random command and the switching relationships between different random commands, the subsequently formed component voting sequence can cover the continuous change process of facial components towards the overall liveness response state driven by random commands. This makes the voting relationship between the local images and facial response states calculated in subsequent steps more reliable, more accurately reflecting the natural and coordinated changes of a real live body driven by random commands, and avoiding distortion of the structured relationship expression caused by data noise or temporal discrepancies.

[0072] Optionally, in the authentication method provided in Embodiment 1 of this application, multiple local images of facial components are extracted from the interaction response sequence, facial response states are determined based on the interaction response sequence and random instructions, and a voting relationship is established between the local images and the facial response states to obtain a component voting sequence. This includes: performing frame alignment processing on facial response frames in the interaction response sequence based on the position information of multiple facial components; extracting multiple local images of facial components from each aligned frame to obtain multiple first feature capsules corresponding to each facial response frame; determining the facial response state corresponding to the random instruction based on the correspondence between the random instruction and the facial response frames in the interaction response sequence and the multiple first feature capsules corresponding to each facial response frame, and constructing a second feature capsule based on the facial response state; establishing transformation weights between the multiple first feature capsules and the second feature capsules corresponding to each facial response frame, and constructing a component voting state for each facial response frame based on the transformation weights of the multiple first feature capsules corresponding to each facial response frame; and sorting the component voting states of each facial response frame according to the time order of the interaction response sequence to obtain a component voting sequence.

[0073] In this embodiment 1, in order to transform the visual features of local facial components into structured data that characterizes the relationship between local and overall living state, a dynamic voting mechanism from low-level feature capsules to high-level feature capsules can be constructed and a temporally ordered component voting sequence can be generated, thereby forming a structured feature sequence that reflects the continuous evolution of facial components toward the overall response state.

[0074] Specifically, for interactive response sequences Each In facial response frames The six facial features are aligned in a fixed order, and the set of feature indices is denoted as [insert set here]. ,in Corresponding to the eyelid, Corresponding to a quarrel, Corresponding to the lip line, Corresponding to the nasal wing, Corresponding to the lower jaw, Corresponding to the cheek boundary. Indexing each area. From facial response frames Extract the corresponding local area In one alternative embodiment, each local region Unified adjustment to Pixel data. To maintain temporal continuity in the interactive response sequence, adjacent facial response frames are... and Index of the same parts The local areas use the same positioning order and the same clipping rules, so that... and The images of the same facial features at adjacent moments correspond to each other, thus forming the source of local states of the eyelids, corners of the mouth, lips, nose, jaw, and cheeks.

[0075] For each local region Using a shared convolutional extraction layer Perform mapping. This represents the network parameters of the shared convolutional extraction layer. The shared convolutional extraction layer is executed in a fixed layer order: first, 32 layers are used... Convolutional kernels are used to extract local regions; then 64 kernels are used. The convolutional kernel continues to extract the output of the previous layer; then, 128 fully connected neurons compress the convolutional output to obtain the local state vector. In an alternative embodiment, each local state vector It is 16-dimensional. The local state vector... As the corresponding low-layer capsule The input representation makes each lower capsule For each facial response frame, a fixed facial region is identified. Based on six low-level capsules, multiple first feature capsules, or low-level capsules, are determined. These six low-level capsules are arranged according to a region index set. Maintaining a one-to-one correspondence. Instead of directly generating a single appearance feature from the entire facial response frame, this process transforms the eyelids, corners of the mouth, lip edges, nose wings, jawline, and cheek boundaries into individually referable low-level capsules, facilitating the expression of the different contributions of multiple facial parts to the overall liveness response state at the same time.

[0076] After the formation of the lower-level capsule, based on random instruction identification The set of low-level capsules corresponding to the same facial response frame Constructing high-level capsule collections This refers to the second characteristic capsule mentioned above. Four high-level capsules. Used to represent different stages of change in the overall liveness response state under the current random command, not to represent static identity categories. Random command identifier Its function is to limit the semantic scope of changes in the entire face at the current moment, low-level capsule set Its function is to provide site-specific evidence of the overall in vivo response of the eyelids, corners of the mouth, lip margins, nasal alae, jawline, and cheek boundaries. For each lower-layer capsule... Voting vectors are generated by transforming weights connected to the four higher-level capsules. ,in In one alternative embodiment, each low-level capsule contains 16 neurons, and each high-level capsule contains 24 neurons, so each voting vector This corresponds to the voting result of a facial feature pointing to a stage of overall liveness response state change. All voting vectors at the same moment are organized into component voting states. , recorded as Component voting status It is no longer a simple splicing of features from individual parts, but a structured mapping result from facial parts to the overall liveness response state.

[0077] After completing the construction of the component voting state at each time point, proceed according to the interaction response sequence. Arrange the voting status of all components in chronological order to form a component voting sequence. During the arrangement process, two types of correspondences remain unchanged: one is the continuous correspondence between indices of the same part at adjacent times, and the other is the random instruction identifier. The defined semantics of the overall living response state and the set of high-level capsules The correspondence.

[0078] After completing the arrangement of the component voting sequence, proceed from the component voting sequence. Extracting consecutive component voting status pairs at adjacent time points ,in Each component voting state pair simultaneously retains the continuous change relationship between the component voting state at the previous time step and the component voting state at the current time step. This continuous change relationship is directly used as the state input for subsequent learning and calculation of the achievable change range of the living organism using the Koopman operator.

[0079] Existing technologies typically classify real and fake facial images directly from consecutive video frames or only detect the completion of a single action. They lack explicit modeling of the contribution of facial components to the overall liveness response state, making it difficult to capture the collaborative evolution of multiple parts over consecutive moments. The steps described above, by establishing transformation weights and voting relationships between the bottom-level capsule (first feature capsule) and the top-level capsule (second feature capsule), can convert the interactive response sequence into a component voting sequence. This ensures that the continuous changes in facial components driven by random instructions towards the overall liveness response state fall on calculable and referential component voting states. This provides a clear input object for the liveness reachability range constraint in dynamic routing, achieving the technical effect of providing structured input for subsequent calculations of the liveness reachability range.

[0080] Optionally, in the authentication method provided in Embodiment 1 of this application, the range of liveness prediction states is determined based on the component voting states at different times in the component voting sequence, and the voting relationship between the local image and the facial response state is updated based on the range of liveness prediction states to generate a liveness state sequence. This includes: extracting the first component voting state corresponding to the first time and the second component voting state corresponding to the second time from the component voting sequence in chronological order, wherein the first time and the second time are adjacent times; inputting the first component voting state into a pre-trained Kopman operator model to obtain the range of liveness prediction states at the second time; comparing the range of liveness prediction states with the second component voting state to obtain a comparison result; performing a weighted summation of the transformation weights of multiple first feature capsules corresponding to the second time in the component voting sequence based on the comparison result to obtain the liveness feature capsule response state at the second time; and generating a liveness state sequence based on the liveness feature capsule response state corresponding to each time in the component voting sequence.

[0081] In this embodiment 1, in order to constrain and correct the voting behavior of facial components driven by random instructions by utilizing the continuous law of the evolution of real living facial states, the voting state of components at adjacent time moments can be extracted, and the Koopman operator can be used to learn and predict the achievable change range of real living component voting at the current time moment (i.e., the range of living predicted state). This range is then used as a constraint condition for dynamic routing to update the voting aggregation relationship of local images to facial response states, thereby forming a living state sequence that conforms to the natural evolution law of living organisms.

[0082] Specifically, it receives the component voting sequence obtained from the previous step. ,in This indicates the number of moments in the component voting sequence. Indicates the first The voting status of each component at any given moment. The voting status of each component. It consists of voting vectors generated by 6 lower-level capsules to 4 higher-level capsules. To ensure that the Koopman operator learning and dynamic route updates use the same numerical object, the voting state of each component is recorded. Expanded into a voting state vector in a fixed order The fixed order is as follows: first, arrange by eyelids, corners of mouth, lip edges, nose wings, jawline, and cheek boundaries; then, arrange by four high-level capsules; and finally, arrange by the 24 components within a single voting vector. Following this order, any component... Each corresponds to a unique facial feature, a unique high-level capsule, and a unique voting dimension; therefore, the voting state vector... This can be directly used as numerical input for learning the Koopman operator. In a fixed example where the interaction response sequence contains 20 frames of facial response frames, each time step contains... Each group of voting vectors is 24-dimensional, therefore the voting state vector... It has 576 dimensions.

[0083] For each Extract the voting status of the components at the previous moment. (i.e., the voting status of the first component at the first moment) and the voting status of the component at the current moment. (i.e., the voting state of the second component corresponding to the second time step), and simultaneously extract the voting state vector of the previous time step. and the voting state vector at the current moment The voting state vector at the previous moment. Entering the mapping network , This represents the parameters of the mapping network. Mapping network. It contains a two-level fully connected structure. The first-level mapping layer receives a 576-dimensional voting state vector from the previous time step. Each mapping layer neuron simultaneously receives 576 voting components and performs weighted summation and nonlinear mapping to form a 128-dimensional intermediate representation. 128-dimensional intermediate representation Each component is calculated using all 576 voting components, and does not correspond to a single facial part, but rather to a comprehensive representation of the voting relationship between all facial parts and the overall liveness response state. The second-level state layer receives a 128-dimensional intermediate representation. Each state layer neuron simultaneously receives 128 intermediate representation components and performs weighted summation and nonlinear mapping to form a 64-dimensional state vector. 64-dimensional state vector It is a compressed numerical representation of the continuous changes in the voting state of the components at the previous moment. Each state component can be directly calculated from the output of the previous layer of neurons.

[0084] Next, the 64-dimensional state vector Enter the 64×64 linear state update layer. The 64×64 linear state update layer stores the Koopman state transformation matrix and updates the 64-dimensional state vector. Perform a linear state update to generate a 64-dimensional trend vector. Trend vector Then enter the decoding network , This indicates the parameters of the decoding network. Decoding network. It consists of two sets of fully connected output layers in parallel, each receiving a 64-dimensional trend vector. The first output layer generates a 576-dimensional lower bound vector. The second output layer generates a 576-dimensional upper bound vector. Lower bound vector and upper bound vector With voting state vector Using the exact same permutation order, therefore the lower bound vector and upper bound vector Each component can be correlated with the voting state vector at the current moment. The corresponding components are directly compared. The lower bound vector is used. and upper bound vector The range of possible changes for the actual living parts voting at the current moment was jointly determined. That is, the range of the predicted living state at the second moment mentioned above.

[0085] To achieve the range of variation achievable by living organisms To directly inject dynamic route updates, first use the 576-dimensional lower bound vector. 576-dimensional upper bound vector And the 576-dimensional voting state vector at the current moment Write back in the same order as three component objects: the current moment component voting state component. Lower bound component and upper bound components .in, Corresponding to the eyelids, corners of the mouth, lip edges, nose wings, jawline, and cheek boundaries. Corresponding to 4 high-level capsules, This corresponds to 24 components within a single voting vector. Subsequently, a dynamic routing comparison result is calculated for the voting relationship between each group of lower-level capsules and higher-level capsules. The calculation results are shown in Formula 1.

[0086] (1)

[0087] In the formula, Indicates the first At the [time]th moment The lower-layer capsule to the first The dynamic routing comparison results obtained when voting in a high-level capsule; The location index ranges from 1 to 6, corresponding to the eyelid, corner of the mouth, lip edge, nose wing, jawline, and cheek boundary, respectively. This indicates a high-level capsule index, with values ​​ranging from 1 to 4; This represents the internal dimension index of a single voting vector, with a value ranging from 1 to 24. Represents the time index, with values ​​ranging from 1 to... ; This represents the numerical components of the corresponding part, the corresponding high-level capsule, and the corresponding dimension in the current component voting status. Indicates the range of possible changes in living organisms. The corresponding lower bound numerical component; Indicates the range of possible changes in living organisms. The corresponding upper bound numerical component; This indicates taking the maximum value among the terms within the parentheses; 24 in the formula represents the number of components in a single voting vector. According to this definition, when... Falling into the lower bound components above the upper bound When the deviation is between 0 and 1, the deviation in the corresponding dimension is 0; when When the deviation falls outside the reachable range of a living organism, the deviation in the corresponding dimension is determined by the distance from the current component's voting state component to the nearest boundary component.

[0088] This is obtained It is a dimensionless scalar, with a value range from 0 to 1. The larger the value, the higher the value. The lower-layer capsule to the first The more consistent the current voting status of the components in each high-level capsule with the range of changes achievable by the living organism, the better. (This will involve...) Composition of dynamic route comparison results This refers to the comparison results described above. Dynamic routing comparison results. It is not an independent output object, but is directly used to update the dynamic routing coefficient matrix at the current moment. Before the first round of dynamic routing begins, assign the same initial value to the four aggregation relations corresponding to each lower-level capsule, so that... to The initial values ​​are equal. Then, the index of the fixed part is... Comparison results of the four dynamic routes to Normalization is performed, and the ratio of each dynamic route comparison result to the sum of the four dynamic route comparison results is written into the corresponding... and keep to The sum is 1. Following this method, voting relationships with larger dynamic routing comparison results result in larger aggregation relationships, and voting relationships with smaller dynamic routing comparison results result in smaller aggregation relationships. In an optional embodiment, dynamic routing is executed continuously for three rounds, with each round using the dynamic routing comparison results from the same point in time. For the dynamic routing coefficient matrix Update.

[0089] Complete the dynamic routing coefficient matrix for each round. After the update, the number will be... The six sets of 24-dimensional voting vectors corresponding to each high-level capsule are... A weighted sum is performed to form a 24-dimensional aggregated vector; then, a capsule compression process is applied to the 24-dimensional aggregated vector to keep its direction unchanged and limit its magnitude to between 0 and 1, forming the vector at the current time step. Individual living capsule The four living state capsules are arranged according to the high-level capsule index to form the living state capsule at the current moment. This refers to the liveness feature capsule response state at the second time point mentioned above. In an optional embodiment, each liveness state capsule... It is 24-dimensional, therefore the current living state capsule for Numerical structures. For all The same process was performed sequentially to obtain living capsules arranged in chronological order. When it is necessary to maintain the same number of moments as the interactive response sequence, The same dynamic routing process is executed once as initialization input, resulting in... Then, they are arranged in chronological order to form a live capsule sequence. This refers to the aforementioned live-state sequence. Live-state capsule sequence Each All Numerical structure, therefore, living state capsule sequence It can be written directly as The timing input is used for calculating the position of state changes.

[0090] By introducing the liveness prediction state range (i.e., the range of liveness reachable changes) at the dynamic routing level, the liveness state capsule not only retains the ability of the capsule network to express the relationship between components and the whole, but also includes the constraint on the natural continuity of time steps. This enables a more effective distinction between the differences between real liveness responses and deep pseudo-synthetic responses in terms of transition process, part coupling relationship and state arrival mode, thereby enhancing the robustness of identification and improving the accuracy of deep pseudo-identity discrimination.

[0091] Optionally, in the authentication method provided in Embodiment 1 of this application, determining the position of state change in the interaction response sequence that exceeds the predicted state range based on the liveness state sequence and extracting abnormal segments includes: aligning the liveness state sequence and the facial response frames in the interaction response sequence in chronological order to obtain a state correspondence sequence; extracting the liveness feature capsule response states at adjacent times from the state correspondence sequence, and determining the candidate positions of state changes based on the change relationship between the liveness feature capsule response states at adjacent times; comparing the liveness predicted state range corresponding to the candidate positions of state changes with the liveness change results at the candidate positions of state changes to obtain position deviation information; determining the candidate positions of state changes as state change positions when the position deviation information meets preset deviation conditions, thus obtaining a set of state change positions; merging and connecting the temporally continuous state change positions in the set of state change positions based on the position distribution information of the set of state change positions in the interaction response sequence to obtain a set of candidate abnormal segments; filtering candidate abnormal segments that overlap with the range of random command based on the interaction response sequence segments corresponding to each state change position in the set of candidate abnormal segments, and determining abnormal segments based on the filtered candidate abnormal segments.

[0092] In this embodiment 1, in order to locate and quantify time segments that deviate from the natural evolution of a real living organism during random command response, abnormal segments can be extracted by aligning the living organism state sequence and the interaction response sequence, comparing the state changes at adjacent times with the predicted range of the living organism, thereby generating verifiable deep fake identity presentation criteria based on spatiotemporal distribution.

[0093] Specifically, the live capsule sequence is denoted as ,in, Indicates the first Capsule representing the living state at a given moment. Organized into 4 high-level capsules and 24 components within each high-level capsule. Numerical structure; interactive response sequence denoted as The sequence of achievable ranges of change in living organisms corresponding to adjacent time points is denoted as... ,in, It consists of a 576-dimensional lower bound vector and a 576-dimensional upper bound vector corresponding to the voting state of the components at the current moment. and Indexed by the same time Alignment is performed to form a state correspondence sequence. ,in, In one optional embodiment, the interaction response sequence includes 20 facial response frames and a state-corresponding sequence. It contains 20 time points, corresponding to 19 adjacent state change positions. The 19 adjacent state change positions correspond to... .

[0094] For each ,Will As a comparison input for candidate state change locations, the state change location calculation layer is configured with a comparison neuron structure consistent with the number of adjacent state change locations; in an optional embodiment, the state change location calculation layer contains 19 comparison neurons, each receiving a living state capsule from the previous time step. Current living state capsule and the corresponding range of variation that can be achieved in living organisms The comparison of neurons first involves... and The state is expanded in a fixed order according to the high-level capsule index and the 24 components within the capsule, resulting in a 96-dimensional previous state vector and a 96-dimensional current state vector. Then, the 96-dimensional current state vector is subtracted dimension by dimension from the 96-dimensional previous state vector to obtain the 96-dimensional state change result. , Each component represents the direct numerical change between adjacent time points within the same high-level capsule and its internal location. Comparison of neurons simultaneously considers the achievable range of variation in living organisms. The fixed correspondence between the eyelids, corners of the mouth, lip edges, nasal wings, jawline, cheek boundaries, and four high-level capsules is restored as follows: The upper and lower bound components are then aggregated according to the component voting relationship used when forming the living state capsule at the current moment. The upper and lower bound components are aggregated into capsules that correspond one-to-one with the living state capsule at the current moment. Current reachable lower bound matrix and Current reachable upper bound matrix To ensure consistency in the comparison objects, the comparison neurons then... and Capsules in their previous living state Subtracting dimension by dimension, we get Reachable change lower bound matrix and Reachable change upper bound matrix After this adjustment, the state change results are as follows: With the lower bound matrix of reachable change achievable change upper bound matrix It corresponds perfectly to the high-level capsule index and the location within the capsule.

[0095] The comparison neurons perform boundary comparisons dimension-by-dimensionally according to the 96 corresponding components. When A component is less than the corresponding lower bound component of reachable change, or When a component is greater than the corresponding upper bound of reachable change, the corresponding component is recorded as the deviation component; when When a component falls between the corresponding lower bound component and the upper bound component of reachable change, the corresponding component is recorded as a consistent component. The number of deviation components among the 96 components is recorded as... , Indicates the first The deviation count of each state change candidate position. Deviation from preset threshold Comparison: When Less than At that time, compare the deviation of the neuron output position from the result. ;when Not less than At that time, compare the deviation of the neuron output position from the result. The positional deviation result sequence is denoted as... This refers to the aforementioned positional deviation information. Based on the positional deviation result sequence... Determine the set of locations of state changes Set of state change locations Each index in All correspond to the interactive response sequence from arrive A position of a state change that deviates from the range of change achievable by a living organism.

[0096] Obtain the set of state change locations Then, the set of state change locations is arranged according to index continuity. Perform interval merging. Merge all state change position indices that satisfy the integer continuous relationship into continuous intervals. ,in, Indicates the first The initial state change position of a continuous interval Indicates the first The position of the end state change of a series of consecutive intervals. For any two adjacent consecutive intervals... and When the index interval between two consecutive intervals is less than a preset frame number threshold When two consecutive intervals are connected, they are considered as the same candidate interval for an abnormal segment; when the index interval between two consecutive intervals is not less than a preset frame number threshold. At the same time, the two consecutive intervals are kept separate. After the connection is completed, a set of candidate abnormal segments is formed. That is, the aforementioned set of candidate abnormal fragments, where, This indicates the number of candidate intervals for abnormal segments. Subsequently, based on the state-correspondence sequence... In each abnormal segment candidate interval Extract the corresponding interactive response sequence fragments. And read the random instruction identifiers corresponding to each facial response frame within the interval. For each abnormal segment candidate interval When the interval covers at least one start response position corresponding to a random instruction and the interval covers at least a preset coverage frame threshold both before and after the start response position. When viewing facial response frames, candidate regions for abnormal segments are selected. Reserved; when the interval falls only in the stable response segment of the same random instruction and does not cover the change process triggered by the random instruction, the candidate interval for the abnormal segment is reserved. Elimination. The remaining intervals are mapped back to the interaction response sequence in chronological order, forming anomalous fragments. ,in, , Indicates the number of abnormal segments.

[0097] By locating the state change positions that deviate from the predicted liveness state range and extracting abnormal segments covering the random instruction triggering process, quantitative judgment criteria can be generated based on the spatiotemporal distribution of abnormal segments. This allows the judgment results to correspond to the abnormal change segments in the specific instruction response process, reflecting not only the abnormality of the coordinated changes between local components and the overall state, but also improving the interpretability of the judgment. This better meets the application requirements of financial institutions for remote identity verification in terms of interaction authenticity, process continuity, and result verifiability.

[0098] Optionally, in the authentication method provided in Embodiment 1 of this application, determining the authentication result of the facial video based on the abnormal segment includes: determining the start state change position and end state change position corresponding to the abnormal segment in the interaction response sequence, and counting the number of state change positions in the abnormal segment to obtain the number of abnormalities; determining the percentage of abnormalities based on the number of abnormalities and the total number of state change positions in the interaction response sequence; if the percentage of abnormalities is greater than a preset abnormal threshold, determining the criteria for deep fake identity determination based on the time distribution information of the abnormal segment in the interaction response sequence, the number of abnormalities, and the percentage of abnormalities; if the percentage of abnormalities is less than or equal to the preset abnormal threshold, determining the criteria for real liveness determination based on the time distribution information of the abnormal segment in the interaction response sequence, the number of abnormalities, and the percentage of abnormalities; determining the target determination criteria based on the criteria for deep fake identity determination or the criteria for real liveness determination, and generating an authentication result for the user based on the target determination criteria.

[0099] In this embodiment 1, in order to quantify the risk assessment based on the coverage of abnormal fragments during the interaction process and output the final identity verification result, the number and proportion of state change locations covered by abnormal fragments can be statistically analyzed and compared with a preset threshold to generate a basis for determining deep fake or real liveness, thereby enabling the output of verifiable identity verification results based on abnormal spatiotemporal distribution characteristics.

[0100] Specifically, the interactive response sequence is denoted as ,in, Indicates the first One facial response frame. This indicates the number of facial response frames in the interactive response sequence; abnormal segments are denoted as... ,in, Indicates the first An abnormal segment, This indicates the number of abnormal segments. Each abnormal segment... It consists of a continuous sequence of state changes within the interactive response sequence. To ensure that the temporal distribution of anomalous segments within the interactive response sequence can directly participate in the determination, each anomalous segment... Scan all covered state change position indices and take the minimum state change position index as the starting state change position. Take the index of the maximum state change position as the end state change position. In one alternative embodiment, the interaction response sequence comprises 20 facial response frames, so all state change locations fall within... to Within the index range. The positions of all abnormal segments are changed according to their initial state. The segments are sorted in ascending order to ensure that the temporal distribution of anomalous segments within the interactive response sequence remains consistent with the sequence itself. Subsequently, the coverage length is calculated for each anomalous segment. Coverage length Take the position as the change from the initial state. To the position of the final state change The number of state change locations covered. As the first A fragment record for each abnormal fragment is used to preserve the location boundaries and coverage of the abnormal fragment in the interactive response sequence.

[0101] After obtaining all fragment records, the coverage length corresponding to all abnormal fragments is... Accumulation results in an abnormal number. Number of anomalies Using the number of state change locations as the unit of measurement, instead of the number of abnormal segments, thus reducing the number of abnormalities... This directly corresponds to the actual coverage of the abnormal segments at the locations of state changes in the interaction response sequence. It is based on the number of facial response frames in the interaction response sequence. Determine the total number of all state change locations. The total number of all state change locations The total number of state change locations that can be formed between adjacent facial response frames is taken as the total number of all state change locations. for In one alternative embodiment, the total number of all state change locations. The number is 19. (This refers to the number of anomalies.) Total number of positions with all state changes Perform ratio calculations to determine the percentage of abnormalities. Abnormal percentage It directly represents the coverage ratio of abnormal segments in all state change locations, thereby compressing the temporal distribution and coverage of abnormal segments in the interactive response sequence into a single comparable value.

[0102] abnormal proportion With abnormal threshold The comparisons are made to generate anomaly comparison results. Anomaly threshold. A preset threshold is used to distinguish whether the coverage ratio of abnormal segments meets the criteria for deep fake identity presentation. When the abnormal percentage Meets the abnormal threshold At that time, the abnormal number Abnormal proportion Together with all the fragment records arranged in chronological order, they form the basis for determining deep fakes. Criteria for determining deep fakeness In the middle, the number of anomalies Used to represent the total number of locations of state changes that deviate from the range of changes achievable by a living organism, and the percentage of abnormalities. This is used to represent the proportion of deviations within all state change locations, and the fragment record is used to represent the temporal distribution of abnormal fragments in the interaction response sequence. When the abnormal proportion... The abnormal threshold is not met. At that time, the abnormal number Abnormal proportion Together with all the fragmented records arranged in chronological order, they form the basis for determining whether a person is truly alive. Criteria for determining a living organism Criteria for determining deep fake Using the same data composition method, the only difference lies in the judgment type corresponding to the anomaly comparison results. Through this process, anomaly fragments are first quantified into the number of anomalies. Then normalized to the proportion of abnormalities Finally, the temporal distribution of abnormal fragments in the interactive response sequence together forms the basis for determining deep pseudo-speculiarities. Or the basis for determining whether a person is actually alive .

[0103] By statistically analyzing the coverage ratio (abnormal percentage) of anomalous fragments in the interactive response sequence and limiting these fragments to changes triggered by random commands, the judgment criteria directly reflect the degree of abnormality in the co-evolution of local and overall facial liveness states. This quantitative judgment based on spatiotemporal distribution and coverage ratio can more accurately capture the characteristics of inconsistencies between local components and the overall state in deepfake attacks, thereby solving the problem of misjudgment caused by the lack of structured correspondence and achieving the technical effect of improving the accuracy and verifiability of remote identity verification.

[0104] Optionally, in the identity verification method provided in Embodiment 1 of this application, determining the target determination basis based on the deepfake determination basis or the real liveness determination basis, and generating an identity verification result for the user based on the target determination basis, includes: if the target determination basis is a deepfake determination basis, determining the deepfake identity determination result based on the determination rule corresponding to the deepfake determination basis and the deepfake determination basis; if the target determination basis is a real liveness determination basis, determining the real liveness determination result based on the determination rule corresponding to the real liveness determination basis and the real liveness determination basis; and determining the identity verification result for the user based on the deepfake identity determination result or the real liveness determination result.

[0105] In this embodiment 1, in order to transform the quantified abnormal distribution features into the final identity verification conclusion, a deep fake or real liveness determination result can be generated based on the target determination, the type and corresponding determination rule, so as to output a bank remote identity verification result with clear basis and verifiability.

[0106] Specifically, the criteria for determining deep falsehoods are denoted as follows: The criteria for determining a truly living person are recorded as follows: . or All due to the number of anomalies Abnormal proportion and abnormal fragment location boundary sequence Composition, in which, This indicates the number of state change locations covered by the abnormal fragment. This indicates the percentage of anomalies among all state change locations. Indicates the first The starting state change positions corresponding to each abnormal segment. Indicates the first The position of the end state change corresponding to each abnormal segment. Indicates the number of abnormal segments. Receive or Next, the object category of the receiving object is read to form the type of judgment criterion. When the receiving object is At that time, the determination will be based on the type. This is determined to be the type of basis for deep fake detection; when the receiving object is At that time, the determination will be based on the type. This is determined to be the type of basis for determining a living organism. (Type of basis for determination) Additional information that is not generated independently, the determination is based on type. Directly from or The object category is used to ensure that the content of the judgment criteria and the direction of the result mapping are consistent.

[0107] In terms of the type of judgment criteria Once confirmed, extract. or The numerical content actually involved in the result mapping. The data used in the result mapping includes the number of anomalies. Abnormal proportion and abnormal fragment location boundary sequence The determination will be based on the type of... Number of abnormalities Abnormal proportion and abnormal fragment location boundary sequence Combined into a result mapping object , where the result mapping object This is used to hold all the input fields required for the result mapping. Result Mapping Object Redundant fields not involved in the result mapping are no longer retained, thus making the result mapping object... You can directly proceed to result mapping processing. Result mapping object The construction order is fixed as writing the judgment criteria type first. Then write the number of exceptions. Abnormal proportion and abnormal fragment location boundary sequence This ensures that the order of the result fields in the bank's remote identity verification is fixed.

[0108] Map the object based on the result. When performing result mapping, first read the result mapping object. The type of judgment criteria in When the judgment is based on type When determining the type of deepfake identity, the deepfake identity presentation identifier is written into the result type field, and the result mapping object is set. Number of anomalies Abnormal proportion and abnormal fragment location boundary sequence This is bound to the deepfake identity presentation judgment identifier to form the deepfake identity presentation judgment result. Among them, the results of the deep fake identity presentation judgment. The result type field corresponds to the deepfake identity presentation judgment result. The corresponding judgment content field , and When the judgment is based on type When the type used for determining true liveness is specified, the true liveness determination identifier is written into the result type field, and the result mapping object is set. Number of anomalies Abnormal proportion and abnormal fragment location boundary sequence Bind to the real liveness detection identifier to generate a real liveness detection result. Among them, the results of the actual liveness determination The result type field corresponds to the actual liveness determination result. The corresponding judgment content field , and .

[0109] After completing the result mapping, the deepfake identity determination result will be presented. Or the result of actual liveness determination Write the judgment return record according to the judgment return format of the bank's remote identity verification. The result type field in the judgment return record can only take either a deep fake identity presentation judgment result or a real liveness judgment result. The judgment content field in the judgment return record is fixed in terms of the number of exceptions. Abnormal proportion and abnormal fragment location boundary sequence This enables banks to directly read the corresponding deepfake identity verification result or real liveness verification result based on the same judgment return record, thus allowing remote identity verification.

[0110] By generating corresponding judgment results based on specific data (such as the percentage of abnormalities and time distribution) in the criteria for judging deep falsehoods or real living organisms, the accuracy of the judgment results is not only improved, but also the interpretability and verifiability of the results are enhanced by retaining the abnormal distribution information. This effectively solves the problems of misjudgment and low credibility caused by the lack of structured correspondence expression.

[0111] Optionally, in this embodiment 1, Figure 4 This is a schematic diagram of the network structure of an optional deep pseudo-identity presentation determination method based on capsule network and Koopman operator learning, provided in Embodiment 1 of this application. Figure 4As shown, firstly, the input includes six local regions: eyelids, corners of the mouth, lip edges, nasal alae, jaw, and cheek boundaries. These regions are mapped to 6×16 bottom-level capsules through two 3×3 convolutional layers (Conv) and a 1×128 fully connected layer (FC). These capsules then vote on the results to four higher-level capsules, forming a 6×4×24 dimensional component voting state, which is then expanded into a 576-dimensional vector. Next, based on the voting state from the previous time step, the state is mapped to 64 dimensions through a 1×128 fully connected layer (FC) and a 1×64 fully connected layer (FC). After a 64×64 dimensional linear update and two 1×576 fully connected layers (FC) for decoding, a 576-dimensional lower bound vector and a 576-dimensional upper bound vector are generated, determining the achievable range of variation for the living organism. Secondly, the voting vector at the current time step (6×4×24, 576 dimensions) is compared dimension-by-dimensionally with the upper bound vector (576 dimensions) and the lower bound vector (576 dimensions) to update the dynamic routing weights. After three rounds of weighted aggregation and compression (Squash), a living state capsule (4×24 dimensions) at time t is obtained, thus forming a complete living state sequence containing 20 time steps. Each time step corresponds to a living state capsule with a dimension of 4×24, forming a 20×4×24 time-series data sequence. By comparing neurons, the living state capsules at adjacent time steps in the state sequence are compared pairwise. Since the sequence contains 20 time steps, a total of 19 comparison positions at adjacent time steps are generated, i.e., 19 candidate positions for state changes. The number of abnormal positions that meet the preset deviation conditions among these 19 positions is counted, and the abnormality rate is calculated based on the total number of positions (19). The calculated anomaly percentage is compared with a preset anomaly threshold. If the anomaly percentage meets a specific condition, a basis for deep fake identity determination is generated; otherwise, a basis for real liveness determination is generated, thereby outputting the final bank remote identity verification result, realizing a closed loop from time-series state comparison to quantitative determination.

[0112] Optionally, in this embodiment 1, Figure 5 This is a schematic diagram of the deviation of optional time and facial components in two-dimensional space according to Embodiment 1 of this application. Figure 5 As shown, the horizontal axis represents the time frames t1 to t8 of the interactive response sequence, and the vertical axis represents components such as eyelids, corners of the mouth, lips, nose, jaw, and cheek boundaries. Each grid represents the degree of deviation between the voting and the corresponding component at that time: white indicates that the voting falls within the range of possible variation in a living organism, which is normal; light gray indicates slight deviation; and diagonal grid patterns indicate significant deviation. The brackets below the figure show the time period corresponding to a certain random instruction. It can be seen that within this time period, light gray and grid patterns appear continuously on the eyelids, corners of the mouth, lips, and cheek boundaries, indicating that these local areas experienced systematic deviations when executing the instruction, thus locating deep pseudo-abnormal segments.

[0113] Optionally, in this embodiment 1, Figure 6This is a schematic diagram of the optional achievable range of variation in living organisms provided in Embodiment 1 of this application. For example... Figure 6 As shown, the contour lines illustrating the reachable range and deviation of the live organism, learned through the Koopman operator, are displayed in a low-dimensional state space. The horizontal axis represents the predicted state, the vertical axis represents the observed state, the central ellipse is the core reachable region, and the regions outside it are successively smaller and larger deviations. The outer diagonal lines represent high deviation areas outside the contour lines. Solid dots represent real live organism samples, mostly distributed within the area enclosed by the inner contour lines, indicating that the trained reachable range can cover normal live organism variations. Crosses represent deep false samples, some of which fall within the textured area or outside the ellipse, and are judged as abnormal states deviating from the reachable range of the live organism.

[0114] Optionally, in this embodiment 1, Figure 7 This is a schematic diagram of an optional three-dimensional deviation intensity surface map constructed based on time frames and component indices, according to Embodiment 1 of this application. Figure 7 As shown, a three-dimensional coordinate system is constructed based on time frames and component indices to display the three-dimensional distribution of deviation intensity. The horizontal coordinate of the bottom grid represents time t1 to t3, the vertical coordinate represents the component index (e.g., component 1, component 2, etc.), and the vertical coordinate represents the deviation intensity under that time-component combination. The surface formed by the polyline grid reflects the trend of deviation changes with time and components. The polygonal area filled with diagonal lines in the center represents an abnormal area where the deviation intensity exceeds the threshold, corresponding to the segment in the interaction response sequence that is judged as a deepfake attack. Through this three-dimensional view, the peak deviation positions of different components at different times and the connectivity of abnormal areas in time and space can be observed simultaneously, thus providing an intuitive basis for the overall judgment of attack segments.

[0115] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0116] In summary, the identity verification method provided in this application transforms the pixel features of local facial components into capsule sequences that perform structured voting on the overall liveness response state. It then uses the liveness prediction state range to dynamically constrain and update the voting relationship. This explicitly models the co-evolutionary relationship between local components such as eyelids, corners of the mouth, lip edges, nose wings, jawline, and cheek boundaries and the overall liveness state during the detection process. Even if a deepfake attack can simulate a single command action, it can identify anomalies by detecting unnatural deviations in the voting process from local components to the overall state. This avoids misjudgments caused by a lack of structured correspondence between local and overall components, achieving the technical effect of accurately distinguishing between genuine liveness and deepfake identity presentation. Therefore, it can solve the technical problem in related technologies where the structured correspondence between local facial locations and the overall liveness state is insufficiently expressed, making it difficult to reflect the true co-changes of various facial components and leading to inaccurate detection results during remote identity verification. This achieves the technical effect of improving the security and accuracy of remote identity verification.

[0117] Example 2

[0118] This application also provides an authentication device. It should be noted that the authentication device of this application can be used to execute the authentication method provided in this application. The authentication device provided in this application is described below.

[0119] According to an embodiment of this application, an apparatus for implementing the above-described authentication method is also provided. Figure 8 This is a schematic diagram of an authentication device provided according to Embodiment 2 of this application. Figure 8 As shown, the device includes: an extraction unit 801, a voting unit 802, a prediction unit 803, and a verification unit 804.

[0120] Specifically, the extraction unit 801 is used to receive a facial video returned by the user according to a random command when the user authorizes the collection of biometric information, extract the response video frame corresponding to the random command from the facial video, and obtain an interactive response sequence.

[0121] Voting unit 802 is used to extract local images of multiple facial components from the interaction response sequence, determine the facial response state based on the interaction response sequence and random instructions, and establish a voting relationship between the local images and the facial response state to obtain the component voting sequence.

[0122] The prediction unit 803 is used to determine the range of liveness prediction states based on the voting states of components at different times in the component voting sequence, update the voting relationship between local images and facial response states based on the range of liveness prediction states, and generate a liveness state sequence.

[0123] The verification unit 804 is used to determine the position of the state change that exceeds the predicted state range in the interaction response sequence based on the liveness state sequence, extract abnormal segments, and determine the identity verification result of the facial video based on the abnormal segments.

[0124] The identity verification device provided in this application transforms the pixel features of local facial components into capsule sequences that perform structured voting on the overall liveness response state. It then dynamically constrains and updates the voting relationship using the liveness prediction state range. This explicitly models the co-evolutionary relationship between local components such as eyelids, corners of the mouth, lip edges, nose wings, jawline, and cheek boundaries and the overall liveness state during the detection process. Even if a deepfake attack can simulate a single command action, it can identify anomalies by detecting unnatural deviations in the voting process from local components to the overall state. This avoids misjudgments caused by a lack of structured correspondence between local and overall components, achieving the technical effect of accurately distinguishing between genuine liveness and deepfake identity presentation. Therefore, it can solve the technical problem in related technologies where the structured correspondence between local facial locations and the overall liveness state is insufficiently expressed, making it difficult to reflect the true co-changes of various facial components and leading to inaccurate detection results during remote identity verification. This achieves the technical effect of improving the security and accuracy of remote identity verification.

[0125] Optionally, in the identity verification device provided in Embodiment 2 of this application, the extraction unit 801 includes: a first determining subunit, used to determine the display order of random instructions and the timestamp information of random instructions, wherein the timestamp information includes: start display time and end display time; a first extraction subunit, used to extract continuous facial video frames corresponding to random instructions from the facial video according to the timestamp information, and sort them according to the time order to obtain a candidate response frame sequence; a first filtering subunit, used to identify the facial region and facial parts of each frame in the candidate response frame sequence based on the target detection algorithm, and filter facial video frames with continuous facial movement and complete facial parts in the candidate response frame sequence according to the facial region and facial parts to obtain facial response frames; and a first sorting subunit, used to sort the facial response frames according to the display order of random instructions and the time information of facial response frames to obtain an interactive response sequence.

[0126] Optionally, in the authentication device provided in Embodiment 2 of this application, the voting unit 802 includes: a first alignment subunit, configured to perform frame alignment processing on facial response frames in the interaction response sequence based on the position information of multiple facial components, extract local images of multiple facial components in each aligned frame, and obtain multiple first feature capsules corresponding to each facial response frame; a second determination subunit, configured to determine the facial response state corresponding to the random instruction based on the correspondence between the random instruction and the facial response frames in the interaction response sequence and the multiple first feature capsules corresponding to each facial response frame, and construct a second feature capsule based on the facial response state; a construction subunit, configured to establish the transformation weights between the multiple first feature capsules and the second feature capsules corresponding to each facial response frame, and construct the component voting state of each facial response frame based on the transformation weights of the multiple first feature capsules corresponding to each facial response frame; and a second sorting subunit, configured to sort the component voting states of each facial response frame according to the time order of the interaction response sequence to obtain a component voting sequence.

[0127] Optionally, in the identity verification device provided in Embodiment 2 of this application, the prediction unit 803 includes: a second extraction subunit, configured to extract the first component voting state corresponding to the first time moment and the second component voting state corresponding to the second time moment from the component voting sequence in chronological order, wherein the first time moment and the second time moment are adjacent times; a first comparison subunit, configured to input the first component voting state into a pre-trained Kopman operator model to obtain the liveness prediction state range at the second time moment, and compare the liveness prediction state range with the second component voting state to obtain a comparison result; a first processing subunit, configured to perform a weighted summation of the transformation weights of multiple first feature capsules corresponding to the second time moment in the component voting sequence based on the comparison result to obtain the liveness feature capsule response state at the second time moment; and a generation subunit, configured to generate a liveness state sequence based on the liveness feature capsule response state corresponding to each time moment in the component voting sequence.

[0128] Optionally, in the identity verification device provided in Embodiment 2 of this application, the verification unit 804 includes: a second alignment subunit, used to align the facial response frames in the live state sequence and the interaction response sequence in chronological order to obtain a state correspondence sequence; a second comparison subunit, used to extract the live feature capsule response states at adjacent times from the state correspondence sequence, and determine the candidate positions of state changes based on the change relationship between the live feature capsule response states at adjacent times, and compare the live predicted state range corresponding to the candidate positions of state changes with the live change results of the candidate positions of state changes to obtain position deviation information; a third determination subunit, used to determine the candidate positions of state changes as state change positions when the position deviation information meets the preset deviation conditions, to obtain a set of state change positions; a second processing subunit, used to merge and connect the temporally continuous state change positions in the set of state change positions based on the position distribution information of the set of state change positions in the interaction response sequence to obtain a set of candidate abnormal segments; and a second filtering subunit, used to filter candidate abnormal segments that overlap with the range of random command based on the interaction response sequence segments corresponding to each state change position in the set of candidate abnormal segments, and determine abnormal segments based on the filtered candidate abnormal segments.

[0129] Optionally, in the identity verification device provided in Embodiment 2 of this application, the aforementioned verification unit 805 includes: a statistics subunit, used to determine the start state change position and end state change position corresponding to the abnormal segment in the interaction response sequence, and to count the number of state change positions in the abnormal segment to obtain the number of abnormalities; a fourth determination subunit, used to determine the percentage of abnormalities based on the number of abnormalities and the total number of state change positions in the interaction response sequence; a fifth determination subunit, used to determine the criteria for deep fake identity determination based on the time distribution information of the abnormal segment in the interaction response sequence, the number of abnormalities, and the percentage of abnormalities when the percentage of abnormalities is greater than a preset abnormal threshold; a sixth determination subunit, used to determine the criteria for real liveness determination based on the time distribution information of the abnormal segment in the interaction response sequence, the number of abnormalities, and the percentage of abnormalities when the percentage of abnormalities is less than or equal to a preset abnormal threshold; and a seventh determination subunit, used to determine the target determination criteria based on the criteria for deep fake identity determination or the criteria for real liveness determination, and to generate an identity verification result for the user based on the target determination criteria.

[0130] Optionally, in the identity verification device provided in Embodiment 2 of this application, the seventh determining subunit includes: a first determining module, used to determine the deepfake identity determination result based on the determination rules corresponding to the deepfake determination basis and the deepfake determination basis when the target determination basis belongs to the deepfake determination basis; a second determining module, used to determine the real liveness determination result based on the determination rules corresponding to the real liveness determination basis and the real liveness determination basis when the target determination basis belongs to the real liveness determination basis; and a third determining module, used to determine the identity verification result for the user based on the deepfake identity determination result or the real liveness determination result.

[0131] It should be noted that the extraction unit 801, voting unit 802, prediction unit 803, and verification unit 804 mentioned above correspond to steps S201 to S204 in Embodiment 1. The two modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules or units can be hardware or software components stored in memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n). The above modules can also be part of a device and run in the computer terminal 10 provided in Embodiment 1.

[0132] Example 3

[0133] Embodiments of this application may provide an electronic device. Figure 9 This is a schematic diagram of an authentication electronic device provided according to Embodiment 3 of this application. Figure 9 As shown, the electronic device may include: one or more ( Figure 9 (Only one is shown) processor 902, memory 904, memory controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.

[0134] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the methods and apparatus in the embodiments of this application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby implementing the above-described methods. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0135] The processor can access information and applications stored in memory via a transmission device to perform the following steps: Upon detecting user authorization to collect biometric information, it receives a facial video returned by the user according to random instructions; extracts response video frames corresponding to the random instructions from the facial video to obtain an interactive response sequence; extracts local images of multiple facial components from the interactive response sequence; determines the facial response state based on the interactive response sequence and random instructions; establishes a voting relationship between the local images and the facial response states to obtain a component voting sequence; determines the liveness prediction state range based on the component voting states at different times in the component voting sequence; updates the voting relationship between the local images and the facial response states based on the liveness prediction state range to generate a liveness state sequence; determines the positions of state changes exceeding the predicted state range in the interactive response sequence based on the liveness state sequence; extracts abnormal segments; and determines the identity verification result of the facial video based on the abnormal segments.

[0136] The processor can access information and applications stored in memory via a transmission device to execute the following steps: extracting response video frames corresponding to random instructions from facial video to obtain an interactive response sequence, including: determining the display order of random instructions and the timestamp information of random instructions, wherein the timestamp information includes: start display time and end display time; extracting continuous facial video frames corresponding to random instructions from the facial video based on the timestamp information, and sorting them according to time order to obtain a candidate response frame sequence; identifying the facial region and facial parts of each frame in the candidate response frame sequence based on a target detection algorithm, and selecting facial video frames with continuous facial movement and complete facial parts from the candidate response frame sequence based on the facial region and facial parts to obtain facial response frames; sorting the facial response frames according to the display order of random instructions and the time information of the facial response frames to obtain an interactive response sequence.

[0137] The processor can invoke information and application programs stored in memory via a transmission device to execute the following steps: extracting local images of multiple facial components from the interactive response sequence, determining the facial response state based on the interactive response sequence and random instructions, and establishing a voting relationship between the local images and the facial response state to obtain a component voting sequence, including: performing frame alignment processing on facial response frames in the interactive response sequence based on the positional information of multiple facial components; extracting local images of multiple facial components from each aligned frame to obtain multiple first feature capsules corresponding to each facial response frame; determining the facial response state corresponding to the random instruction based on the correspondence between the random instruction and the facial response frames in the interactive response sequence and the multiple first feature capsules corresponding to each facial response frame, and constructing a second feature capsule based on the facial response state; establishing transformation weights between the multiple first feature capsules and the second feature capsules corresponding to each facial response frame, and constructing the component voting state of each facial response frame based on the transformation weights of the multiple first feature capsules corresponding to each facial response frame; and sorting the component voting states of each facial response frame according to the time order of the interactive response sequence to obtain a component voting sequence.

[0138] The processor can invoke information and applications stored in the memory via a transmission device to execute the following steps: determining the range of liveness prediction states based on the component voting states at different times in the component voting sequence; updating the voting relationship between the local image and facial response states based on the range of liveness prediction states; and generating a liveness state sequence, including: extracting the first component voting state corresponding to the first time moment and the second component voting state corresponding to the second time moment from the component voting sequence in chronological order, wherein the first time moment and the second time moment are adjacent times; inputting the first component voting state into a pre-trained Kopman operator model to obtain the range of liveness prediction states at the second time moment; comparing the range of liveness prediction states with the second component voting state to obtain a comparison result; performing a weighted summation of the transformation weights of multiple first feature capsules corresponding to the second time moment in the component voting sequence based on the comparison result to obtain the liveness feature capsule response state at the second time moment; and generating a liveness state sequence based on the liveness feature capsule response state corresponding to each time moment in the component voting sequence.

[0139] The processor can access information and applications stored in memory via a transmission device to execute the following steps: Based on the liveness state sequence, determine the location of state changes exceeding the predicted state range in the interaction response sequence, and extract abnormal segments, including: aligning the liveness state sequence and facial response frames in the interaction response sequence in chronological order to obtain a state correspondence sequence; extracting the liveness feature capsule response states at adjacent time points from the state correspondence sequence, and determining candidate positions for state changes based on the change relationship between the liveness feature capsule response states at adjacent time points; comparing the predicted liveness state range corresponding to the candidate positions with the liveness change results at the candidate positions to obtain position deviation information; if the position deviation information meets preset deviation conditions, determining the candidate positions for state changes as state change positions to obtain a set of state change positions; merging and connecting temporally continuous state change positions in the set of state change positions based on the positional distribution information of the set of state change positions in the interaction response sequence to obtain a set of candidate abnormal segments; based on the interaction response sequence segments corresponding to each state change position in the set of candidate abnormal segments, filtering candidate abnormal segments that overlap with the range of random instructions, and determining abnormal segments based on the filtered candidate abnormal segments.

[0140] The processor can invoke information and applications stored in the memory via a transmission device to perform the following steps: determining the identity verification result of the facial video based on abnormal segments, including: determining the start and end state change positions corresponding to the abnormal segments in the interaction response sequence, and counting the number of state change positions in the abnormal segments to obtain the number of abnormalities; determining the percentage of abnormalities based on the number of abnormalities and the total number of state change positions in the interaction response sequence; if the percentage of abnormalities is greater than a preset abnormality threshold, determining the criteria for deep fake identity determination based on the time distribution information of the abnormal segments in the interaction response sequence, the number of abnormalities, and the percentage of abnormalities; if the percentage of abnormalities is less than or equal to the preset abnormality threshold, determining the criteria for real liveness determination based on the time distribution information of the abnormal segments in the interaction response sequence, the number of abnormalities, and the percentage of abnormalities; determining the target determination criteria based on the criteria for deep fake identity determination or the criteria for real liveness determination, and generating an identity verification result for the user based on the target determination criteria.

[0141] The processor can invoke information and applications stored in the memory via a transmission device to perform the following steps: determining the target determination basis based on either deepfake determination basis or real liveness determination basis; generating an authentication result for the user based on the target determination basis, including: if the target determination basis is a deepfake determination basis, determining a deepfake identity determination result based on the determination rules corresponding to the deepfake determination basis and the deepfake determination basis; if the target determination basis is a real liveness determination basis, determining a real liveness determination result based on the determination rules corresponding to the real liveness determination basis and the real liveness determination basis; and determining an authentication result for the user based on either the deepfake identity determination result or the real liveness determination result.

[0142] This application provides an identity verification method that transforms pixel features of local facial components into capsule sequences that perform structured voting on the overall liveness response state. The voting relationship is dynamically constrained and updated using the liveness prediction state range. This explicitly models the co-evolutionary relationship between local components such as eyelids, corners of the mouth, lip edges, nose wings, jawline, and cheek boundaries and the overall liveness state during the detection process. Even if a deepfake attack can simulate a single command action, anomalies can be identified by detecting unnatural deviations during the voting process from local components to the overall state. This avoids misjudgments caused by a lack of structured correspondence between local and overall components, achieving the technical effect of accurately distinguishing between genuine liveness and deepfake identity presentation. Therefore, it solves the technical problem in related technologies where the structured correspondence between local facial locations and the overall liveness state is insufficiently expressed, making it difficult to reflect the true co-changes of various facial components and leading to inaccurate detection results during remote identity verification. This improves the security and accuracy of remote identity verification.

[0143] Those skilled in the art will understand that Figure 9 The structure shown is for illustrative purposes only. Electronic devices can also be smartphones, tablets, handheld computers, mobile internet devices (MIDs), PADs, and other terminal devices. Figure 9 This does not limit the structure of the aforementioned electronic device. For example, electronic devices may also include components that are more... Figure 9 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 9 The different configurations shown.

[0144] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0145] Example 4

[0146] Embodiments of this application also provide a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the authentication method provided in Embodiment 1.

[0147] Optionally, in this embodiment, the storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.

[0148] This application also provides a computer program product that, when executed on a data processing device, is suitable for performing authentication method steps.

[0149] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0150] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0151] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0152] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0153] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0154] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0155] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. An authentication method, characterized in that, include: When the user authorizes the collection of biometric information, the system receives a facial video returned by the user according to a random instruction, extracts the response video frame corresponding to the random instruction from the facial video, and obtains an interactive response sequence. Multiple partial images of facial components are extracted from the interaction response sequence. The facial response state is determined based on the interaction response sequence and the random command. A voting relationship is established between the partial images and the facial response state to obtain the component voting sequence. The range of liveness prediction states is determined based on the component voting states at different times in the component voting sequence. The voting relationship between the local image and the facial response state is updated based on the range of liveness prediction states to generate a liveness state sequence. Based on the liveness state sequence, determine the position of the state change that exceeds the predicted state range in the interaction response sequence, extract abnormal segments, and determine the identity verification result of the facial video based on the abnormal segments.

2. The method according to claim 1, characterized in that, Extracting the response video frame corresponding to the random command from the facial video yields an interactive response sequence, including: The display order of the random instructions and the timestamp information of the random instructions are determined, wherein the timestamp information includes: the start display time and the end display time; Based on the timestamp information, extract consecutive facial video frames corresponding to the random instruction from the facial video, and sort them in chronological order to obtain a candidate response frame sequence; The facial region and facial parts in each frame of the candidate response frame sequence are identified based on the target detection algorithm. Based on the facial region and facial parts, facial video frames with continuous facial movement and complete facial parts are selected from the candidate response frame sequence to obtain facial response frames. The facial response frames are sorted according to the display order of the random instructions and the time information of the facial response frames to obtain the interactive response sequence.

3. The method according to claim 1, characterized in that, Multiple partial images of facial components are extracted from the interaction response sequence. Facial response states are determined based on the interaction response sequence and the random command. A voting relationship is established between the partial images and the facial response states to obtain a component voting sequence, including: Based on the position information of the multiple facial components, the facial response frames in the interactive response sequence are frame aligned. Based on each aligned frame, local images of the multiple facial components are extracted to obtain multiple first feature capsules corresponding to each facial response frame. The facial response state corresponding to the random instruction is determined based on the correspondence between the random instruction and the facial response frames in the interaction response sequence and the multiple first feature capsules corresponding to each facial response frame, and a second feature capsule is constructed based on the facial response state. Establish transformation weights between multiple first feature capsules and second feature capsules corresponding to each facial response frame, and construct the component voting state of each facial response frame based on the transformation weights of multiple first feature capsules corresponding to each facial response frame; The component voting status of each facial response frame is sorted according to the time sequence of the interaction response sequence to obtain the component voting sequence.

4. The method according to claim 1, characterized in that, Based on the component voting states at different times in the component voting sequence, a liveness prediction state range is determined. Based on this liveness prediction state range, the voting relationship between the local image and the facial response state is updated to generate a liveness state sequence, including: Extract the first component voting state corresponding to the first moment and the second component voting state corresponding to the second moment from the component voting sequence in chronological order, wherein the first moment and the second moment are adjacent moments; The voting state of the first component is input into the pre-trained Kopman operator model to obtain the range of the liveness prediction state at the second time step. The range of the liveness prediction state is compared with the voting state of the second component to obtain the comparison result. Based on the comparison results, the transformation weights of multiple first feature capsules corresponding to the second time step in the component voting sequence are weighted and summed to obtain the live feature capsule response state at the second time step; The living state sequence is generated based on the living feature capsule response state corresponding to each moment in the component voting sequence.

5. The method according to claim 1, characterized in that, Based on the liveness state sequence, determine the locations of state changes in the interaction response sequence that exceed the predicted state range, and extract abnormal segments, including: Align the liveness state sequence with the facial response frames in the interaction response sequence in chronological order to obtain a state correspondence sequence; Extract the live feature capsule response states at adjacent time points from the state-corresponding sequence, and determine the candidate positions of state changes based on the change relationship between the live feature capsule response states at adjacent time points. Compare the live predicted state range corresponding to the candidate positions of state changes with the live change results at the candidate positions of state changes to obtain position deviation information. If the position deviation information meets the preset deviation conditions, the candidate position of the state change is determined as the state change position, and a set of state change positions is obtained. Based on the positional distribution information of the set of state change locations in the interactive response sequence, the temporally continuous state change locations in the set of state change locations are merged and connected to obtain a set of candidate abnormal segments. Based on the interactive response sequence fragments corresponding to each state change position in the candidate abnormal fragment set, candidate abnormal fragments that overlap with the scope of the random instruction are selected, and the abnormal fragment is determined based on the selected candidate abnormal fragments.

6. The method according to claim 1, characterized in that, Determining the identity verification result of the facial video based on the abnormal fragments includes: In the interactive response sequence, determine the starting state change position and the ending state change position corresponding to the abnormal segment, and count the number of state change positions in the abnormal segment to obtain the number of abnormalities; The percentage of anomalies is determined based on the number of anomalies and the total number of state change positions in the interaction response sequence; When the proportion of anomalies is greater than a preset anomaly threshold, the criteria for determining deep fake news are determined based on the time distribution information of the anomaly fragments in the interaction response sequence, the number of anomalies, and the proportion of anomalies. When the percentage of abnormalities is less than or equal to the preset abnormality threshold, the criteria for determining the true liveness are determined based on the time distribution information of the abnormal segments in the interactive response sequence, the number of abnormalities, and the percentage of abnormalities. The target determination criteria are determined based on the deep fake determination criteria or the real liveness determination criteria, and the identity verification result for the user is generated based on the target determination criteria.

7. The method according to claim 6, characterized in that, Based on the deepfake detection criteria or the real liveness detection criteria, a target detection criterion is determined, and based on the target detection criterion, an authentication result for the user is generated, including: If the target determination criteria belong to the deepfake determination criteria, the deepfake identity determination result is determined according to the determination rules corresponding to the deepfake determination criteria and the deepfake determination criteria. If the target determination criteria belong to the true liveness determination criteria, the true liveness determination result is determined according to the determination rule corresponding to the true liveness determination criteria and the true liveness determination criteria. The authentication result for the user is determined based on the deepfake identity determination result or the real liveness determination result.

8. An authentication device, characterized in that, include: The extraction unit is used to receive a facial video returned by the user according to a random instruction when the user authorizes the collection of biometric information, and extract the response video frame corresponding to the random instruction from the facial video to obtain an interactive response sequence. A voting unit is used to extract partial images of multiple facial components from the interaction response sequence, determine the facial response state based on the interaction response sequence and the random instruction, and establish a voting relationship between the partial images and the facial response state to obtain a component voting sequence. The prediction unit is used to determine the range of liveness prediction states based on the voting states of components at different times in the component voting sequence, update the voting relationship between the local image and the facial response state based on the range of liveness prediction states, and generate a liveness state sequence. The verification unit is used to determine the position of the state change that exceeds the predicted state range in the interaction response sequence based on the liveness state sequence, extract abnormal segments, and determine the identity verification result of the facial video based on the abnormal segments.

9. An electronic device, characterized in that, include: Memory, which stores executable programs; A processor for running the program, wherein the program, when running, performs the authentication method according to any one of claims 1 to 7.

10. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the steps of the authentication method according to any one of claims 1 to 7.

Images