Identity verification method and system based on text-image multi-modal feature fusion

Abstract

The invention relates to the technical field of identity authentication, and discloses an identity authentication method and system based on text-image multi-modal feature fusion. The method comprises the steps that character interval time and input behavior characteristics are encoded into embedded vectors, and text behavior representation vectors are obtained through processing of an encoder; calculating inter-frame dynamic characteristics based on the image time sequence data, and identifying a micro-expression triggering moment to obtain an image behavior anchor point; extracting a text behavior anchor point from the text time sequence data, and calculating the time sequence matching degree of the text behavior anchor point and the image behavior anchor point to obtain a cross-modal consistency score; and adjusting the fusion weight of the text behavior representation vector and the image feature vector according to the cross-modal consistency score, and outputting a verification result after feature fusion. According to the method, a cross-modal behavior anchor point time sequence alignment mechanism and a self-adaptive fusion strategy are constructed, so that the detection capability of time decoupling attacks and the overall robustness of a verification system are improved.

Description

Technical Field

[0001] This application relates to the field of identity authentication technology, and in particular to an identity authentication method and system based on text-image multimodal feature fusion. Background Technology

[0002] Existing authentication methods mainly include password-based authentication, biometric recognition, and multimodal feature fusion authentication. Multimodal feature fusion authentication improves the accuracy and security of authentication by combining multiple data sources such as text input and image acquisition. A typical implementation is to extract the semantic features of the text and the visual features of the image separately, and then fuse them through linear weighting with fixed weights or simple feature concatenation. Finally, the data is input into a classifier for identity determination. Some methods introduce attention mechanisms to learn the importance weights of different modal features, but these methods mainly focus on the fusion of features at the spatial level.

[0003] However, text feature extraction only focuses on the semantic information of the input content, ignoring the micro-behavioral features of users during the input process, such as typing rhythm, pause patterns, and error correction habits, which are individualized behavioral genes. This allows attackers to forge identities by stealing text content. Secondly, image feature extraction is mainly based on the static appearance features of a single frame or a few frames, lacking the capture of dynamic changes in facial micro-expressions during the verification process, and cannot identify pre-recorded images or deepfake video attacks. Thirdly, feature fusion strategies use fixed weights or simple attention weighting, failing to adaptively adjust the contribution of each modality according to different verification scenarios and user states. When the data quality of a certain modality deteriorates, the fusion result is significantly affected.

[0004] More critically, existing technologies completely ignore the temporal correlation verification between text input and image acquisition during multimodal fusion. Feature extraction and fusion of text and image modalities are performed independently, lacking a modeling and verification mechanism for the temporal consistency of the two modalities. Attackers exploit this vulnerability by using pre-recorded legitimate user images with real-time forged text input, or by using stolen historical text data with deepfake real-time images. Because existing methods cannot detect whether there is a temporal decoupling anomaly between the cognitive pause in text input and the micro-expression triggering time in image acquisition, such temporal attacks can successfully bypass verification. Further analysis reveals that even if some methods simultaneously collect timestamp information from text and images, they only use timestamps as auxiliary attributes during feature fusion without deeply mining the temporal behavioral pattern correlation across modalities. This results in the inability to identify the inconsistency between text input behavior and facial micro-expression behavior on the timeline. This lack of temporal verification capability makes existing multimodal authentication methods insufficiently resistant to temporal decoupling attacks. Summary of the Invention

[0005] This application provides an authentication method and system based on text-image multimodal feature fusion, which solves the problems of lack of cross-modal temporal consistency verification, fixed feature fusion weights, and neglect of user micro-behavioral features in the existing text-image multimodal authentication. By constructing a cross-modal behavior anchor temporal alignment mechanism and an adaptive fusion strategy, the detection capability against temporal decoupling attacks and the overall robustness of the authentication system are improved.

[0006] Firstly, this application provides an authentication method based on text-image multimodal feature fusion, the authentication method based on text-image multimodal feature fusion comprising:

[0007] Simultaneously acquire text input sequences and image sequences, record the character input time and character interval time, extract facial key points from image frames, and obtain text temporal data and image temporal data;

[0008] The character interval time and input behavior features in the text time series data are encoded into an embedding vector, which is then processed by an encoder to obtain a text behavior representation vector.

[0009] Based on the image time-series data, calculate the inter-frame dynamic features, identify the micro-expression trigger time, and obtain the image behavior anchor point;

[0010] Text behavior anchors are extracted from the text time series data, and the time series matching degree between the text behavior anchors and the image behavior anchors is calculated to obtain the cross-modal consistency score.

[0011] The fusion weights of the text behavior representation vector and the image feature vector are adjusted based on the cross-modal consistency score, and the verification result is output after feature fusion.

[0012] Secondly, this application provides an authentication system based on text-image multimodal feature fusion, the authentication system based on text-image multimodal feature fusion comprising:

[0013] The acquisition module is used to simultaneously acquire text input sequences and image sequences, record the character input time and character interval time, extract facial key points from image frames, and obtain text temporal data and image temporal data;

[0014] The encoding module is used to encode the character interval time and input behavior features in the text time series data into an embedding vector, which is then processed by the encoder to obtain the text behavior representation vector.

[0015] The recognition module is used to calculate inter-frame dynamic features based on the image time-series data, identify the trigger time of micro-expressions, and obtain image behavior anchor points;

[0016] The calculation module is used to extract text behavior anchors from the text time series data, calculate the time series matching degree between the text behavior anchors and the image behavior anchors, and obtain a cross-modal consistency score.

[0017] The fusion module is used to adjust the fusion weights of the text behavior representation vector and the image feature vector based on the cross-modal consistency score, and output the verification result after feature fusion.

[0018] Thirdly, an authentication device based on text-image multimodal feature fusion is provided, comprising: a memory and at least one processor, wherein the memory stores instructions; the at least one processor invokes the instructions in the memory to cause the authentication device based on text-image multimodal feature fusion to execute the aforementioned authentication method based on text-image multimodal feature fusion.

[0019] Fourthly, a computer-readable storage medium is provided, wherein instructions are stored therein, which, when executed on a computer, cause the computer to perform the aforementioned authentication method based on text-image multimodal feature fusion.

[0020] The technical solution provided in this application establishes a temporal correlation between text and image modalities by synchronously acquiring text input sequences and image sequences, recording character input times and character intervals, and extracting facial key points from image frames. Compared to existing technologies that only acquire static content while ignoring temporal information, this method lays the foundation for subsequent cross-modal temporal alignment during the data acquisition stage. By encoding character intervals and input behavior features in text temporal data into embedding vectors and processing them through an encoder to obtain text behavior representation vectors, this method deeply integrates users' micro-input behaviors, such as typing rhythm, pause patterns, and error correction habits, into the text feature representation. This ensures that the extracted features not only contain semantic information but also individual behavioral genes that are difficult for attackers to imitate. This effectively solves the problem in existing technologies where text feature extraction only focuses on semantics while ignoring behavioral features, allowing attackers to forge identities by stealing text content. By calculating inter-frame dynamic features based on image temporal data and identifying micro-expression trigger times to obtain image behavior anchors, this method captures naturally occurring facial micro-dynamic changes during the verification process. These micro-expressions are individually different and difficult to consciously control. Compared to existing technologies based on single frames, this method provides a more comprehensive approach. This static feature method, combined with a dynamic feature extraction mechanism, effectively identifies pre-recorded image and deepfake video attacks. Its core innovation lies in extracting text behavior anchors from temporal text data and calculating the temporal matching degree between these anchors and image behavior anchors to obtain a cross-modal consistency score. This mechanism establishes a precise temporal correspondence between the cognitive pause moments of text input and the micro-expression trigger moments of image acquisition, filling a gap in existing technologies for cross-modal temporal consistency verification. It effectively detects temporal decoupling attacks where attackers use pre-recorded images with real-time text or stolen text with real-time images. By adjusting the fusion weights of the text behavior representation vector and image feature vector based on the cross-modal consistency score and performing feature fusion, the verification result is output. This adaptive fusion strategy overcomes the limitations of existing technologies that use fixed weights, making them unable to cope with different verification scenarios and variations in modal data quality. It automatically increases the temporal feature weights when cross-modal temporal consistency is high and dynamically adjusts the weight allocation when the quality of a particular modality data declines. The overall solution, in the field of identity verification, deeply integrates temporal behavior analysis with multimodal feature fusion, constructing a complete technical chain from data acquisition, feature extraction, temporal alignment to adaptive fusion.

[0021] Specifically, in the specific application domain of text-image multimodal authentication, this application introduces algorithmic features such as a keyboard dynamics-enhanced encoder, a micro-expression-aware convolutional network, a behavior anchor temporal alignment algorithm, and a gating network-driven adaptive fusion mechanism. This allows the entire verification process to not only focus on the static biometric and knowledge features of user identity but also delve deeper into the dynamic behavioral features and cross-modal temporal consistency features exhibited by users during verification. The encoder encodes behavioral features such as typing intervals, error corrections, and pause patterns into embedding vectors and fuses them with semantic embeddings. This enables the Transformer's self-attention mechanism to simultaneously learn the contextual dependencies of text content and the temporal patterns of user behavior. This deep fusion is more comprehensive than existing technologies that treat behavioral features as independent channels. The anchor point temporal alignment algorithm finds the optimal matching path between textual cognitive pauses and image micro-expression triggers through dynamic temporal warping. This algorithm considers the differences in individual response delays and uses adaptive temporal sensitivity parameter settings to ensure that temporal matching maintains sensitivity to anomalies while tolerating normal individual differences. The gating network dynamically outputs fusion weights based on the feature distribution and cross-modal consistency scores of the current verification samples. This data-driven weight adjustment mechanism is more adaptable than manually set fixed weights or methods that rely solely on attention scores. The synergistic effect of these algorithmic features makes the verification method of this application exhibit stronger robustness and discriminative ability in the face of complex and ever-changing attack methods, especially in scenarios where existing technologies are difficult to deal with, such as temporal decoupling attacks, deepfake attacks, and behavior imitation attacks. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a schematic diagram of an embodiment of the identity verification method based on text-image multimodal feature fusion in this application.

[0024] Figure 2 This is a schematic diagram of the cross-modal behavior anchor point timing alignment process in an embodiment of this application;

[0025] Figure 3 This is a schematic diagram of an embodiment of an identity verification system based on text-image multimodal feature fusion in this application.

[0026] Figure 4This is a schematic block diagram of the structure of an identity verification device based on text-image multimodal feature fusion in an embodiment of the present invention. Detailed Implementation

[0027] This application provides an authentication method and system based on text-image multimodal feature fusion. The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data used can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" or "having" and any variations thereof are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0028] For ease of understanding, the specific process of the embodiments of this application is described below. Please refer to [link / reference]. Figure 1 One embodiment of the authentication method based on text-image multimodal feature fusion in this application includes:

[0029] Step S101: Synchronously acquire text input sequence and image sequence, record character input time and character interval time, extract facial key points of image frames, and obtain text temporal data and image temporal data;

[0030] Step S102: Encode the character interval time and input behavior features in the text time series data into an embedding vector, and process it through the encoder to obtain the text behavior representation vector;

[0031] Step S103: Calculate inter-frame dynamic features based on image temporal data, identify the trigger time of micro-expressions, and obtain image behavior anchor points;

[0032] Step S104: Extract text behavior anchors from text time series data, calculate the time series matching degree between text behavior anchors and image behavior anchors, and obtain cross-modal consistency score;

[0033] Step S105: Adjust the fusion weights of the text behavior representation vector and the image feature vector based on the cross-modal consistency score, and output the verification results after feature fusion.

[0034] It is understood that the executing entity of this application can be an identity verification system based on text-image multimodal feature fusion, or it can be a terminal or a server; the specific implementation is not limited here. This application's embodiments use a server as an example for illustration.

[0035] Specifically, when a user enters text on the verification interface, the acquisition module records the ASCII code value of each character and the millisecond-level timestamp of the input time. It calculates the character interval by subtracting the previous character's timestamp from the current character's timestamp, organizing the character encoding, timestamp, and character interval into a text input sequence. Simultaneously, error correction detection listens for backspace key events, recording the timestamp of the backspace occurrence and the number of characters deleted, categorizing them into single-character correction, multi-character correction, and continuous correction types, forming an error correction feature sequence. The pause recognition module iterates through all character intervals, marking intervals exceeding a pause threshold (set to 500 milliseconds) as pause events, recording the pause start time and duration, and constructing a pause feature sequence. The text input sequence, error correction feature sequence, and pause feature sequence are integrated into text time-series data. Image acquisition uses a front-facing camera to capture facial image sequences at a fixed frame rate. A multi-task cascaded convolutional neural network is used to detect the facial region in each frame, extracting the coordinates of facial key points including the corners of the eyes, the tip of the nose, and the corners of the mouth, and recording the acquisition timestamp for each frame. Inter-frame difference calculation quantifies the overall change in the facial region by subtracting the pixel values ​​of the current frame from those of the previous frame point by point and taking the square root of the sum of the squares. It organizes keypoint coordinates, frame timestamps, and inter-frame difference values ​​into image temporal data. For example, when a user enters the password "Pass123", the input time of the character "P" is t1 = 1000 milliseconds, the input time of the character "a" is t2 = 1180 milliseconds, so the character interval is 180 milliseconds. The input time of the character "s" is t3 = 1350 milliseconds, with an interval of 170 milliseconds. The first "s" input time is t4 = 2400 milliseconds. Since the interval of 1050 milliseconds exceeds the threshold, it is marked as a pause event, with a pause time of 1350 milliseconds and a duration of 1050 milliseconds. The key point coordinates of the 10th frame in the image sequence with a timestamp of 1200 milliseconds include the coordinates of the left eye corner (45, 78) and the right eye corner (102, 76). The key point coordinates of the 11th frame with a timestamp of 1233 milliseconds are the coordinates of the left eye corner (45, 79) and the right eye corner (102, 77). The inter-frame difference is obtained by calculating the square root of the sum of the squares of the gray level differences of all pixel positions, and the difference value is 12.3, which represents the intensity of facial micro-movements during this time period.

[0036] Behavioral features in textual time-series data are transformed into deep semantic representations. Character interval time mapping employs a positional encoding function, which uses a combination of sine and cosine functions to map continuous time values ​​to a high-dimensional space. Specifically, different frequencies of trigonometric functions are applied to the normalized character interval time (normalized by dividing by the maximum interval time), generating a 128-dimensional typing interval embedding vector. Error correction embedding converts error correction types (single-character correction, multi-character correction, continuous correction) into one-hot vectors (a three-dimensional vector with only one position set to 1 and the rest to 0), and then maps them to a 64-dimensional error correction embedding vector using a learnable linear transformation matrix. Pause pattern embedding categorizes pause duration into three types based on duration: short pauses (less than 1000 milliseconds), medium pauses (1000 to 3000 milliseconds), and long pauses (greater than 3000 milliseconds), and maps them to a 32-dimensional pause pattern embedding vector using a piecewise function. Semantic character encoding uses a pre-trained vocabulary to convert characters into 768-dimensional character semantic embedding vectors. The semantic embedding vectors, typing interval embedding vectors, error correction embedding vectors, and pause pattern embedding vectors corresponding to the same character position are concatenated according to their dimensions to form a composite embedding vector with dimensions of 768+128+64+32=992. The composite embedding vector sequence is input into a Transformer encoder, which contains 12 layers. Each layer performs self-attention calculation, quantifying the association strength between characters through matrix operations on the query vector, key vector, and value vector. Each element of the attention score matrix is ​​obtained by dividing the dot product of the query vector and the key vector by the square root of the dimension. After softmax normalization, it is multiplied by the value vector to produce a self-attention output. This output is processed by a feedforward neural network (containing two layers of linear transformation and ReLU activation function), residually connected with the input, and layer normalization is performed to obtain the hidden state vector of that layer. After passing through 12 layers, the hidden state vectors of layers 6 and 12 are extracted and concatenated. The importance weight of each character position is calculated through the behavior attention fusion module. This weight is obtained by linearly transforming the hidden state vector, applying the tanh activation function, and then performing another linear transformation to obtain a scalar score. The scores of all positions are normalized by softmax to obtain the attention weight. This weight is used to perform weighted average pooling on the hidden state vectors of all character positions to obtain a 128-dimensional text behavior representation vector.For example, when processing an input sequence containing 7 characters, the composite embedding vector of the 4th character is input into the first layer of the encoder. When calculating the query matrix, the weight matrix is ​​multiplied by the embedding vector. The key matrix and value matrix are calculated similarly. The query matrix is ​​multiplied by the transpose of the key matrix to obtain a 7×7 attention score matrix. The element in the 4th row and 2nd column of this matrix represents the attention level of the 4th character to the 2nd character. After softmax normalization of all elements in the 4th row, each element is multiplied by the corresponding column of the value matrix and summed to obtain the self-attention output vector of the 4th character. This vector is then fed forward. The first linear transformation of the network expands the dimension from 992 to 3072. After applying ReLU, non-negative values ​​are retained and negative values ​​are set to zero. The second linear transformation compresses the dimension back to 992. After residual connection with the input, layer normalization is performed to obtain the first hidden state, which is then passed to the subsequent 11 layers. The hidden states of the 6th and 12th layers are extracted and concatenated to obtain a dimension of 1984. The attention weight of the 4th character is calculated to be 0.18 through the attention fusion module. The hidden states of the 7 character positions are multiplied by their respective attention weights and then summed to obtain the text behavior representation vector.

[0037] Micro-expression trigger moments are extracted from temporal image data as behavioral anchors. Keypoint extraction for the eye region involves filtering facial keypoint coordinates from indices 37 to 48, and keypoint extraction for the mouth region involves indices 49 to 68. The positional change of corresponding keypoint coordinates in adjacent frames is calculated by subtracting the coordinates of the keypoint in the previous frame from the coordinates of the keypoint in the current frame, resulting in a two-dimensional displacement vector. All displacement vectors of the eye keypoints are organized into an eye motion vector sequence, and the mouth motion vector sequence is obtained similarly. The optical flow algorithm employs the Farneback dense optical flow method, calculating an optical flow vector at each pixel position for adjacent frames. This vector contains a horizontal displacement component u and a vertical displacement component v. The optical flow amplitude is calculated by taking the square root of the square of u plus the square of v. The average optical flow amplitude of all pixels within the rectangular area of ​​the eye is used to obtain the micro-dynamic intensity value of the eye in that frame. The same method is used to calculate the micro-dynamic intensity value of the mouth region. The micro-dynamic intensity values ​​of the eye in all frames are organized temporally into an eye micro-dynamic intensity sequence, and the mouth micro-dynamic intensity sequence is constructed similarly. The dynamic threshold calculation first calculates the mean of the eye micro-dynamic intensity sequence by summing the intensity values ​​of all frames and dividing by the number of frames. Then, the standard deviation is calculated by summing the squares of the intensity values ​​of each frame minus the mean, dividing by the number of frames, and then taking the square root. The dynamic threshold is set to the mean plus 1.5 times the standard deviation. Peak detection iterates through the eye micro-dynamic intensity sequence. When the intensity value of a frame is greater than the intensity values ​​of the adjacent frames and exceeds the dynamic threshold, it is marked as a local maximum point, and the corresponding frame index and frame timestamp are recorded. Local maximum points are identified similarly for the mouth micro-dynamic intensity sequence. All local maximum points identified for the eyes and mouth are summarized into a set of micro-expression trigger candidate points. During local feature vector extraction, for each candidate point, the eye or mouth region of the five frames (two frames before and two frames after the current frame) are extracted. The optical flow amplitude, keypoint displacement, and inter-frame difference values ​​of the five frames are concatenated to form a local feature vector. This feature vector is associated with the corresponding frame timestamp to obtain the image behavior anchor point.For example, in an image sequence containing 60 frames, the intensity value of frame 15 in the eye micro-dynamic intensity sequence is 8.2, frame 16 is 11.5, frame 17 is 9.1, and frame 18 is 7.8. The mean value calculated by iterating through all frames is 6.3, and the standard deviation is 2.1. The dynamic threshold is 6.3 plus 1.5 multiplied by 2.1, which equals 9.45. The intensity value of frame 16 (11.5) is greater than that of adjacent frames and exceeds the threshold, so it is marked as a local maximum. The corresponding frame timestamp is 533 milliseconds. Frames 14 to 18 (before and after frame 16) are extracted. The eye region features (2 frames each) are analyzed. The average optical flow amplitude in frame 14 is 7.1, the keypoint displacement is 1.2, and the inter-frame difference is 10.3. The corresponding values ​​in frame 15 are 8.2, 1.5, and 11.8. The values ​​in frame 16 are 11.5, 2.3, and 14.2. The values ​​in frame 17 are 9.1, 1.8, and 12.1. The values ​​in frame 18 are 7.8, 1.3, and 10.5. These 15 values ​​(5 frames × 3 features) are concatenated to form a local feature vector, which is then associated with a timestamp of 533 milliseconds to obtain an image behavior anchor point.

[0038] Text behavior anchor extraction involves filtering from the pause feature sequence, traversing all pause events. When the pause duration exceeds the cognitive pause threshold (set to 800 milliseconds), the pause time of that event is used as the text behavior anchor timestamp. Typing interval embedding vectors, error correction embedding vectors, and pause pattern embedding vectors within three character ranges before and after the pause are extracted and concatenated to form a local behavior feature vector. This feature vector is then associated with the timestamp to obtain the text behavior anchor. Assuming m text behavior anchors and n image behavior anchors are identified, temporal similarity calculation traverses all text and image anchor pairings. The absolute value of the time difference between the timestamp of the i-th text anchor and the timestamp of the j-th image anchor is calculated. The temporal sensitivity parameter is set based on the average response delay between text input and image acquisition times in the user's historical verification data, specifically calculated as 2.0 divided by (average response delay plus 200 milliseconds). The absolute value of the time difference is multiplied by the temporal sensitivity parameter and negative, serving as the input to an exponential function. The output of the exponential function is the temporal similarity component. Behavioral similarity calculation involves performing cosine similarity calculations on the local feature vectors of the i-th text anchor point and the j-th image anchor point. The cosine similarity is obtained by dividing the dot product of the two vectors by the product of their magnitudes. The dot product is the sum of the product of corresponding elements, and the magnitude is the square root of the sum of the squares of all vector elements. This yields the behavioral similarity component. The pairing score is calculated by multiplying the temporal similarity component by the behavioral similarity component. All pairing scores are filled into an m x n cross-modal pairing matrix, where the element in the i-th row and j-th column represents the pairing score between the i-th text anchor point and the j-th image anchor point. The dynamic time warping algorithm initializes the cumulative cost matrix with the first element of the pairing matrix. For the cumulative cost in the i-th row and j-th column, the minimum value among the left element (i-th row, j-1 column), the top element (i-1 row, j-1 column), and the top-left element (i-1 row, j-1 column) is taken and added to the pairing score at the current position to obtain the cumulative cost at the current position. Starting from the bottom right element of the cumulative cost matrix and going back to the top left element, the position with the smallest cumulative cost among three adjacent positions is selected as the path node each time. All position index pairs contained in the backtracking path constitute the optimal matching path. The scores of all pairs on this path are summed and divided by the path length to obtain the cross-modal consistency score.For example, the timestamps of three text action anchors are identified as 1350 ms, 3200 ms, and 5100 ms, and the timestamps of two image action anchors are 1400 ms and 5050 ms. The user's historical average response latency is 300 ms. The temporal sensitivity parameter is 2.0 divided by 500, which equals 0.004. The absolute value of the time difference between the first text anchor and the first image anchor is 50 ms. The temporal similarity is the exponential function e raised to the power of -0.2, which is approximately 0.819. The dot product of the local feature vectors of the two anchors is 23.5. The magnitude of the feature vector of the first text anchor is 8.2, and the magnitude of the feature vector of the first image anchor is 7.6. The cosine similarity is 23.5 divided by 8.2 multiplied by 7.6, which is approximately 0.377. The pairing matching score is 0.819 multiplied by 0. 377 is approximately equal to 0.309. Similarly, calculate the other pairing scores to fill the 3x2 pairing matrix. The first row and first column of the cumulative cost matrix are initialized to 0.309. The second row and second column are the sum of 0.309 from the first row and first column plus the current pairing score of 0.156, which equals 0.465. The first row and first column are the sum of 0.309 from the first row and first column plus the current pairing score of 0.092, which equals 0.401. The second row and second column take the minimum of the three values: 0.465 from the left, 0.401 from the top, and 0.309 from the top left, which is 0.309, plus the current pairing score of 0.284, which equals 0.593. Fill the cumulative cost matrix sequentially up to the second row and second column. Backtrack from this position to select the path with the minimum cumulative cost. The optimal path contains 3 pairing scores. After summing them, divide by 3 to obtain the cross-modal consistency score.

[0039] The text behavior representation vector (128-dimensional), image feature vector (64-dimensional), and cross-modal consistency score (scalar) are concatenated dimensionally to form a 193-dimensional input vector, which is then input into the first fully connected layer of the gating network. This layer contains 256 neurons. Each neuron calculates the dot product of the input vector and its weight vector, plus a bias term. The ReLU activation function is applied to the outputs of the 256 neurons, setting negative values ​​to zero and retaining positive values, resulting in a 256-dimensional first-layer output. The first-layer output is then input into the second fully connected layer, which contains 128 neurons. Each neuron calculates the dot product of the 256-dimensional input and its weight vector, plus a bias term. The Tanh activation function is applied to the 128 outputs, mapping the input to a range from -1 to +1, resulting in a 128-dimensional second-layer output. The second-layer output is then input into the third fully connected layer, which contains three neurons corresponding to text weights, image weights, and temporal weights, respectively. The outputs of these three neurons are normalized using softmax. Softmax calculates the exponent value of each neuron's output, divides it by the sum of the three exponent values, and obtains three normalized weights, the sum of which is 1. The text behavior representation vector is multiplied element-wise with the text weights to obtain weighted text features. The image feature vector is multiplied with the image weights to obtain weighted image features. The temporal feature vector (extended from the cross-modal consistency score to have the same dimension as other features) is multiplied with the temporal weights to obtain weighted temporal features. The three weighted features are summed dimensionally to obtain the fused feature vector. The fused feature vector is input into the first fully connected layer of the classifier, which contains 64 neurons. After linear transformation and ReLU activation, dropout is applied to randomly set the output of 30% of the neurons to zero. The outputs of the remaining neurons are input into the second layer, which contains two neurons, corresponding to the real user class and the attack sample class, respectively. The outputs of the two neurons are normalized by softmax to obtain the probability of the real user class and the probability of the attack sample class. When the probability of the real user class exceeds the decision threshold of 0.85, the output validation is passed; otherwise, the output validation is rejected.For example, if a certain dimension of the text behavior representation vector is 0.52, the corresponding dimension of the image feature vector is 0.78, and the cross-modal consistency score is 0.83, the concatenated 193-dimensional vector is input into a gating network. The dot product of the weight vector of the 10th neuron in the first layer with the input vector is 45.3 plus a bias of 2.1, which equals 47.4. After ReLU activation, this value remains at 47.4. The outputs of all 256 neurons constitute the output of the first layer. The dot product of the 5th neuron in the second layer with the output of the first layer is 12.8 plus a bias of -1.2, which equals 11. After Tanh activation, the value is approximately 0.999. The outputs of all 128 neurons constitute the output of the second layer. The outputs of the three neurons in the third layer are 2.3, 1.8, and 1.5, respectively. Softmax normalization calculates e to the power of 2.3 as approximately 9.97, e to the power of 1.8 as approximately 6.05, and e to the power of 1.5 as approximately 4.48. The sum of these three is 20.5. The text weight is 9.97 divided by 20.5, approximately 0.486; the image weight is 6.05 divided by 20.5, approximately 0.295; and the temporal weight is 4.48 divided by... 20.5 is approximately 0.219. The text feature dimension (0.52 x 0.486) is approximately 0.253. The image feature dimension (0.78 x 0.295) is approximately 0.230. The temporal feature dimension (0.83 x 0.219) is approximately 0.182. Summing these three values ​​(0.253 + 0.230 + 0.182) equals 0.665, which is used as the value for that dimension of the fusion feature. This fusion feature is input into the third neuron of the first layer of the classifier, where the dot product is calculated to be 8.7. Adding the bias of 0.5 equals 9.2, and after ReLU, the value is maintained at 9.2. After dropout and random zeroing, the output of the neuron is retained. The outputs of all 64 neurons are input into the second layer. The output of the real user neuron is 1.9, and the output of the attack sample neuron is 0.6. After softmax normalization, the probability of the real user is approximately 6.69 divided by 1.9 of e plus 0.6 of e, approximately 6.69 divided by 8.51, approximately 0.786. The probability of the attack sample is approximately 0.214. The probability of the real user is 0.786, which is less than the decision threshold of 0.85. The validation rejection result is output.

[0040] In one specific embodiment, the process of performing step S101 may specifically include the following steps:

[0041] Collect each character input by the user and its corresponding input time, calculate the time interval between adjacent characters, and obtain a text input sequence containing character encoding, timestamp, and character interval time;

[0042] The error correction feature sequence is obtained by detecting the backspace key usage time and the number of correction characters in the text input sequence, recording the timestamps of error correction events;

[0043] Identify time intervals exceeding the pause threshold in the character interval time, record the pause occurrence time and pause duration to obtain the pause feature sequence, and combine the text input sequence, error correction feature sequence and pause feature sequence into text time series data;

[0044] A series of face image frames are acquired using an image acquisition device. The facial key point coordinates of each frame are extracted using a face detection algorithm. The image difference value between adjacent frames is calculated to obtain image time-series data containing key point coordinates, frame timestamps, and inter-frame difference values.

[0045] Specifically, when a key press event is detected, the ASCII code value corresponding to the key is obtained as the character encoding. At the same time, the timestamp function is called to obtain the millisecond-level timestamp of the current moment. The character encoding and timestamp are stored as a pair of data. When the next character is received, the timestamp of the current character is subtracted from the timestamp of the previous character to obtain the character interval time. The character encoding, timestamp, and character interval time are organized into a triplet and arranged in the input order to form a text input sequence. This sequence records the user's complete text input behavior trajectory.

[0046] The system continuously monitors the keyboard event stream. When the backspace key is detected, the timestamp of that moment is recorded as the error correction event timestamp. At the same time, the number of characters deleted by the backspace operation is counted. The number of deleted characters is determined by analyzing the changes in the text input sequence before and after the backspace operation. The error correction event timestamp is associated with the number of deleted characters and stored. Based on the number of deleted characters, error correction is divided into three types: single-character correction, multi-character correction, and continuous correction. Single-character correction refers to the deletion of one character, multi-character correction refers to the deletion of multiple characters but only one backspace operation, and continuous correction refers to multiple consecutive presses of the backspace key in a short period of time. All error correction events are organized into an error correction feature sequence according to their time sequence.

[0047] The process iterates through all character intervals in the text input sequence, comparing each character interval with a preset pause threshold. When a character interval exceeds the threshold, it is considered a pause event. The timestamp of the preceding character is recorded as the pause occurrence time, and the pause duration is the value of the character interval itself. The pause occurrence time and pause duration are stored in pairs. After the iteration is complete, a pause feature sequence is obtained, which contains all pause events and their time attributes. The text input sequence, error correction feature sequence, and pause feature sequence are organized into a unified text temporal data object at the data structure level. This object fully preserves the semantic content, temporal characteristics, and behavioral features of the user-input text.

[0048] The image acquisition device continuously captures images of the user's facial area at a fixed frame rate. Each captured frame is immediately recorded with its timestamp. The image data is then associated with the timestamp and fed into a face detection algorithm. This algorithm first locates the rectangular region of the face in the image. Within this region, a keypoint detection network is applied to extract the pixel coordinates of facial feature points. These keypoints include anatomical features such as the corners of the eyes, the ends of the eyebrows, the tip of the nose, and the corners of the mouth. Each keypoint is represented by a coordinate pair consisting of an abscissa and a ordinate. All keypoint coordinates are organized into a keypoint coordinate set. The algorithm receives two adjacent frames and calculates the pixel value difference for each corresponding pixel position. The absolute value of the difference between the current and previous frames is subtracted from the current frame's pixel value. The squares of all pixel differences are summed and then the square root is taken to obtain an image difference value representing the overall degree of change between the two frames. The keypoint coordinate set, frame timestamp, and image difference value are associated to form a single-frame image temporal data unit. All image temporal data units are arranged chronologically to form complete image temporal data.

[0049] This data structure addresses the problem of existing technologies that only collect static images while ignoring temporal dynamic information. By synchronously recording the timestamps of text input and image acquisition, it establishes a temporal correlation between the two modalities. The character interval time in the text temporal data reflects the user's input rhythm and cognitive processing. The error correction feature sequence reveals the user's hesitation and error correction behavior patterns during the input process. The pause feature sequence marks the key moments of user thinking and memory retrieval. These micro-behavioral features are manifestations of the user's individual behavioral genes and are difficult for attackers to imitate by simply stealing text content. The key point coordinates in the image temporal data capture the spatial structural information of the face, the inter-frame difference value quantifies the dynamic change features of the face, and the timestamp sequence of consecutive frames provides a temporal location basis for subsequent identification of micro-expression trigger moments. The synchronous acquisition mechanism of text temporal data and image temporal data makes subsequent cross-modal temporal alignment analysis possible, solving the technical deficiency of existing technologies that lack temporal correlation verification when fusing text features and image features.

[0050] In one specific embodiment, the process of performing step S102 may specifically include the following steps:

[0051] The character interval time is mapped to the typing interval embedding vector through the position encoding function. The error correction event type in the error correction feature sequence is converted into an error correction embedding vector after one-hot encoding. The pause duration in the pause feature sequence is classified according to the duration range and mapped to the pause pattern embedding vector.

[0052] Semantic encoding is performed on the characters in the text input sequence to obtain character semantic embedding vectors. The character semantic embedding vectors are then concatenated with the typing interval embedding vector, error correction embedding vector, and pause pattern embedding vector to obtain a composite embedding vector.

[0053] The composite embedding vector is input into a multi-layer Transformer encoder, and the contextual dependencies and temporal behavior patterns between characters are calculated through a bidirectional self-attention mechanism to obtain the hidden state vector at each character position.

[0054] The hidden state vectors of the intermediate and final layers of the encoder are extracted. The attention weights at each position are calculated through the behavior attention fusion module. The hidden state vectors of all character positions are weighted average pooled to obtain the text behavior representation vector.

[0055] Specifically, the character interval time mapping uses a positional encoding function, which takes the normalized character interval time as input. Normalization is achieved by dividing the original interval time by the maximum interval time in the sequence. The normalized value is between zero and one. The positional encoding function uses a combination of sine and cosine functions to extend this scalar value to a higher-dimensional space. Different frequencies of trigonometric functions are used for different dimension indices, with the frequency decreasing with the dimension index. This allows low-dimensional elements to capture rapid changes while high-dimensional elements capture slow changes. The value of each dimension is calculated by multiplying the normalized interval time by the corresponding frequency and then substituting it into the sine or cosine function. Even-numbered dimensions use the sine function, and odd-numbered dimensions use the cosine function. Finally, a fixed-length typing interval embedding vector is generated.

[0056] Error correction event types include three categories: single-character correction, multi-character correction, and continuous correction. One-hot encoding maps these three types to three-dimensional vectors. The vector corresponding to single-character correction has one in the first dimension and zero in the rest. The vector corresponding to multi-character correction has one in the second dimension and zero in the rest. The vector corresponding to continuous correction has one in the third dimension and zero in the rest. This three-dimensional one-hot vector is multiplied by a learnable weight matrix to obtain the error correction embedding vector. Each column of the weight matrix corresponds to an embedding representation of a correction type. The matrix multiplication operation selects the column corresponding to the unique non-zero element in the one-hot vector as the embedding vector.

[0057] The pause duration classification uses a duration threshold to divide the pause. The duration is compared with two thresholds. If it is less than the first threshold, it is classified as a short pause; if it is between the first and second thresholds, it is classified as a medium pause; and if it is greater than the second threshold, it is classified as a long pause. The three categories are also converted into three-dimensional vectors through one-hot encoding and then multiplied by the corresponding weight matrix to map into pause pattern embedding vectors.

[0058] The semantic encoding of characters uses a pre-trained vocabulary, which contains all possible characters and their corresponding high-dimensional vector representations. The corresponding vector is found in the vocabulary based on the ASCII code value of the character and serves as the character semantic embedding vector. This vector encodes the semantic information of the character. The semantic embedding vectors, typing interval embedding vectors, error correction embedding vectors, and pause pattern embedding vectors corresponding to the same character position are concatenated end to end in the dimensional direction to form a longer vector. This concatenation operation is completed by arranging the elements of each vector into the new vector in order. The length of the concatenated vector is equal to the sum of the lengths of each component vector, resulting in a composite embedding vector. This vector contains both semantic information and behavioral feature information.

[0059] The composite embedding vector sequence is input to the Transformer encoder, which contains multiple stacked identical structures. Each layer receives the output of the previous layer as input. The first layer receives the composite embedding vector sequence. Within each layer, self-attention calculation is first performed. Each vector in the input vector sequence is transformed into a query vector, a key vector, and a value vector through three different linear transformations. The query vector is multiplied by all key vectors to obtain an attention score. The dot product operation multiplies corresponding elements of two vectors and sums them. The resulting scalar reflects the association strength between two positions. The dot product of a query vector and all key vectors is divided by the square root of the vector dimension for scaling. The scaled score sequence is converted into a probability distribution through the softmax function. The softmax function calculates the exponent value of each score and divides it by the sum of all exponent values ​​to obtain a normalized attention weight sequence. This weight sequence is weighted and summed with the value vector sequence. Each value vector is multiplied by its corresponding attention weight and accumulated to obtain the self-attention output vector at that position. This process is performed in parallel at all positions to obtain the self-attention output vector sequence.

[0060] The self-attention output vector sequence is input into a feedforward neural network. This network contains two linear transformation layers and an activation function. The first linear transformation layer multiplies the input vector by a weight matrix and adds a bias vector. The number of rows in the weight matrix is ​​greater than the input dimension to achieve dimensionality expansion. The activation function acts independently on each element of the expanded vector, setting negative values ​​to zero and positive values ​​to remain unchanged. The second linear transformation layer multiplies the activated vector by another weight matrix and adds a bias. The number of columns in this matrix is ​​equal to the original input dimension to achieve dimensionality compression. The compressed vector is added element-wise to the initial input vector of this layer to obtain the residual connection output. The vector after the residual connection is normalized by layer. Layer normalization calculates the mean and variance of all elements of the vector. Each element is subtracted from the mean and divided by the square root of the variance. The normalized vector is multiplied by a learnable scaling parameter and added with a learnable bias parameter to obtain the final output of this layer, which is the hidden state vector. This hidden state vector sequence is passed to the next layer encoder. Multiple encoders extract more abstract feature representations layer by layer.

[0061] Hidden state vector sequences are extracted from the intermediate and final layers of the encoder. The vectors extracted from the intermediate layer retain features at a medium level of abstraction, while the vectors extracted from the final layer contain highly abstract features. The vectors from the two layers are concatenated dimensionally to form an enhanced hidden state vector. The concatenation operation links the elements of the intermediate and final layer vectors end-to-end. The input behavior is then fused with attention. This fusion calculates the importance weight for each character position. Specifically, the enhanced hidden state vector is multiplied by the weight vector and a tanh activation function is applied to obtain an intermediate scalar. This scalar is then multiplied by another weight scalar to obtain the original attention score. Softmax normalization is applied to the original scores for all character positions to obtain the attention weight distribution. The enhanced hidden state vector for each position is multiplied by the corresponding attention weight. All weighted vectors are summed and divided by the sum of the weights to obtain the weighted average pooling result. This result is a fixed-length vector that does not depend on the length of the input sequence. This vector is the text behavior representation vector, which encodes... This approach integrates the semantic content of the entire text input sequence with the user's input behavior patterns, addressing the problem in existing technologies where text feature extraction focuses solely on semantics while neglecting user behavior features. By deeply integrating keyboard dynamics features into the embedding layer and attention mechanism of a pre-trained language model, the extracted feature vectors not only contain the semantic information of the text but also include behavioral genes such as the user's unique input rhythm, error correction habits, and cognitive pause patterns. These behavioral features are individualized and difficult for attackers to imitate. Even if an attacker obtains the correct text content, they cannot reproduce the micro-input behavior patterns of a real user. The bidirectional self-attention mechanism of the multi-layer Transformer encoder captures the long-distance dependencies between characters and the contextual associations of temporal behavior patterns. Behavioral attention fusion achieves adaptive feature aggregation by learning the contribution of different character positions to the overall behavioral features. Compared to the simple feature concatenation or average pooling of existing technologies, this method generates more compact text behavior representation vectors with stronger discriminative power.

[0062] In one specific embodiment, the process of inputting the composite embedding vector into a multi-layer Transformer encoder and calculating the contextual dependencies and temporal behavior patterns between characters through a bidirectional self-attention mechanism can specifically include the following steps:

[0063] The composite embedding vector is input into the first layer of the Transformer encoder to calculate the query matrix, key matrix, and value matrix. The attention score matrix is ​​obtained through matrix multiplication. After the attention score matrix is ​​normalized by softmax, it is multiplied with the value matrix to obtain the self-attention output vector of the first layer.

[0064] The self-attention output vector is input into the feedforward neural network. After linear transformation and activation function processing, residual connection and layer normalization are performed to obtain the hidden state vector of the first layer.

[0065] The hidden state vector of the first layer is passed as input to the encoder of each subsequent layer. The self-attention calculation and feedforward network processing are repeated to update the hidden state vector layer by layer, so as to obtain the hidden state vector sequence of each layer of the encoder.

[0066] Extract the intermediate layer output and the final layer output from the hidden state vector sequence, and concatenate the intermediate layer output and the final layer output to obtain the hidden state vector of each character position containing multi-level semantic and behavioral features.

[0067] Specifically, the composite embedded vector sequence is input into the first layer of the Transformer encoder. This layer independently performs three linear transformations on the vector at each position in the sequence. The first transformation generates the query vector by multiplying the composite embedded vector by the query weight matrix. The second transformation generates the key vector by multiplying it by the key weight matrix. The third transformation generates the value vector by multiplying it by the value weight matrix. All three weight matrices have the same dimension. Matrix multiplication involves multiplying each element of the input vector by the corresponding element of the weight matrix and summing the results to obtain the corresponding element of the output vector. After generating the query vector, key vector, and value vector for each position in the sequence, the query vectors at all positions are arranged in rows to form the query matrix, the key vectors are arranged in rows to form the key matrix, and the value vectors are arranged in rows to form the value matrix. The product of the query matrix and the transpose of the key matrix is ​​calculated to obtain the attention score matrix. The matrix transpose swaps the rows and columns of the key matrix. When multiplying the two matrices, the first matrix... Each row is multiplied by the corresponding element of each column of the second matrix and then summed to obtain the element at the corresponding position in the resulting matrix. The number of rows and columns of the attention score matrix are equal to the sequence length. The element in the i-th row and j-th column of the matrix represents the attention level of the i-th position to the j-th position. Each element of the attention score matrix is ​​divided by the square root of the query vector dimension to scale it. The scaling operation is completed by traversing all elements of the matrix and dividing them by this scalar. The scaled matrix is ​​then normalized independently for each row by softmax. Softmax first calculates the exponent value for all elements in a row, and then divides each exponent value by the sum of all exponent values ​​in that row. After normalization, the sum of the elements in each row is one and all elements are non-negative. The normalized attention score matrix is ​​multiplied by the value matrix to obtain the self-attention output matrix. This matrix multiplication uses each row of the attention score matrix as a weight to perform a weighted summation of all rows of the value matrix. Each row of the output matrix is ​​the self-attention output vector at the corresponding position.

[0068] The self-attention output vector sequence is input into a feedforward neural network. This network contains two linear transformation layers with an activation function inserted in between. The first linear transformation layer multiplies the input vector by a first weight matrix and adds a first bias vector. The weight matrix has a much larger number of rows than columns, achieving dimensionality expansion. Matrix multiplication and bias addition are performed element-wise. Each element of the expanded vector is independently processed by the ReLU activation function. The ReLU function determines the sign of the element value; positive values ​​remain unchanged, while negative and zero values ​​are uniformly set to zero. The activated vector is then input into the second linear transformation layer, multiplied by a second weight matrix and added with a second bias vector. The second weight matrix has the same number of rows as the first weight matrix, and the number of columns equals the dimension of the original input vector, achieving dimensionality compression. The vector is shrunk back to its original size, and the compressed vector is added element-wise to the self-attention output vector to achieve residual connection. The addition operation adds the corresponding elements of the two vectors separately. The vector after residual connection is then normalized. This normalization first calculates the arithmetic mean of all elements of the vector as the mean, then calculates the square root of the sum of the squares of the differences between each element and the mean, divided by the number of elements, to obtain the standard deviation. Each element of the vector is subtracted from the mean and then divided by the standard deviation to obtain the normalized vector. Each element of the normalized vector is multiplied by the corresponding learnable scaling parameter and the corresponding learnable offset parameter is added. The scaling parameter and offset parameter are updated through backpropagation during training. The vector sequence after layer normalization is the hidden state vector sequence of the first layer.

[0069] The hidden state vector sequence of the first layer serves as the input to the second layer. The second layer performs the same operations as the first layer, including querying the key-value matrix, calculating the attention score, softmax normalization, multiplying with the value matrix to obtain the self-attention output, two-layer linear transformation and ReLU activation of the feedforward network, residual connections, and layer normalization. The weight matrix and bias vector of the second layer are independent and not shared with the first layer. The hidden state vector sequence output by the second layer is passed to the third layer, and so on, layer by layer. Each layer extracts higher-level abstract features based on the previous layer. The self-attention mechanism enables each layer to capture the dependency relationship between any two positions in the sequence. Shallow layers extract local patterns, and deep layers extract global patterns. The feedforward network performs nonlinear transformations on the features at each position to enhance expressive power. Residual connections ensure that the gradient can flow directly through multiple layers during backpropagation. Layer normalization stabilizes the input distribution of each layer and accelerates training convergence. After stacking multiple layers, each layer of the encoder outputs a corresponding hidden state vector sequence. The outputs of all layers constitute the set of hidden state vector sequences of each layer of the encoder.

[0070] From this set, select the hidden state vector sequences of the intermediate layer and the final layer. The selection of the intermediate layer depends on the total number of layers in the encoder, typically choosing a layer close to half the depth. The output of the intermediate layer retains features with a moderate level of abstraction, containing both low-level details and some semantic abstraction. The output of the final layer contains the highest level of semantic and behavioral pattern abstraction. For each position in the sequence, extract the hidden state vector in the intermediate layer and the hidden state vector in the final layer. Concatenate the two vectors in the dimensional direction. The concatenation operation arranges all elements of the intermediate layer vector and all elements of the final layer vector into a new vector. The length of the new vector is equal to the sum of the lengths of the two vectors. The concatenated vector contains both intermediate layer features and final layer features. For all positions in the sequence... Performing this concatenation operation yields an enhanced hidden state vector sequence for each character position. This sequence addresses the problem of incomplete information caused by using only single-layer features in existing technologies. Through layer-by-layer abstraction of multi-layer Transformers and feature fusion of intermediate and final layers, the encoder can simultaneously model the semantic content of the text and the user's input behavior patterns within a unified framework. The bidirectional self-attention mechanism ensures that the representation of each character depends not only on the preceding context but also on the following context, comprehensively capturing the contextual dependencies of the input sequence. Compared to existing technologies using unidirectional language models or simple recurrent neural networks, this method extracts richer features with stronger discriminative power. The concatenation and fusion of multi-layer features avoids information loss during deep network transmission and preserves feature representations at different levels of abstraction.

[0071] In one specific embodiment, the process of executing step S103 may specifically include the following steps:

[0072] Based on the facial key point coordinates in the image time series data, key points in the eye region and key points in the mouth region are extracted. The positional change of the eye key point coordinates and mouth key point coordinates in adjacent frames is calculated to obtain the eye motion vector sequence and mouth motion vector sequence.

[0073] The optical flow algorithm is used to calculate the pixel displacement field between adjacent frames in the temporal data of the image. The sum of squares of the horizontal displacement component and the vertical displacement component is calculated based on the pixel displacement field. The optical flow amplitude is statistically analyzed for the eye region and the mouth region respectively to obtain the micro-dynamic intensity sequence of the eye and the micro-dynamic intensity sequence of the mouth.

[0074] The mean and standard deviation of the micro-dynamic intensity sequences of the eyes and mouth are calculated. The dynamic threshold is set as the sum of the products of the mean and standard deviation. Local maxima points that exceed the dynamic threshold in the micro-dynamic intensity sequences of the eyes and mouth are identified to obtain a set of candidate points for micro-expression triggering.

[0075] Extract the corresponding frame timestamps from the set of micro-expression trigger candidate points, associate the frame timestamps with the corresponding local feature vectors, and obtain image behavior anchors containing timestamps and feature descriptions.

[0076] Specifically, the facial key point coordinate set contains points that mark the positions of various facial features. Based on the index range of the key points, the eye region and the mouth region are divided. The key points in the eye region include the coordinates of the left and right corners of the eye, the upper eyelid, and the lower eyelid. The key points in the mouth region include the coordinates of the corners of the mouth, the upper lip, and the lower lip. The extraction operation is completed by filtering the coordinates of the key points within a specific index range. The eye key point coordinates of adjacent frames are paired. The horizontal displacement of a key point in the current frame is obtained by subtracting the horizontal coordinate of the corresponding key point in the previous frame from the horizontal coordinate of the key point in the current frame. The vertical displacement is obtained by subtracting the vertical coordinate. The horizontal and vertical displacements constitute the two-dimensional displacement vector of the key point. This calculation is performed on all key points of the eye to obtain the set of eye key point displacement vectors in the current frame. The eye key point displacement vectors of all frames are arranged in time sequence to form the eye motion vector sequence. The mouth motion vector sequence is obtained by performing the same coordinate difference calculation on the mouth key points.

[0077] The optical flow algorithm takes two adjacent complete frames as input. It assumes that the grayscale values ​​of pixels remain constant but their positions shift between adjacent frames. It calculates the displacement vector for each pixel by solving the optical flow equation, which is based on the assumption of constant image brightness and spatial smoothness constraints. The algorithm outputs a pixel displacement field, a two-dimensional matrix of the same size as the image. Each position in the matrix stores the displacement vector of the corresponding pixel, containing two components: a horizontal displacement component and a vertical displacement component. The square of the horizontal component plus the square of the vertical component is calculated for each position in the displacement field to obtain the squared displacement amplitude of that pixel. The displacement amplitude is obtained by taking the square root of the square root. Based on the rectangular bounding box coordinates of the eye region in the image, the displacement amplitude of all pixels within the rectangular region is extracted from the pixel displacement field. The displacement amplitudes of all pixels within the region are added together and divided by the total number of pixels to obtain the average optical flow amplitude of the eye region. This average value quantifies the overall motion intensity of the eye region between adjacent frames. This average value is used as the eye micro-dynamic intensity value of the current frame. This calculation is performed on all adjacent frame pairs. The eye micro-dynamic intensity values ​​of all frames are arranged in time sequence to form the eye micro-dynamic intensity sequence. The mouth micro-dynamic intensity sequence is obtained by extracting the optical flow amplitude of the rectangular region of the mouth and statistically analyzing it.

[0078] The mean calculation iterates through all elements of the eye micro-dynamic intensity sequence, summing all intensity values ​​and dividing by the total number of elements to obtain the sequence mean. The standard deviation calculation first calculates the difference between each intensity value and the mean, squares the differences, sums them, and then takes the square root of the sum divided by the total number of elements to obtain the standard deviation. The dynamic threshold is calculated by multiplying the mean, standard deviation, and a preset coefficient. This threshold adaptively depends on the statistical characteristics of the current sequence. Peak detection iterates through the eye micro-dynamic intensity sequence, determining whether the intensity value at each position is greater than the intensity values ​​at the previous and next positions. If the condition is met and the intensity value exceeds the dynamic threshold, the position is marked as a local maximum point, and the timestamp of the corresponding frame is recorded as the candidate point timestamp. The same peak detection is performed on the mouth micro-dynamic intensity sequence, and all local maximum points detected in the eyes and mouth are summarized into a set of micro-expression trigger candidate points.

[0079] For each candidate point in the micro-expression trigger candidate point set, the corresponding frame timestamp is extracted, and a local feature vector is constructed for that candidate point. This local feature vector extracts dynamic features from the frame containing the candidate point and several frames before and after it, including optical flow amplitude, keypoint displacement, and inter-frame difference values. These feature values ​​from several consecutive frames are then concatenated in temporal order into a high-dimensional vector. This vector describes the dynamic change pattern of the face within a time window surrounding the candidate point. The frame timestamp and the local feature vector are stored as a pair of data, forming an image behavior anchor point. All image behavior anchor points corresponding to candidate points constitute an image behavior anchor point set. This set addresses the problem in existing technologies where image feature extraction focuses only on static appearance while ignoring dynamic behavior. It captures dynamic behavior through optical flow algorithms and keypoint displacement calculations. The system captures subtle dynamic changes in the face, including natural physiological micro-expressions such as blinking, slight eyebrow twitching, and slight opening or closing of the mouth, which spontaneously occur during the verification process. The triggering time and intensity of these micro-expressions vary from person to person and are difficult for attackers to consciously control and imitate. The adaptive setting of the dynamic threshold takes into account the variability of micro-expression intensity under different users and different verification scenarios. The local maxima identified by peak detection correspond to the turning point of the micro-expression from calm to arousal. These moments are intrinsically related to the user's cognitive processing on the timeline. The local feature vector encodes the dynamic evolution trajectory of the micro-expression before and after triggering, containing richer identity discrimination information than single-frame static features. The timestamp of the image behavior anchor provides a precise time positioning benchmark for subsequent cross-modal temporal alignment.

[0080] In one specific embodiment, the process of executing step S104 may specifically include the following steps:

[0081] From the pause feature sequence, select pause events whose pause duration exceeds the cognitive pause threshold, extract the pause time of the pause event as the text behavior anchor timestamp, extract the local behavior feature vector within a preset character range before and after the pause event, and obtain the text behavior anchor containing the timestamp and local features.

[0082] The absolute value of the time difference between the text behavior anchor timestamp and the image behavior anchor timestamp is calculated. The time sensitivity parameter is set based on the response delay baseline value calculated from the user's historical verification data. The product of the absolute value of the time difference and the time sensitivity parameter is used as the negative input of the exponential function to obtain the time similarity component.

[0083] Calculate the cosine similarity between the local feature vectors of text behavior anchors and the local feature vectors of image behavior anchors to obtain the behavior similarity component. Multiply the temporal similarity component with the behavior similarity component to obtain the pairing matching score between each text behavior anchor and each image behavior anchor, and construct a cross-modal pairing matrix.

[0084] The cumulative cost matrix is ​​calculated using a dynamic time warping algorithm based on the cross-modal pairing matrix. The optimal matching path is determined by backtracking path optimization. The average matching scores on the optimal matching path are then calculated to obtain the cross-modal consistency score.

[0085] Specifically, the process iterates through all pause events in the pause feature sequence, comparing the duration of each pause event with a cognitive pause threshold. This threshold is set as a critical point representing the duration of deep thinking or memory retrieval by the user. If the pause duration is greater than the threshold, the pause event is retained; if it is less than or equal to the threshold, the event is filtered out. After filtering, a subset of pause events meeting the criteria is obtained. From each retained pause event, the pause moment is extracted as a timestamp for the text behavior anchor. This timestamp marks the precise moment when the user experiences a cognitive pause during text input. The extraction range of the local behavior feature vector is determined by a preset character range, which defines how many characters before and after the pause event. For each pause event, its position index in the text input sequence is located within the window. A preset number of character positions are calculated forward to obtain the window start index, and a preset number of character positions are calculated backward to obtain the window end index. Typing interval embedding vector, error correction embedding vector, and pause pattern embedding vector corresponding to all characters within the window are extracted. The embedding vectors of all characters within the window are concatenated in sequence or aggregated through average pooling into a fixed-length local behavior feature vector. This vector encodes the input behavior pattern before and after the pause event. The timestamp and the local behavior feature vector are paired and stored to obtain a text behavior anchor point. The anchor points corresponding to all pause events that meet the conditions constitute the text behavior anchor point set.

[0086] The temporal correlation between text action anchors and image action anchors is calculated by iterating through all pairs of text action anchor sets and image action anchor sets. For each pair of anchors, its timestamp is extracted. The time difference is obtained by subtracting the image anchor timestamp from the text anchor timestamp. The absolute value of the time difference is taken to eliminate positive and negative directions, retaining only the magnitude of the time interval. The response latency baseline value is statistically obtained from the user's historical verification data. This baseline value reflects the typical time offset between the user's text input time and image acquisition time during normal verification. The statistical method includes extracting the time differences between text anchors and image anchors from all historical verification records, calculating the median or weighted average of these time differences as the baseline, and dividing the temporal sensitivity parameter by a numerical constant. The parameter is calculated by adding a fixed offset to the baseline value. The value of this parameter decreases as the baseline value increases. When the user's habitual response delay is long, the sensitivity parameter takes a small value, making the temporal similarity less sensitive to changes in the time difference. When the user's response delay is short, the sensitivity parameter takes a large value, making the temporal similarity more sensitive to the time difference. The scaled time difference is obtained by multiplying the absolute value of the time difference by the temporal sensitivity parameter. The scaled time difference is negative and used as the input of the exponential function. The exponential function uses the natural constant as the base. When the input is negative, the output is between zero and one. The larger the absolute value of the input, the closer the output is to zero. When the input is zero, the output is one. The output of this exponential function is the temporal similarity component, which quantifies the degree of matching between the two anchor points in the time dimension.

[0087] Cosine similarity calculation first extracts the local behavioral feature vectors of text anchors and the local feature vectors of image anchors. The dot product of the two vectors is obtained by multiplying the corresponding elements one by one and then summing them. The magnitude of the text feature vector is calculated by summing the squares of all elements of the vector and then taking the square root. The magnitude of the image feature vector is calculated in the same way. The cosine similarity is equal to the dot product divided by the product of the two magnitudes. This similarity is between negative one and positive one. The closer the value is to positive one, the closer the two vectors are in direction, that is, the more similar the feature patterns are. This similarity is the behavioral similarity component. The pairing matching score is calculated by multiplying the temporal similarity component by the behavioral similarity component. This score takes into account the proximity in the time dimension and the similarity in the feature space. All text anchors and image anchors are paired, and the matching score of each pair is calculated. Assuming that the text anchor set contains m anchors and the image anchor set contains n anchors, an m-row n-column cross-modal pairing matrix is ​​constructed. The element in the i-th row and j-th column of the matrix stores the pairing matching score of the i-th text anchor and the j-th image anchor.

[0088] The Dynamic Time Warping (RTW) algorithm takes a cross-modal pairing matrix as input. It finds the optimal alignment path between text anchor sequences and image anchor sequences. The cumulative cost matrix has the same size as the pairing matrix. The elements in the first row and first column of the matrix are initialized to the values ​​of the corresponding positions in the pairing matrix. For other columns in the first row, the cumulative cost equals the pairing score of that column plus the cumulative cost of the left-adjacent column. For other rows in the first column, the cumulative cost equals the pairing score of that row plus the cumulative cost of the top-adjacent row. For other positions, the cumulative cost equals the pairing score of the current position plus the minimum cumulative cost among the three adjacent positions to the left, top, and top left. This recursive calculation fills the entire cumulative cost matrix row by row or column by column from the top left corner. Backtracking path optimization starts from the bottom right corner of the matrix, recording the current position as a path node. It examines the cumulative costs of the three adjacent positions to the left, top, and top left, selecting the position with the minimum cumulative cost as the predecessor node. It moves to the predecessor node and records it as a path node, repeating this process until it backtracks to the top left corner of the matrix. All nodes on the backtracking path constitute the optimal pairing path. The matching path defines the optimal pairing relationship between text anchors and image anchors. The matching scores for all nodes on the optimal path are extracted, and these scores are summed and divided by the total number of path nodes to calculate the average score, which is the cross-modal consistency score. This score addresses the lack of cross-modal temporal correlation verification in existing technologies. By establishing a precise temporal correspondence between the cognitive pause time of text input and the micro-expression trigger time of image acquisition, the consistency of the two modalities in the temporal dimension is verified. The exponential decay characteristic of the temporal similarity component rapidly reduces the pairing score with large time offsets. The behavioral similarity component verifies the consistency of behavioral features by comparing the input behavioral patterns before and after the pause with the facial dynamic patterns around the micro-expressions. The dynamic time warping algorithm allows nonlinear scaling and misalignment of the two sequences at local time scales, overcoming the small-amplitude time drift that may exist between different modalities. The average score of the optimal matching path comprehensively reflects the global temporal consistency level of text behavior and image behavior throughout the verification process.

[0089] Figure 2This is a schematic diagram of the cross-modal behavior anchor timing alignment process in an embodiment of this application. The diagram illustrates the temporal correspondence between text input behavior and image micro-expression triggering. The solid curve represents the character interval time feature extracted from the text temporal data, and the dashed curve represents the eye or mouth micro-dynamic intensity feature extracted from the image temporal data. The horizontal axis represents the time axis of the verification process, the left vertical axis represents the millisecond value of the character interval time, and the right vertical axis represents the quantified value of the micro-expression intensity. It can be clearly observed from the diagram that at 1350 milliseconds and 2400 milliseconds, the text input exhibited prolonged pauses exceeding the cognitive pause threshold (character interval times were 170 milliseconds and 1050 milliseconds, respectively). These pauses were marked as text behavior anchors. At the same time, local peaks appeared in the image micro-expression intensity curve within the corresponding time window (intensity values ​​of approximately 9.1 and 7.8, respectively). These peak moments were marked as image behavior anchors. The temporal matching degree between the two sets of anchors was calculated using a dynamic time warping algorithm. When the text pause moments and the image micro-expression trigger moments showed high synchronization on the time axis, it indicated that the cross-modal consistency score of the verification sample was high, which was consistent with the natural cognitive-physiological response pattern of real users. Conversely, if there was obvious decoupling or misalignment between the two in time, it indicated that there might be attacks in the time dimension.

[0090] In one specific embodiment, the process of executing step S105 may specifically include the following steps:

[0091] The text behavior representation vector, image feature vector and cross-modal consistency score are input into the gated network, and after fully connected layer and softmax normalization, the text weight, image weight and temporal weight are output.

[0092] The text behavior representation vector is multiplied by the text weight, the image feature vector is multiplied by the image weight, and the temporal feature vector is multiplied by the temporal weight. Then, the results are weighted and summed to obtain the fused feature vector.

[0093] The fused feature vector is input into the classifier, which outputs the probability of the true user class. When the probability of the true user class exceeds the decision threshold, the validation pass result is output; otherwise, the validation rejection result is output.

[0094] Specifically, the text behavior representation vector, image feature vector, and cross-modal consistency score are concatenated dimensionally to form a unified input vector. This concatenation operation arranges all elements of the three data points end-to-end into a new vector. The concatenated vector is input into the first fully connected layer of the gating network. This layer contains several neurons. Each neuron calculates the dot product of the input vector and its corresponding weight vector and adds a bias scalar. The dot product operation multiplies each element of the input vector with the corresponding element of the weight vector and then sums them. The outputs of all neurons constitute the first layer output vector. This vector is input into the second fully connected layer and undergoes a similar linear transformation. The second layer output is input into the third fully connected layer, which contains three neurons corresponding to text weights, image weights, and temporal weights, respectively. The outputs of the three neurons are three scalars. Softmax normalization is applied to these three scalars. Softmax first calculates the exponent value of each scalar, and then divides each exponent value by the sum of the three exponent values. The three normalized values ​​are the text weight, image weight, and temporal weight. The sum of the three is always one and all are non-negative. This normalization ensures that the weights have a probability distribution property.

[0095] The temporal feature vector is derived from the cross-modal consistency score, which is a scalar. This scalar value is extended to the same dimension as the text behavior representation vector and the image feature vector by repeating the scalar value or multiplying it with the learnable parameter vector. Each element of the text behavior representation vector is multiplied by the text weight to obtain the weighted text feature vector. This multiplication is a scalar-vector multiplication. The scalar is multiplied by each element of the vector. Each element of the image feature vector is multiplied by the image weight to obtain the weighted image feature vector. Each element of the temporal feature vector is multiplied by the temporal weight to obtain the weighted temporal feature vector. The three weighted vectors are added element-wise, that is, the elements at corresponding positions are added to obtain the fused feature vector. The length of this vector is the same as the length of the three component vectors. The element at each position integrates the weighted contributions of the three modalities in this dimension.

[0096] The first fully connected layer of the classifier, which integrates feature vectors, contains several neurons. Each neuron performs a linear transformation on the input vector and then applies the ReLU activation function. The ReLU function retains non-negative values ​​and sets negative values ​​to zero. The activated vector is then input into a dropout layer. During training, the dropout layer randomly sets the outputs of some neurons to zero, while maintaining the outputs of all neurons during testing. The dropout vector is then input into a second fully connected layer, which contains two neurons corresponding to the real user class and the attack sample class, respectively. The output scalars of the two neurons are normalized using softmax. After normalization, the exponents of the two scalars are calculated and divided by the sum of their exponents. The first normalized value is the probability of the real user class, and the second normalized value is the probability of the attack sample class. The sum of the two probabilities is one. The probability of the real user class is compared with a preset decision threshold. If the probability is greater than the threshold, the user is identified as a real user, and the verification result is output as passed. If the probability is less than or equal to the threshold, the user is identified as an attack sample. This method outputs a validation rejection result and solves the problem of fixed fusion strategies in existing technologies. The gating network dynamically adjusts the fusion weights based on the feature distribution of the current validation samples and the cross-modal consistency score. When the cross-modal consistency score is high, the gating network automatically increases the temporal weights and decreases the text and image weights because high consistency indicates good temporal alignment and strong temporal feature discrimination. When the cross-modal consistency score is low, the gating network decreases the temporal weights. This adaptive mechanism is more flexible than existing technologies that use fixed weights or simple attention weighting. The fusion feature vectors reduce feature redundancy by weighted summation rather than simple concatenation. The two-layer structure of the classifier and dropout regularization prevent overfitting. The probability distribution output by softmax provides confidence information for the validation decision. The decision threshold balances the false recognition rate and the rejection rate. A higher threshold reduces the risk of false recognition but increases the risk of rejection, and a lower threshold has the opposite effect. The threshold is adjusted according to the security requirements of the actual application scenario.

[0097] The above describes the authentication method based on text-image multimodal feature fusion in the embodiments of this application. The following describes the authentication system based on text-image multimodal feature fusion in the embodiments of this application. Please refer to [link to relevant documentation]. Figure 3 One embodiment of the identity verification system based on text-image multimodal feature fusion in this application includes:

[0098] The acquisition module is used to simultaneously acquire text input sequences and image sequences, record the character input time and character interval time, extract facial key points from image frames, and obtain text temporal data and image temporal data;

[0099] The encoding module is used to encode the character interval time and input behavior features in the text time series data into an embedding vector, which is then processed by the encoder to obtain the text behavior representation vector.

[0100] The recognition module is used to calculate inter-frame dynamic features based on the image time-series data, identify the trigger time of micro-expressions, and obtain image behavior anchor points;

[0101] The calculation module is used to extract text behavior anchors from the text time series data, calculate the time series matching degree between the text behavior anchors and the image behavior anchors, and obtain a cross-modal consistency score.

[0102] The fusion module is used to adjust the fusion weights of the text behavior representation vector and the image feature vector based on the cross-modal consistency score, and output the verification result after feature fusion.

[0103] above Figure 3 The authentication system based on text-image multimodal feature fusion in this embodiment of the invention will be described in detail from the perspective of modular functional entities. The authentication device based on text-image multimodal feature fusion in this embodiment of the invention will be described in detail from the perspective of hardware processing.

[0104] Reference Figure 4 This invention also provides an authentication device based on text-image multimodal feature fusion. This authentication device can be a server, and its internal structure can be as follows: Figure 4 As shown, the text-image multimodal feature fusion-based authentication device includes a processor, memory, display screen, input device, network interface, and database connected via a system bus. The processor, designed as a computer, provides computational and control capabilities. The memory of the text-image multimodal feature fusion-based authentication device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database of the text-image multimodal feature fusion-based authentication device stores the data corresponding to this embodiment. The network interface of the text-image multimodal feature fusion-based authentication device is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements the above-described method.

[0105] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present invention and does not constitute a limitation on the identity verification device based on text-image multimodal feature fusion applied thereto.

[0106] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, wherein the computer-readable storage medium stores instructions that, when the instructions are executed on a computer, cause the computer to perform the steps of the authentication method based on text-image multimodal feature fusion.

[0107] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0108] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an authentication device (which may be a personal computer, server, or network device, etc.) based on text-image multimodal feature fusion to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0109] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An authentication method based on text-image multimodal feature fusion, characterized in that, The method includes: Simultaneously acquire text input sequences and image sequences, record the character input time and character interval time, extract facial key points from image frames, and obtain text temporal data and image temporal data; The character interval time and input behavior features in the text time series data are encoded into an embedding vector, which is then processed by an encoder to obtain a text behavior representation vector. Based on the image time-series data, calculate the inter-frame dynamic features, identify the micro-expression trigger time, and obtain the image behavior anchor point; Text behavior anchors are extracted from the text time series data, and the time series matching degree between the text behavior anchors and the image behavior anchors is calculated to obtain the cross-modal consistency score. The fusion weights of the text behavior representation vector and the image feature vector are adjusted based on the cross-modal consistency score, and the verification result is output after feature fusion.

2. The authentication method based on text-image multimodal feature fusion according to claim 1, characterized in that, The synchronous acquisition of text input sequences and image sequences records the character input time and character interval time, extracts facial key points from image frames, and obtains text temporal data and image temporal data, including: Collect each character input by the user and its corresponding input time, calculate the time interval between adjacent characters, and obtain a text input sequence containing character encoding, timestamp, and character interval time; The backspace key usage time and the number of correction characters in the text input sequence are detected, and the timestamps of error correction events are recorded to obtain the error correction feature sequence; Identify time intervals exceeding the pause threshold in the character interval time, record the pause occurrence time and pause duration to obtain a pause feature sequence, and combine the text input sequence, the error correction feature sequence and the pause feature sequence into the text time series data; A series of face image frames are acquired by an image acquisition device. The facial key point coordinates of each frame are extracted using a face detection algorithm. The image difference value between adjacent frames is calculated to obtain the image time series data containing key point coordinates, frame timestamps and inter-frame difference values.

3. The authentication method based on text-image multimodal feature fusion according to claim 2, characterized in that, The step of encoding the character interval time and input behavior features in the text time series data into an embedding vector, and then processing it through an encoder to obtain a text behavior representation vector, includes: The character interval time is mapped to a typing interval embedding vector through a positional encoding function. The error correction event type in the error correction feature sequence is converted into an error correction embedding vector after one-hot encoding. The pause duration in the pause feature sequence is classified according to the duration range and mapped to a pause pattern embedding vector. The characters in the text input sequence are semantically encoded to obtain a character semantic embedding vector. The character semantic embedding vector is then concatenated with the typing interval embedding vector, the error correction embedding vector, and the pause pattern embedding vector to obtain a composite embedding vector. The composite embedding vector is input into a multi-layer Transformer encoder, and the contextual dependencies and temporal behavior patterns between characters are calculated through a bidirectional self-attention mechanism to obtain the hidden state vector at each character position. The hidden state vectors of the intermediate and final layers of the encoder are extracted, and the attention weights at each position are calculated through the behavior attention fusion module. The hidden state vectors of all character positions are then weighted and averaged to obtain the text behavior representation vector.

4. The authentication method based on text-image multimodal feature fusion according to claim 3, characterized in that, The process involves inputting the composite embedding vector into a multi-layer Transformer encoder, calculating the contextual dependencies and temporal behavior patterns between characters through a bidirectional self-attention mechanism, and obtaining the hidden state vector for each character position, including: The composite embedding vector is input into the first layer of the Transformer encoder to calculate the query matrix, key matrix, and value matrix. The attention score matrix is ​​obtained through matrix multiplication. After the attention score matrix is ​​normalized by softmax, it is multiplied by the value matrix to obtain the self-attention output vector of the first layer. The self-attention output vector is input into the feedforward neural network. After linear transformation and activation function processing, residual connection and layer normalization are performed to obtain the hidden state vector of the first layer. The hidden state vector of the first layer is passed as input to the encoder of each subsequent layer in sequence. The self-attention calculation and feedforward network processing are repeatedly performed to update the hidden state vector layer by layer, so as to obtain the hidden state vector sequence of each layer of the encoder. The intermediate layer output and the final layer output are extracted from the hidden state vector sequence, and the intermediate layer output and the final layer output are concatenated to obtain the hidden state vector of each character position containing multi-level semantic and behavioral features.

5. The authentication method based on text-image multimodal feature fusion according to claim 4, characterized in that, The step of calculating inter-frame dynamic features based on the image temporal data, identifying the micro-expression trigger time, and obtaining image behavior anchor points includes: Based on the facial key point coordinates in the image time series data, key points in the eye region and key points in the mouth region are extracted. The positional change of the eye key point coordinates and mouth key point coordinates in adjacent frames is calculated to obtain the eye motion vector sequence and the mouth motion vector sequence. The optical flow algorithm is used to calculate the pixel displacement field between adjacent frames in the image time series data. The sum of squares of the horizontal displacement component and the vertical displacement component is calculated based on the pixel displacement field. The optical flow amplitude is statistically analyzed for the eye region and the mouth region respectively to obtain the eye micro-dynamic intensity sequence and the mouth micro-dynamic intensity sequence. The mean and standard deviation of the micro-dynamic intensity sequence of the eyes and the micro-dynamic intensity sequence of the mouth are calculated. The dynamic threshold is set as the sum of the products of the mean and the standard deviation. Local maxima points in the micro-dynamic intensity sequences of the eyes and the mouth that exceed the dynamic threshold are identified to obtain a set of micro-expression triggering candidate points. Extract the corresponding frame timestamp from the set of micro-expression triggering candidate points, associate the frame timestamp with the corresponding local feature vector, and obtain the image behavior anchor point containing the timestamp and feature description.

6. The authentication method based on text-image multimodal feature fusion according to claim 5, characterized in that, The step of extracting text behavior anchors from the text time-series data, calculating the time-series matching degree between the text behavior anchors and the image behavior anchors, and obtaining a cross-modal consistency score includes: From the pause feature sequence, select pause events whose pause duration exceeds the cognitive pause threshold, extract the pause time of the pause event as the text behavior anchor timestamp, extract the local behavior feature vector within a preset character range before and after the pause event, and obtain the text behavior anchor containing the timestamp and local features. Calculate the absolute value of the time difference between the text behavior anchor timestamp and the image behavior anchor timestamp, set the time sensitivity parameter based on the response delay baseline value calculated from the user's historical verification data, and use the product of the absolute value of the time difference and the time sensitivity parameter as the negative input of the exponential function to obtain the time similarity component. Calculate the cosine similarity between the local feature vector of the text behavior anchor and the local feature vector of the image behavior anchor to obtain the behavior similarity component. Multiply the temporal similarity component with the behavior similarity component to obtain the pairing matching score between each text behavior anchor and each image behavior anchor, and construct a cross-modal pairing matrix. Based on the cross-modal pairing matrix, the cumulative cost matrix is ​​calculated using the dynamic time warping algorithm. The optimal matching path is determined by backtracking path optimization. The average matching scores on the optimal matching path are then calculated to obtain the cross-modal consistency score.

7. The authentication method based on text-image multimodal feature fusion according to claim 1, characterized in that, The step of adjusting the fusion weights of the text behavior representation vector and the image feature vector based on the cross-modal consistency score, and outputting the verification result after feature fusion includes: The text behavior representation vector, the image feature vector, and the cross-modal consistency score are input into a gating network. After passing through a fully connected layer and softmax normalization, the text weight, image weight, and temporal weight are output. The text behavior representation vector is multiplied by the text weight, the image feature vector is multiplied by the image weight, and the temporal feature vector is multiplied by the temporal weight, and then a weighted sum is obtained to obtain the fused feature vector. The fused feature vector is input into the classifier, which outputs the probability of the true user category. When the probability of the true user category exceeds the decision threshold, the validation pass result is output; otherwise, the validation rejection result is output.

8. An identity verification system based on text-image multimodal feature fusion, characterized in that, For implementing the authentication method based on text-image multimodal feature fusion as described in any one of claims 1-7, the authentication system based on text-image multimodal feature fusion comprises: The acquisition module is used to simultaneously acquire text input sequences and image sequences, record the character input time and character interval time, extract facial key points from image frames, and obtain text temporal data and image temporal data; The encoding module is used to encode the character interval time and input behavior features in the text time series data into an embedding vector, which is then processed by the encoder to obtain the text behavior representation vector. The recognition module is used to calculate inter-frame dynamic features based on the image time-series data, identify the trigger time of micro-expressions, and obtain image behavior anchor points; The calculation module is used to extract text behavior anchors from the text time series data, calculate the time series matching degree between the text behavior anchors and the image behavior anchors, and obtain a cross-modal consistency score. The fusion module is used to adjust the fusion weights of the text behavior representation vector and the image feature vector based on the cross-modal consistency score, and output the verification result after feature fusion.

9. An identity verification device based on text-image multimodal feature fusion, characterized in that, The device includes a memory and a processor, the memory storing a computer program that can run on the processor, and the processor executing the computer program to implement the authentication method based on text-image multimodal feature fusion as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is run by the processor, it causes the processor to execute the authentication method based on text-image multimodal feature fusion as described in any one of claims 1 to 7.

Images