Enterprise video authentication consistency verification method and system based on multi-dimensional data fusion

Abstract

The invention relates to an enterprise video authentication consistency verification method and system based on multi-dimensional data fusion, and the method comprises the steps: guiding a user to carry out multi-mode interaction with an authentication real object, a challenge code and an environment in a real physical space through issuing a dynamic visual challenge code and an acoustic interaction instruction; after audio and video streams are synchronously received, a time sequence feature sequence is extracted in real time, continuous frames are abstracted into a multi-element time-space relation graph sequence, a time sequence graph neural network is used for deeply analyzing the spatial adjacency, motion collaboration and semantic association relation between elements, and feature vectors representing global consistency are output. And on the basis, checking of four dimensions is executed in parallel, and finally, the score of each dimension is fused through an adaptive decision model to output an authentication result. According to the method and the device, the spanning from single face feature verification to panoramic relation verification is realized, the situations of virtual camera injection, video pre-recording, deep scene counterfeiting and the like can be effectively prevented, and the authenticity of video authentication is greatly improved.

Description

Technical Field

[0001] This application relates to the field of video anti-counterfeiting technology, and in particular to a method and system for verifying the consistency of enterprise video authentication based on multi-dimensional data fusion. Background Technology

[0002] Video authentication, as a key means of remote identity verification and business scenario auditing, has been widely used in areas such as financial account opening, enterprise services, and online contract signing. Its core purpose is to ensure that the operator is the actual person performing the operation in a real physical environment and business scenario, thereby preventing identity theft and virtual fraud.

[0003] Currently, mainstream enhanced video authentication technologies focus on face liveness detection. This method typically requires users to perform randomly specified actions (such as blinking or shaking their heads) during the authentication process, and uses computer vision algorithms to analyze subtle changes, texture features, or optical flow responses of the face in the video to determine whether the face in front of the camera is a real, live person.

[0004] The aforementioned technology only protects against the single element of the face, and cannot verify the authenticity of the scene beyond the face. Therefore, it cannot verify whether the key scene elements claimed by the user, such as the office environment and company logo, actually exist and are consistent with the person's behavior in time and space, thus affecting the company's video authentication service.

[0005] Based on this, this application provides a method and system for verifying the consistency of enterprise video authentication based on multi-dimensional data fusion. Summary of the Invention

[0006] To address the issue that protection is limited to a single element like the face, and cannot verify the authenticity of scenes beyond the face, thus failing to verify whether key scene elements such as the user's claimed office environment and company logo actually exist and are consistent with personnel behavior in time and space, affecting the enterprise's video authentication service, this application provides an enterprise video authentication consistency verification method and system based on multi-dimensional data fusion.

[0007] Firstly, this application provides a method for verifying the consistency of enterprise video authentication based on multi-dimensional data fusion, employing the following technical solution: including: A dynamic visual challenge code and a random acoustic interaction command are sent to the client. The visual attributes of the dynamic visual challenge code in the display area change over time according to a predetermined pattern. The acoustic interaction command includes content that requires the user to operate a specified authentication object and associate it with the dynamic visual challenge code. The system receives a synchronous video stream and an audio stream returned by the client, which contain responses to the dynamic visual challenge code and the acoustic interaction command. From each frame of the synchronous video stream, the system extracts in real time the sequence of human features, the sequence of scene objects, the sequence of environmental text, and the sequence of visual states of the dynamic visual challenge code. From the audio stream, the system extracts in real time the sequence of acoustic features and identifies the completed content of the acoustic interaction command. For multiple consecutive frames of a synchronized video stream, various elements extracted from each frame are used as nodes to construct a multi-element spatiotemporal relationship graph. The nodes include at least personnel nodes, physical object nodes, challenge code nodes, and text nodes. The edges between nodes encode at least spatial adjacency relationships, motion coordination relationships, and semantic association relationships, generating a relationship graph sequence. The relation graph sequence is input into a pre-trained spatiotemporal graph neural network. The spatiotemporal graph neural network aggregates edge-based node relation information within the same frame through graph convolutional layers, and learns the evolution law of node and edge states in the relation graph sequence through temporal modeling layers, and outputs a first feature vector representing the global spatiotemporal consistency of the entire interaction process. Based on the personnel feature sequence and the acoustic feature sequence, the naturalness of personnel movement and lip-verbal synchronization are verified to generate a first verification score; based on the correspondence between the visual state sequence of the dynamic visual challenge code and the response actions in the personnel feature sequence, the real-time performance and accuracy of the interactive response are verified to generate a second verification score; based on the environmental text sequence and the preset authentication declaration information, the logical consistency of the scene text is verified to generate a third verification score; based on the first feature vector extracted by the spatiotemporal graph neural network, the physical rationality of the movement and spatial relationship between multiple elements is verified to generate a fourth verification score. The first verification score, the second verification score, the third verification score, and the fourth verification score are input into the adaptive decision model. The model is weighted, fused, and judged according to the preset risk strategy and threshold. If the comprehensive judgment result exceeds the preset threshold, a video authentication pass signal is output; otherwise, a signal indicating authentication failure or manual review is output.

[0008] Preferably, the receiving client returns a synchronized video stream and an audio stream that respond to the dynamic visual challenge code and the acoustic interaction command; from each frame of the synchronized video stream, in real time, it extracts personnel feature sequences, scene object feature sequences, environmental text sequences, and the visual state sequence of the dynamic visual challenge code; from the audio stream, it extracts acoustic feature sequences in real time and identifies the completed content of the acoustic interaction command, including: Receive the synchronous video stream and audio stream returned by the client, decode the synchronous video stream to obtain a sequence of video frames arranged in chronological order, and decode the audio stream to obtain the audio signal; For each frame in the video frame sequence, a face detection and alignment model is used to locate the face region in the current frame image; a facial landmark detection model is used to extract multiple sets of facial landmark coordinates from the aligned face region; a human pose estimation model is used to extract multiple sets of human body key point coordinates in the current frame image; the sets of facial landmark coordinates and human body key point coordinates are merged to form the personnel features of the corresponding frame. Using a pre-trained object detection model, at least one preset authentication-related object is detected and located in the current frame image. The authentication-related object includes the specified authentication object and environmental markers. For each detected object, its bounding box coordinates and category label are obtained, and a visual feature vector is extracted from the bounding box region of the object using a convolutional neural network. The bounding box coordinates, category label and visual feature vector are combined to form the scene object features of the corresponding frame. Within a preset display area of ​​the current frame image, the dynamic visual challenge code is detected; the visual attributes of the dynamic visual challenge code in the current frame are extracted, and the visual attributes include at least its overall color histogram, outline shape descriptor and center pixel coordinates within the frame; the color histogram, shape descriptor and center pixel coordinates are used to construct the visual state information of the dynamic visual challenge code in the corresponding frame. In the current frame image, optical character recognition is performed on the preset text interest area to obtain the recognized text content and its position coordinates in the image; the text content and position coordinates are used to construct the environmental text features of the corresponding frame. The audio signal is divided into frames and windowed to extract the acoustic features of each frame. The acoustic features include Mel frequency cepstral coefficients and speaker embedding vectors. The extracted acoustic features are arranged in chronological order to form the acoustic feature sequence. The audio signal is input into the speech recognition model to obtain the transcribed text; based on the semantic template of the acoustic interaction command, key information is extracted from the transcribed text as the completion content of the acoustic interaction command. The personnel features, scene object features, visual state information of the dynamic visual challenge code, and environmental text features extracted from each frame of the video frame sequence are arranged in chronological order according to the corresponding frames to obtain the personnel feature sequence, the scene object feature sequence, the visual state sequence of the dynamic visual challenge code, and the environmental text sequence.

[0009] Preferably, for each frame in the video frame sequence, a face detection and alignment model is used to locate the face region in the current frame image; a facial landmark detection model is used to extract multiple sets of facial landmark coordinates from the aligned face region; a human pose estimation model is used to extract multiple sets of human body keypoint coordinates in the current frame image; and the facial landmark coordinate sets and human body keypoint coordinate sets are merged to form the personnel features of the corresponding frame, including: For the current frame image in the video frame sequence, a face detection model is used to detect whether there is a face region in the image; if multiple faces are detected, the face with the largest area or located in the center of the image is selected as the target authentication face, and the initial bounding box of the target authentication face is obtained. Based on the initial bounding box, extract the face region image; use the face alignment model to perform rotation, scaling and translation transformations on the face region image to generate a pose-normalized standard face image; A standard face image is input into a pre-trained facial landmark detection model to obtain a set of two-dimensional coordinate points that represent the contours of key facial features, thus forming a set of facial landmark coordinates. Within the global scope of the current frame image, a human pose estimation model is used to detect and locate the two-dimensional coordinates of the main joints of the target human body, forming a set of human body key point coordinates; the human body key points include at least the corresponding points of the neck and shoulders; The set of facial landmark coordinates extracted from the current frame is merged with the set of human body key point coordinates to form the preliminary personnel feature data of the current frame. In the processing of continuous video frame sequences, the preliminary personnel feature data of the current frame is verified based on the temporal continuity of the facial landmark coordinate set. If the motion pattern of the facial landmark coordinate set changes abruptly between adjacent frames, it is determined that the corresponding frame feature extraction is abnormal, and interpolation or frame reference mechanism is used for correction to ensure that the final personnel features are smooth and continuous in the time dimension.

[0010] Preferably, for the continuous multiple frames of the synchronized video stream, various elements extracted from each frame are used as nodes to construct a multi-element spatiotemporal relationship graph. The nodes include at least personnel nodes, object nodes, challenge code nodes, and text nodes. The edges between nodes encode at least spatial adjacency relationships, motion coordination relationships, and semantic association relationships, generating a relationship graph sequence, including: For the t-th frame of the synchronized video stream, where t is a natural number, based on the extracted feature sequences and the personnel features of the t-th frame, the geometric center of the set of facial landmark coordinates or the statistical features of the predefined set of key points are encoded into the initial feature vector of the personnel node. For each object detected in the scene object features of frame t, an initial feature vector for the corresponding physical node is generated based on its visual feature vector and its category label. Based on the visual state information of the dynamic visual challenge code in frame t, an initial feature vector of the challenge code node is generated. For each text segment identified in the environmental text features of frame t, its text content is converted into a semantic vector through a word embedding model, and combined with its position coordinates, an initial feature vector for the corresponding text node is generated. In the graph constructed in frame t, three types of edges are established and encoded between nodes: calculate the normalized distance or bounding box overlap between any two nodes in the two-dimensional space of the image; if the distance is less than the spatial threshold or there is overlap, establish a spatial adjacency edge between the two nodes and encode the corresponding spatial relationship as the attribute of the corresponding edge. For movable nodes, calculate their motion trajectory in multiple consecutive frames. By analyzing the correlation between the motion trajectory directions of different nodes, if the correlation exceeds the motion coordination threshold, establish motion coordination edges between the corresponding nodes. The weight or attribute of the motion coordination edge is encoded by the strength value of the motion direction correlation. Based on the preset knowledge base or rules, establish semantic association edges between logically related nodes. Repeat the node instantiation and relation edge construction steps to construct a corresponding multi-factor spatiotemporal relation graph for each consecutive frame in the video frame sequence, resulting in the relation graph sequence arranged in chronological order.

[0011] Preferably, the step of inputting the relationship graph sequence into a pre-trained spatiotemporal graph neural network (SPLM) involves the SPLM aggregating edge-based node relationship information within the same frame through graph convolutional layers and learning the evolution of node and edge states in the relationship graph sequence through temporal modeling layers. The SPLM outputs a first feature vector representing the global spatiotemporal consistency of the entire interaction process, including: Based on the spatiotemporal graph neural network, the input relationship graph sequence is updated frame by frame in chronological order and the temporal information is fused: For each frame of the graph, the network performs multi-layer graph convolution operation so that each node aggregates feature information from its neighboring nodes according to the spatial adjacency relationship edge, motion cooperation relationship edge and semantic association relationship edge connected to it, and generates updated node features containing local association context information. Based on the single-frame node feature update, for each node, its features in different frames of the entire relation graph sequence are arranged into a sequence along the time dimension; a temporal convolutional layer is used to process the corresponding sequence to capture the dynamic pattern and long-term dependency relationship of the target node features over time, and output the temporal enhanced features of the target node. After obtaining the temporal enhancement features of all nodes, a global pooling operation is performed on the temporal enhancement features of all nodes in the same frame to generate a frame-level global feature vector that represents the overall scene relationship of the corresponding frame. The frame-level global feature vectors of each frame are arranged in chronological order, and then global pooling is performed again along the time dimension to gather the time clues of the entire interaction process and generate a preliminary sequence-level global feature vector. Based on the temporal enhancement features of nodes, the importance weights of different node types for the true / false discrimination task are calculated. According to the importance weights, the temporal enhancement features of nodes are weighted and fused to generate a set of node feature representations that have been calibrated for importance. The weighted and fused node feature representations are concatenated with the sequence-level global feature vectors, and then nonlinear transformation and dimensionality reduction are performed through one or more fully connected layers to compress and output a first feature vector of fixed dimension.

[0012] Preferably, the first feature vector, which aggregates information from all nodes across all frames based on the spatiotemporal graph neural network and outputs a fixed-dimensional feature vector representing the overall pattern of multi-element spatiotemporal relationships throughout the entire interaction process, includes: The node feature representation generated after node type attention weighted fusion is concatenated with the sequence-level global feature vector to obtain a concatenated feature vector that integrates fine-grained node attention information and global spatiotemporal context information. The spliced ​​feature vector is input into a feature transformation module consisting of at least one fully connected layer. Based on the feature transformation module, the spliced ​​feature vector is nonlinearly projected through the first fully connected layer to expand or transform its feature representation space. A non-linear activation function is applied to the output of the first fully connected layer, and the activated features are then normalized to stabilize the training process and accelerate convergence. The normalized high-dimensional features are mapped to a significantly lower-dimensional feature space through the second fully connected layer, thereby achieving feature compression and refinement. The feature vector that has undergone dimensionality reduction is subjected to a final linear projection, which compresses it to a preset fixed dimension, and the output is used as the first feature vector.

[0013] Preferably, before inputting the relationship graph sequence into a pre-trained spatiotemporal graph neural network, whereby the spatiotemporal graph neural network aggregates edge-based node relationship information within the same frame through graph convolutional layers and learns the evolution law of node and edge states in the relationship graph sequence through temporal modeling layers, and outputs a first feature vector representing the global spatiotemporal consistency of the entire interaction process, the method further includes: A large number of real video authentication interaction samples were collected as positive samples, and corresponding fake video authentication interaction samples were generated as negative samples by injecting virtual camera data, using pre-recorded videos, or tampering with scene elements. For each training sample, its multimodal feature sequence is extracted, and its corresponding spatiotemporal relationship graph sequence labeled with true and false labels is constructed to form a training dataset. Construct a spatiotemporal graph neural network, which at least includes the following sequentially connected components: a graph convolutional module for spatial relation aggregation, a temporal convolutional module for temporal evolution modeling, a node type attention weighted fusion module, and a feature compression and output module; initialize the parameters of each module of the network using a preset initialization strategy; The training dataset is input into the initialized spatiotemporal graph neural network for iterative training. In each iteration, the spatiotemporal graph neural network outputs the predicted first feature vector based on the input relationship graph sequence. A joint loss function is constructed to optimize the spatiotemporal graph neural network. The joint loss function includes at least: the difference loss between the true and false classification results calculated based on the first feature vector and the true and false labels of the samples. The model is encouraged to learn the high temporal consistency of the motion coordination relationship and spatial relationship among multiple elements for positive samples, while amplifying the relationship consistency loss of inconsistencies in the relationship for negative samples. The performance of the trained spatiotemporal graph neural network is evaluated using an independent validation dataset. Once the performance meets the preset accuracy and generalization metrics, the model training is completed. The trained spatiotemporal graph neural network is embedded as a fixed module in the video authentication and anti-counterfeiting system for extracting the first feature vector and performing consistency analysis on the real-time acquired interactive video stream.

[0014] Secondly, this application discloses an enterprise video authentication consistency verification device based on multi-dimensional data fusion, which adopts the following technical solution, including: The interactive verification module is used to send a dynamic visual challenge code and a random acoustic interaction command to the client. The visual attributes of the dynamic visual challenge code in the display area change over time according to a predetermined pattern. The acoustic interaction command includes content that requires the user to operate a specified authentication object and associate it with the dynamic visual challenge code. The feature extraction module is used to receive a synchronous video stream and an audio stream returned by the client, which contain responses to the dynamic visual challenge code and the acoustic interaction command; extract personnel feature sequences, scene object feature sequences, environmental text sequences, and the visual state sequence of the dynamic visual challenge code from each frame of the synchronous video stream in real time; and extract acoustic feature sequences from the audio stream in real time and identify the completed content of the acoustic interaction command. The spatiotemporal correlation module is used to construct a multi-element spatiotemporal relationship graph for multiple consecutive frames of a synchronized video stream, using various elements extracted from each frame as nodes. The nodes include at least personnel nodes, physical object nodes, challenge code nodes, and text nodes. The edges between nodes encode at least spatial adjacency relationships, motion coordination relationships, and semantic relationships, generating a sequence of relationship graphs. The feature evolution module is used to input the relationship graph sequence into a pre-trained spatiotemporal graph neural network. The spatiotemporal graph neural network aggregates the node relationship information based on edges within the same frame through graph convolutional layers, and learns the evolution law of node and edge states in the relationship graph sequence through temporal modeling layers, and outputs a first feature vector that represents the global spatiotemporal consistency of the entire interaction process. The feature verification module is used to verify the naturalness of personnel movement and lip-verb synchronization based on the personnel feature sequence and the acoustic feature sequence, and generate a first verification score; to verify the real-time performance and accuracy of interactive response based on the correspondence between the visual state sequence of the dynamic visual challenge code and the response actions in the personnel feature sequence, and generate a second verification score; to verify the logical consistency of scene text based on the environmental text sequence and preset authentication declaration information, and generate a third verification score; and to verify the physical rationality of the movement and spatial relationship between multiple elements based on the first feature vector extracted by the spatiotemporal graph neural network, and generate a fourth verification score. The video authentication module is used to input the first verification score, the second verification score, the third verification score and the fourth verification score into the adaptive decision model, and perform weighted fusion and judgment according to the preset risk strategy and threshold. If the comprehensive judgment result exceeds the preset threshold, the video authentication pass signal is output; otherwise, the authentication failure signal or the need for manual review signal is output.

[0015] Thirdly, this application also provides a control device, the device comprising: It includes a memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and executed, such as the enterprise video authentication consistency verification method based on multi-dimensional data fusion described above.

[0016] Fourthly, this application also provides a computer-readable storage medium storing a computer program that can be loaded by a processor and executed as described above regarding the enterprise video authentication consistency verification method based on multi-dimensional data fusion.

[0017] In summary, this application uses a server to send dynamic visual challenge codes and random acoustic interaction commands to the client, actively guiding the user to engage in multimodal interactions with designated certified objects, challenge codes, and the environment in a real physical space. After synchronously receiving the audio and video streams, the system extracts multi-dimensional data in real time, including personnel feature sequences, scene object feature sequences, environmental text sequences, and challenge code status sequences. The core of the solution lies in abstracting each frame into a multi-element spatiotemporal relationship graph, where personnel, objects, challenge codes, and text serve as nodes. The edges between nodes dynamically encode their spatial adjacency, motion coordination, and semantic association relationships, thereby constructing a temporal graph sequence describing the entire interaction process. This graph sequence is fed into a pre-trained spatiotemporal graph neural network for deep analysis. The network aggregates element relationships within the same frame through graph convolution and captures cross-frame evolution patterns through temporal modeling, ultimately outputting a first feature vector that integrates global spatiotemporal consistency information. Building upon this foundation, multi-level consistency checks are performed in parallel: verifying the naturalness of the operator's movements and the synchronization of their voice and lips; verifying the real-time performance and accuracy of their responses to dynamic challenges; verifying the logical consistency between the scene text and the declared information; and verifying the physical rationality of the motion and spatial relationships between multiple elements based on the first feature vector. Finally, the system adaptively integrates the various verification scores and outputs an authentication decision. Thus, by leaping from single facial feature verification to panoramic relationship verification, it organically combines proactive challenges, multimodal perception, graph relationship modeling, and multi-level logical analysis, constructing a complete framework capable of simultaneously defending against virtual camera injection, pre-recorded videos, deepfake scenarios, and "real person, fake scene" attacks. While ensuring the operator is a real, living entity, it further strongly verifies the authenticity of the physical scene, the compliance of business elements, and the logical consistency of the entire interaction process, thereby providing reliable protection for remote identity and real-world scenario authentication with high security requirements. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating a method for verifying the consistency of enterprise video authentication based on multi-dimensional data fusion.

[0019] Figure 2 This is a structural block diagram of an enterprise video authentication consistency verification device based on multi-dimensional data fusion. Detailed Implementation

[0020] The following combination Figure 1 - Figure 2 This application will be described in further detail.

[0021] Video authentication, as a crucial step in verifying identity and context in enterprise services, directly impacts business operations, financial security, commercial integrity, and social trust. With the increasing realism of deepfake technology and the proliferation of virtual camera tools, attack methods have expanded from forging a single identity to constructing entire virtual scenarios, posing a serious threat to traditional authentication systems from new composite attacks such as "real person, fake scene." Currently, mainstream technologies rely on facial liveness detection and voiceprint / lip movement synchronization for personnel feature verification. While these can effectively identify pre-recorded videos or still photos, their scope remains limited to the "person" in the image, completely failing to determine whether the "scene" in which the person is situated is real or whether the interaction between the person and the scene conforms to physical logic. This fundamental flaw allows attackers to bypass authentication simply by synthesizing a fake background in a real facial video, rendering the verification of critical business elements such as the office environment and corporate identity meaningless. To this end, this application proposes a completely new technical paradigm that shifts the focus of authentication from "isolated individuals" to "a dynamic unity between individuals and scenarios." Through proactive interaction, multimodal perception, and spatiotemporal relationship modeling, it constructs a complete authentication system that can simultaneously verify the authenticity of individuals and the reality of scenarios.

[0022] Reference Figure 1 The embodiments of this application include at least steps S10 to S60.

[0023] S10: Send a dynamic visual challenge code and a random acoustic interaction command to the client. The visual attributes of the dynamic visual challenge code in the display area change over time according to a predetermined pattern. The acoustic interaction command contains content that requires the user to operate a specified authentication object and associate it with the dynamic visual challenge code.

[0024] S20: Receive the synchronous video stream and audio stream returned by the client, which contain responses to the dynamic visual challenge code and acoustic interaction instructions; extract in real time the personnel feature sequence, scene object feature sequence, environmental text sequence, and visual state sequence of the dynamic visual challenge code from each frame of the synchronous video stream; extract the acoustic feature sequence in real time from the audio stream and identify the completed content of the acoustic interaction instructions.

[0025] S30: For multiple consecutive frames of a synchronized video stream, various elements extracted from each frame are used as nodes to construct a multi-element spatiotemporal relationship graph. The nodes include at least personnel nodes, physical object nodes, challenge code nodes, and text nodes. The edges between nodes encode at least spatial adjacency relationships, motion coordination relationships, and semantic association relationships, generating a relationship graph sequence.

[0026] S40, the relation graph sequence is input into the pre-trained spatiotemporal graph neural network. The spatiotemporal graph neural network aggregates the node relation information based on the edge in the same frame through the graph convolutional layer, and learns the evolution law of the node and edge state in the relation graph sequence through the temporal modeling layer, and outputs the first feature vector representing the global spatiotemporal consistency of the entire interaction process.

[0027] S50, based on personnel feature sequences and acoustic feature sequences, verifies the naturalness of personnel movement and lip-verbal synchronization, generating a first verification score; based on the correspondence between the visual state sequence of the dynamic visual challenge code and the response actions in the personnel feature sequence, verifies the real-time performance and accuracy of the interactive response, generating a second verification score; based on the environmental text sequence and the preset authentication declaration information, verifies the logical consistency of the scene text, generating a third verification score; based on the first feature vector extracted by the spatiotemporal graph neural network, verifies the physical rationality of the movement and spatial relationship between multiple elements, generating a fourth verification score.

[0028] S60: Input the first verification score, the second verification score, the third verification score, and the fourth verification score into the adaptive decision model, and perform weighted fusion and judgment according to the preset risk strategy and threshold. If the comprehensive judgment result exceeds the preset threshold, output a video authentication pass signal; otherwise, output a signal that authentication fails or requires manual review.

[0029] Specifically, the system first proactively constructs a verification scenario that relies on real physical space and object interaction by issuing dynamic visual challenge codes and acoustic interaction commands, thus laying the foundation for defense. Subsequently, the system synchronously receives and parses the response audio and video streams, extracting the state sequences of personnel, objects, text, and challenge codes in real time, providing structured multimodal data input for subsequent analysis. The core step is to construct a multi-element spatiotemporal relationship graph sequence from these continuous frame data and perform deep analysis using a temporal graph neural network to extract the first feature vector representing the inherent logic and collaborative patterns of the entire interaction process. Based on this, the solution performs parallel verification in four dimensions: personnel movement and lip-verbal synchronization verification, challenge-response real-time and accuracy verification, scene text logic consistency verification, and multi-element physical relationship rationality verification based on the first feature vector. Finally, all verification scores are fused through adaptive decision-making to output the authentication result. This achieves a fundamental leap from passive identification to active verification, and from single feature verification to panoramic relationship analysis. Its core function is to simultaneously and indivisibly ensure the authenticity of the operator, the authenticity of the physical scene, and the spatiotemporal logical consistency among all interactive elements, thereby building a comprehensive defense capability against advanced attacks such as virtual camera injection, pre-recorded videos, and deepfake scenarios.

[0030] In some embodiments, step S20 specifically includes the following steps: receiving a synchronous video stream and an audio stream returned by the client; decoding the synchronous video stream to obtain a video frame sequence arranged in chronological order; decoding the audio stream to obtain an audio signal; for each frame image in the video frame sequence, using a face detection and alignment model to locate the face region in the current frame image; using a facial landmark detection model to extract multiple sets of facial landmark coordinates from the aligned face region; using a human pose estimation model to extract multiple sets of human keypoint coordinates in the current frame image; merging the set of facial landmark coordinates and the set of human keypoint coordinates to form the personnel features of the corresponding frame; using a pre-trained object detection model to detect and locate at least one preset authentication-related object in the current frame image, the authentication-related object including designated authentication objects and environmental landmarks; for each detected object, obtaining its bounding box coordinates and category label, and using a convolutional neural network to extract a visual feature vector from the bounding box region of the object; combining the bounding box coordinates, category label, and visual feature vector to form the scene object features of the corresponding frame; Within a preset display area of ​​the current frame image, a dynamic visual challenge code is detected; the visual attributes of the dynamic visual challenge code in the current frame are extracted, including at least its overall color histogram, outline shape descriptor, and center pixel coordinates within the frame; the color histogram, shape descriptor, and center pixel coordinates are used to construct the visual state information of the dynamic visual challenge code in the corresponding frame; in the current frame image, optical character recognition is performed on a preset text interest area to obtain the recognized text content and its position coordinates in the image; the text content and position coordinates are used to construct the environmental text features of the corresponding frame; the audio signal is divided into frames and windowed to extract the acoustic features of each frame, including Mel-frequency cepstral coefficients and speaker embedding vectors; the extracted acoustic features are arranged in chronological order to form an acoustic feature sequence; the audio signal is input into a speech recognition model to obtain transcribed text; based on the semantic template of the acoustic interaction command, key information is extracted from the transcribed text as the completion content of the acoustic interaction command; The personnel features, scene object features, visual state information of the dynamic visual challenge code, and environmental text features extracted from each frame of the video frame sequence are arranged in the time order of the corresponding frames to obtain the personnel feature sequence, scene object feature sequence, visual state sequence of the dynamic visual challenge code, and environmental text sequence.

[0031] Specifically, the received synchronous audio and video streams are first decoded and separated to obtain the original video frame sequence and audio signal. Then, for each video frame, four types of feature extraction are performed in parallel: motion key points constituting personnel features are extracted using face and human pose models; scene object features, including bounding boxes, categories, and visual features, are extracted using object detection and convolutional neural networks; the color, shape, and positional visual state of the dynamic challenge code are precisely extracted in a preset region; and optical character recognition is performed on a preset text region to obtain the environmental text and its location. Audio processing simultaneously extracts acoustic feature sequences and recognizes voice commands. Finally, all features are strictly arranged in chronological order, forming personnel feature sequences, scene object feature sequences, visual state sequences of the dynamic visual challenge code, environmental text sequences, and acoustic feature sequences. Its core function is to systematically and in real-time deconstruct a continuous authentication video stream containing rich interactive actions into a set of time-aligned, structured multimodal feature sequences. This provides a synchronous and complete data foundation for subsequent construction of spatiotemporal relationship graphs and multi-level consistency analysis, serving as an indispensable prerequisite for the entire authentication system to achieve advanced semantic understanding from raw data.

[0032] Furthermore, considering the issues of personnel feature extraction and verification processes, the corresponding processing steps are as follows: For the current frame image in the video frame sequence, a face detection model is used to detect whether a face region exists in the image; if multiple faces are detected, the face with the largest area or located in the center of the image is selected as the target authentication face, and the initial bounding box of the target authentication face is obtained; based on the initial bounding box, the face region image is extracted; using a face alignment model, the face region image is rotated, scaled, and translated to generate a pose-normalized standard face image; the standard face image is input into a pre-trained facial landmark detection model to obtain a set of two-dimensional coordinate points representing the contours of key facial parts, forming a set of facial landmark coordinates; in the global range of the current frame image... Within the scope, a human pose estimation model is used to detect and locate the two-dimensional coordinates of the main joints of the target human body, forming a set of human key point coordinates. The human key points include at least the corresponding points on the neck and shoulders. The set of facial landmark coordinates extracted in the current frame is merged with the set of human key point coordinates to form the preliminary personnel feature data of the current frame. In the continuous video frame sequence processing, the preliminary personnel feature data of the current frame is verified based on the temporal continuity of the set of facial landmark coordinates. If the motion pattern of the set of facial landmark coordinates shows a non-physical abrupt change between adjacent frames, it is determined that the corresponding frame feature extraction is abnormal, and interpolation or previous and next frame reference mechanisms are used for correction to ensure that the final personnel features are smooth and continuous in the time dimension.

[0033] Specifically, the process begins by ensuring accurate selection and pose normalization of the target face through face detection and alignment. Then, a specialized model is used to extract detailed facial landmark coordinates and key body point coordinates, which are then combined to form preliminary feature data representing the person's pose and movement. This process involves extracting the two-dimensional coordinates of N facial landmarks from a standardized face image, which can be formally represented as follows: For frame t, obtain the set of facial landmark coordinates LP_t = { (x_t^1, y_t^1), (x_t^2, y_t^2), ..., (x_t^N, y_t^N)}, where (x_t^i, y_t^i) represents the horizontal and vertical coordinates of the i-th landmark in the image.

[0034] Its core innovation lies in the introduction of a temporal continuity verification mechanism, which diagnoses feature extraction anomalies in real time by analyzing whether non-physical abrupt changes occur in the motion patterns of facial landmarks between adjacent frames. This mechanism is based on the calculation and analysis of motion vectors. First, the motion vector ΔP_t^i = (x_{t+1}^i - x_t^i, y_{t+1}^i - y_t^i) of the i-th landmark is calculated between adjacent frames. Then, by analyzing the changes in the direction and amplitude of the motion vector ΔP_t^i in consecutive frames, it is determined whether it violates the smoothness constraint of natural motion. If a sudden change occurs in the motion pattern, it is determined to be an abnormal frame. The correlation of motion directions between landmarks, cor_{mn}, is calculated to identify cooperative motion patterns, and its formula is expressed as: ; in, This represents the correlation of motion directions between marker m and marker n, with a value ranging from 0 to 1. A larger value indicates stronger coordination. T' represents the number of effective frames in which the motion vectors of marker m and n are both non-zero. and Let m and n represent the motion vectors of marker points m and n from frame i to frame i+1, respectively; the symbol · indicates the dot product operation of the vectors; ||·| indicates the magnitude (length) of the vector; and |·| indicates taking the absolute value. This formula quantifies the correlation between the motions of different facial regions. In real video, the correlation between related marker points... The values ​​should remain relatively stable, but this pattern is easily disrupted in forged videos. Based on this principle, the system can automatically trigger interpolation or reference to previous and subsequent frames for correction, ensuring the smoothness and continuity of the final constructed sequence of personnel features over time. In some embodiments, step S30 specifically includes the following steps: For the t-th frame of the synchronized video stream, where t is a natural number, based on the extracted feature sequences and the personnel features of the t-th frame, the geometric center of the set of facial landmark coordinates or the statistical features of the predefined key point set are encoded into the initial feature vector of the personnel node; For each object detected in the scene object features of the t-th frame, its visual feature vector is used as the basis, combined with its category label, to generate the initial feature vector of the corresponding physical object node; Based on the visual state information of the dynamic visual challenge code in the t-th frame, the initial feature vector of the challenge code node is generated; For each piece of text identified in the environmental text features of the t-th frame, its text content is converted into a semantic vector through a word embedding model, and combined with its position coordinates, the initial feature vector of the corresponding text node is generated; In the graph constructed in the t-th frame, three types of edges are established and encoded between nodes: the normalized distance or bounding box overlap between any two nodes in the two-dimensional space of the image is calculated, and if the distance is less than the spatial threshold or there is overlap, a spatial adjacency edge is established between the two nodes, and the corresponding spatial relationship is encoded into the attribute of the corresponding edge; For movable nodes, their motion trajectories are calculated across multiple consecutive frames. By analyzing the correlation between the directions of different node motion trajectories, if the correlation exceeds a motion coordination threshold, motion coordination edges are established between the corresponding nodes. The weight or attribute of the motion coordination edge is encoded by the strength value of the motion direction correlation. Based on a preset knowledge base or rules, semantic association edges are established between logically related nodes. For example, strong semantic association edges are established between a "personnel node" and its actual "physical object node" (such as an employee badge), between a "physical object node" and its surface "text node" (such as the name on the employee badge), and between an "environmental text node" (such as a wall sign) and the claimed company name. The attributes of this edge are encoded by the association type.

[0035] Repeat the above steps to construct a corresponding multi-factor spatiotemporal relationship graph for each consecutive frame in the video frame sequence, resulting in a sequence of relationship graphs arranged in chronological order.

[0036] Specifically, the heterogeneous features extracted from each frame, such as people, objects, challenge codes, and text, are uniformly encoded into different types of nodes in the graph. Their complex relationships are dynamically modeled using three types of edges: spatial adjacency edges describe the visual layout, motion collaboration edges quantify the statistical correlation of motion between nodes, and semantic association edges inject prior business logic. The core function of this scheme is to systematically transform a multimodal video frame sequence into a graph sequence that preserves its spatiotemporal structure and semantics and can be directly processed by deep networks. This transforms the problem of verifying the authenticity of authentication videos into a pattern recognition problem of dynamic interaction relationships within the graph, providing a unique and necessary structured input for subsequent deep consistency analysis based on graph neural networks.

[0037] In some embodiments, step S40 specifically includes the following steps: updating node features and fusing temporal information frame by frame based on the spatiotemporal graph neural network: For each frame of the graph, the network performs multi-layer graph convolution operations to enable each node to aggregate feature information from its neighboring nodes based on the spatial adjacency edges, motion cooperation edges, and semantic association edges connected to it, generating updated node features containing local association context information; Based on the completion of single-frame node feature updates, for each node, its features in different frames of the entire relational graph sequence are arranged into a sequence along the time dimension; Temporal convolutional layers are used to process the corresponding sequences to capture the dynamic patterns and long-term dependencies of the target node features over time, and output the temporal enhanced features of the target node; After obtaining the temporal augmentation features of all nodes, a global pooling operation is performed on the temporal augmentation features of all nodes within the same frame to generate a frame-level global feature vector representing the overall scene relationship of the corresponding frame. The frame-level global feature vectors of each frame are arranged in chronological order and global pooling is performed again along the time dimension to gather the temporal clues of the entire interaction process and generate a preliminary sequence-level global feature vector. Based on the temporal augmentation features of the nodes, the importance weights of different node types for the authenticity discrimination task are calculated. According to the importance weights, the temporal augmentation features of the nodes are weighted and fused to generate a set of node feature representations calibrated by importance. The weighted and fused node feature representations are concatenated with the sequence-level global feature vector, and then nonlinear transformation and dimensionality reduction are performed through one or more fully connected layers to compress and output a first feature vector of fixed dimension.

[0038] Specifically, the Spatiotemporal Graph Neural Network (ST-GNN) processes the input sequence of relational graphs... Feature extraction is performed through hierarchical processing. First, spatial, motion, and semantic relationships between nodes within a single frame are aggregated using graph convolutional layers. The graph convolutional operation of each layer can be formally represented as: ; in, This represents the feature matrix of all nodes in the l-th layer; This represents the feature matrix of the (l+1)th layer nodes after graph convolution. It is an adjacency matrix with added self-loops, where A is the original adjacency matrix. The self-loops ensure that nodes retain their own characteristics when aggregating neighbor information. yes The degree matrix, whose diagonal elements represent pairs. Perform a power-1 (negative two) operation for normalization; It is the trainable weight matrix of the l-th layer; This represents a non-linear activation function (such as ReLU). This operation allows each node to fuse feature information from its neighboring nodes (connected by spatial adjacency edges, motion collaboration edges, and semantic association edges).

[0039] Subsequently, the system gradually fuses spatial and temporal information through a two-layer pooling operation to generate a sequence-level global feature vector. Simultaneously, it applies attention weighting to the enhanced features based on node type importance to highlight key authentication elements. Finally, the network concatenates the weighted node features with the global features and compresses them into a fixed-dimensional first feature vector through a nonlinear transformation. Thus, through the collaborative processing of spatial aggregation, temporal modeling, hierarchical fusion, and attention refinement, a feature representation rich in discriminative information is distilled from the dynamic relationship graph sequence. This representation comprehensively encodes the collaborative movement patterns, spatial structural relationships, and temporal evolution laws of multiple elements in the authentication interaction.

[0040] In some embodiments, considering the refinement process of generating the final discriminative features from the spatiotemporal graph neural network, the corresponding processing steps are as follows: The node feature representation generated after node type attention weighted fusion is concatenated with the sequence-level global feature vector to obtain a concatenated feature vector that integrates fine-grained node attention information and global spatiotemporal context information; the concatenated feature vector is input into a feature transformation module consisting of at least one fully connected layer; based on the feature transformation module, the concatenated feature vector is nonlinearly projected through the first fully connected layer to expand or transform its feature representation space; a nonlinear activation function is applied to the output of the first fully connected layer, and the activated features are subjected to layer normalization to stabilize the training process and accelerate convergence; the normalized high-dimensional features are mapped to a significantly lower-dimensional feature space through the second fully connected layer to achieve feature compression and refinement; the feature vector after dimensionality reduction is finally linearly projected to compress it to a preset fixed dimension, and the output is used as the first feature vector.

[0041] Specifically, firstly, it concatenates attention-weighted fine-grained node features with sequence-level global features representing the overall scene, ensuring that the output simultaneously encompasses key local elements and global spatiotemporal context. Subsequently, this concatenated feature is enhanced with nonlinear projection, activation, and layer normalization to improve expressive power and training stability. It then undergoes dimensionality reduction through a fully connected layer to remove redundant information, achieving feature refinement. Finally, it is compressed to a fixed dimension via linear projection. Its core function is to distill high-dimensional, complex spatiotemporal interaction information into a highly discriminative first feature vector through this series of structured transformations. This vector serves as a consistency metric for the entire system, providing the crucial mathematical representation for subsequent judgments on whether the interaction process conforms to real physical and logical laws.

[0042] In some embodiments, the construction and training process of the spatiotemporal graph neural network is as follows: A large number of real video authentication interaction samples are collected as positive samples, and corresponding fake video authentication interaction samples are generated as negative samples by injecting virtual camera data, using pre-recorded videos, or tampering with scene elements; for each training sample, its multimodal feature sequence is extracted, and its corresponding multi-element spatiotemporal relationship graph sequence labeled with true and false tags is constructed to form a training dataset; a spatiotemporal graph neural network is constructed, which at least includes, in sequence: a graph convolutional module for spatial relationship aggregation, a temporal convolutional module for temporal evolution modeling, a node type attention weighted fusion module, and a feature compression and output module; the parameters of each module of the network are initialized using a preset initialization strategy; The training dataset is input into the initialized spatiotemporal graph neural network for iterative training. In each iteration, the spatiotemporal graph neural network outputs the predicted first feature vector based on the input relation graph sequence. A joint loss function is constructed to optimize the spatiotemporal graph neural network end-to-end. This loss function consists of two weighted parts, aiming to simultaneously optimize classification accuracy and relation consistency. ; in, Indicates the total loss; This represents the classification loss, used to measure the difference between the model's prediction and the true label; Represents the relation consistency loss, used to constrain the spatiotemporal relation patterns learned by the model; It is an adjustable hyperparameter used to balance the contributions of the two losses. 1. Classification Loss: Standard cross-entropy loss is used. Let the model predict the fake probability for a batch of samples as follows: If the corresponding real label is y (1 for real video, 0 for fake video), then: ; Where N represents the number of samples in the batch; and Let represent the true label and predicted probability of the i-th sample, respectively.

[0043] 2. Relationship Consistency Loss: For positive samples (real videos), the model is encouraged to maintain high temporal consistency in the relationships between nodes (especially those modeled through motion-cooperative edges); for negative samples (fake videos), inconsistencies in these relationships are amplified. One approach is to encourage the temporal enhancement features extracted by the model to approximate a "consistent prototype" in positive samples and deviate from it in negative samples. This can be formally implemented using the idea of ​​contrastive learning. ; in, Let N and N represent the sets of indices for positive and negative samples in a batch, respectively. This represents the first feature vector extracted from the i-th sample after passing through the spatiotemporal graph neural network; It is a learnable "consistent prototype" vector; ||·|| represents the L2 norm (Euclidean distance) of the vector. This loss function drives the feature vectors of positive samples to cluster around the prototype. The feature vectors of negative samples are pushed away, thereby enhancing the model's ability to discriminate inconsistencies in relationships.

[0044] The performance of the trained spatiotemporal graph neural network is evaluated using an independent validation dataset. Once the performance meets the preset accuracy and generalization metrics, the model training is completed. The trained spatiotemporal graph neural network is then embedded as a fixed module into the video authentication and anti-counterfeiting system for extracting the first feature vector and performing consistency analysis on the real-time acquired interactive video stream.

[0045] Specifically, a dataset containing real samples and various forgery methods is constructed, and their corresponding spatiotemporal relationship graph sequences are labeled to provide the learning foundation for the model. The network architecture integrates core modules such as spatial graph convolution, temporal modeling, attention fusion, and feature compression. By jointly optimizing classification loss and relationship consistency loss, the model learns natural and fluent collaborative patterns from positive samples and identifies contradictory relationships that violate physical and logical principles from negative samples. The fully trained network is deployed as the core analysis engine in the online system, enabling it to automatically extract the first feature vector from real-time video streams. This vector condenses the multidimensional consistency information of the interaction process, thus providing crucial deep semantic criteria for the final authenticity decision of the entire authentication system.

[0046] Furthermore, step S50 specifically includes the following steps: The model receives the first to fourth verification scores generated from the previous process. These scores quantify the confidence levels of personnel authenticity, response timeliness, logical consistency, and physical rationality, respectively. Subsequently, the model dynamically assigns weights to each score according to a preset, configurable risk strategy. For example, in a high-security enterprise onboarding scenario, the weights of the scenario text logical consistency and physical relationship rationality are increased, while in a daily check-in scenario, the emphasis may be more on personnel authenticity and response timeliness. After weight allocation, the model calculates a comprehensive judgment value through weighted summation or a more complex nonlinear function. This comprehensive judgment value is compared with a set of preset thresholds: if it exceeds the pass threshold, the system automatically outputs an authentication pass signal; if it is below the rejection threshold, an authentication failure signal is output; if it falls into the "gray area" between the pass and rejection thresholds, a signal requiring manual review is output. Thus, by integrating multi-dimensional evidence through a configurable strategy, a balance between flexibility and accuracy in authentication decisions is achieved. This avoids the risk of misjudgment based on a single dimension and can adaptively adjust according to the risk tolerance of different business scenarios, thereby optimizing user experience and operational efficiency while ensuring security.

[0047] The implementation principle of the enterprise video authentication consistency verification method based on multi-dimensional data fusion in this application embodiment is as follows: The server sends dynamic visual challenge codes and random acoustic interaction commands to the client, actively guiding the user to generate multimodal interactions with designated authentication objects, challenge codes, and the environment in a real physical space. After synchronously receiving the audio and video streams, the system extracts multi-dimensional data in real time, including personnel feature sequences, scene object feature sequences, environmental text sequences, and challenge code status sequences. Next, the core of the solution lies in abstracting each frame into a multi-element spatiotemporal relationship graph, where personnel, objects, challenge codes, and text serve as nodes. The edges between nodes dynamically encode their spatial adjacency, motion coordination, and semantic association relationships, thereby constructing a temporal graph sequence describing the entire interaction process. This graph sequence is fed into a pre-trained spatiotemporal graph neural network for deep analysis. The network aggregates the element relationships within the same frame through graph convolution and captures cross-frame evolution patterns through temporal modeling, ultimately outputting a first feature vector that integrates global spatiotemporal consistency information. Building upon this foundation, multi-level consistency checks are performed in parallel: verifying the naturalness of the operator's movements and the synchronization of their voice and lips; verifying the real-time performance and accuracy of their responses to dynamic challenges; verifying the logical consistency between the scene text and the declared information; and verifying the physical rationality of the motion and spatial relationships between multiple elements based on the first feature vector. Finally, the system adaptively integrates the various verification scores and outputs an authentication decision. Thus, by leaping from single facial feature verification to panoramic relationship verification, it organically combines proactive challenges, multimodal perception, graph relationship modeling, and multi-level logical analysis, constructing a complete framework capable of simultaneously defending against virtual camera injection, pre-recorded videos, deepfake scenarios, and "real person, fake scene" attacks. While ensuring the operator is a real, living entity, it further strongly verifies the authenticity of the physical scene, the compliance of business elements, and the logical consistency of the entire interaction process, thereby providing reliable protection for remote identity and real-world scenario authentication with high security requirements.

[0048] Figure 1 This is a flowchart illustrating an enterprise video authentication consistency verification method based on multi-dimensional data fusion in one embodiment. It should be understood that, although... Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows; unless explicitly stated otherwise, there is no strict order requirement for the execution of these steps, and they can be executed in other orders; and Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0049] Based on the same technical concept, referring to Figure 2 This application also provides an enterprise video authentication consistency verification device based on multi-dimensional data fusion, which adopts the following technical solution: The device includes: The interactive verification module is used to send a dynamic visual challenge code and a random acoustic interaction command to the client. The visual attributes of the dynamic visual challenge code in the display area change over time according to a predetermined pattern. The acoustic interaction command contains content that requires the user to operate a specified authentication object and associate it with the dynamic visual challenge code. The feature extraction module is used to receive synchronous video and audio streams returned by the client, which contain responses to dynamic visual challenge codes and acoustic interaction commands; extract personnel feature sequences, scene object feature sequences, environmental text sequences, and visual state sequences of dynamic visual challenge codes in real time from each frame of the synchronous video stream; and extract acoustic feature sequences in real time from the audio stream and identify the completed content of acoustic interaction commands. The spatiotemporal correlation module is used to construct a multi-element spatiotemporal relationship graph for multiple consecutive frames of a synchronized video stream, using various elements extracted from each frame as nodes. The nodes include at least personnel nodes, physical object nodes, challenge code nodes, and text nodes. The edges between nodes encode at least spatial adjacency relationships, motion coordination relationships, and semantic relationships, generating a sequence of relationship graphs. The feature evolution module is used to input the relation graph sequence into the pre-trained spatiotemporal graph neural network. The spatiotemporal graph neural network aggregates the node relation information based on the edge in the same frame through the graph convolutional layer, and learns the evolution law of the node and edge state in the relation graph sequence through the temporal modeling layer, and outputs the first feature vector representing the global spatiotemporal consistency of the entire interaction process. The feature verification module is used to verify the naturalness of personnel movement and lip-verb synchronization based on personnel feature sequences and acoustic feature sequences, generating a first verification score; to verify the real-time performance and accuracy of interactive responses based on the correspondence between the visual state sequence of the dynamic visual challenge code and the response actions in the personnel feature sequence, generating a second verification score; to verify the logical consistency of scene text based on the environmental text sequence and the preset authentication declaration information, generating a third verification score; and to verify the physical rationality of the movement and spatial relationships between multiple elements based on the first feature vector extracted by the spatiotemporal graph neural network, generating a fourth verification score. The video authentication module is used to input the first verification score, the second verification score, the third verification score, and the fourth verification score into the adaptive decision model. It performs weighted fusion and judgment based on the preset risk strategy and threshold. If the comprehensive judgment result exceeds the preset threshold, it outputs a video authentication pass signal; otherwise, it outputs a signal indicating authentication failure or requiring manual review.

[0050] In some embodiments, the feature extraction module is specifically used to receive the synchronous video stream and audio stream returned by the client, decode the synchronous video stream to obtain a video frame sequence arranged in chronological order, and decode the audio stream to obtain an audio signal; For each frame in the video frame sequence, a face detection and alignment model is used to locate the face region in the current frame image; a facial landmark detection model is used to extract multiple sets of facial landmark coordinates from the aligned face region; a human pose estimation model is used to extract multiple sets of human body key point coordinates in the current frame image; the sets of facial landmark coordinates and human body key point coordinates are merged to form the personnel features of the corresponding frame. Using a pre-trained object detection model, at least one preset authentication-related object is detected and located in the current frame image. The authentication-related object includes the specified authentication object and environmental markers. For each detected object, its bounding box coordinates and category label are obtained, and a visual feature vector is extracted from the bounding box region of the object using a convolutional neural network. The bounding box coordinates, category label and visual feature vector are combined to form the scene object features of the corresponding frame. Within the preset display area of ​​the current frame image, detect the dynamic visual challenge code; extract the visual attributes of the dynamic visual challenge code in the current frame, the visual attributes include at least its overall color histogram, outline shape descriptor and center pixel coordinates in the frame; combine the color histogram, shape descriptor and center pixel coordinates to form the visual state information of the dynamic visual challenge code in the corresponding frame. In the current frame image, optical character recognition is performed on the preset text interest area to obtain the recognized text content and its position coordinates in the image; the text content and position coordinates are used to construct the environmental text features of the corresponding frame. The audio signal is divided into frames and windowed to extract the acoustic features of each frame. The acoustic features include Mel frequency cepstral coefficients and speaker embedding vectors. The extracted acoustic features are arranged in chronological order to form an acoustic feature sequence. The audio signal is input into the speech recognition model to obtain the transcribed text; based on the semantic template of the acoustic interaction command, key information is extracted from the transcribed text as the completion content of the acoustic interaction command. The personnel features, scene object features, visual state information of the dynamic visual challenge code, and environmental text features extracted from each frame of the video frame sequence are arranged in the time order of the corresponding frames to obtain the personnel feature sequence, scene object feature sequence, visual state sequence of the dynamic visual challenge code, and environmental text sequence.

[0051] In some embodiments, the feature extraction module is specifically used to detect whether there is a face region in the current frame image in the video frame sequence using a face detection model; if multiple faces are detected, the face with the largest area or located in the center region of the image is selected as the target authentication face, and the initial bounding box of the target authentication face is obtained. Based on the initial bounding box, extract the face region image; use the face alignment model to perform rotation, scaling and translation transformations on the face region image to generate a pose-normalized standard face image; A standard face image is input into a pre-trained facial landmark detection model to obtain a set of two-dimensional coordinate points that represent the contours of key facial features, thus forming a set of facial landmark coordinates. Within the global scope of the current frame image, a human pose estimation model is used to detect and locate the two-dimensional coordinates of the main joints of the target human body, forming a set of human body key point coordinates; the human body key points include at least the corresponding points of the neck and shoulders; The set of facial landmark coordinates extracted from the current frame is merged with the set of human body key point coordinates to form the preliminary personnel feature data of the current frame. In the processing of continuous video frame sequences, the preliminary personnel feature data of the current frame is verified based on the temporal continuity of the facial landmark coordinate set. If the motion pattern of the facial landmark coordinate set changes abruptly between adjacent frames, it is determined that the corresponding frame feature extraction is abnormal, and interpolation or frame reference mechanism is used for correction to ensure that the final personnel features are smooth and continuous in the time dimension.

[0052] In some embodiments, the spatiotemporal correlation module is specifically used for the t-th frame of the synchronized video stream, where t is a natural number, to encode the geometric center of the set of facial landmark coordinates or the statistical features of a predefined set of key points into the initial feature vector of the personnel node based on the extracted feature sequences and the personnel features of the t-th frame. For each object detected in the scene object features of frame t, an initial feature vector for the corresponding physical node is generated based on its visual feature vector and its category label. Based on the visual state information of the dynamic visual challenge code in frame t, the initial feature vector of the challenge code node is generated. For each text segment identified in the environmental text features of frame t, its text content is converted into a semantic vector through a word embedding model, and combined with its position coordinates, an initial feature vector for the corresponding text node is generated. In the graph constructed in frame t, three types of edges are established and encoded between nodes: calculate the normalized distance or bounding box overlap between any two nodes in the two-dimensional space of the image; if the distance is less than the spatial threshold or there is overlap, establish a spatial adjacency edge between the two nodes and encode the corresponding spatial relationship as the attribute of the corresponding edge. For movable nodes, calculate their motion trajectory in multiple consecutive frames. By analyzing the correlation between the motion trajectory directions of different nodes, if the correlation exceeds the motion coordination threshold, establish motion coordination edges between the corresponding nodes. The weight or attribute of the motion coordination edge is encoded by the strength value of the motion direction correlation. Based on the preset knowledge base or rules, establish semantic association edges between logically related nodes. Repeat the node instantiation and relation edge construction steps to construct a corresponding multi-factor spatiotemporal relation graph for each consecutive frame in the video frame sequence, resulting in a relation graph sequence arranged in chronological order.

[0053] In some embodiments, the feature evolution module is specifically used to update node features and fuse temporal information frame by frame in chronological order based on the spatiotemporal graph neural network for the input relation graph sequence: For each frame of the graph, the network uses multi-layer graph convolution operations to enable each node to aggregate feature information from its neighboring nodes based on the spatial adjacency relationship edges, motion cooperation relationship edges and semantic association relationship edges connected to it, and generate updated node features containing local association context information. Based on the single-frame node feature update, for each node, its features in different frames of the entire relation graph sequence are arranged into a sequence along the time dimension; a temporal convolutional layer is used to process the corresponding sequence to capture the dynamic pattern and long-term dependency relationship of the target node features over time, and output the temporal enhanced features of the target node. After obtaining the temporal enhancement features of all nodes, a global pooling operation is performed on the temporal enhancement features of all nodes in the same frame to generate a frame-level global feature vector that represents the overall scene relationship of the corresponding frame. The frame-level global feature vectors of each frame are arranged in chronological order, and then global pooling is performed again along the time dimension to gather the time clues of the entire interaction process and generate a preliminary sequence-level global feature vector. Based on the temporal enhancement features of nodes, the importance weights of different node types for the true / false discrimination task are calculated. According to the importance weights, the temporal enhancement features of nodes are weighted and fused to generate a set of node feature representations that have been calibrated for importance. The weighted and fused node feature representations are concatenated with the sequence-level global feature vectors, and then nonlinear transformation and dimensionality reduction are performed through one or more fully connected layers to compress and output a first feature vector of fixed dimension.

[0054] In some embodiments, the feature evolution module is specifically used to concatenate the node feature representation generated after node type attention weighted fusion with the sequence-level global feature vector to obtain a concatenated feature vector that integrates fine-grained node attention information and global spatiotemporal context information. The concatenated feature vector is input into a feature transformation module consisting of at least one fully connected layer. Based on the feature transformation module, the concatenated feature vector is nonlinearly projected through the first fully connected layer to expand or transform its feature representation space. A non-linear activation function is applied to the output of the first fully connected layer, and the activated features are then normalized to stabilize the training process and accelerate convergence. The normalized high-dimensional features are mapped to a significantly lower-dimensional feature space through the second fully connected layer, thereby achieving feature compression and refinement. The feature vectors that have undergone dimensionality reduction are subjected to a final linear projection, which compresses them to a preset fixed dimension, and the output is used as the first feature vector.

[0055] In some embodiments, the feature evolution module is also used to collect a large number of real video authentication interaction samples as positive samples, and generate corresponding fake video authentication interaction samples as negative samples by injecting virtual camera data, using pre-recorded videos, or tampering with scene elements. For each training sample, extract its multimodal feature sequence and construct its corresponding multi-element spatiotemporal relationship graph sequence labeled with true and false labels to form a training dataset; Construct a spatiotemporal graph neural network, which at least includes the following sequentially connected components: a graph convolutional module for spatial relation aggregation, a temporal convolutional module for temporal evolution modeling, a node type attention weighted fusion module, and a feature compression and output module; initialize the parameters of each module of the network using a preset initialization strategy; The training dataset is input into the initialized spatiotemporal graph neural network for iterative training. In each iteration, the spatiotemporal graph neural network outputs the predicted first feature vector based on the input relationship graph sequence. A joint loss function is constructed to optimize the spatiotemporal graph neural network. The joint loss function includes at least: the true / false classification result calculated based on the first feature vector, and the difference loss between the true / false labels of the samples. The model is encouraged to learn the high temporal consistency of the motion coordination relationship and spatial relationship among multiple elements for positive samples, while the relationship consistency loss of the inconsistency in the relationship is amplified for negative samples. The performance of the trained spatiotemporal graph neural network is evaluated using an independent validation dataset. Once the performance meets the preset accuracy and generalization metrics, the model training is completed. The trained spatiotemporal graph neural network is embedded as a fixed module in the video authentication and anti-counterfeiting system for extracting the first feature vector and performing consistency analysis on the real-time acquired interactive video stream.

[0056] This application also discloses a control device.

[0057] Specifically, the control device includes a memory and a processor. The memory stores a computer program that can be loaded by the processor and executed to perform the aforementioned enterprise video authentication consistency verification method based on multi-dimensional data fusion.

[0058] This application also discloses a computer-readable storage medium.

[0059] Specifically, the computer-readable storage medium stores a computer program that can be loaded by a processor and executed, such as the enterprise video authentication consistency verification method based on multi-dimensional data fusion described above. The computer-readable storage medium includes, for example, various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0060] The above are all preferred embodiments of this application, and are not intended to limit the scope of protection of this application. Therefore, all equivalent changes made in accordance with the structure, shape and principle of this application should be covered within the scope of protection of this application.

Claims

1. A method for verifying the consistency of enterprise video authentication based on multi-dimensional data fusion, characterized in that, include: A dynamic visual challenge code and a random acoustic interaction command are sent to the client. The visual attributes of the dynamic visual challenge code in the display area change over time according to a predetermined pattern. The acoustic interaction command includes content that requires the user to operate a specified authentication object and associate it with the dynamic visual challenge code. The system receives a synchronous video stream and an audio stream returned by the client, which contain responses to the dynamic visual challenge code and the acoustic interaction command. From each frame of the synchronous video stream, the system extracts in real time the sequence of human features, the sequence of scene objects, the sequence of environmental text, and the sequence of visual states of the dynamic visual challenge code. From the audio stream, the system extracts in real time the sequence of acoustic features and identifies the completed content of the acoustic interaction command. For multiple consecutive frames of a synchronized video stream, various elements extracted from each frame are used as nodes to construct a multi-element spatiotemporal relationship graph. The nodes include at least personnel nodes, physical object nodes, challenge code nodes, and text nodes. The edges between nodes encode at least spatial adjacency relationships, motion coordination relationships, and semantic association relationships, generating a relationship graph sequence. The relation graph sequence is input into a pre-trained spatiotemporal graph neural network. The spatiotemporal graph neural network aggregates edge-based node relation information within the same frame through graph convolutional layers, and learns the evolution law of node and edge states in the relation graph sequence through temporal modeling layers, and outputs a first feature vector representing the global spatiotemporal consistency of the entire interaction process. Based on the personnel feature sequence and the acoustic feature sequence, the naturalness of personnel movement and vocal-lip synchronization are verified to generate a first verification score; Based on the correspondence between the visual state sequence of the dynamic visual challenge code and the response actions in the personnel feature sequence, the real-time performance and accuracy of the interactive response are verified, and a second verification score is generated. Based on the environmental text sequence and the preset authentication statement information, the logical consistency of the scene text is verified, and a third verification score is generated; based on the first feature vector extracted by the spatiotemporal graph neural network, the physical rationality of the motion and spatial relationship between multiple elements is verified, and a fourth verification score is generated. The first verification score, the second verification score, the third verification score, and the fourth verification score are input into the adaptive decision model. The model is weighted, fused, and judged according to the preset risk strategy and threshold. If the comprehensive judgment result exceeds the preset threshold, a video authentication pass signal is output; otherwise, a signal indicating authentication failure or manual review is output.

2. The enterprise video authentication consistency verification method based on multi-dimensional data fusion according to claim 1, characterized in that, The receiving client returns a synchronized video stream and an audio stream that respond to the dynamic visual challenge code and the acoustic interaction command; from each frame of the synchronized video stream, the personnel feature sequence, scene object feature sequence, environmental text sequence, and visual state sequence of the dynamic visual challenge code are extracted in real time. From the audio stream, acoustic feature sequences are extracted in real time, and the completion content of the acoustic interaction commands is identified, including: Receive the synchronous video stream and audio stream returned by the client, decode the synchronous video stream to obtain a sequence of video frames arranged in chronological order, and decode the audio stream to obtain the audio signal; For each frame in the video frame sequence, a face detection and alignment model is used to locate the face region in the current frame image; a facial landmark detection model is used to extract multiple sets of facial landmark coordinates from the aligned face region; a human pose estimation model is used to extract multiple sets of human body key point coordinates in the current frame image; the sets of facial landmark coordinates and human body key point coordinates are merged to form the personnel features of the corresponding frame. Using a pre-trained object detection model, at least one preset authentication-related object is detected and located in the current frame image. The authentication-related object includes the specified authentication object and environmental markers. For each detected object, its bounding box coordinates and category label are obtained, and a visual feature vector is extracted from the bounding box region of the object using a convolutional neural network. The bounding box coordinates, category label and visual feature vector are combined to form the scene object features of the corresponding frame. Within a preset display area of ​​the current frame image, the dynamic visual challenge code is detected; the visual attributes of the dynamic visual challenge code in the current frame are extracted, and the visual attributes include at least its overall color histogram, outline shape descriptor and center pixel coordinates within the frame; the color histogram, shape descriptor and center pixel coordinates are used to construct the visual state information of the dynamic visual challenge code in the corresponding frame. In the current frame image, optical character recognition is performed on the preset text interest area to obtain the recognized text content and its position coordinates in the image; the text content and position coordinates are used to construct the environmental text features of the corresponding frame. The audio signal is divided into frames and windowed to extract the acoustic features of each frame. The acoustic features include Mel frequency cepstral coefficients and speaker embedding vectors. The extracted acoustic features are arranged in chronological order to form the acoustic feature sequence. The audio signal is input into the speech recognition model to obtain the transcribed text; based on the semantic template of the acoustic interaction command, key information is extracted from the transcribed text as the completion content of the acoustic interaction command. The personnel features, scene object features, visual state information of the dynamic visual challenge code, and environmental text features extracted from each frame of the video frame sequence are arranged in chronological order according to the corresponding frames to obtain the personnel feature sequence, the scene object feature sequence, the visual state sequence of the dynamic visual challenge code, and the environmental text sequence.

3. The enterprise video authentication consistency verification method based on multi-dimensional data fusion according to claim 2, characterized in that, For each frame in the video frame sequence, a face detection and alignment model is used to locate the face region in the current frame image; a facial landmark detection model is used to extract multiple sets of facial landmark coordinates from the aligned face region. Using a human pose estimation model, extract the set of coordinates of multiple key points of the human body in the current frame image; The set of facial landmark coordinates is merged with the set of human body key point coordinates to form the personnel features of the corresponding frame, including: For the current frame image in the video frame sequence, a face detection model is used to detect whether there is a face region in the image; if multiple faces are detected, the face with the largest area or located in the center of the image is selected as the target authentication face, and the initial bounding box of the target authentication face is obtained. Based on the initial bounding box, extract the face region image; use the face alignment model to perform rotation, scaling and translation transformations on the face region image to generate a pose-normalized standard face image; A standard face image is input into a pre-trained facial landmark detection model to obtain a set of two-dimensional coordinate points that represent the contours of key facial features, thus forming a set of facial landmark coordinates. Within the global scope of the current frame image, a human pose estimation model is used to detect and locate the two-dimensional coordinates of the main joints of the target human body, forming a set of human body key point coordinates; the human body key points include at least the corresponding points of the neck and shoulders; The set of facial landmark coordinates extracted from the current frame is merged with the set of human body key point coordinates to form the preliminary personnel feature data of the current frame. In the processing of continuous video frame sequences, the preliminary personnel feature data of the current frame is verified based on the temporal continuity of the facial landmark coordinate set. If the motion pattern of the facial landmark coordinate set changes abruptly between adjacent frames, it is determined that the corresponding frame feature extraction is abnormal, and interpolation or frame reference mechanism is used for correction to ensure that the final personnel features are smooth and continuous in the time dimension.

4. The enterprise video authentication consistency verification method based on multi-dimensional data fusion according to claim 3, characterized in that, For the continuous multiple frames of the synchronized video stream, various elements extracted from each frame are used as nodes to construct a multi-element spatiotemporal relationship graph. The nodes include at least personnel nodes, object nodes, challenge code nodes, and text nodes. The edges between nodes encode at least spatial adjacency relationships, motion coordination relationships, and semantic association relationships, generating a relationship graph sequence, including: For the t-th frame of the synchronized video stream, where t is a natural number, based on the extracted feature sequences and the personnel features of the t-th frame, the geometric center of the set of facial landmark coordinates or the statistical features of the predefined set of key points are encoded into the initial feature vector of the personnel node. For each object detected in the scene object features of frame t, an initial feature vector for the corresponding physical node is generated based on its visual feature vector and its category label. Based on the visual state information of the dynamic visual challenge code in frame t, an initial feature vector of the challenge code node is generated. For each text segment identified in the environmental text features of frame t, its text content is converted into a semantic vector through a word embedding model, and combined with its position coordinates, an initial feature vector for the corresponding text node is generated. In the graph constructed in frame t, three types of edges are established and encoded between nodes: calculate the normalized distance or bounding box overlap between any two nodes in the two-dimensional space of the image; if the distance is less than the spatial threshold or there is overlap, establish a spatial adjacency edge between the two nodes and encode the corresponding spatial relationship as the attribute of the corresponding edge. For movable nodes, calculate their motion trajectory in multiple consecutive frames. By analyzing the correlation between the motion trajectory directions of different nodes, if the correlation exceeds the motion coordination threshold, establish motion coordination edges between the corresponding nodes. The weight or attribute of the motion coordination edge is encoded by the strength value of the motion direction correlation. Based on the preset knowledge base or rules, establish semantic association edges between logically related nodes. Repeat the node instantiation and relation edge construction steps to construct a corresponding multi-factor spatiotemporal relation graph for each consecutive frame in the video frame sequence, resulting in the relation graph sequence arranged in chronological order.

5. The enterprise video authentication consistency verification method based on multi-dimensional data fusion according to claim 4, characterized in that, The relationship graph sequence is input into a pre-trained spatiotemporal graph neural network. The spatiotemporal graph neural network aggregates edge-based node relationship information within the same frame through graph convolutional layers, and learns the evolution law of node and edge states in the relationship graph sequence through temporal modeling layers. It outputs a first feature vector representing the global spatiotemporal consistency of the entire interaction process, including: Based on the spatiotemporal graph neural network, the input relationship graph sequence is updated frame by frame in chronological order and the temporal information is fused: For each frame of the graph, the network performs multi-layer graph convolution operation so that each node aggregates feature information from its neighboring nodes according to the spatial adjacency relationship edge, motion cooperation relationship edge and semantic association relationship edge connected to it, and generates updated node features containing local association context information. Based on the single-frame node feature update, for each node, its features in different frames of the entire relation graph sequence are arranged into a sequence along the time dimension; a temporal convolutional layer is used to process the corresponding sequence to capture the dynamic pattern and long-term dependency relationship of the target node features over time, and output the temporal enhanced features of the target node. After obtaining the temporal enhancement features of all nodes, a global pooling operation is performed on the temporal enhancement features of all nodes in the same frame to generate a frame-level global feature vector that represents the overall scene relationship of the corresponding frame. The frame-level global feature vectors of each frame are arranged in chronological order, and then global pooling is performed again along the time dimension to gather the time clues of the entire interaction process and generate a preliminary sequence-level global feature vector. Based on the temporal enhancement features of nodes, the importance weights of different node types for the true / false discrimination task are calculated. According to the importance weights, the temporal enhancement features of nodes are weighted and fused to generate a set of node feature representations that have been calibrated for importance. The weighted and fused node feature representations are concatenated with the sequence-level global feature vectors, and then nonlinear transformation and dimensionality reduction are performed through one or more fully connected layers to compress and output a first feature vector of fixed dimension.

6. The enterprise video authentication consistency verification method based on multi-dimensional data fusion according to claim 5, characterized in that, The spatiotemporal graph neural network aggregates information from all nodes across all frames and outputs a first feature vector of fixed dimension representing the overall pattern of multi-element spatiotemporal relationships throughout the entire interaction process, including: The node feature representation generated after node type attention weighted fusion is concatenated with the sequence-level global feature vector to obtain a concatenated feature vector that integrates fine-grained node attention information and global spatiotemporal context information. The spliced ​​feature vector is input into a feature transformation module consisting of at least one fully connected layer. Based on the feature transformation module, the spliced ​​feature vector is nonlinearly projected through the first fully connected layer to expand or transform its feature representation space. A non-linear activation function is applied to the output of the first fully connected layer, and the activated features are then normalized to stabilize the training process and accelerate convergence. The normalized high-dimensional features are mapped to a significantly lower-dimensional feature space through the second fully connected layer, thereby achieving feature compression and refinement. The feature vector that has undergone dimensionality reduction is subjected to a final linear projection, which compresses it to a preset fixed dimension, and the output is used as the first feature vector.

7. The enterprise video authentication consistency verification method based on multi-dimensional data fusion according to claim 5, characterized in that, Before the step of inputting the relationship graph sequence into a pre-trained spatiotemporal graph neural network, where the spatiotemporal graph neural network aggregates edge-based node relationship information within the same frame through graph convolutional layers, learns the evolution rules of node and edge states in the relationship graph sequence through temporal modeling layers, and outputs a first feature vector representing the global spatiotemporal consistency of the entire interaction process, the process further includes: A large number of real video authentication interaction samples were collected as positive samples, and corresponding fake video authentication interaction samples were generated as negative samples by injecting virtual camera data, using pre-recorded videos, or tampering with scene elements. For each training sample, its multimodal feature sequence is extracted, and its corresponding spatiotemporal relationship graph sequence labeled with true and false labels is constructed to form a training dataset. Construct a spatiotemporal graph neural network, which at least includes the following sequentially connected components: a graph convolutional module for spatial relation aggregation, a temporal convolutional module for temporal evolution modeling, a node type attention weighted fusion module, and a feature compression and output module; initialize the parameters of each module of the network using a preset initialization strategy; The training dataset is input into the initialized spatiotemporal graph neural network for iterative training. In each iteration, the spatiotemporal graph neural network outputs the predicted first feature vector based on the input relationship graph sequence. A joint loss function is constructed to optimize the spatiotemporal graph neural network. The joint loss function includes at least: the difference loss between the true and false classification results calculated based on the first feature vector and the true and false labels of the samples. The model is encouraged to learn the high temporal consistency of the motion coordination relationship and spatial relationship among multiple elements for positive samples, while amplifying the relationship consistency loss of inconsistencies in the relationship for negative samples. The performance of the trained spatiotemporal graph neural network is evaluated using an independent validation dataset. Once the performance meets the preset accuracy and generalization metrics, the model training is completed. The trained spatiotemporal graph neural network is embedded as a fixed module in the video authentication and anti-counterfeiting system for extracting the first feature vector and performing consistency analysis on the real-time acquired interactive video stream.

8. A device for verifying the consistency of enterprise video authentication based on multi-dimensional data fusion, characterized in that, The device includes: The interactive verification module is used to send a dynamic visual challenge code and a random acoustic interaction command to the client. The visual attributes of the dynamic visual challenge code in the display area change over time according to a predetermined pattern. The acoustic interaction command includes content that requires the user to operate a specified authentication object and associate it with the dynamic visual challenge code. The feature extraction module is used to receive a synchronous video stream and an audio stream returned by the client, which contain responses to the dynamic visual challenge code and the acoustic interaction command; extract personnel feature sequences, scene object feature sequences, environmental text sequences, and the visual state sequence of the dynamic visual challenge code from each frame of the synchronous video stream in real time; and extract acoustic feature sequences from the audio stream in real time and identify the completed content of the acoustic interaction command. The spatiotemporal correlation module is used to construct a multi-element spatiotemporal relationship graph for multiple consecutive frames of a synchronized video stream, using various elements extracted from each frame as nodes. The nodes include at least personnel nodes, physical object nodes, challenge code nodes, and text nodes. The edges between nodes encode at least spatial adjacency relationships, motion coordination relationships, and semantic relationships, generating a sequence of relationship graphs. The feature evolution module is used to input the relationship graph sequence into a pre-trained spatiotemporal graph neural network. The spatiotemporal graph neural network aggregates the node relationship information based on edges within the same frame through graph convolutional layers, and learns the evolution law of node and edge states in the relationship graph sequence through temporal modeling layers, and outputs a first feature vector that represents the global spatiotemporal consistency of the entire interaction process. The feature verification module is used to verify the naturalness of personnel movement and lip-verb synchronization based on the personnel feature sequence and the acoustic feature sequence, and generate a first verification score; to verify the real-time performance and accuracy of interactive response based on the correspondence between the visual state sequence of the dynamic visual challenge code and the response actions in the personnel feature sequence, and generate a second verification score; to verify the logical consistency of scene text based on the environmental text sequence and preset authentication declaration information, and generate a third verification score; and to verify the physical rationality of the movement and spatial relationship between multiple elements based on the first feature vector extracted by the spatiotemporal graph neural network, and generate a fourth verification score. The video authentication module is used to input the first verification score, the second verification score, the third verification score and the fourth verification score into the adaptive decision model, and perform weighted fusion and judgment according to the preset risk strategy and threshold. If the comprehensive judgment result exceeds the preset threshold, the video authentication pass signal is output; otherwise, the authentication failure signal or the need for manual review signal is output.

9. A control device, characterized in that, The device includes: A memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and executed as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer program is stored that can be loaded by a processor and executed as described in any one of claims 1 to 7.

Images