Video authentication method and device, electronic equipment and readable storage medium

Abstract

The application discloses a video authentication method and device, electronic equipment and a readable storage medium. The method comprises the following steps: obtaining multi-dimensional features of a target person in a video; wherein the multi-dimensional features comprise at least two of the following: appearance features, sound features, photoplethysmography (PPG) physiological features, and forgery traceability features; fusing the multi-dimensional features to obtain target fusion features; and performing authentication processing on the video based on the target fusion features to determine an authentication result of the video.

Description

Technical Field

[0001] This application belongs to the field of video detection technology, specifically relating to a video authentication method, device, electronic device, and readable storage medium. Background Technology

[0002] In existing video authentication technologies, single-modal authentication is mainly achieved by extracting spatiotemporal features from the video and combining them with frame difference analysis, or by collecting visual and audio modal data from the video and then performing authentication based on features that integrate both visual and audio modalities. However, these methods rely on a single feature dimension for video authentication, primarily depending on static features such as spatiotemporal, visual, or audio characteristics. This makes them unable to handle a wide variety of forgery techniques or scenarios, resulting in low accuracy in video authentication. Summary of the Invention

[0003] The purpose of this application is to provide a video authentication method, device, electronic device, and readable storage medium, which can solve the problem that related video authentication technologies have only one feature dimension for video authentication, cannot cope with a variety of different forgery technologies or scenarios, and have a low accuracy rate in video authentication.

[0004] In a first aspect, embodiments of this application provide a video authentication method, which includes: acquiring multidimensional features of a target person in a video; wherein the multidimensional features include at least two of the following: appearance features, voice features, photoplethysmography (PPG) physiological features, and forgery tracing features; fusing the multidimensional features to obtain target fused features; and performing authentication processing on the video based on the target fused features to determine the authentication result of the video.

[0005] Secondly, embodiments of this application provide a video authentication device, which includes: an acquisition module for acquiring multidimensional features of a target person in a video; wherein the multidimensional features include at least two of the following: appearance features, voice features, photoplethysmography (PPG) physiological features, and forgery tracing features; a fusion module for fusing the multidimensional features to obtain target fused features; and an authentication module for performing authentication processing on the video based on the target fused features to determine the authentication result of the video.

[0006] Thirdly, embodiments of this application provide an electronic device including a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the method described in the first aspect.

[0007] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0008] Fifthly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the steps of the method described in the first aspect.

[0009] In a sixth aspect, embodiments of this application provide a computer program product, the computer program product including a computer program stored on a non-transitory computer-readable storage medium, the computer program including a program or instructions, which, when executed, implement the steps of the method described in the first aspect.

[0010] In this embodiment, multi-dimensional features are acquired from the target person in the video, including appearance features, voice features, photoplethysmography (PPG) physiological features, and forgery tracing features. These multi-dimensional features are then fused to obtain a target fusion feature. This target fusion feature integrates multi-dimensional features, enabling multi-dimensional analysis and evaluation of the video to cover various forgery techniques or scenarios. Based on this target fusion feature, the video is subjected to forgery detection processing to determine the forgery detection result, thereby comprehensively identifying the authenticity of the video and achieving more accurate video forgery detection. Attached Figure Description

[0011] Figure 1 This is a flowchart illustrating a video authentication method provided in an embodiment of this application; Figure 2 This is a flowchart illustrating another video authentication method provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of a video authentication device provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0012] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0013] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0014] The video authentication method, apparatus, electronic device, and readable storage medium provided in this application will be described in detail below with reference to the accompanying drawings and through specific embodiments and application scenarios.

[0015] Figure 1 The diagram illustrates a flowchart of a video authentication method provided in an embodiment of this application. This method can be executed by an electronic device. See also... Figure 1 The method may include the following steps.

[0016] Step 102: Obtain multidimensional features of the target person in the video; wherein, the multidimensional features include at least two of the following: appearance features, voice features, photoplethysmography (PPG) physiological features, and forgery tracing features.

[0017] The appearance features can include RGB facial image frames with a resolution of 1080P / 720P and three-dimensional facial pose parameters (pitch angle, yaw angle, roll angle) with an accuracy of ±0.5°. The audio features can include 20-dimensional Mel-Frequency Cepstral Coefficients (MFCC) audio features with a sampling rate of 16kHz. The photoplethysmography (PPG) physiological features can include the acquisition of dynamic subcutaneous blood flow signals by a photoplethysmography (PPG) sensor at a sampling rate of 20Hz to capture the temporal pattern of blood pulsation, and can also include the acquisition of heart rate variability (HRV) signals by a heart rate sensor. The 20Hz sampling rate is based on the main frequency range of human blood pulsation signals (0.8-2Hz), and 20Hz satisfies the Nyquist sampling theorem (sampling rate ≥ 2 times the highest signal frequency) to avoid signal aliasing. Forgery traceability features may include frame block effect parameters generated from the video, H.264 / H.265 compression coding anomalies, and metadata verification information, such as verification results after timestamp continuity checks and frame sequence cyclic redundancy check (CRC) verification results.

[0018] The video acquisition can employ a combined acquisition mode of "3-second keyframes + time-series segments". The duration of each time-series segment is set to 10 seconds, with a frame interval consistent with the keyframes (500ms), meaning each 10-second time-series segment contains 20 frames. Furthermore, the data is encrypted during the acquisition process, using algorithms such as the national standard SM2, to ensure the data's traceability and tamper-proof nature, providing high-quality data support for subsequent video recognition.

[0019] Step 104: Fuse the multidimensional features to obtain the target fused features.

[0020] In some embodiments, the multidimensional features can be preprocessed and enhanced before fusion. This mainly includes static processing and dynamic processing, as well as noise interference removal and enhancement of key anti-counterfeiting features.

[0021] In static processing, for appearance features, such as RGB facial images, the RetinaFace algorithm is used to extract 68 facial feature points. Simultaneously, an Enhanced Super-Resolution Generative Adversarial Network (ESRGAN) super-resolution algorithm with a 2x magnification is used to optimize facial detail texture and improve feature recognition. For PPG physiological features, such as PPG blood flow signals, wavelet thresholding based on the Db4 wavelet basis (threshold set to 3) is used for denoising. To eliminate environmental interference, the signal amplitude is then uniformly normalized within the 0-1 range, thereby enhancing the stability of physiological characteristics. The calculation employs a noise standard deviation estimation method: the standard deviation of pixel grayscale values ​​is calculated for the first 100 sampling points of the PPG blood flow signal (segments without significant physiological signal interference in the initial acquisition stage), using the following formula: , Let i be the signal value at the i-th sampling point. The average value of the signals from the first 100 sampling points is calculated, where N=100. Then multiply by 3 to get the final noise reduction threshold.

[0022] In dynamic processing, spatiotemporal consistency analysis is performed on the video, and a spatiotemporal difference index is calculated. The spatiotemporal consistency of the video is quantified by the spatiotemporal difference index, and a spatiotemporal difference heatmap of the video is determined based on the spatiotemporal difference index to mark the abnormal areas of the video.

[0023] In some embodiments, the temporal and spatial difference index (TDI) can be determined using the following formula: The larger the TDI, the higher the probability of forgery in that spatiotemporal region. There is no fixed value range; a threshold needs to be set based on the business scenario. For example, a TDI > 50 is considered a suspicious region.

[0024] This represents the time step index, corresponding to the temporal order of the video frames, and its value range is... An integer is used to locate the position of multidimensional features in the time dimension.

[0025] This represents the total time step, the total number of video frames involved in the calculation. It can be set to 30, corresponding to a 3-second keyframe acquisition cycle (1 frame is extracted every 500ms) to ensure complete timing coverage.

[0026] This represents the spatial block index, corresponding to the pixel segmentation block of a single frame image, with a value range of... An integer is used to locate the position of multidimensional features in the spatial dimension.

[0027] This represents the total number of spatial blocks, the total number of blocks after a single frame image is divided into blocks of a fixed size. S can be set to 224×224, corresponding to a 224×224 pixel Region of Interest (ROI) for the face, and divided into 1×1 pixel granularity to ensure the accuracy of spatial feature localization.

[0028] Indicates the first Time step, number The feature vector of the spatial block contains multi-dimensional features such as RGB texture and dynamic optical flow. Its dimension depends on the type of feature after preprocessing. For example, the feature dimension after static processing is 256, and the feature dimension after dynamic processing is 128.

[0029] The marker representing authenticity or forgery is used to distinguish the true attributes of multidimensional features; it is a binary variable, where... Representing true characteristics, This indicates forged features, ensuring attribute matching during both forward and reverse calculations.

[0030] This represents the feature vector calculated in reverse order, i.e., obtained by traversing the spatiotemporal sequence from back to front. Reconstructed values ​​and With consistent dimensions, its core function is to verify temporal continuity and capture the abnormal characteristics of "temporal breaks" in fake videos.

[0031] This represents the feature vector calculated in the forward direction, i.e., obtained by traversing the spatiotemporal sequence from front to back. Original value, and Dimensional consistency serves as a benchmark for temporal coherence and is used to compare with reverse-reconstructed values ​​to quantify differences.

[0032] L2 norm (Euclidean norm) is used to calculate the Euclidean distance between two features, quantifying the difference between features. It takes the value of a non-negative real number. The larger the value, the more significant the difference between the two features, and it directly reflects the degree of consistency between spatiotemporal features.

[0033] The threshold conditions for TDI can be dynamically adjusted under different application scenarios: In one embodiment, for a financial account opening scenario (high security requirements), TDI > 45 is set as a suspicious area, based on the fact that the spatiotemporal inconsistencies between forged video frames are usually more concealed in this scenario, requiring a lower threshold to increase sensitivity. In another embodiment, for an application registration scenario (normal requirements), TDI > 55 is set as a suspicious area, balancing the false detection rate and detection efficiency. The above values ​​were determined through testing and verification with 10,000 real or forged samples from different application scenarios.

[0034] In this embodiment of the application, multidimensional features are fused, and the feature retention rate is... satisfy This is used to ensure that key forgery traces are not lost after fusion, thus guaranteeing the effectiveness of the target fused features in forgery detection. Among them, The dimension representing the target fusion feature is fixed at 2048. This dimension is obtained through dimensionality reduction optimization using Principal Component Analysis (PCA), which reduces computational complexity while retaining more than 98% of the information, thus balancing accuracy and efficiency. This represents the total dimension of all input features before fusion, which is the sum of the dimensions of each feature, namely, the dimension of appearance and voice features (512) + the dimension of PPG physiological features (256) + the dimension of forgery tracing features (1024) + the dimension of spatiotemporal difference heatmap features (512). It comprehensively covers the dimensional information of multi-dimensional features.

[0035] Step 106: Perform anti-spoofing processing on the video based on the target fusion features to determine the anti-spoofing result of the video.

[0036] The target fusion features can be processed through two paths. First, the target fusion features can be used by a classifier to determine whether they are real or fake. Second, the source of the fake can be identified based on the target fusion features, thus enabling traceability of video authentication.

[0037] In this embodiment, multi-dimensional features are acquired from the target person in the video, including appearance features, voice features, photoplethysmography (PPG) physiological features, and forgery tracing features. These multi-dimensional features are then fused to obtain a target fusion feature. This target fusion feature integrates multi-dimensional features, enabling multi-dimensional analysis and evaluation of the video to cover various forgery techniques or scenarios. Based on this target fusion feature, the video is subjected to forgery detection processing to determine the forgery detection result, thereby comprehensively identifying the authenticity of the video and achieving more accurate video forgery detection.

[0038] In one implementation, step 104 above, which fuses the multidimensional features to obtain target fused features, may include the following steps.

[0039] Step 1041: The appearance features and the sound features are fused using an attention network to determine the primary fused features.

[0040] In some embodiments, the attention network includes: one Sigmoid activation function layer (input is appearance features and voice features, dimension 512+64=576), one fully connected layer (256 neurons, weights initialized using a normal distribution), and one fusion layer. The Sigmoid activation function layer outputs attention weights in the range of 0-1. The fully connected layer element-wise multiplies the original features with the attention weights to enhance high-discrimination features. The fusion layer swaps appearance features and voice features in a 3:1 ratio and then fuses them to obtain a primary fused feature.

[0041] Step 1042: Perform spatiotemporal consistency analysis on the video to determine the spatiotemporal difference heatmap of the video.

[0042] Among them, tg performs spatiotemporal consistency analysis on the video, calculates the spatiotemporal difference index, and determines the spatiotemporal difference heatmap of the video based on the spatiotemporal difference index.

[0043] Step 1043: The primary fusion feature is fused with the photoplethysmography (PPG) physiological feature, the forgery tracing feature, and the spatiotemporal difference heatmap to determine the target fusion feature.

[0044] Specifically, by analyzing the time dependence of the primary fusion features with the photoplethysmography (PPG) physiological features, the forgery tracing features, and the spatiotemporal difference heatmap, the target fusion features are determined by fusing them.

[0045] In this embodiment, multidimensional features are fused in a hierarchical manner. First, appearance features and sound features are fused through an attention network. Then, the fused primary features are fused with photoplethysmography (PPG) physiological features, forgery tracing features, and spatiotemporal difference heatmaps to determine the target fused features. By fusing static related features to dynamic related features, the fusion of multidimensional features is made more reasonable, avoiding the loss of core features used for video forgery detection.

[0046] In one implementation, step 106 above, which performs anti-counterfeiting processing on the video based on the target fusion features and determines the anti-counterfeiting result of the video, may include: inputting the target fusion features into a lightweight feature classifier for classification processing to obtain the anti-counterfeiting result of the video, wherein the anti-counterfeiting result includes: real or fake.

[0047] The lightweight feature classifier compares the confidence level of predicting the video as fake with the target fake video detection rate. If the confidence level is greater than or equal to the target fake video detection rate, the video is determined to be fake. The target fake video detection rate is the proportion of fake video samples in the video samples within a first preset time period.

[0048] The lightweight feature classifier comprises: an input layer (2048-dimensional feature vectors), a batch normalization layer, a fully connected layer 1 (1024 neurons, ReLU activation function), a dropout layer (dropout probability 0.3), a fully connected layer 2 (256 neurons, ReLU activation function), and an output layer (2 neurons, corresponding to real or fake data, with Softmax activation function). The lightweight feature classifier has 185M parameters, meeting the lightweight requirement of less than 200M.

[0049] In this embodiment, a lightweight feature classifier is used to process the target fusion features. The lightweight feature classifier will give a prediction result for the target fusion features, namely the confidence that the video is real and the confidence that the video is fake. Then, the confidence that the video is fake is compared with the detection rate of the target fake video. By comparing, it is determined whether the prediction of the target fusion features by the lightweight feature classifier is accurate, and the accuracy of the lightweight feature classifier is confirmed, thereby obtaining an accurate fake detection result.

[0050] In some embodiments, the target fake video detection rate is the proportion of fake video samples in the video samples within a first preset time period, which can be determined by the following formula: .

[0051] in, The target is to fake the detection rate of the video, with a value range of [value missing]. This refers to the threshold that the confidence level needs to meet in the current application scenario. This value can be dynamically adjusted according to the risk of the application scenario.

[0052] Represents a timestamp, corresponding to The update time is set to update at 60-second intervals (i.e., ...). ),and Synchronize statistical periods to avoid Judgment bias caused by the asynchronous updating and risk statistics.

[0053] This represents the baseline threshold, i.e., the initial threshold when there is no risk of falsification, and is a fixed value. =0.90, determined by the historical average forgery rate (usually 5%-8%), serves as a benchmark for dynamic adjustment of the threshold.

[0054] express The real-time fake video detection rate is the percentage of fake video samples counted every 60 seconds within the past hour, with a value range of [value missing]. (Unit: %), calculated statistically through a sliding window (window size = 60 seconds), reflecting the current level of counterfeiting risk in real time.

[0055] This represents the expected counterfeit detection rate, which is the stable detection rate that the business expects to maintain. The fixed value is 95%, which meets the core requirement of "precise interception" in the real-name registration business and is the target orientation for threshold adjustment.

[0056] This represents the threshold adjustment coefficient, used to control the sensitivity of the threshold to fluctuations in the detection rate. A fixed value of 0.002 can prevent small fluctuations in the detection rate (fluctuations) from affecting the threshold. (At times) this leads to frequent threshold adjustments, ensuring the stability of threshold determination.

[0057] In some embodiments, when time t =92% (lower than) =95%), then Th(t)=0.90+0.02×(95%-92%)=0.96, at this time the judgment rule is updated to judge it as forgery when the confidence level Cr≥0.96; in some other embodiments, when =96% (higher than) ),but =0.90+0.02×(95%-96)=0.88, ensuring that the threshold is appropriately lowered to reduce false judgments when the risk is low.

[0058] In one implementation, the above method may further include the following steps.

[0059] Step 108: Obtain the fake detection accuracy of the lightweight feature classifier.

[0060] The accuracy of the fake detection measure the probability that the lightweight feature classifier correctly predicts whether a video is real or fake over a period of time, thus illustrating the performance of the lightweight feature classifier.

[0061] Step 110: If the reduction in the counterfeit detection accuracy of the lightweight feature classifier during the second preset time period is greater than a preset threshold, update the parameters of the lightweight feature classifier and / or the parameters of the attention network; wherein the attention network is used to fuse the multidimensional features.

[0062] Wherein, the reduction in the counterfeit detection accuracy of the lightweight feature classifier within a second preset time period is greater than a preset threshold, which can be that the counterfeit detection accuracy of the lightweight feature classifier decreases by more than 5% or 3% within seven days.

[0063] In this embodiment, after the lightweight feature classifier classifies the target fused features and determines the video's authentication result, the performance evaluation of the lightweight feature classifier, i.e., its authentication accuracy, is analyzed. The changes in authentication accuracy over a second preset time period are analyzed. When the authentication accuracy decreases and the decrease exceeds a preset threshold, it indicates that the current lightweight feature classifier's performance is suboptimal, thus triggering an update to the classifier's parameters. Furthermore, since the input to the lightweight feature classifier is the target fused feature obtained by fusing multi-dimensional features using an attention network, when the authentication accuracy indicates that the current lightweight feature classifier's performance is suboptimal, the parameters of the attention network can also be updated. This avoids a decrease in authentication accuracy due to multiple factors related to the attention network and the lightweight feature classifier, comprehensively addressing situations where authentication accuracy declines and improving overall accuracy.

[0064] In one implementation, the above method may further include step 112: matching the target fusion feature with a preset forgery tool encoding feature library to determine the target forgery technology type corresponding to the video.

[0065] The pre-defined feature library for forgery tools can include the following types of forgery tools: ① FaceSwap face swapping, ② Generative Adversarial Network (GAN) generation (including StyleGAN and ProGAN), ③ DeepFake v2 / v3, ④ FaceShifter, ⑤ NeuralTextures, ⑥ 3D model reconstruction forgery, ⑦ Video splicing forgery, and ⑧ AI-generated content (AIGC) text-generated faces. The weight coefficients for GAN generation / DeepFake v3 range from 0.4 to 0.5, the weight coefficients for FaceSwap / FaceShifter range from 0.3 to 0.4, and the weight coefficients for other types range from 0.2 to 0.3.

[0066] In some embodiments, the forgery tools in the preset forgery tool coding feature library can be dynamically updated as technology develops, so as to ensure the timeliness and reliability of the preset forgery tool coding feature library.

[0067] In this embodiment, the authentication result of the video can be determined by classifying the target fusion features. Furthermore, the target fusion features can be matched and analyzed with a preset forgery tool encoding feature library to determine the target forgery technology type corresponding to the video, thereby enabling the tracing of the forgery technology used in the forged video.

[0068] In some embodiments, when the target fused features are matched with a preset forgery tool encoding feature library, if no forgery technique type corresponding to the forged video is matched, the preset forgery tool encoding feature library can be updated to determine whether the failure to match is due to the preset forgery tool encoding feature library not covering the latest forgery tools. Alternatively, the parameters of the aforementioned lightweight feature classifier and / or attention network can be updated to determine whether the failure to match is due to an error in forgery detection.

[0069] In one implementation, the above method may further include the following steps.

[0070] Step 114: In response to the target person's operation on the preset instruction, perform a spatiotemporal consistency analysis on the operation to determine the continuity of the operation; wherein, the preset instruction is determined based on the risk coefficient of the application scenario of the video.

[0071] The preset instructions may include nodding three times in a row, slowly opening your mouth for five seconds, shaking your head left and right twice, blinking five times, raising your head once, lowering your head once, raising the corners of your mouth twice, and closing your eyes for three seconds.

[0072] For example, in applications with high security requirements, feedback on the target's actions in response to preset commands can be added. Two types of commands can be selected as preset commands to prevent fixed commands from being cracked. Spatiotemporal consistency analysis of the operations can be performed to determine the continuity of the operations. This can be achieved by determining that a nodding action requires a pitch angle change range of ≥30° and an interval of 0.5-1 second between nods, and that a mouth-opening action requires a distance change of ≥20 pixels between mouth feature points (points 62-68 out of 68 feature points extracted by RetinaFace).

[0073] Step 116: Based on the continuity of the operation and the physiological characteristics of the PPG, determine the authentication result of the target person, wherein the authentication result includes: real or fake.

[0074] In this application embodiment, for application scenarios with high security requirements, it can also respond to the target person's operation on preset instructions, and then perform spatiotemporal consistency analysis on the operation to determine the continuity of the operation. Based on the continuity of the operation and the PPG physiological characteristics, the authentication result of the target person can be determined, thereby realizing differentiated processing for different application scenarios and improving the accuracy of video authentication.

[0075] In some embodiments, for application scenarios with ordinary security requirements, a lightweight feature classifier can be used to classify the target fused features, and the confidence level of the predicted video being fake can be compared with the dynamically determined target fake video detection rate to determine whether the video is real or fake.

[0076] In some embodiments, after determining the authentication result, confidence levels for different application scenarios can be calculated. These confidence levels measure the degree of matching between the authentication result and the security requirements of different application scenarios, thereby further confirming the authentication result. The confidence levels for different application scenarios can be determined using the following formula: .

[0077] in, The range of values ​​is (Precision is 0.01). In the embodiments of this application, it is set that... The authenticity verification result is confirmed to be reliable.

[0078] This represents the confidence level of the aforementioned authentication result, with a value range of [value missing]. (Precision is 0.01), used to reflect the authenticity credibility at the target fusion feature level, without considering the security differences of application scenarios.

[0079] This represents the scenario risk coefficient, set based on the historical video forgery rate for each application scenario. The application reflects the different security requirements of different scenarios, with high-security scenarios (such as financial account opening and government authentication) having a value range of [value missing]. In typical scenarios (such as app registration), the value range is: A higher risk coefficient indicates that the application scenario has more stringent security requirements.

[0080] 0.1 represents the risk coefficient adjustment factor, which is used to balance the application scenario risk and confidence sensitivity to avoid over-calibration leading to misjudgment. This value is a fixed empirical value determined through more than 10,000 application scenario tests, which can ensure that the false detection rate after calibration is less than 3%, taking into account both security and user experience.

[0081] Figure 2 This diagram illustrates a flowchart of another video authentication method provided in an embodiment of this application. This method can be executed by an electronic device, which may include cloud devices and edge devices. Video authentication is achieved through collaborative analysis of the video by the cloud device and the edge device. See also... Figure 2 The method may include the following steps.

[0082] Step 201: Collect multidimensional features of the video.

[0083] The multidimensional features include at least two of the following: appearance features, voice features, photoplethysmography (PPG) physiological features, and forgery tracing features.

[0084] The appearance features may include RGB facial image frames with a resolution of 1080P / 720P and three-dimensional facial pose parameters (pitch angle, yaw angle, roll angle) with an accuracy of ±0.5°.

[0085] The sound features may include 20-dimensional Mel-Frequency Cepstral Coefficients (MFCC) audio features with a 16kHz sampling rate.

[0086] Among them, the physiological characteristics of photoplethysmography (PPG) can include the use of a photoplethysmography (PPG) sensor to collect dynamic signals of subcutaneous blood flow in a person at a sampling rate of 20 Hz, and to capture the temporal pattern of blood pulsation.

[0087] Among them, forgery tracing features may include frame block effect parameters generated based on video, H.264 / H.265 compression coding anomalies, and metadata verification information, such as verification results after timestamp continuity check and frame sequence cyclic redundancy check (CRC) verification results.

[0088] Step 202: Preprocess and enhance the multidimensional features.

[0089] Specifically, for appearance features, such as RGB facial images, the RetinaFace algorithm was used to extract 68 facial feature points. Simultaneously, an Enhanced Super-Resolution Generative Adversarial Network (ESRGAN) super-resolution algorithm with a 2x magnification was used to optimize facial detail texture and improve feature recognition. For PPG physiological features, such as PPG blood flow signals, wavelet thresholding based on the Db4 wavelet basis was used for noise reduction (threshold set to 3). To eliminate environmental interference, the signal amplitude is then normalized within the 0-1 range to unify the signal amplitude and enhance the stability of physiological characteristics.

[0090] This includes performing spatiotemporal consistency analysis on the video, calculating spatiotemporal difference indices, quantifying the spatiotemporal consistency of the video using these indices, and determining a spatiotemporal difference heatmap of the video based on these indices to mark abnormal areas of the video.

[0091] Step 203: Perform hierarchical fusion of multidimensional features to determine the target fusion features.

[0092] The hierarchical fusion of multidimensional features employs collaborative processing between edge devices and cloud devices. First, the edge device uses an attention network to initially fuse appearance and voice features from the preprocessed and enhanced multidimensional features. The attention network outputs a 512-dimensional primary fused feature after adaptive weight allocation and redundant data removal. Then, the cloud device performs cross-modal fusion of the primary fused feature with PPG physiological features, the forgery tracing features, and the spatiotemporal difference heatmap. The weights of the fused features in the cloud device are determined based on a pre-defined forgery tool encoding feature library, ultimately determining a 2048-dimensional target fused feature with a feature retention rate ≥98%.

[0093] Step 204: Perform dual-path anti-spoofing on the video based on target fusion features.

[0094] The dual-path approach comprises two parts: video forgery determination and forgery technology tracing. Forgery determination utilizes a lightweight feature classifier. Target fused features are input into the lightweight feature classifier for classification. The confidence level of the lightweight feature classifier predicting a forged video is compared with the dynamically determined target forged video detection rate to determine the forgery authentication result.

[0095] Among them, the forgery technology tracing method identifies the forgery technology type that matches the target fusion feature by matching the target fusion feature with the forgery tools in the preset forgery tool coding feature library, thereby enabling the tracing of the forgery technology of the forged video.

[0096] In the embodiments of this application, the order of the two parts of the dual-path authentication is not specifically limited. They can be executed in parallel, or the forgery determination can be performed first and then the forgery technical traceability can be performed first, or the forgery technical traceability can be performed first and then the forgery determination can be performed.

[0097] Step 205: In response to the update trigger condition, update the parameters of the video authentication model.

[0098] The video authentication model includes an attention network for fusing multi-dimensional features and a lightweight feature classifier for authentication. Update triggering conditions may include a decrease in the authentication accuracy of the lightweight feature classifier exceeding a preset threshold within a second preset time period; for example, a decrease in the authentication accuracy of the lightweight feature classifier exceeding 5% within seven days. Alternatively, update triggering conditions may include failing to match the corresponding forgery technique type when matching the target fused features with forgery tools in a preset forgery tool encoding feature library. Meeting any of the above update triggering conditions will trigger an update to the parameters of the lightweight feature classifier and / or the attention network, thereby improving authentication accuracy.

[0099] In some embodiments, the update process first collects gradient data of the attention network at the edge through a federated data aggregation mechanism. Based on differential privacy technology with a privacy budget ε=1.5, the gradient data is transmitted using AES-256 encryption (transmission rate ≥10Mbps) to avoid leakage of the original data. Secondly, a new sample library is built, with samples uniformly using YUV420P format (resolution 1080P / 720P). Each update adds ≥1000 samples covering different lighting and angle scenarios.

[0100] The federated data aggregation mechanism uses the FedAvg algorithm, and the specific process is as follows: ① Each edge device (assuming 100 edge nodes) calculates the gradient locally and then adds Laplace noise (the noise intensity is calculated based on ε=1.5 / 1.2 and the gradient data range, the formula is noise intensity=Δ / ε, where Δ is the global sensitivity of the gradient data, and here Δ=0.1); ② The edge devices upload the encrypted gradient data to the cloud device; ③ The cloud device performs a weighted average of the gradients of the 100 edge devices according to the node data volume weight (weight=number of local samples / total number of samples) to obtain the global gradient. During the aggregation process, the gradient transmission rate is maintained at ≥10Mbps through QoS guarantee.

[0101] Finally, incremental training was used to iterate the parameters of the lightweight feature classifier and / or attention network. The updated lightweight feature classifier and / or attention network were verified through a gray-scale release (initially with 10% of the traffic), and fully deployed after 72 hours without any anomalies. This ensured the continuous adaptability of the updated lightweight feature classifier and / or attention network to new forgery methods, providing a high-performance video forgery detection model to support video forgery detection.

[0102] Furthermore, the hyperparameters for incremental training can be set as follows: learning rate = 0.001 (using cosine annealing for learning rate scheduling, initial learning rate 0.001, decaying by 10% every 5 epochs), batch size = 32, training epochs = 20; the training termination condition is: validation set accuracy improves by ≤0.1% for 3 consecutive epochs or reaches 20 epochs; the cross-entropy loss function is used during training (the loss function is...). , For genuine labels (including genuine or fake). Calculate the loss value for the predicted probability.

[0103] Step 206: Video authentication adapted to application scenarios.

[0104] The application scenarios can include those with high security requirements and those with ordinary security requirements. For high-security scenarios, in response to the target's operation on a preset command, a spatiotemporal consistency analysis is performed on the operation to determine its continuity. Based on the continuity of the operation and the PPG physiological characteristics, the authentication result of the target is determined, thereby achieving differentiated processing for this application scenario and improving the accuracy of video authentication.

[0105] For applications with general security requirements, a lightweight feature classifier can be used to classify the target fusion features. The confidence level of the predicted video being fake is compared with the dynamically determined target fake video detection rate to determine whether the video is real or fake.

[0106] In some embodiments, different application scenarios can be further subdivided, such as application scenarios with ultra-high security requirements, application scenarios with high security requirements, and application scenarios with low security requirements.

[0107] In some optional embodiments, application scenarios may include, but are not limited to: finance (remote account opening, identity verification for large-amount transfers), government affairs (social security authentication, facial verification for household registration), internet (APP real-name registration, live streaming platform anchor authentication), and telecom operators (supplementary real-name registration for mobile phone numbers).

[0108] Step 207: Organize and output the entire process information of video authentication based on its acceptability.

[0109] The entire process of video authentication includes: multi-dimensional features, target fusion features, authentication results, and the results of tracing the source of fake videos.

[0110] In this embodiment, by incorporating photoplethysmography (PPG) physiological features and forgery tracing features into the acquisition scope, the feature dimensions are increased. This allows for multi-dimensional analysis and evaluation of the video using target fusion features, covering various forgery techniques or scenarios. This embodiment can employ a collaborative approach between edge devices and cloud devices for video authentication, improving response speed. Dual-path authentication not only determines the authentication result by classifying the target fusion features but also matches them with a pre-defined forgery tool encoding feature library to determine the corresponding forgery technique type, enabling traceability of the forgery technique used in the video. Based on the performance evaluation of dual-path authentication, the video authentication model is dynamically updated, continuously improving the accuracy of video authentication. Furthermore, by implementing video authentication methods tailored to different application scenarios, the adaptability of video authentication to various application scenarios is improved.

[0111] In some optional embodiments, the video authentication method provided in this application can also be applied to three terminals: terminal device, edge device, and cloud device. The terminal device is used to collect multi-dimensional features of the video, the edge device is used to preprocess and enhance the multi-dimensional features, and the cloud device is used to fuse the multi-dimensional features and perform dual-path authentication.

[0112] It should be noted that the video authentication method provided in this application can be executed by a video authentication device or a control module within that device for executing the video authentication method. This application uses the execution method of a video authentication device as an example to illustrate the video authentication device provided in this application.

[0113] Figure 3 This paper shows a schematic diagram of the structure of a video authentication device provided in an embodiment of this application. See also: Figure 3 The device 300 may include: an acquisition module 31, a fusion module 32, and an authentication module 33.

[0114] The acquisition module 31 is used to acquire multi-dimensional features of the target person in the video; wherein the multi-dimensional features include at least two of the following: appearance features, voice features, photoplethysmography (PPG) physiological features, and forgery tracing features; the fusion module 32 is used to fuse the multi-dimensional features to obtain target fused features; and the forgery detection module 33 is used to perform forgery detection processing on the video based on the target fused features to determine the forgery detection result of the video.

[0115] In one implementation, the aforementioned fusion module 32 can be used to: fuse the appearance features and the sound features through an attention network to determine a primary fusion feature; perform spatiotemporal consistency analysis on the video to determine a spatiotemporal difference heatmap of the video; and fuse the primary fusion feature with the photoplethysmography (PPG) physiological features, the forgery tracing features, and the spatiotemporal difference heatmap to determine the target fusion feature.

[0116] In one implementation, the aforementioned anti-counterfeiting module 33 can be used to: input the target fusion features into a lightweight feature classifier for classification processing to obtain the anti-counterfeiting result of the video, the anti-counterfeiting result including: real or fake; wherein, the lightweight feature classifier is used to compare the confidence level of predicting that the video is fake with the target fake video detection rate, and if the confidence level is greater than or equal to the target fake video detection rate, determine that the anti-counterfeiting result of the video is fake; the target fake video detection rate is the proportion of fake video samples in the video samples within a first preset time period.

[0117] In one implementation, the aforementioned apparatus 300 may further include a training module for obtaining the fake detection accuracy of the lightweight feature classifier; and updating the parameters of the lightweight feature classifier and / or the parameters of the attention network when the decrease in the fake detection accuracy of the lightweight feature classifier within a second preset time period is greater than a preset threshold; wherein the attention network is used to fuse the multidimensional features.

[0118] In one implementation, the aforementioned apparatus 300 may further include a matching module for matching the target fusion features with a preset forgery tool encoding feature library to determine the target forgery technology type corresponding to the video.

[0119] In one implementation, the aforementioned authentication module 33 can also be used to: respond to the target person's operation on a preset instruction, perform spatiotemporal consistency analysis on the operation, and determine the continuity of the operation; wherein the preset instruction is determined based on the risk coefficient of the application scenario of the video; and determine the authentication result of the target person based on the continuity of the operation and the PPG physiological characteristics, wherein the authentication result includes: real or fake.

[0120] The video authentication device in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the specific devices.

[0121] The video authentication device in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit it.

[0122] The video authentication device provided in this application embodiment can achieve... Figures 1 to 2 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.

[0123] Based on the same technical concept, this application also provides an electronic device for performing the above-described video authentication method. Figure 4 This is a schematic diagram of the structure of an electronic device to implement the various embodiments of this application. The electronic device can vary significantly due to differences in configuration or performance, and may include a processor 401, a communications interface 402, a memory 403, and a communication bus 404. The processor 401, communications interface 402, and memory 403 communicate with each other via the communication bus 404. The processor 401 can call a computer program stored in the memory 403 and executable on the processor 401 to perform the various steps of the above-described video authentication method embodiments, achieving the same technical effects. To avoid repetition, further details are omitted here.

[0124] It should be noted that the electronic devices in the embodiments of this application include servers, terminals, or other devices besides terminals. For example, automobiles, robots, and handheld devices.

[0125] The above electronic device structure does not constitute a limitation on the electronic device. An electronic device may include more or fewer components than illustrated, or combine certain components, or arrange them differently. For example, an input unit may include a Graphics Processing Unit (GPU) and a microphone, and a display unit may use a liquid crystal display (LCD), organic light-emitting diode (OLED), or other similar display panels. User input units include at least one of a touch panel and other input devices. A touch panel is also called a touchscreen. Other input devices may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be elaborated further here.

[0126] Memory can be used to store software programs and various data. Memory can primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area can store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, memory can include volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (Synchlink DRAM, SLDRAM), and direct memory bus RAM (DRRAM).

[0127] The processor may include one or more processing units; optionally, the processor integrates an application processor and a modem processor, wherein the application processor mainly handles operations related to the operating system, user interface, and applications, while the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into the processor.

[0128] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described video authentication method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0129] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0130] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described video authentication method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0131] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0132] This application also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes a program or instructions. When the program or instructions are executed, they implement the various processes of the above-described video authentication method embodiments and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0133] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0134] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0135] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A video authentication method, characterized in that, include: Obtain multidimensional features of the target person in the video; wherein, the multidimensional features include at least two of the following: appearance features, voice features, photoplethysmography (PPG) physiological features, and forgery tracing features; The multidimensional features are fused to obtain the target fused features; The video is subjected to anti-spoofing processing based on the target fusion features to determine the anti-spoofing result of the video.

2. The method according to claim 1, characterized in that, The process of fusing the multidimensional features to obtain the target fused features includes: The appearance features and the sound features are fused using an attention network to determine the initial fusion features; Spatiotemporal consistency analysis was performed on the video to determine the spatiotemporal difference heatmap of the video; The primary fusion feature is fused with the photoplethysmography (PPG) physiological features, the forgery tracing features, and the spatiotemporal difference heatmap to determine the target fusion feature.

3. The method according to claim 1, characterized in that, The step of performing anti-counterfeiting processing on the video based on the target fusion features and determining the anti-counterfeiting result of the video includes: The target fusion features are input into a lightweight feature classifier for classification processing to obtain the authentication result of the video, which includes: real or fake. The lightweight feature classifier is used to compare the confidence level of predicting that the video is fake with the target fake video detection rate. If the confidence level is greater than or equal to the target fake video detection rate, the video is determined to be fake. The target fake video detection rate is the proportion of fake video samples in the video samples within a first preset time period.

4. The method according to claim 3, characterized in that, The method further includes: Obtain the fake detection accuracy of the lightweight feature classifier; If the decrease in the counterfeit detection accuracy of the lightweight feature classifier is greater than a preset threshold within a second preset time period, the parameters of the lightweight feature classifier and / or the parameters of the attention network are updated; wherein, the attention network is used to fuse the multidimensional features.

5. The method according to claim 1, characterized in that, The method further includes: The target fusion features are matched with a preset forgery tool encoding feature library to determine the target forgery technology type corresponding to the video.

6. The method according to claim 1, characterized in that, The method further includes: In response to the target person's operation of a preset instruction, a spatiotemporal consistency analysis is performed on the operation to determine the continuity of the operation; wherein, the preset instruction is determined based on the risk coefficient of the application scenario of the video; Based on the continuity of the operation and the physiological characteristics of the PPG, the authentication result of the target person is determined, and the authentication result includes: real or fake.

7. A video authentication device, characterized in that, include: The acquisition module is used to acquire multidimensional features of the target person in the video; wherein, the multidimensional features include at least two of the following: appearance features, voice features, photoplethysmography (PPG) physiological features, and forgery tracing features; The fusion module is used to fuse the multidimensional features to obtain the target fused features; The anti-counterfeiting module is used to perform anti-counterfeiting processing on the video based on the target fusion features and determine the anti-counterfeiting result of the video.

8. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the video authentication method as described in any one of claims 1 to 6.

9. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the video authentication method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product includes a computer program stored on a non-transitory computer-readable storage medium, the computer program including programs or instructions that, when executed, implement the steps of the video authentication method as described in any one of claims 1 to 6.

Images