Evaluation Method and System for Multimodal Deep Spoofing Algorithm

Abstract

This invention provides an evaluation method and system for a multimodal deep fake data identification algorithm. The method includes: after each update of the judgment matrix, calculating the cosine similarity of the weight vectors of the two most recent rounds; if it is lower than a preset threshold, performing a positional median fusion and consistency check on the weight vectors of the two rounds; if it meets the threshold, generating target weights; training the identification algorithm evaluation model using an evaluation dataset; during training, applying a dynamic mask to each training sample of the evaluation dataset according to the target weights; marking positions where the modality consistency score is lower than the preset weighted mean as perturbable regions; adding perturbation information and recording the model's identification results; accumulating the difference in identification results before and after perturbation to obtain a robustness increment, and adding it to the corresponding index weighted score; iterating cyclically; after the model converges, using the model to evaluate the multimodal deep fake data identification algorithm to obtain the evaluation results, so as to accurately measure the actual performance and generalization ability of the identification algorithm.

Description

Technical Field

[0001] This invention relates to the field of deep fake data identification technology, and in particular to an evaluation method and system for a multimodal deep fake data identification algorithm. Background Technology

[0002] With the rapid development of artificial intelligence generation technology, multimodal deepfake data, represented by text, images, audio, and video, is becoming increasingly rampant. Deepfake data refers to multimedia content generated, altered, or synthesized using deep learning technology, and its core characteristic is that it is difficult for the human eye or ear to distinguish.

[0003] Currently, the barrier to entry for creating deepfake content based on technologies such as generative adversarial networks, diffusion models, and Transformers is constantly decreasing, and its realism is continuously improving, enabling the synthesis of cross-modal fake information. For example, it can generate highly realistic fake images through text descriptions, and swap faces in videos while simultaneously matching and forging audio. While these multimodal deepfake data bring convenience to entertainment and creative design, they also pose serious threats to social security, information dissemination, and personal privacy. For instance, fake news, using multimodal formats, is more likely to mislead the public and undermine social trust; maliciously altered audio and video may be used for defamation, fraud, and other illegal activities, damaging the reputation of individuals and organizations. Therefore, it is necessary to identify multimodal deepfake data.

[0004] However, current techniques for identifying deep fake data in multimodal scenarios still face numerous challenges. Due to the differences in characteristics among different modalities and the complex relationships involved in multimodal fusion, single-modal identification methods are insufficient to effectively address the identification needs of deep fake data in multimodal scenarios. Furthermore, the rapid iteration of deep fake techniques necessitates continuous updates to identification algorithms to adapt to new attack methods, while existing evaluation systems struggle to accurately measure the actual performance and generalization ability of identification algorithms. Summary of the Invention

[0005] This invention provides an evaluation method and system for multimodal deep fake data identification algorithms to accurately measure the actual performance and generalization ability of the identification algorithms.

[0006] To solve the above problems, the present invention adopts the following technical solution:

[0007] This invention provides an evaluation method for a multimodal deep fake data identification algorithm, comprising:

[0008] Construct an evaluation index system for multimodal deep fake data identification algorithms;

[0009] Based on the aforementioned evaluation index system, a cross-round consistency constraint process is introduced within the framework of the analytic hierarchy process: after each update of the judgment matrix, the cosine similarity between the current weight vector and the weight vector of the previous round is calculated. If the cosine similarity is lower than a preset threshold, a backtracking mechanism is triggered, and the matrix elements of the current round are fused with the matrix elements of the previous round bit by bit, and the consistency check is performed again until the cosine similarity is higher than the preset threshold and the consistency ratio meets the requirements, thus generating the target weight.

[0010] An evaluation dataset for a multimodal deep fake data identification algorithm is constructed. The evaluation model of the identification algorithm is trained using the evaluation dataset. During the training process, a dynamic mask is applied to each training sample of the evaluation dataset according to the target weight. The mask region is driven in real time by the modality consistency score. The evaluation model of the identification algorithm is a neural network model, which is used to comprehensively evaluate the multimodal deep fake data identification algorithm.

[0011] The positions where the modal consistency score is lower than the preset weighted mean are marked as perturbed regions. Perturbation information is added to the perturbed regions and the identification results of the identification algorithm evaluation model are recorded. The difference between the identification results of the identification algorithm evaluation model before and after the perturbation is accumulated to obtain the robustness increment. The robustness increment is added to the corresponding index weighted score to complete one evaluation iteration.

[0012] The identification algorithm evaluation model is iterated repeatedly. After the model converges, the multimodal deep pseudo data identification algorithm is evaluated using the identification algorithm evaluation model to obtain the evaluation result.

[0013] Preferably, the evaluation index system for constructing the multimodal deep fake data identification algorithm includes:

[0014] The algorithm for identifying multimodal deep fake data is constructed by establishing basic classification metrics and multimodal fusion specificity metrics. The basic classification metrics include accuracy, precision, recall, F1 score, AUC and ROC. The multimodal fusion specificity metrics include modality consistency score and cross-modal transferability metrics.

[0015] The target layer of the identification algorithm evaluation model is set as comprehensive performance evaluation, and the criterion layer and scheme layer are set as the basic classification index and multimodal fusion specific index.

[0016] Furthermore, before introducing the cross-round consistency constraint process within the analytic hierarchy process framework, it also includes:

[0017] Based on the 1-9 scaling method, pairwise comparisons are made between the indicators of the criterion layer and the scheme layer to form a judgment matrix;

[0018] The columns of the judgment matrix are normalized and summed, and then normalized again to obtain the eigenvectors. The eigenvectors are used as the initial weight vectors.

[0019] Preferably, generating the target weights includes:

[0020] The target weight is obtained by multiplying the weights of the criterion layer with the weights of the scheme layer under the corresponding criteria and then summing them.

[0021] Preferably, the evaluation dataset for constructing the multimodal deep fake data identification algorithm includes:

[0022] Collect the first training samples, which include text-image fusion, audio-video fusion, and deep pseudo data with cross-modal manipulation;

[0023] Obtain deep pseudo-data generated based on generative adversarial networks, variational autoencoders, diffusion models, and Transformer models to obtain the second training samples;

[0024] The first and second training samples are assigned real or deep pseudo labels, and the generated model type, tampered area and data source are labeled to form an evaluation dataset.

[0025] Preferably, the step of evaluating the multimodal deep fake data identification algorithm using the identification algorithm evaluation model to obtain the evaluation result includes:

[0026] The multimodal deep fake data identification algorithm is run on the same evaluation dataset and the index values ​​are recorded using the identification algorithm evaluation model. Each index value is sorted to generate a sorting table. The sorting table is converted into a score matrix. The score matrix is ​​multiplied element-wise with a preset comprehensive weight vector to obtain a weighted score matrix.

[0027] The weighted score matrix is ​​summed row by row to obtain the overall performance score of the multimodal deep pseudo data identification algorithm. The overall performance scores are then reordered from high to low to form a performance difference sequence. The performance difference sequence is output in the form of a stepped bar chart, where the height of the steps corresponds to the difference in the total score, the width of the steps is fixed, and adjacent steps are distinguished by solid color filling.

[0028] The inference speed, number of model parameters, and computational resource consumption of the multimodal deep fake data identification algorithm are evaluated. The performance differences, inference speed, number of model parameters, and computational resource consumption are compared with corresponding standard values, and the evaluation results are output.

[0029] Preferably, the step of performing digit-by-digit median fusion of the elements of the current matrix with the elements of the previous matrix includes:

[0030] The elements of the current matrix are matched one-to-one with the elements of the previous matrix according to their row and column coordinates to form a sequence of element pairs. For each sequence of element pairs, the arithmetic mean of the two values ​​is selected as the benchmark value. The interval between the benchmark value and the larger value is divided into three segments, and the upper limit of the first segment is used as the median extraction point.

[0031] The values ​​of the median extraction points are written into the corresponding positions of the new matrix. After all positions are written, an initial fusion matrix is ​​formed. For each column of the initial fusion matrix, the maximum value of the column is used as the denominator, and each element in the column is replaced with a value proportional to the maximum value to generate the target fusion matrix.

[0032] Preferably, applying a dynamic mask to each training sample of the evaluation dataset according to the target weights includes:

[0033] The target weights are written into a read-only memory in bit order, with the bit order as the base address and the weight value as the storage content within the base address, to form a weight-address mapping table.

[0034] For each modal channel of each training sample, the original modal data and the remodeled modal data are compared bit by bit through an XOR logic array, the number of high-level bits is output, and the ratio of the number of high-level bits to the total number of bits of the channel is written into the scoring register to obtain the modal consistency score.

[0035] The weight value output from the weight and address mapping table is used as the dividend, and the modal consistency score is used as the divisor to calculate the quotient. The integer part of the quotient is used as the mask length and written into the length register.

[0036] A sliding window is deployed on the modal channel of each training sample. The window length is provided in real time by the length register. The window sliding step is fixed at one clock cycle. The data position is set to zero where the window goes, thus completing the application of dynamic mask.

[0037] Preferably, the step of iteratively evaluating the discrimination algorithm model includes:

[0038] Construct a mirror twin evaluation loop for the identification algorithm evaluation model, and deploy an identification algorithm evaluation model with the same structure and completely identical initial weights in the loop, denoted as the reference identification algorithm evaluation model;

[0039] After each evaluation iteration, the weights of the reference discrimination algorithm evaluation model are flipped element by element to obtain the flipped weights, based on the weights of the discrimination algorithm evaluation model.

[0040] The flipped weights are written back to the discrimination algorithm evaluation model, and the discrimination algorithm evaluation model and the reference discrimination algorithm evaluation model are trained in parallel using the same batch of training samples to obtain the first output sequence and the second output sequence, respectively.

[0041] Perform a bitwise XOR operation on the first output sequence and the second output sequence to obtain an XOR sequence. Count the number of true bits in the XOR sequence to obtain the sign-consistent number.

[0042] If the number of symbol consistency is equal to the length of the output sequence, then all convolutional kernels of the discrimination algorithm evaluation model and the reference discrimination algorithm evaluation model are expanded into row vectors, forming a first row vector set and a second row vector set. Pairwise Hamming distance is calculated on the first row vector set and the second row vector set to obtain a Hamming distance set. The maximum Hamming distance in the Hamming distance set is extracted.

[0043] If the maximum Hamming distance is zero, the discrimination algorithm evaluation model is determined to have converged, and the iteration is terminated.

[0044] If the maximum Hamming distance is not zero, the discrimination algorithm evaluation model is determined to have not converged, and the iteration continues.

[0045] This invention also provides an evaluation system for a multimodal deep fake data identification algorithm, comprising:

[0046] The module is used to build an evaluation index system for multimodal deep fake data identification algorithms;

[0047] The constraint module is used to introduce a cross-round consistency constraint process within the framework of the analytic hierarchy process based on the evaluation index system: after each judgment matrix update, the cosine similarity between the current weight vector and the weight vector of the previous round is calculated. If the cosine similarity is lower than a preset threshold, a backtracking mechanism is triggered to perform digit-by-digit median fusion of the matrix elements of the current round and the matrix elements of the previous round, and the consistency check is performed again until the cosine similarity is higher than the preset threshold and the consistency ratio meets the requirements, and the target weight is generated.

[0048] The training module is used to construct an evaluation dataset for the multimodal deep fake data identification algorithm. The evaluation model of the identification algorithm is trained using the evaluation dataset. During the training process, a dynamic mask is applied to each training sample of the evaluation dataset according to the target weight. The mask region is driven in real time by the modality consistency score. The evaluation model of the identification algorithm is a neural network model used to comprehensively evaluate the multimodal deep fake data identification algorithm.

[0049] The marking module is used to mark the positions where the modal consistency score is lower than the preset weighted mean as perturbed regions, add perturbation information to the perturbed regions and record the identification results of the identification algorithm evaluation model, accumulate the difference in the identification results of the identification algorithm evaluation model before and after perturbation to obtain the robustness increment, and add the robustness increment to the corresponding index weighted score to complete one evaluation iteration.

[0050] The evaluation module is used to iteratively evaluate the identification algorithm evaluation model. After the model converges, the evaluation model is used to evaluate the multimodal deep pseudo-data identification algorithm to obtain the evaluation result.

[0051] Compared with the prior art, the technical solution of the present invention has at least the following advantages:

[0052] This invention provides an evaluation method for a multimodal deep fake data identification algorithm. It embeds cross-round consistency constraints within the analytic hierarchy process (AHP) framework, using cosine similarity to compare weight vectors from adjacent rounds in real time. Combined with position-by-position median fusion and consistency checks, this ensures high similarity between the weight update direction and historical trends, eliminating accuracy drift caused by subjective fluctuations. Secondly, a dynamic mask driven by target weights is introduced during the training phase. The mask region changes in real time with the modality consistency score, forcing the network to focus on the most easily forged local features with each parameter update, thus breaking local optima and improving the model's convergence speed. Furthermore, by marking regions with modality consistency scores below the weighted mean as perturbed areas, online perturbations are injected, and the differences in identification results before and after the perturbation are accumulated to form robust increments. These increments are directly superimposed on the corresponding weighted scores, enabling parallel evaluation of the model while training and robustness are achieved. This significantly reduces offline retesting and improves the overall processing efficiency of the model. Finally, the evaluation model is used to evaluate the multimodal deep fake data identification algorithm, allowing the trained evaluation model to accurately measure the actual performance and generalization ability of the algorithm. Attached Figure Description

[0053] Figure 1 This is a flowchart of an embodiment of an evaluation method for a multimodal deep fake data identification algorithm according to the present invention;

[0054] Figure 2 This is a flowchart of another embodiment of the evaluation method for a multimodal deep fake data identification algorithm of the present invention;

[0055] Figure 3 This is a flowchart of another embodiment of the evaluation method for a multimodal deep fake data identification algorithm of the present invention;

[0056] Figure 4 This is a structural block diagram of an embodiment of an evaluation system for a multimodal deep pseudo-data identification algorithm according to the present invention. Detailed Implementation

[0057] To enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0058] Please refer to Figure 1 This invention provides an evaluation method for a multimodal deep fake data identification algorithm, comprising the following steps:

[0059] S11. Construct an evaluation index system for multimodal deep fake data identification algorithms;

[0060] S12. Based on the evaluation index system, a cross-round consistency constraint process is introduced within the framework of the analytic hierarchy process: After each judgment matrix update, the cosine similarity between the current weight vector and the previous weight vector is calculated. If the cosine similarity is lower than a preset threshold, a backtracking mechanism is triggered to perform digit-by-digit median fusion of the elements of the current matrix and the elements of the previous matrix, and the consistency check is performed again until the cosine similarity is higher than the preset threshold and the consistency ratio meets the requirements, and the target weight is generated.

[0061] S13. Construct an evaluation dataset for the multimodal deep fake data identification algorithm, and use the evaluation dataset to train the evaluation model of the identification algorithm. During the training process, apply a dynamic mask to each training sample of the evaluation dataset according to the target weight. The mask region is driven in real time by the modality consistency score. The evaluation model of the identification algorithm is a neural network model, which is used to comprehensively evaluate the multimodal deep fake data identification algorithm.

[0062] S14. Mark the positions where the modal consistency score is lower than the preset weighted mean as perturbed regions. Add perturbation information to the perturbed regions and record the identification results of the identification algorithm evaluation model. Accumulate the differences in the identification results of the identification algorithm evaluation model before and after the perturbation to obtain the robustness increment. Add the robustness increment to the corresponding index weighted score to complete one evaluation iteration.

[0063] S15. Iterate the evaluation model of the identification algorithm. After the model converges, use the evaluation model of the identification algorithm to evaluate the multimodal deep pseudo data identification algorithm and obtain the evaluation result.

[0064] In this embodiment, the system can quantify forgery traces in four modalities—text, image, audio, and video—into an evaluation index system, including semantic consistency between text and images, temporal synchronization, spectral anomaly, tampering residue, cross-modal correlation strength, and so on.

[0065] After each update of the judgment matrix in the analytic hierarchy process (AHP), the cosine similarity between the current weight vector and the previous weight vector is immediately calculated. If the cosine similarity is lower than a preset threshold, it indicates that expert experience or environmental changes have caused excessive weight drift. In this case, a backtracking mechanism is triggered, fusing the elements of the current matrix with the median of the elements of the previous matrix to form a fused matrix, and then performing a consistency check again. This process is repeated until both the cosine similarity and the consistency ratio meet the target, at which point the target weight is output. This backtracking mechanism can fix weight mutations within an allowable range, synchronizing the evaluation system with the iteration rhythm of deep pseudotechnology and preventing the evaluation benchmark from becoming invalid due to changes in the generation strategy.

[0066] Dynamic masks are applied to each training sample in the evaluation dataset using target weights: first, modal consistency scores for each region of the sample are calculated; regions with low scores are masked in real time, while regions with high scores are preserved. The mask positions are refreshed as the neural network's parameters are updated, forcing the neural network to continuously learn the most easily forged local features, avoiding feature forgetting and local optima caused by fixed masking, significantly accelerating convergence speed and improving generalization ability.

[0067] The modality consistency score is a quantitative value that measures whether the four modalities (text, image, audio, and video) within the same sample conform to logical consistency at the semantic, temporal, spectral, and pixel levels. A high score indicates that the modalities support each other, while a low score indicates suspected forgery. It is used to drive dynamic masking and perturbable region selection.

[0068] Then, the locations where the modal consistency score is lower than the weighted mean are marked as perturbable regions. Small perturbation information is injected into these regions, and the model's discrimination difference before and after the perturbation is recorded. The difference is accumulated as a robustness increment and directly added to the weighted score of the corresponding index. This embeds the anti-attack test into the main training process, realizes online robustness update, eliminates the offline batch retesting step, and completes the evaluation and training simultaneously, which greatly shortens the overall computing cycle of the evaluation model.

[0069] The perturbable region is a sample area whose modality consistency score is lower than the weighted mean. This region is identified as a sensitive location where the network has not yet fully learned and has a high risk of being forged. It is used to inject small perturbation information and observe changes in the model output, thereby collecting robustness information online.

[0070] The discrimination algorithm evaluation model is iteratively evaluated. After model convergence is confirmed, the final parameters are locked, and a comprehensive evaluation result is output for any multimodal deep pseudo-data to be tested. The convergence mechanism ensures that the weights no longer drift, and the evaluation results are stable and reproducible.

[0071] This invention provides an evaluation method for a multimodal deep fake data identification algorithm. It embeds cross-round consistency constraints within the analytic hierarchy process (AHP) framework, using cosine similarity to compare weight vectors from adjacent rounds in real time. Combined with position-by-position median fusion and consistency checks, this ensures high similarity between the weight update direction and historical trends, eliminating accuracy drift caused by subjective fluctuations. Secondly, a dynamic mask driven by target weights is introduced during the training phase. The mask region changes in real time with the modality consistency score, forcing the network to focus on the most easily forged local features with each parameter update, thus breaking local optima and improving the model's convergence speed. Furthermore, by marking regions with modality consistency scores below the weighted mean as perturbed areas, online perturbations are injected, and the differences in identification results before and after the perturbation are accumulated to form robust increments. These increments are directly superimposed on the corresponding weighted scores, enabling parallel evaluation of the model while training and robustness are achieved. This significantly reduces offline retesting and improves the overall processing efficiency of the model. Finally, the evaluation model is used to evaluate the multimodal deep fake data identification algorithm, allowing the trained evaluation model to accurately measure the actual performance and generalization ability of the algorithm.

[0072] In one embodiment, please refer to Figure 2 The evaluation index system for constructing the multimodal deep fake data identification algorithm includes:

[0073] S111. Construct the basic classification indicators and multimodal fusion specificity indicators for the multimodal deep fake data identification algorithm. The basic classification indicators include accuracy, precision, recall, F1 score, AUC and ROC. The multimodal fusion specificity indicators include modality consistency score and cross-modal transferability index.

[0074] S112. Set the target layer of the identification algorithm evaluation model to comprehensive performance evaluation, and set the criterion layer and scheme layer to the basic classification index and multimodal fusion specific index.

[0075] In this embodiment, basic classification metrics are first established: accuracy, precision, recall, F1 score, AUC, and ROC. Then, multimodal fusion-specific metrics are introduced, including modality consistency score and cross-modal transferability metric. The two types of metrics complement each other, covering traditional statistical measures while fixing deep pseudo-cross-modal vulnerabilities and avoiding evaluation blind spots.

[0076] Precision is the overall percentage of correctly identified samples. Accuracy is the percentage of samples predicted as "deep fake" that were actually "deep fake," measuring the risk of misclassifying real data. Recall is the percentage of samples actually being "deep fake" that were correctly identified, measuring the risk of missing deep fake data. The F1 score is the harmonic mean of precision and recall, balancing the trade-off between the two. AUC and ROC are the areas under the ROC curve, reflecting the algorithm's ability to distinguish between "real" and "deep fake" samples at different thresholds.

[0077] Modality consistency score evaluates the ability of a discrimination algorithm to discriminate logical consistency between multimodal data (such as audio-lip-sync matching in video, or text-to-image content matching). Cross-modality transferability metric measures the ability of a discrimination algorithm to generalize to other modalities (such as video frames) after being trained on one modality (such as images).

[0078] The evaluation objective is placed at the top layer, while the criterion layer and scheme layer contain basic classification indicators and multimodal fusion-specific indicators. The scheme layer also contains the identification algorithm to be tested. Through this three-layer structure, the abstract comprehensive performance requirements are decomposed layer by layer into computable indicator nodes, so that subsequent weight allocation and consistency verification have a clear framework, preventing weight drift caused by mixed indicators.

[0079] This embodiment can first split and then layer the evaluation system, which retains the comparability of traditional statistical indicators and introduces cross-modal specificity measures, so that deep pseudo data can be identified in both the "single-modal flaw" and "inter-modal inconsistency" dimensions.

[0080] In one embodiment, prior to introducing the cross-round consistency constraint process within the analytic hierarchy process framework, the method further includes:

[0081] Based on the 1-9 scaling method, pairwise comparisons are made between the indicators of the criterion layer and the scheme layer to form a judgment matrix;

[0082] The columns of the judgment matrix are normalized and summed, and then normalized again to obtain the eigenvectors. The eigenvectors are used as the initial weight vectors.

[0083] In this embodiment, evaluation experts can be invited to conduct pairwise comparisons based on the importance of the indicators, and a judgment matrix can be constructed using a 1-9 scale (1 indicates equal importance, and 9 indicates extreme importance). For example, for the criterion layer, if the experts believe that the basic classification indicator is more important than the multimodal fusion specific indicator, with an importance level of 3, then the corresponding position in the judgment matrix is ​​3; otherwise, it is 1 / 3.

[0084] Divide each column of the judgment matrix by its sum to make the column elements between 0 and 1, eliminating the difference in dimensions; sum the normalized matrix row by row to obtain the sum vector of each row; normalize the sum vector again to make its length one, and the result is the initial weight vector. It can quickly approximate the largest eigenvector without iteration, with low computational cost, and is suitable as the starting point for cross-round consistency constraints.

[0085] The 1-9 scale method uses nine discrete levels to describe the relative importance of two indicators. The higher the level, the more prominent the relative advantage. The reciprocal corresponds to the inverse comparison, ensuring matrix reciprocity.

[0086] The judgment matrix is ​​in the form of a square matrix, with rows and columns corresponding to the same set of indicators. The element values ​​reflect the importance ratio of the row indicators to the column indicators, and it is the only data source for weight extraction and consistency verification.

[0087] In this embodiment, the expert experience is first solidified using the 1-9 scale method, and then the initial weights are obtained quickly and accurately through double normalization, so that the starting point of the weights has both the interpretability of the humanities and meets the requirements of mathematical normalization.

[0088] Preferably, generating the target weights includes:

[0089] The target weight is obtained by multiplying the weights of the criterion layer with the weights of the scheme layer under the corresponding criteria and then summing them.

[0090] After cross-round consistency constraints, each indicator in the criterion layer (such as accuracy, modal consistency score, etc.) has obtained a unique and stable weight value, which represents the relative importance of the criterion in the comprehensive evaluation.

[0091] Under the same criterion, each discrimination algorithm forms a sub-judgment matrix through pairwise comparisons. After normalization, the local weights of the scheme layer are obtained, which reflect the order of merit of different discrimination algorithms within the same evaluation dimension.

[0092] The local weights of each discrimination algorithm at the scheme layer are multiplied pairwise with the criterion layer weights of their respective criteria. These multiplications are then added to the classification algorithm's name, forming a weighted sum from the bottom-level indicators to the top-level objective. This is the final target weight of the discrimination algorithm. Through hierarchical multiplication of criterion layer weights and scheme layer local weights, the abstract comprehensive performance requirements are mapped to each discrimination algorithm in one step, achieving hierarchical transmission of importance and one-time quantification of superiority and inferiority. This process does not introduce additional hyperparameters, and the computational path is clear and traceable. It retains the advantages of the analytic hierarchy process (AHP) in combining subjective and objective methods while avoiding weight drift. The final target weight can be directly used for subsequent dynamic masking and robust incremental calculations, ensuring that the evaluation results are updated synchronously with the deep pseudotechnology iterations.

[0093] In one embodiment, the evaluation dataset for constructing the multimodal deep fake data identification algorithm includes:

[0094] Collect the first training samples, which include text-image fusion, audio-video fusion, and deep pseudo data with cross-modal manipulation;

[0095] Obtain deep pseudo-data generated based on generative adversarial networks, variational autoencoders, diffusion models, and Transformer models to obtain the second training samples;

[0096] The first and second training samples are assigned real or deep pseudo labels, and the generated model type, tampered area and data source are labeled to form an evaluation dataset.

[0097] This embodiment can collect real deepfake data that has been circulating from public platforms and authorized partners, such as image and text splicing, audio and video synchronous replacement, and cross-modal tampering, as the first training sample. It covers scenarios such as news forgery, advertising face swapping, and voice cloning to ensure that the sample background is complex and the attack methods are diverse.

[0098] Four mainstream generation engines—Generative Adversarial Networks, Variational Autoencoders, Diffusion Models, and Transformers—are invoked to resynthesize a batch of deep pseudo-data on a controlled computing cluster, resulting in a second training sample. During generation, multiple cross-modal links such as text-driven image, image-driven audio, and video-driven speech are covered, and the latest attack modes are artificially supplemented.

[0099] The first and second training samples are assigned real or deep pseudo labels, and the generation model type, tampered area and data source are labeled to form a labeled evaluation dataset, which can be directly used for subsequent dynamic masking and robust incremental calculation.

[0100] In this embodiment, dual-channel acquisition enables the dataset to simultaneously contain both exposed and potential attack features, with the coverage expanding synchronously with the generation technology. The annotation mechanism ensures that each sample carries information about the attack source and tampering location, providing accurate coordinates for subsequent modality consistency score calculation and perturbable region localization, significantly reducing evaluation bias caused by annotation noise.

[0101] In one embodiment, please refer to Figure 3 The evaluation of the multimodal deep fake data identification algorithm using the identification algorithm evaluation model, and the resulting evaluation, includes:

[0102] S151. The multimodal deep fake data identification algorithm is run on the same evaluation dataset using the identification algorithm evaluation model and the index values ​​are recorded. Each index value is sorted to generate a sorting table. The sorting table is converted into a score matrix. The score matrix is ​​multiplied element-wise with a preset comprehensive weight vector to obtain a weighted score matrix.

[0103] S152. Sum the weighted score matrix row by row to obtain the overall performance score of the multimodal deep pseudo data identification algorithm. Re-sort the overall performance scores from high to low to form a performance difference sequence. Output the performance difference sequence in the form of a stepped bar chart. The height of the steps corresponds to the difference in the total score. The width of the steps is fixed. Adjacent steps are distinguished by solid color filling.

[0104] S153. Evaluate the inference speed, number of model parameters, and computational resource consumption of the multimodal deep fake data identification algorithm. Compare the performance differences, inference speed, number of model parameters, and computational resource consumption with the corresponding standard values ​​and output the evaluation results.

[0105] In this embodiment, the discrimination algorithm is used to evaluate the model. First, the discrimination algorithm is run on the same evaluation dataset to record the values ​​of each indicator, such as accuracy, modality consistency score, and cross-modal transferability. The values ​​of each indicator are sorted according to the indicator dimension to form a sorting table. The sorting table is linearly mapped to the score according to the ranking and then assembled into a score matrix. Then, the score matrix is ​​multiplied element by element with the comprehensive weight vector to obtain the weighted score matrix.

[0106] The weighted score matrix is ​​summed using the algorithm's row vectors to obtain the overall performance score. The scores are then rearranged in descending order to form a performance difference sequence. This sequence is then output as a stepped bar chart, with the height of each step corresponding to the difference in the total score. The width of each step is fixed, and adjacent steps are filled with different solid colors, making the differences in performance immediately apparent and allowing for intuitive comparison without additional text.

[0107] The inference speed, model parameter count, and computational resource consumption of each identification algorithm are collected synchronously, compared with industry standard values, and the evaluation results are output.

[0108] This embodiment generates a ranking table by sorting each indicator value, converts the ranking table into a score matrix, and multiplies it element-wise with a preset comprehensive weight vector to obtain a weighted score matrix. The weighted score matrix is ​​then summed row-wise to obtain the overall performance score. The algorithms are then reordered in descending order of their overall performance scores to form a performance difference sequence, thereby accurately evaluating the performance of each identification algorithm. Furthermore, the multi-dimensional indicators can be compressed into a single ladder chart, allowing evaluators to simultaneously read relative rankings and absolute differences within a single chart. Adding speed, parameter count, and resource consumption checks ensures that the selected algorithms meet both accuracy requirements and deployment constraints, achieving a scientific, rapid, and interpretable algorithm selection process.

[0109] In one embodiment, the step of merging the current matrix elements with the previous matrix elements by digit-by-digit median includes:

[0110] The elements of the current matrix are matched one-to-one with the elements of the previous matrix according to their row and column coordinates to form a sequence of element pairs. For each sequence of element pairs, the arithmetic mean of the two values ​​is selected as the benchmark value. The interval between the benchmark value and the larger value is divided into three segments, and the upper limit of the first segment is used as the median extraction point.

[0111] The values ​​of the median extraction points are written into the corresponding positions of the new matrix. After all positions are written, an initial fusion matrix is ​​formed. For each column of the initial fusion matrix, the maximum value of the column is used as the denominator, and each element in the column is replaced with a value proportional to the maximum value to generate the target fusion matrix.

[0112] In this embodiment, each element of the current round's judgment matrix is ​​paired with each element of the previous round's judgment matrix according to row and column coordinates, resulting in a sequentially arranged sequence of element pairs to ensure accurate positional correspondence. For each element pair sequence, the arithmetic mean is first taken as the baseline value. Then, the interval between the baseline value and the larger value is divided into three segments, and the upper limit of the first segment is selected as the median extraction point. This point is always located at the conservative end, absorbing new information while suppressing extreme preferences.

[0113] The median extraction points are filled into the corresponding coordinates of the new matrix. After all positions are filled, the initial fusion matrix is ​​obtained, completing the first round of fusion. Using the maximum value of each column as the denominator, each element in the same column is replaced with its ratio to the maximum value to form the target fusion matrix. This brings columns with different dimensions to the same scale, ensuring that the weights of each column are comparable during subsequent consistency checks.

[0114] This embodiment uses the arithmetic mean of each element's pair with the sequence as a benchmark value. The interval between the benchmark and larger values ​​is divided into three segments, with the upper limit of the first segment used as the median extraction point. This prevents the new matrix from suddenly biasing towards any particular round of expert opinions, effectively preventing weight drift. Furthermore, column scaling further eliminates dimensional differences, ensuring that cross-round consistency constraints operate on a unified scale, thus improving the stability and reliability of the analytic hierarchy process in dynamic environments.

[0115] In one embodiment, applying a dynamic mask to each training sample of the evaluation dataset according to the target weights includes:

[0116] The target weights are written into a read-only memory in bit order, with the bit order as the base address and the weight value as the storage content within the base address, to form a weight-address mapping table.

[0117] For each modal channel of each training sample, the original modal data and the remodeled modal data are compared bit by bit through an XOR logic array, the number of high-level bits is output, and the ratio of the number of high-level bits to the total number of bits of the channel is written into the scoring register to obtain the modal consistency score.

[0118] The weight value output from the weight and address mapping table is used as the dividend, and the modal consistency score is used as the divisor to calculate the quotient. The integer part of the quotient is used as the mask length and written into the length register.

[0119] A sliding window is deployed on the modal channel of each training sample. The window length is provided in real time by the length register. The window sliding step is fixed at one clock cycle. The data position is set to zero where the window goes, thus completing the application of dynamic mask.

[0120] In this embodiment, the target weights can be written sequentially into a read-only memory, where the bit order is the base address and the weight value is the stored content, forming a weight-address mapping table. After power failure, the content can be kept unchanged, ensuring that the weights start at the same point when the model restarts training, thereby preventing baseline drift caused by random initialization.

[0121] For each modal channel of each training sample, the remodeled modal data of the original modal number model is compared bit by bit using an XOR logic array, and the number of high-level bits is output. The more high-level bits, the greater the difference and the worse the consistency. The ratio of the number of high-level bits to the total number of bits of the channel is stored in the score register and used as the modal consistency score.

[0122] Using the weight value output from the weight and address mapping table as the dividend, and the modal consistency score obtained in the previous step as the divisor, perform integer division. The integer part of the calculated quotient is directly written into the length register. This achieves automatic adjustment so that the higher the weight and the worse the consistency, the longer the mask will be, without the need for additional table lookup or multiplication operations.

[0123] A sliding window is deployed in each modal channel. The window length is updated in real time by the length register, and the step size is fixed at one clock cycle. The corresponding data point is immediately set to zero when the window reaches a certain position, forming a dynamic masking to achieve online masking of areas with poor consistency.

[0124] In this embodiment, after the target weights are written to the read-only memory, training restarts and power outages do not require reloading, eliminating the latency and consistency risks associated with traditional memory copying. The mask decision-making side always reads the same authoritative version, ensuring that the evaluation benchmark does not drift. Furthermore, the bitwise XOR operation between the original mode and the remodeled mode is entirely performed by a combinational logic array, yielding the number of high-level bits in a single clock cycle. The mode consistency score is generated synchronously with data loading, eliminating the need for software post-processing. Secondly, the integer quotient obtained by dividing the weights by the consistency score directly determines the mask length, enabling precise adjustment of the occlusion region. In addition, since the occlusion length is proportional to the weights and inversely proportional to the consistency, the gradient can only pass through the unoccluded, highly consistent region during backpropagation, forcing the model to continuously focus on the most difficult-to-spoof and most important local features for evaluation, thus improving convergence speed and reducing the risk of local optima.

[0125] In one embodiment, the iterative evaluation of the discrimination algorithm model includes:

[0126] Construct a mirror twin evaluation loop for the identification algorithm evaluation model, and deploy an identification algorithm evaluation model with the same structure and completely identical initial weights in the loop, denoted as the reference identification algorithm evaluation model;

[0127] After each evaluation iteration, the weights of the reference discrimination algorithm evaluation model are flipped element by element to obtain the flipped weights, based on the weights of the discrimination algorithm evaluation model.

[0128] The flipped weights are written back to the discrimination algorithm evaluation model, and the discrimination algorithm evaluation model and the reference discrimination algorithm evaluation model are trained in parallel using the same batch of training samples to obtain the first output sequence and the second output sequence, respectively.

[0129] Perform a bitwise XOR operation on the first output sequence and the second output sequence to obtain an XOR sequence. Count the number of true bits in the XOR sequence to obtain the sign-consistent number.

[0130] If the number of symbol consistency is equal to the length of the output sequence, then all convolutional kernels of the discrimination algorithm evaluation model and the reference discrimination algorithm evaluation model are expanded into row vectors, forming a first row vector set and a second row vector set. Pairwise Hamming distance is calculated on the first row vector set and the second row vector set to obtain a Hamming distance set. The maximum Hamming distance in the Hamming distance set is extracted.

[0131] If the maximum Hamming distance is zero, the discrimination algorithm evaluation model is determined to have converged, and the iteration is terminated.

[0132] If the maximum Hamming distance is not zero, the discrimination algorithm evaluation model is determined to have not converged, and the iteration continues.

[0133] This embodiment can replicate a reference model (reference identification algorithm evaluation model) with the same structure and initial weights as the main model (discrimination algorithm evaluation model) to form a closed-loop hardware circuit; the two models share a data bus, but the weight update paths are independent, which facilitates subsequent sign flipping comparison.

[0134] The mirror twin evaluation loop is a hardware-software co-designed convergence monitoring structure used to determine in real time whether the evaluation model of the discrimination algorithm has reached a stable state. Its core idea is to create a copy of the main model with identical structure and initial weights. The two models share the input data stream, but their weight update paths are independent, forming a closed-loop comparison system. After each iteration, the system performs operations such as sign flipping, XOR operations, and Hamming distance calculations on the weights or outputs of the main and reference models. If the two models are completely identical in decision signs and weight values, the main model is considered to have converged, and training can be terminated; otherwise, iteration continues.

[0135] After each iteration, using the weights of the discrimination algorithm evaluation model as a benchmark, the weights of the reference discrimination algorithm evaluation model are inverted element by element to obtain the flipped weights, and these flipped weights are immediately written back to the discrimination algorithm evaluation model. The same batch of training samples are simultaneously fed into both the discrimination algorithm evaluation model with the flipped weights written back and the reference discrimination algorithm evaluation model, each completing one forward propagation to obtain the first output sequence and the second output sequence. Parallel execution ensures that environmental noise and data perturbations are completely consistent, eliminating the interference of external differences on the comparison results.

[0136] Perform an XOR operation bit by bit on the two output sequences: if the corresponding bits are the same, the output is true; otherwise, the output is false. Count the number of true bits to obtain the sign consistency number, which directly reflects the degree of overlap between the two models at the decision sign level.

[0137] When the number of identical signs equals the sequence length, all convolutional kernels are unrolled as row vectors, forming the first set of row vectors and the second set of row vectors. Hamming distances are then calculated for each pair to obtain a set of Hamming distances. The maximum Hamming distance is extracted from this set. If the maximum Hamming distance is zero, it means that the weight signs and values ​​are completely identical, and the discrimination algorithm evaluation model is considered to have converged. If the maximum Hamming distance is not zero, the discrimination algorithm evaluation model is iteratively trained until the maximum Hamming distance is also zero.

[0138] This embodiment utilizes sign flipping to write the generated flipped weights back to the discrimination algorithm to evaluate the model. Without introducing additional hyperparameters or random noise, it can apply the maximum magnitude of decision-reverse perturbation to the main model. If the main model still outputs the exact same sign sequence as the reference model, it indicates that its decision hyperplane is stable enough to be immune to sign changes, completely eliminating the risk of false convergence. Secondly, comparing the number of identical signs with the length of the output sequence quickly filters out non-convergent states of the model. Only when the signs are completely identical is the convolution kernel unwound for fine-grained Hamming distance verification, minimizing computational load while ensuring decision accuracy, achieving efficient and precise convergence monitoring. Furthermore, the mirror twin evaluation loop adopts a homogeneous parallel architecture; sign flipping, XOR operations, and Hamming accumulation are all completed in the same clock domain, ensuring strict alignment between the convergence signal and the training cycle. This eliminates the need to pause training or use additional validation sets, shortening the overall iteration cycle. Finally, a maximum Hamming distance of zero means that the weight signs and values ​​are both identical, and the main model will not change during subsequent training. Parameters can be directly frozen for deployment, avoiding performance fluctuations caused by traditional early stopping and improving online stability.

[0139] Please refer to Figure 4 The embodiments of the present invention also provide an evaluation system for a multimodal deep fake data identification algorithm, comprising:

[0140] Module 11 is used to construct an evaluation index system for the multimodal deep fake data identification algorithm;

[0141] Constraint module 12 is used to introduce a cross-round consistency constraint process within the framework of the analytic hierarchy process based on the evaluation index system: after each judgment matrix update, the cosine similarity between the current weight vector and the previous weight vector is calculated. If the cosine similarity is lower than a preset threshold, a backtracking mechanism is triggered to perform digit-by-digit median fusion of the matrix elements of the current round and the matrix elements of the previous round, and the consistency check is performed again until the cosine similarity is higher than the preset threshold and the consistency ratio meets the requirements, and the target weight is generated.

[0142] Training module 13 is used to construct an evaluation dataset for a multimodal deep fake data identification algorithm. The evaluation dataset is used to train the evaluation model of the identification algorithm. During the training process, a dynamic mask is applied to each training sample of the evaluation dataset according to the target weight. The mask region is driven in real time by the modality consistency score. The evaluation model of the identification algorithm is a neural network model used to comprehensively evaluate the multimodal deep fake data identification algorithm.

[0143] The marking module 14 is used to mark the positions where the modal consistency score is lower than the preset weighted mean as perturbed regions, add perturbation information to the perturbed regions and record the identification results of the identification algorithm evaluation model, accumulate the difference in the identification results of the identification algorithm evaluation model before and after perturbation to obtain the robustness increment, and add the robustness increment to the corresponding index weighted score to complete one evaluation iteration.

[0144] The evaluation module 15 is used to iteratively evaluate the identification algorithm evaluation model. After the model converges, the evaluation module is used to evaluate the multimodal deep pseudo data identification algorithm to obtain the evaluation result.

[0145] Regarding the system in the above embodiments, the specific ways in which each module performs operations have been described in detail in the embodiments related to the method, and will not be elaborated here.

[0146] In one embodiment, the present invention also provides a storage medium storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the evaluation method of the multimodal deep fake data identification algorithm described above. The storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, or optical data storage device, etc.

[0147] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a storage medium, and when executed, it can include the processes of the embodiments of the methods described above. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, optical disk, or read-only memory (ROM), or random access memory (RAM).

[0148] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0149] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. An evaluation method for a multimodal deep fake data identification algorithm, characterized in that, include: Construct an evaluation index system for multimodal deep fake data identification algorithms; Based on the aforementioned evaluation index system, a cross-round consistency constraint process is introduced within the framework of the analytic hierarchy process: after each update of the judgment matrix, the cosine similarity between the current weight vector and the weight vector of the previous round is calculated. If the cosine similarity is lower than a preset threshold, a backtracking mechanism is triggered, and the matrix elements of the current round are fused with the matrix elements of the previous round bit by bit, and the consistency check is performed again until the cosine similarity is higher than the preset threshold and the consistency ratio meets the requirements, thus generating the target weight. An evaluation dataset for a multimodal deep fake data identification algorithm is constructed. The evaluation model of the identification algorithm is trained using the evaluation dataset. During the training process, a dynamic mask is applied to each training sample of the evaluation dataset according to the target weight. The mask region is driven in real time by the modality consistency score. The evaluation model of the identification algorithm is a neural network model, which is used to comprehensively evaluate the multimodal deep fake data identification algorithm. The positions where the modal consistency score is lower than the preset weighted mean are marked as perturbed regions. Perturbation information is added to the perturbed regions and the identification results of the identification algorithm evaluation model are recorded. The difference between the identification results of the identification algorithm evaluation model before and after the perturbation is accumulated to obtain the robustness increment. The robustness increment is added to the corresponding index weighted score to complete one evaluation iteration. The identification algorithm evaluation model is iterated repeatedly. After the model converges, the multimodal deep pseudo data identification algorithm is evaluated using the identification algorithm evaluation model to obtain the evaluation result.

2. The method according to claim 1, characterized in that, The evaluation index system for constructing a multimodal deep fake data identification algorithm includes: The algorithm for identifying multimodal deep fake data is constructed by establishing basic classification metrics and multimodal fusion specificity metrics. The basic classification metrics include accuracy, precision, recall, F1 score, AUC and ROC. The multimodal fusion specificity metrics include modality consistency score and cross-modal transferability metrics. The target layer of the identification algorithm evaluation model is set as comprehensive performance evaluation, and the criterion layer and scheme layer are set as the basic classification index and multimodal fusion specific index.

3. The method according to claim 2, characterized in that, Before introducing cross-round consistency constraints within the analytic hierarchy process framework, the process also includes: Based on the 1-9 scaling method, pairwise comparisons are made between the indicators of the criterion layer and the scheme layer to form a judgment matrix; The columns of the judgment matrix are normalized and summed, and then normalized again to obtain the eigenvectors. The eigenvectors are used as the initial weight vectors.

4. The method according to claim 2, characterized in that, The generation of target weights includes: The target weight is obtained by multiplying the weights of the criterion layer with the weights of the scheme layer under the corresponding criteria and then summing them.

5. The method according to claim 1, characterized in that, The evaluation dataset for constructing the multimodal deep fake data identification algorithm includes: Collect the first training samples, which include text-image fusion, audio-video fusion, and deep pseudo data with cross-modal manipulation; Obtain deep pseudo-data generated based on generative adversarial networks, variational autoencoders, diffusion models, and Transformer models to obtain the second training samples; The first and second training samples are assigned real or deep pseudo labels, and the generated model type, tampered area and data source are labeled to form an evaluation dataset.

6. The method according to claim 1, characterized in that, The evaluation model using the aforementioned identification algorithm is used to evaluate the multimodal deep fake data identification algorithm, and the evaluation results are obtained, including: The multimodal deep fake data identification algorithm is run on the same evaluation dataset and the index values ​​are recorded using the identification algorithm evaluation model. Each index value is sorted to generate a sorting table. The sorting table is converted into a score matrix. The score matrix is ​​multiplied element-wise with a preset comprehensive weight vector to obtain a weighted score matrix. The weighted score matrix is ​​summed row by row to obtain the overall performance score of the multimodal deep pseudo data identification algorithm. The overall performance scores are then reordered from high to low to form a performance difference sequence. The performance difference sequence is output in the form of a stepped bar chart, where the height of the steps corresponds to the difference in the total score, the width of the steps is fixed, and adjacent steps are distinguished by solid color filling. The inference speed, number of model parameters, and computational resource consumption of the multimodal deep fake data identification algorithm are evaluated. The performance differences, inference speed, number of model parameters, and computational resource consumption are compared with corresponding standard values, and the evaluation results are output.

7. The method according to claim 1, characterized in that, The step of merging the matrix elements of the current round with the matrix elements of the previous round by digit-by-digit median includes: The elements of the current matrix are matched one-to-one with the elements of the previous matrix according to their row and column coordinates to form a sequence of element pairs. For each sequence of element pairs, the arithmetic mean of the two values ​​is selected as the benchmark value. The interval between the benchmark value and the larger value is divided into three segments, and the upper limit of the first segment is used as the median extraction point. The values ​​of the median extraction points are written into the corresponding positions of the new matrix. After all positions are written, an initial fusion matrix is ​​formed. For each column of the initial fusion matrix, the maximum value of the column is used as the denominator, and each element in the column is replaced with a value proportional to the maximum value to generate the target fusion matrix.

8. The method according to claim 1, characterized in that, Applying a dynamic mask to each training sample of the evaluation dataset according to the target weights includes: The target weights are written into a read-only memory in bit order, with the bit order as the base address and the weight value as the storage content within the base address, to form a weight-address mapping table. For each modal channel of each training sample, the original modal data and the remodeled modal data are compared bit by bit through an XOR logic array, the number of high-level bits is output, and the ratio of the number of high-level bits to the total number of bits of the channel is written into the scoring register to obtain the modal consistency score. The weight value output from the weight and address mapping table is used as the dividend, and the modal consistency score is used as the divisor to calculate the quotient. The integer part of the quotient is used as the mask length and written into the length register. A sliding window is deployed on the modal channel of each training sample. The window length is provided in real time by the length register. The window sliding step is fixed at one clock cycle. The data position is set to zero where the window goes, thus completing the application of dynamic mask.

9. The method according to claim 1, characterized in that, The iterative evaluation of the discrimination algorithm model includes: Construct a mirror twin evaluation loop for the identification algorithm evaluation model, and deploy an identification algorithm evaluation model with the same structure and completely identical initial weights in the loop, denoted as the reference identification algorithm evaluation model; After each evaluation iteration, the weights of the reference discrimination algorithm evaluation model are flipped element by element to obtain the flipped weights, based on the weights of the discrimination algorithm evaluation model. The flipped weights are written back to the discrimination algorithm evaluation model, and the discrimination algorithm evaluation model and the reference discrimination algorithm evaluation model are trained in parallel using the same batch of training samples to obtain the first output sequence and the second output sequence, respectively. Perform a bitwise XOR operation on the first output sequence and the second output sequence to obtain an XOR sequence. Count the number of true bits in the XOR sequence to obtain the sign-consistent number. If the number of symbol consistency is equal to the length of the output sequence, then all convolutional kernels of the discrimination algorithm evaluation model and the reference discrimination algorithm evaluation model are expanded into row vectors, forming a first row vector set and a second row vector set. Pairwise Hamming distance is calculated on the first row vector set and the second row vector set to obtain a Hamming distance set. The maximum Hamming distance in the Hamming distance set is extracted. If the maximum Hamming distance is zero, the discrimination algorithm evaluation model is determined to have converged, and the iteration is terminated. If the maximum Hamming distance is not zero, the discrimination algorithm evaluation model is determined to have not converged, and the iteration continues.

10. An evaluation system for a multimodal deep fake data identification algorithm, characterized in that, include: The module is used to build an evaluation index system for multimodal deep fake data identification algorithms; The constraint module is used to introduce a cross-round consistency constraint process within the framework of the analytic hierarchy process based on the evaluation index system: after each judgment matrix update, the cosine similarity between the current weight vector and the weight vector of the previous round is calculated. If the cosine similarity is lower than a preset threshold, a backtracking mechanism is triggered to perform digit-by-digit median fusion of the matrix elements of the current round and the matrix elements of the previous round, and the consistency check is performed again until the cosine similarity is higher than the preset threshold and the consistency ratio meets the requirements, and the target weight is generated. The training module is used to construct an evaluation dataset for the multimodal deep fake data identification algorithm. The evaluation model of the identification algorithm is trained using the evaluation dataset. During the training process, a dynamic mask is applied to each training sample of the evaluation dataset according to the target weight. The mask region is driven in real time by the modality consistency score. The evaluation model of the identification algorithm is a neural network model used to comprehensively evaluate the multimodal deep fake data identification algorithm. The marking module is used to mark the positions where the modal consistency score is lower than the preset weighted mean as perturbed regions, add perturbation information to the perturbed regions and record the identification results of the identification algorithm evaluation model, accumulate the difference in the identification results of the identification algorithm evaluation model before and after perturbation to obtain the robustness increment, and add the robustness increment to the corresponding index weighted score to complete one evaluation iteration. The evaluation module is used to iteratively evaluate the identification algorithm evaluation model. After the model converges, the evaluation model is used to evaluate the multimodal deep pseudo-data identification algorithm to obtain the evaluation result.

Images