User generated content deepfake detection method, apparatus, device and storage medium

Abstract

The application provides a user-generated content deep fake detection method, device and equipment and a storage medium, which are applied to the field of financial risk control. The method comprises the following steps: generating enhanced data from original data in training video data based on an enhancement strategy; inputting the original data and the enhanced data into a candidate architecture to obtain first classification results and second classification results; updating the weights of the candidate architecture and the enhancement strategy in combination with true and false labels, the first classification results and the second classification results to generate new enhanced data; training the candidate architecture after the weights are updated by using the new enhanced data; repeating the above steps until the training of the candidate architecture is completed, determining the performance evaluation value of the trained candidate architecture, and updating the architecture parameters; selecting a new candidate architecture from a search space based on the updated architecture parameters to perform training, and finally selecting a target architecture with the optimal performance evaluation; and performing fake detection on to-be-processed video data by using the target architecture.

Description

Technical Field

[0001] This invention relates to the field of forgery detection technology, and in particular to a method, apparatus, device, and storage medium for detecting deep forgery of user-generated content. Background Technology

[0002] With the rapid development of deepfake technology, especially the maturity of technologies such as Generative Adversarial Networks (GANs) and diffusion models, the realism of deepfake content has approached a level that is indistinguishable from genuine content. User-generated content (UGC) platforms (such as social media and short video applications) have become the main channels for the spread of deepfakes. However, because UGC is usually processed through compression, transcoding, and the introduction of sensor noise, its distortion characteristics significantly increase the difficulty of detection. Existing detection methods mainly face the following challenges: 1. Insufficient robustness to distortion: Traditional detection models rely on hand-designed features or fixed-architecture CNNs, which are poorly adapted to complex composite distortions in UGC (such as the superposition of H.264 compression and motion blur). For example, detection methods based on frequency domain analysis show a significant performance drop under low bitrate compression.

[0003] 2. Inefficient architecture design: Existing Neural Architecture Search (NAS) methods are mostly optimized for general image classification tasks, without considering the local artifacts and temporal inconsistencies unique to deepfakes. Some studies have attempted to transfer object detection frameworks such as YOLO to deepfake detection, but their search spaces lack specific design for UGC distortion.

[0004] 3. Lack of dynamic augmentation strategies: Current data augmentation methods mostly employ random combinations of basic transformations (such as rotation and pruning), which are difficult to simulate the true distortion distribution of UGC. Although some methods have proposed dynamic augmentation approaches, they are not optimized in conjunction with the NAS process, resulting in insufficient matching between the augmentation strategy and the model architecture. Summary of the Invention

[0005] This invention provides a method, apparatus, device, and storage medium for high-efficiency and high-accuracy deep forgery detection of user-generated content.

[0006] To address the aforementioned technical problems, embodiments of the present invention provide a method for detecting deepfakes in user-generated content, comprising: The raw data in the training video data is augmented using an augmentation strategy to obtain augmented data. The original data and enhanced data are input into the candidate architecture to obtain a first classification result corresponding to the original data and a second classification result corresponding to the enhanced data. The candidate architecture is selected from the search space based on the current architecture parameters, and the architecture parameters characterize the selection probability of each module in the candidate architecture. The weights and enhancement strategies of the candidate architecture are updated by combining the obtained labels of authenticity of the training video data, the first classification result, and the second classification result. New enhancement data is generated based on the updated enhancement strategy and the original data, so as to use the new enhancement data to train the candidate architecture with updated weights. Repeat the above steps until the training of the candidate architecture is completed, determine the performance evaluation value of the trained candidate architecture, and update the current architecture parameters based on the performance evaluation value; Based on the updated architecture parameters, a new candidate architecture is selected from the search space, and the new candidate architecture is trained according to the steps described above. The target architecture is determined based on the performance evaluation values ​​of multiple trained candidate architectures. The target architecture is used to detect forgery in the video data to be processed.

[0007] In one embodiment, the step of inputting the original data and augmented data into a candidate architecture to obtain a first classification result corresponding to the original data and a second classification result corresponding to the augmented data includes: The original data and enhanced data are input into the candidate architecture, which performs multi-scale feature extraction, spatiotemporal modeling, and multi-layer perception on the original data and enhanced data, respectively, to obtain the first classification result and its corresponding confidence score, as well as the second classification result and its corresponding confidence score.

[0008] In one embodiment, the original data includes image data and audio data, and the enhanced data includes enhanced image data and enhanced audio data. The step of inputting the original data and enhanced data into a candidate architecture to obtain a first classification result corresponding to the original data and a second classification result corresponding to the enhanced data includes: The image data and the enhanced image data are input into the first candidate architecture to obtain a first classification result corresponding to the image data and a second classification result corresponding to the enhanced image data; The audio data and enhanced audio data are input into the second candidate architecture to obtain a first classification result corresponding to the audio data and a second classification result corresponding to the enhanced audio data; The determination of the target architecture based on the performance evaluation values ​​of multiple trained candidate architectures includes: Based on the performance evaluation values ​​of the first and second candidate architectures after multiple training sessions, a first target architecture for processing video data and a second target architecture for processing audio data are determined. The forgery detection of the video data to be processed using the target architecture includes: Using the first target architecture and the second target architecture, forgery detection is performed on the image data and audio data in the video to be processed, and the image forgery detection results and corresponding confidence scores are obtained, as well as the audio forgery detection results and corresponding confidence scores. Multimodal fusion is performed on the image forgery detection results and their corresponding confidence scores, and the audio forgery detection results and their corresponding confidence scores to obtain the forgery detection results corresponding to the video data to be processed.

[0009] In one embodiment, updating the weights and enhancement strategy of the candidate architecture by combining the obtained labels of authenticity of the training video data, the first classification result, and the second classification result includes: The total loss is determined by combining the obtained labels of authenticity of the training video data, the first classification result, the second classification result, and the preset computational constraints. The total loss is related to the classification accuracy, the distortion sensitivity of the video data, and the computational cost of the candidate architecture. The weights and enhancement strategies of the candidate architecture are updated based on the total loss.

[0010] In one embodiment, determining the total loss by combining the obtained labels of authenticity of the video data used for training, the first classification result, the second classification result, and preset computational constraints includes: Calculate the cross-entropy loss based on the label, the first classification result, and the second classification result; The sum of squared differences between the original feature vectors of the original data extracted from the candidate architecture and the enhanced feature vectors of the enhanced data are calculated to obtain the distortion sensitivity loss. The computational cost estimate and computational cost constraint are calculated based on the candidate architecture, and the computational cost estimate is related to the structure of the candidate architecture. The total loss is obtained by weighted summation of the cross-entropy loss, distortion sensitivity loss, and computational constraint loss.

[0011] In one embodiment, updating the weights and enhancement strategies of the candidate architecture based on the total loss includes: The weights of the candidate architecture are updated based on the total loss and the gradient descent algorithm. The enhancement policy is updated based on the distortion sensitivity loss in the total loss and the policy gradient algorithm.

[0012] In one embodiment, determining the performance evaluation value of the candidate architecture after training, and updating the current architecture parameters in conjunction with the performance evaluation value, includes: The trained candidate architecture is subjected to multi-fidelity evaluation to obtain the performance evaluation value; The architecture parameters are updated by combining the performance evaluation values ​​with the policy gradient algorithm.

[0013] Another embodiment of the present invention also provides a user-generated content deepfake detection device, comprising: The processing module is used to perform augmentation processing on the raw data in the training video data based on the augmentation strategy to obtain augmented data; The input module is used to input the original data and the enhanced data into the candidate architecture to obtain a first classification result corresponding to the original data and a second classification result corresponding to the enhanced data. The candidate architecture is selected from the search space based on the current architecture parameters, and the architecture parameters represent the selection probability of each module in the candidate architecture. The update module is used to update the weights and enhancement strategies of the candidate architecture by combining the obtained labels representing the authenticity of the training video data, the first classification result, and the second classification result, and to generate new enhanced data based on the updated enhancement strategy and the original data, so as to use the new enhanced data to train the candidate architecture with updated weights. The first determining module is used to repeat the above steps until the training of the candidate architecture is completed, determine the performance evaluation value of the trained candidate architecture, and update the current architecture parameters in combination with the performance evaluation value. The selection module is used to select new candidate architectures from the search space based on the updated architecture parameters, and to train the new candidate architectures according to the above steps. The second determining module is used to determine the target architecture based on the performance evaluation values ​​of multiple trained candidate architectures. The detection module is used to perform forgery detection on the video data to be processed using the target architecture.

[0014] Another embodiment of the present invention also provides an electronic device, comprising: One or more processors; Memory, configured to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the user-generated content deepfake detection method as described in any one of the above descriptions.

[0015] Another embodiment of the present invention provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements the user-generated content deepfake detection method as described above.

[0016] Based on the above, the beneficial effects of this embodiment include constructing a search space comprising multiple candidate architectures, and training the candidate architectures using raw data and augmented data processed according to continuously changing augmentation strategies. This allows the candidate architectures to continuously improve their detection capabilities for various types of forgery, correct detection weaknesses, and ultimately determine a final performance evaluation value. Subsequently, the system selects the optimal architecture based on the performance evaluation values ​​of each candidate architecture, and uses the optimal architecture for subsequent forgery detection of video data. Because the selected optimal architecture has high detection accuracy and can effectively resist complex distortion phenomena, using the optimal architecture for forgery detection of video data has high robustness.

[0017] Other features and advantages of this application will be set forth in the following description. The objectives and other advantages of this application can be realized and obtained through the structures particularly pointed out in the written description and drawings.

[0018] The technical solution of this application will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0019] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating the user-generated content deepfake detection method in an embodiment of the present invention.

[0021] Figure 2 This is a flowchart illustrating the user-generated content deepfake detection method in an application embodiment of the present invention.

[0022] Figure 3 This is a flowchart illustrating a user-generated content deepfake detection method in another application embodiment of the present invention.

[0023] Figure 4 This is a structural block diagram of a user-generated content deepfake detection device in an embodiment of the present invention.

[0024] Figure 5 This is a schematic diagram of the structure of an electronic device in an embodiment of the present invention. Detailed Implementation

[0025] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings, but these are not intended to limit the scope of the invention.

[0026] It should be understood that various modifications can be made to the embodiments disclosed herein. Therefore, the following description should not be considered as limiting, but merely as an example of embodiments. Other modifications within the scope of this disclosure will be apparent to those skilled in the art.

[0027] The accompanying drawings, which are included in and form part of this specification, illustrate embodiments of the present disclosure and, together with the general description of the disclosure given above and the detailed description of the embodiments given below, serve to explain the principles of the disclosure.

[0028] These and other features of the invention will become apparent from the following description of preferred forms of embodiments given as non-limiting examples, with reference to the accompanying drawings.

[0029] It should also be understood that although the invention has been described with reference to some specific examples, those skilled in the art can certainly implement many other equivalent forms of the invention, which have the features described in the claims and are therefore all within the scope of protection defined herein.

[0030] The above and other aspects, features and advantages of this disclosure will become more apparent when taken in conjunction with the accompanying drawings and in view of the following detailed description.

[0031] Specific embodiments of the present disclosure are described thereafter with reference to the accompanying drawings; however, it should be understood that the disclosed embodiments are merely examples of the present disclosure and can be implemented in various ways. Well-known and / or repeated functions and structures are not described in detail to avoid unnecessary or redundant details that could obscure the present disclosure. Therefore, the specific structural and functional details disclosed herein are not intended to be limiting, but merely to serve as the basis and representative basis for the claims to teach those skilled in the art to use the present disclosure in a variety of substantially any suitable detailed structures.

[0032] This specification may use the phrases “in one embodiment,” “in another embodiment,” “in yet another embodiment,” or “in still another embodiment,” all of which may refer to one or more of the same or different embodiments according to this disclosure.

[0033] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0034] like Figure 1 As shown, this embodiment of the invention provides a method for detecting deepfakes in user-generated content, including: S1: The original data in the training video data is augmented based on the augmentation strategy to obtain augmented data; S2: Input the original data and the enhanced data into the candidate architecture to obtain a first classification result corresponding to the original data and a second classification result corresponding to the enhanced data. The candidate architecture is selected from the search space based on the current architecture parameters, and the architecture parameters represent the selection probability of each module in the candidate architecture. S3: Combine the obtained labels of authenticity of the training video data, the first classification result, and the second classification result to update the weights and enhancement strategy of the candidate architecture, and generate new enhancement data based on the updated enhancement strategy and the original data, so as to use the new enhancement data to train the candidate architecture with updated weights. S4: Repeat the above steps until the training of the candidate architecture is completed, determine the performance evaluation value of the trained candidate architecture, and update the current architecture parameters in combination with the performance evaluation value; S5: Select a new candidate architecture from the search space based on the updated architecture parameters, and train the new candidate architecture according to the above steps; S6: Determine the target architecture based on the performance evaluation values ​​of multiple trained candidate architectures; S7: Use the target architecture to perform forgery detection on the video data to be processed.

[0035] The solution in this embodiment can be applied, but is not limited to, financial risk control scenarios to identify whether users are being impersonated for financial fraud, and can also be applied to other online platforms to identify instances of impersonating other public figures to post videos, such as impersonating medical personnel to post misleading videos. For example, the system first needs to construct a search space, such as a NAS search space (NAS stands for Neural Architecture Search), which contains multiple candidate architectures. These candidate architectures can be, but are not limited to, composed of one or more of the following: a frequency domain awareness module (DCT convolutional layer), a spatial adaptation module (deformable convolution v3), and a channel attention module (improved SE block). The selection of candidate architectures can be based on architecture parameters, including the selection probability of each module. Initially, the selection probability of each module in the architecture parameters is evenly distributed. As the architecture parameters are continuously updated, the probabilities of each module in the architecture parameters will change, enabling the system to select candidate architectures that are closer to the target architecture based on the updated architecture parameters. Next, training data is prepared. In this embodiment, the system enhances the training video data according to an enhancement strategy. For example, a dynamic enhancement engine can be used to enhance the training video data, generating enhanced data with specific distortions, such as compression, blurring, and sensor noise. The enhancement strategy in this embodiment is also dynamically changing. The system determines the detection weaknesses of the candidate architecture based on its discrimination performance against the current version of enhanced data, and then adjusts the enhancement strategy to address these weaknesses, enabling the adjusted strategy to generate enhanced data corresponding to the detection weaknesses. By using this data to train the candidate architecture, the detection performance of the candidate architecture can be improved.

[0036] After preparing the training data, the system preprocesses the raw and augmented video data, such as through standardization and normalization, before sending them to candidate architectures selected from the search space. These candidate architectures analyze the raw and augmented data to obtain a first classification result for the raw data and a second classification result for the augmented data—essentially, forgery detection results for the raw and augmented data. The system then performs a comprehensive analysis based on the authenticity labels (identical labels for both sets of data) of the raw and augmented data, along with the first and second classification results. Based on this analysis, the system updates the weights of the candidate architecture and the current augmentation strategy—that is, it updates the architecture weights and augmentation strategy based on the current training round. The system then uses the updated augmentation strategy and the raw data to generate new augmented data, and uses this new augmented data to train the candidate architecture with updated weights, thus entering the next training round. Repeat the above steps until the preset number of training rounds are completed. Then, determine the performance evaluation value of the trained candidate architectures and update the current architecture parameters based on this value. The system will then select a new architecture from the search space based on the updated parameters and train that candidate architecture using the same method until the preset number of candidate architectures have been trained. At this point, the system will have the performance evaluation values ​​for each trained candidate architecture and will select the optimal architecture, i.e., the target architecture, based on these performance evaluation values ​​to process subsequent video data and determine whether the video data is forged.

[0037] The user-generated content deepfake detection method in this embodiment combines Neural Architecture Search (NAS) with enhanced perceptual feature extraction to automatically optimize a deepfake detection architecture robust to common distortions such as compression, blurring, and sensor noise. Simultaneously, the system employs a dynamic augmentation strategy, utilizing reinforcement learning to adaptively select data augmentation methods that expose model weaknesses, thereby optimizing training data and improving training accuracy. Furthermore, by combining candidate architecture parameter updates and architecture performance evaluation, training costs and architecture search costs can be significantly reduced. Therefore, the method provided in this embodiment can significantly improve the robustness of deepfake detection in user-generated content scenarios while ensuring computational efficiency.

[0038] In one embodiment, the step of inputting the original data and augmented data into a candidate architecture to obtain a first classification result corresponding to the original data and a second classification result corresponding to the augmented data includes: S201: Input the original data and enhanced data into the candidate architecture, and the candidate architecture performs multi-scale feature extraction, spatiotemporal modeling and multi-layer perception on the original data and enhanced data respectively to obtain the first classification result and the corresponding confidence score, and the second classification result and the corresponding confidence score.

[0039] For example, such as Figure 2 As shown, after receiving the raw and augmented data, the candidate architecture extracts multi-scale features from each data point, such as frequency domain features, spatial features, and channel attention features. Then, the Transformer encoder head is used to perform spatiotemporal modeling on the extracted features to capture inter-frame forgery traces. The spatiotemporal modeling data is then input into the multilayer sensing head to obtain preliminary classification results, and the corresponding confidence scores are calculated.

[0040] In this embodiment, the Transformer structure performs layer normalization before calculating multi-head attention to center and normalize the input data, making the input distribution of multi-head attention more stable, reducing internal covariate shifts, and avoiding gradient explosion or vanishing, thereby making training more stable.

[0041] Furthermore, the original data includes image data and audio data, and the enhanced data includes enhanced image data and enhanced audio data. The step of inputting the original data and enhanced data into the candidate architecture to obtain a first classification result corresponding to the original data and a second classification result corresponding to the enhanced data includes: 202: Input the image data and the enhanced image data into the first candidate architecture to obtain a first classification result corresponding to the image data and a second classification result corresponding to the enhanced image data; S203: Input the audio data and enhanced audio data into the second candidate architecture to obtain a first classification result corresponding to the audio data and a second classification result corresponding to the enhanced audio data; The determination of the target architecture based on the performance evaluation values ​​of multiple trained candidate architectures includes: S204: Based on the performance evaluation values ​​of the first candidate architecture and the second candidate architecture after multiple training, determine the first target architecture for processing video data and the second target architecture for processing audio data. The forgery detection of the video data to be processed using the target architecture includes: S701: Using the first target architecture and the second target architecture, perform forgery detection on the image data and audio data in the video to be processed, and obtain the image forgery detection result and the corresponding confidence level, and the audio forgery detection result and the corresponding confidence level; S702: Perform multimodal fusion on the image forgery detection results and corresponding confidence scores, and the audio forgery detection results and corresponding confidence scores to obtain the forgery detection results corresponding to the video data to be processed.

[0042] like Figure 3As shown, the video data in this embodiment includes image data and audio data. The system needs to select corresponding candidate architectures for the image data and audio data respectively, and then train them separately until the optimal architecture is selected, which is equivalent to... Figure 3 The image shows an enhanced perception NAS detector and an audio detector. When applying this target architecture to detect video data, image data and audio data are detected separately by their respective target architectures, yielding detection results for the image and audio data. Simultaneously, the confidence level of each detection result can be determined based on a configured confidence estimator. Finally, the system performs multimodal fusion on the image and audio detection results to obtain the final deepfake detection result for the video data.

[0043] In one embodiment, updating the weights and enhancement strategy of the candidate architecture by combining the obtained labels of authenticity of the training video data, the first classification result, and the second classification result includes: S301: Combine the obtained representation of the labels of authenticity of the training video data, the first classification result, the second classification result, and the preset computational constraints to determine the total loss. The total loss is related to the classification accuracy, the distortion sensitivity of the video data, and the computational cost of the candidate architecture. S302: Update the weights and enhancement strategies of the candidate architecture based on the total loss.

[0044] For example, based on the obtained labels and classification results, and preset computational constraints, the total loss for the current training round is calculated. In this embodiment, the total loss includes not only the loss of classification accuracy but also the loss due to the candidate architecture's sensitivity to distorted data, as well as the computational loss used to constrain the scale of the candidate architecture. This aims to obtain a target architecture that is accurate in classification, has low distortion sensitivity, and is lightweight, facilitating its deployment in different scenarios. The computational constraints can be flexibly adjusted according to different deployment needs. For example, when deployed on small computing devices such as mobile phones, the computational constraints can be reduced; if deployed on high computing devices, broader computational constraints can be set. After obtaining the total loss, the system updates the weights of the candidate architecture and the current enhancement strategy based on this total loss to prepare for the next training round.

[0045] Furthermore, the determination of the total loss by combining the obtained labels of authenticity of the video data used for training, the first classification result, the second classification result, and preset computational constraints includes: S303: Calculate the cross-entropy loss based on the label, the first classification result, and the second classification result; S304: Calculate the sum of squared differences between the original feature vectors of the original data extracted from the candidate architecture and the enhanced feature vectors of the enhanced data to obtain the distortion sensitivity loss; S305: Calculate the computational constraint loss based on the estimated computational cost and computational cost constraints of the candidate architecture, wherein the estimated computational cost is related to the structure of the candidate architecture; S306: The cross-entropy loss, distortion sensitivity loss, and computational constraint loss are weighted and summed to obtain the total loss.

[0046] In this embodiment, the loss function characterizing the accuracy of the classification result, i.e., the classification loss, is... The cross-entropy loss is determined by calculating the cross-entropy loss based on the labels, the first classification result, and the second classification result. Distortion sensitivity loss measures how sensitive the architecture is to data augmentation (distortion). Lower sensitivity indicates better robustness of the architecture. This embodiment calculates distortion sensitivity. The formula is: ; in, x The original data, x aug To enhance the data, A i As a candidate architecture, The original features of the original data, Enhanced features to enhance data.

[0047] The computational constraint loss The calculation can be determined by the difference or squared difference between the actual estimated computational cost and the computational cost constraint (such as 2G). The specific method is not unique and other forms of calculation may also be used. The computational cost constraint loss is optional; if no computational cost constraint is involved, this loss can be omitted.

[0048] After obtaining the above-mentioned losses, the system will perform a weighted summation to obtain the total loss. : .

[0049] After obtaining the total loss, the weights and enhancement strategies of the candidate architecture are updated based on the total loss, including: S307: Update the weights of the candidate architecture based on the total loss and gradient descent algorithm; S308: Update the enhancement policy based on the distortion sensitivity loss in the total loss and the policy gradient algorithm.

[0050] For example, the system can determine the weights of each parameter in the candidate architecture based on the total loss and gradient descent algorithm, and update the weights in the current candidate architecture accordingly, that is, use the backpropagation method to update the weights of the candidate architecture. Based on this, the specific parameters of the current candidate architecture can be optimized, thereby making its classification more accurate, its features more robust, and its computational cost closer to the target.

[0051] For augmentation strategies, i.e., augmentation strategy distribution This embodiment uses manual parameters. This controls the intensity of the exploration, gradually leading the enhancement strategy to select enhancement methods that expose the weaknesses of the architecture to generate enhancement data, thereby achieving targeted training of candidate architectures.

[0052] The specific update formulas include:

[0053] in, The reward function is the sensitivity to distortion of the original data. In order to enhance j The reward function for enhancing the distortion sensitivity of the data is determined by combining the distortion sensitivity loss in the total loss.

[0054] After the enhancement strategy is updated, the system can activate and apply the enhancement strategy through the strategy scheduler, so that the augmented data used in the next round of training has the specified type of enhancement.

[0055] After completing the training for a preset number of rounds, the system will determine the performance evaluation value of the candidate architecture after training, and update the current architecture parameters based on the performance evaluation value, including: S501: Perform multi-fidelity evaluation on the trained candidate architecture to obtain the performance evaluation value; S502: Update the current architecture parameters by combining the performance evaluation value with the policy gradient algorithm.

[0056] In this embodiment, after training the candidate architecture, the system performs multi-fidelity evaluation on the candidate architecture, including a two-stage verification strategy. The first stage uses a small amount of test data, such as 20% of the test data (including the original test data and augmented data), to verify the candidate architecture and obtain a first score. Then, all test data (including all types of augmented data) are used to perform fine-grained validation of the candidate architecture, resulting in a second score. Finally, the total score for the performance evaluation is calculated based on the first score and the second score. ; The These are the architecture parameters used to balance efficiency and accuracy. In other words, multi-fidelity evaluation is calculated and determined in conjunction with these architecture parameters. These architecture parameters are like the "accumulated experience" of NAS; through continuous trial and error, they determine which architecture is more suitable for UGC deepfake detection, and then use this experience to guide subsequent architecture searches, ultimately finding the optimal architecture.

[0057] Taking a compressed Deepfake video as an example, which contains compressed and distorted data, the target architecture uses a dynamic enhancement engine to add Gaussian noise during processing. After preprocessing, multi-scale features are extracted. For example, low-frequency components are extracted using a DCT convolutional layer, and high-frequency detail features are captured by a variable convolutional layer. These features are then passed to a Transformer encoder to output spatiotemporal features. Finally, based on these spatiotemporal features, a classification result representing the authenticity of the video and the corresponding confidence level are calculated.

[0058] like Figure 4 As shown, another embodiment of the present invention also provides a user-generated content deepfake detection device, comprising: The processing module is used to perform augmentation processing on the raw data in the training video data based on the augmentation strategy to obtain augmented data; The input module is used to input the original data and the enhanced data into the candidate architecture to obtain a first classification result corresponding to the original data and a second classification result corresponding to the enhanced data. The candidate architecture is selected from the search space based on the current architecture parameters, and the architecture parameters represent the selection probability of each module in the candidate architecture. The update module is used to update the weights and enhancement strategies of the candidate architecture by combining the obtained labels representing the authenticity of the training video data, the first classification result, and the second classification result, and to generate new enhanced data based on the updated enhancement strategy and the original data, so as to use the new enhanced data to train the candidate architecture with updated weights. The first determining module is used to repeat the above steps until the training of the candidate architecture is completed, determine the performance evaluation value of the trained candidate architecture, and update the current architecture parameters in combination with the performance evaluation value. The selection module is used to select new candidate architectures from the search space based on the updated architecture parameters, and to train the new candidate architectures according to the above steps. The second determining module is used to determine the target architecture based on the performance evaluation values ​​of multiple trained candidate architectures. The detection module is used to perform forgery detection on the video data to be processed using the target architecture.

[0059] In one embodiment, the step of inputting the original data and augmented data into a candidate architecture to obtain a first classification result corresponding to the original data and a second classification result corresponding to the augmented data includes: The original data and enhanced data are input into the candidate architecture, which performs multi-scale feature extraction, spatiotemporal modeling, and multi-layer perception on the original data and enhanced data, respectively, to obtain the first classification result and its corresponding confidence score, as well as the second classification result and its corresponding confidence score.

[0060] In one embodiment, the original data includes image data and audio data, and the enhanced data includes enhanced image data and enhanced audio data. The step of inputting the original data and enhanced data into a candidate architecture to obtain a first classification result corresponding to the original data and a second classification result corresponding to the enhanced data includes: The image data and the enhanced image data are input into the first candidate architecture to obtain a first classification result corresponding to the image data and a second classification result corresponding to the enhanced image data; The audio data and enhanced audio data are input into the second candidate architecture to obtain a first classification result corresponding to the audio data and a second classification result corresponding to the enhanced audio data; The determination of the target architecture based on the performance evaluation values ​​of multiple trained candidate architectures includes: Based on the performance evaluation values ​​of the first and second candidate architectures after multiple training sessions, a first target architecture for processing video data and a second target architecture for processing audio data are determined. The forgery detection of the video data to be processed using the target architecture includes: Using the first target architecture and the second target architecture, forgery detection is performed on the image data and audio data in the video to be processed, and the image forgery detection results and corresponding confidence scores are obtained, as well as the audio forgery detection results and corresponding confidence scores. Multimodal fusion is performed on the image forgery detection results and their corresponding confidence scores, and the audio forgery detection results and their corresponding confidence scores to obtain the forgery detection results corresponding to the video data to be processed.

[0061] In one embodiment, updating the weights and enhancement strategy of the candidate architecture by combining the obtained labels of authenticity of the training video data, the first classification result, and the second classification result includes: The total loss is determined by combining the obtained labels of authenticity of the training video data, the first classification result, the second classification result, and the preset computational constraints. The total loss is related to the classification accuracy, the distortion sensitivity of the video data, and the computational cost of the candidate architecture. The weights and enhancement strategies of the candidate architecture are updated based on the total loss.

[0062] In one embodiment, determining the total loss by combining the obtained labels of authenticity of the video data used for training, the first classification result, the second classification result, and preset computational constraints includes: Calculate the cross-entropy loss based on the label, the first classification result, and the second classification result; The sum of squared differences between the original feature vectors of the original data extracted from the candidate architecture and the enhanced feature vectors of the enhanced data are calculated to obtain the distortion sensitivity loss. The computational cost estimate and computational cost constraint are calculated based on the candidate architecture, and the computational cost estimate is related to the structure of the candidate architecture. The total loss is obtained by weighted summation of the cross-entropy loss, distortion sensitivity loss, and computational constraint loss.

[0063] In one embodiment, updating the weights and enhancement strategies of the candidate architecture based on the total loss includes: The weights of the candidate architecture are updated based on the total loss and the gradient descent algorithm. The enhancement policy is updated based on the distortion sensitivity loss in the total loss and the policy gradient algorithm.

[0064] In one embodiment, determining the performance evaluation value of the candidate architecture after training, and updating the current architecture parameters in conjunction with the performance evaluation value, includes: The trained candidate architecture is subjected to multi-fidelity evaluation to obtain the performance evaluation value; The architecture parameters are updated by combining the performance evaluation values ​​with the policy gradient algorithm.

[0065] like Figure 5 As shown, another embodiment of the present invention also provides an electronic device, including: One or more processors; Memory, configured to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the user-generated content deepfake detection method as described in any one of the above descriptions.

[0066] Furthermore, another embodiment of the present invention provides a storage medium having a computer program stored thereon, which is implemented when executed by a processor: The raw data in the training video data is augmented using an augmentation strategy to obtain augmented data. The original data and enhanced data are input into the candidate architecture to obtain a first classification result corresponding to the original data and a second classification result corresponding to the enhanced data. The candidate architecture is selected from the search space based on the current architecture parameters, and the architecture parameters characterize the selection probability of each module in the candidate architecture. The weights and enhancement strategies of the candidate architecture are updated by combining the obtained labels of authenticity of the training video data, the first classification result, and the second classification result. New enhancement data is generated based on the updated enhancement strategy and the original data, so as to use the new enhancement data to train the candidate architecture with updated weights. Repeat the above steps until the training of the candidate architecture is completed, determine the performance evaluation value of the trained candidate architecture, and update the current architecture parameters based on the performance evaluation value; Based on the updated architecture parameters, a new candidate architecture is selected from the search space, and the new candidate architecture is trained according to the steps described above. The target architecture is determined based on the performance evaluation values ​​of multiple trained candidate architectures. The target architecture is used to detect forgery in the video data to be processed.

[0067] It should be understood that the various solutions in this embodiment have the same technical effects as those in the above method embodiments, and will not be repeated here.

[0068] Furthermore, embodiments of the present invention also provide a computer program product, which is tangibly stored on a computer-readable medium and includes computer-readable instructions that, when executed, cause at least one processor to perform a user-generated content deepfake detection method as described in the embodiments above.

[0069] It should be noted that the computer storage medium of the present invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, system, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access storage medium (RAM), a read-only storage medium (ROM), an erasable programmable read-only storage medium (EPROM or flash memory), an optical fiber, a portable compact disk read-only storage medium (CD-ROM), an optical storage medium, a magnetic storage medium, or any suitable combination thereof. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. In the present invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program configured for use by or in connection with an instruction execution system, system, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, antenna, optical fiber, RF, etc., or any suitable combination thereof.

[0070] Furthermore, those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0071] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1One or more processes and / or boxes Figure 1 A system that specifies functions in one or more boxes.

[0072] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including an instruction set implemented in a process. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0073] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of protection of this application is limited to these examples; within the framework of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of one or more embodiments of this application as described above, which are not provided in detail for the sake of brevity.

Claims

1. A method for detecting deepfakes in user-generated content, characterized in that, include: The raw data in the training video data is augmented using an augmentation strategy to obtain augmented data. The original data and enhanced data are input into the candidate architecture to obtain a first classification result corresponding to the original data and a second classification result corresponding to the enhanced data. The candidate architecture is selected from the search space based on the current architecture parameters, and the architecture parameters characterize the selection probability of each module in the candidate architecture. The weights and enhancement strategies of the candidate architecture are updated by combining the obtained labels of authenticity of the training video data, the first classification result, and the second classification result. New enhancement data is generated based on the updated enhancement strategy and the original data, so as to use the new enhancement data to train the candidate architecture with updated weights. Repeat the above steps until the training of the candidate architecture is completed, determine the performance evaluation value of the trained candidate architecture, and update the current architecture parameters based on the performance evaluation value; Based on the updated architecture parameters, a new candidate architecture is selected from the search space, and the new candidate architecture is trained according to the steps described above. The target architecture is determined based on the performance evaluation values ​​of multiple trained candidate architectures. The target architecture is used to detect forgery in the video data to be processed.

2. The method for detecting deepfakes of user-generated content according to claim 1, characterized in that, The step of inputting the original data and enhanced data into the candidate architecture to obtain a first classification result corresponding to the original data and a second classification result corresponding to the enhanced data includes: The original data and enhanced data are input into the candidate architecture, which performs multi-scale feature extraction, spatiotemporal modeling, and multi-layer perception on the original data and enhanced data, respectively, to obtain the first classification result and its corresponding confidence score, as well as the second classification result and its corresponding confidence score.

3. The method for detecting deepfakes of user-generated content according to claim 2, characterized in that, The original data includes image data and audio data, and the enhanced data includes enhanced image data and enhanced audio data. The step of inputting the original data and enhanced data into a candidate architecture to obtain a first classification result corresponding to the original data and a second classification result corresponding to the enhanced data includes: The image data and the enhanced image data are input into the first candidate architecture to obtain a first classification result corresponding to the image data and a second classification result corresponding to the enhanced image data; The audio data and enhanced audio data are input into the second candidate architecture to obtain a first classification result corresponding to the audio data and a second classification result corresponding to the enhanced audio data; The determination of the target architecture based on the performance evaluation values ​​of multiple trained candidate architectures includes: Based on the performance evaluation values ​​of the first and second candidate architectures after multiple training sessions, a first target architecture for processing video data and a second target architecture for processing audio data are determined. The forgery detection of the video data to be processed using the target architecture includes: Using the first target architecture and the second target architecture, forgery detection is performed on the image data and audio data in the video to be processed, and the image forgery detection results and corresponding confidence scores are obtained, as well as the audio forgery detection results and corresponding confidence scores. Multimodal fusion is performed on the image forgery detection results and their corresponding confidence scores, and the audio forgery detection results and their corresponding confidence scores to obtain the forgery detection results corresponding to the video data to be processed.

4. The method for detecting deepfakes of user-generated content according to claim 1, characterized in that, The step of combining the obtained labels of authenticity of the video data used for training with the representation, the first classification result, and the second classification result to update the weights and enhancement strategies of the candidate architecture includes: The total loss is determined by combining the obtained labels of authenticity of the training video data, the first classification result, the second classification result, and the preset computational constraints. The total loss is related to the classification accuracy, the distortion sensitivity of the video data, and the computational cost of the candidate architecture. The weights and enhancement strategies of the candidate architecture are updated based on the total loss.

5. The method for detecting deepfakes of user-generated content according to claim 4, characterized in that, The determination of the total loss by combining the obtained labels of authenticity of the video data used for training with the representation, the first classification result, the second classification result, and preset computational constraints includes: Calculate the cross-entropy loss based on the label, the first classification result, and the second classification result; The sum of squared differences between the original feature vectors of the original data extracted from the candidate architecture and the enhanced feature vectors of the enhanced data are calculated to obtain the distortion sensitivity loss. The computational cost estimate and computational cost constraint are calculated based on the candidate architecture, and the computational cost estimate is related to the structure of the candidate architecture. The total loss is obtained by weighted summation of the cross-entropy loss, distortion sensitivity loss, and computational constraint loss.

6. The method for detecting deepfakes of user-generated content according to claim 4, characterized in that, The step of updating the weights and enhancement strategies of the candidate architecture based on the total loss includes: The weights of the candidate architecture are updated based on the total loss and the gradient descent algorithm. The enhancement policy is updated based on the distortion sensitivity loss in the total loss and the policy gradient algorithm.

7. The method for detecting deepfakes of user-generated content according to claim 1, characterized in that, The step of determining the performance evaluation value of the candidate architecture after training, and updating the current architecture parameters in combination with the performance evaluation value, includes: The trained candidate architecture is subjected to multi-fidelity evaluation to obtain the performance evaluation value; The architecture parameters are updated by combining the performance evaluation values ​​with the policy gradient algorithm.

8. A device for detecting deepfakes of user-generated content, characterized in that, include: The processing module is used to perform augmentation processing on the raw data in the training video data based on the augmentation strategy to obtain augmented data; The input module is used to input the original data and the enhanced data into the candidate architecture to obtain a first classification result corresponding to the original data and a second classification result corresponding to the enhanced data. The candidate architecture is selected from the search space based on the current architecture parameters, and the architecture parameters represent the selection probability of each module in the candidate architecture. The update module is used to update the weights and enhancement strategies of the candidate architecture by combining the obtained labels representing the authenticity of the training video data, the first classification result, and the second classification result, and to generate new enhanced data based on the updated enhancement strategy and the original data, so as to use the new enhanced data to train the candidate architecture with updated weights. The first determining module is used to repeat the above steps until the training of the candidate architecture is completed, determine the performance evaluation value of the trained candidate architecture, and update the current architecture parameters in combination with the performance evaluation value. The selection module is used to select new candidate architectures from the search space based on the updated architecture parameters, and to train the new candidate architectures according to the above steps. The second determining module is used to determine the target architecture based on the performance evaluation values ​​of multiple trained candidate architectures. The detection module is used to perform forgery detection on the video data to be processed using the target architecture.

9. An electronic device, characterized in that, include: One or more processors; Memory, configured to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the user-generated content deepfake detection method as described in any one of claims 1-7.

10. A storage medium having a computer program stored thereon, which, when executed by a processor, implements the user-generated content deepfake detection method as described in any one of claims 1-7.

Images