Deep face forgery detection model training method, device and equipment based on reinforcement learning

Abstract

The invention discloses a deep face forgery detection model training method and device based on reinforcement learning, and the method comprises the steps: firstly obtaining a training set, and constructing a combined state vector containing current state information and historical state information for a sample in the training set; inputting the combined state vector into a tutor agent to output a loss weight; the method comprises the following steps: calculating a loss weight of a student detection model, weighting the original loss of the student detection model according to the loss weight, updating the parameters of the student detection model by using the weighted loss, calculating a reward signal according to the performance change of the student detection model before and after the parameter updating, and updating the strategy of a tutor agent by using the reward signal. The student model is dynamically guided to pay attention to difficulty and key samples through the tutor agent, the robustness and generalization ability of the deep face forgery detection model can be remarkably improved, and therefore a more reliable detection means is provided for deep face forgery.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence security technology, and more specifically, to a method, apparatus, and device for training a deep face forgery detection model based on reinforcement learning. Background Technology

[0002] In recent years, deepfake technology, particularly the technique of "face-swapping" to create fake audio and video content, has seen explosive growth on the internet. This technology has been maliciously used to create fake news, commit identity fraud, or defame others, posing a serious threat to social security, political stability, and public trust. Therefore, developing efficient and robust automated deepfake detection models has become an urgent need in the field of cyberspace security.

[0003] Currently, deepfake detection models primarily rely on standard supervised learning paradigms for training. This traditional approach typically treats all samples in the training set equally during training, assigning each sample the same learning importance or loss weight.

[0004] However, this "one-size-fits-all" training strategy has obvious flaws. In real training datasets, there is an inevitable mix of a large number of "simple samples" (i.e., samples that the model has mastered and can easily classify correctly) and a small number of "difficult samples" (i.e., samples that the model has difficulty distinguishing, is easily confused with, or consistently makes incorrect predictions).

[0005] In the later stages of training, standard supervised learning methods often lead to the model wasting significant computational resources and optimization iterations on simple samples already mastered, while neglecting difficult samples that could truly enhance the model's discriminative ability and generalization boundaries. This results in the final trained model potentially overfitting to the surface features of simple samples, failing to learn more fundamental and robust discriminative features that distinguish between genuine and fake samples. Consequently, its detection performance and generalization ability will be significantly reduced when faced with novel and unseen forgery techniques.

[0006] While some curriculum learning methods exist in the industry that attempt to address this problem, most rely on a pre-designed, static "easy-to-difficult" order of samples. This static curriculum cannot dynamically and adaptively adjust based on the model's real-time learning state during training (e.g., which samples the model begins to "confused" or "forgot"), resulting in poor robustness and generalization ability of the model. Summary of the Invention

[0007] To address the aforementioned technical problems, this invention proposes a training method, apparatus, and device for a deep face forgery detection model based on reinforcement learning. This method can automatically assess the dynamic difficulty of each sample and intelligently and adaptively assign different training priorities to different samples, enabling the model to focus on tackling key and difficult samples, thereby significantly improving the final robustness and generalization ability of the deepfake detection model.

[0008] Specifically, the first aspect of the present invention provides a training method for a deep face forgery detection model based on reinforcement learning. The deep face forgery detection model includes a student detection model and a tutor agent. The training method includes: Obtain a training set and construct a combined state vector containing current state information and historical state information for the samples in the training set. The current state information is extracted from the samples using the student detection model, and the historical state information is obtained from the pre-built state manager. The historical state information includes at least the exponential moving average loss and the number of historical forgettings. The combined state vector is input into the mentor agent, and the mentor agent outputs the loss weights based on the combined state vector and the mentor agent's policy. Based on the loss weights, the original loss of the student detection model is weighted, and the parameters of the student detection model are updated using the weighted loss. Then, a reward signal is calculated based on the performance change of the student detection model before and after the parameter update, and the reward signal is used to update the policy of the tutor agent.

[0009] In one implementation, training samples are acquired, and a combined state vector containing current state information and historical state information is constructed for the training samples, including: Obtain the training samples and corresponding labels in the current batch, and use the student detection model and its feature extraction function for forward propagation to extract the current state information of the training samples; Obtain historical state information of training samples in the current batch from the pre-built state manager, and construct a combined state vector from the current state information and historical state information.

[0010] In one implementation, the current state information of training samples in the current batch is obtained from the state manager, and the current state information and historical state information are used to construct a combined state vector, including: Based on the index of the training sample in the current batch, query the corresponding current state information from the state manager; The current state information obtained from the query is concatenated with the extracted historical state information according to a predetermined dimension to construct a combined state vector.

[0011] In one implementation, the state manager updates during the iteration process as follows:

[0012] in, For smoothing coefficients, Let i be the current loss of sample i in the t-th iteration. Let i be the exponential moving average loss of sample i in round t-1. Let $\frac{i}{t}$ be the exponential moving average loss of sample $i$ during the $t$ iterations, used to characterize the historical learning difficulty trend of the sample during the training process.

[0013] In one implementation, a reward signal is calculated based on the performance change of the student detection model before and after parameter updates, including: Obtain the output probability of the student detection model before parameter update and the output probability after parameter update; Calculate the prediction accuracy before the update, the prediction accuracy after the update, the confidence level of the true label before the update, and the confidence level of the true label after the update based on the output probability of the student detection model before and after the parameter update. Based on the accuracy of the prediction before and after the update, the change in state is determined, and the reward signal is updated and calculated based on the change in state, the confidence level of the true label, and the confidence level of the true label after the update.

[0014] In one implementation, before acquiring the training set and constructing a combined state vector containing current and historical state information for the samples in the training set, the method further includes pre-training the tutor agent, specifically: Load a student detection model trained with supervised learning as an expert model, and use the expert model to traverse all N training samples, collecting the state vector and corresponding loss value of each training sample. Define an expert action for each sample based on a pre-defined heuristic strategy; The collected state vectors are paired with defined expert actions to form a supervised learning dataset with N data points; The policy network of the mentor agent is trained using a supervised learning dataset through behavior cloning.

[0015] In one implementation, updating the policy of the mentor agent using the reward signal includes updating the mentor agent's policy using a proximal policy optimization algorithm, wherein the total loss function used in the policy update process is:

[0016] in, Represents the total loss function. This represents the expectation over time step t. Let c1 and c2 represent the objective function for policy pruning, and c1 and c2 are the corresponding loss coefficients. This represents the mean squared error loss of the value function. The entropy represents the strategy.

[0017] in, This represents the ratio of the old to the new strategies. Indicates the current action. Indicates the current state. For the new strategy, For the old strategy, Let ε be the dominance function and ε be the clipping coefficient. For the cropping operation, The parameters represent the objective function of the strategy pruning;

[0018] in, The value of the current state, For target value.

[0019] Based on the same inventive concept, a second aspect of this invention provides a training method for a deep face forgery detection model based on reinforcement learning. The deep face forgery detection model includes a student detection model and a tutor agent. The training method includes: The state construction module is used to acquire the training set and construct a combined state vector containing current state information and historical state information for the samples in the training set. The current state information is extracted from the samples using the student detection model, and the historical state information is obtained from the pre-built state manager. The historical state information includes at least the exponential moving average loss and the number of historical forgettings. The reinforcement learning module is used to input the combined state vector into the mentor agent, and the mentor agent outputs loss weights based on the combined state vector and the mentor agent's policy. The reward and update module is used to weight the original loss of the student detection model according to the loss weight, update the parameters of the student detection model using the weighted loss, calculate the reward signal based on the performance change of the student detection model before and after the parameter update, and update the policy of the tutor agent using the reward signal.

[0020] Based on the same inventive concept, a third aspect of the present invention provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, provides the training method for a deep face forgery detection model based on reinforcement learning as described in the first aspect.

[0021] Based on the same inventive concept, a fourth aspect of the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the reinforcement learning-based deep face forgery detection model training method described in the first aspect.

[0022] Compared with the prior art, the advantages and beneficial technical effects of the present invention are as follows: (1) This invention transforms the training course from static to dynamic and adaptive. The system can dynamically adjust the training strategy according to the real-time learning status of the student model (rather than manually preset rules), which significantly improves the training efficiency and the final robustness of the model.

[0023] (2) The present invention assesses the difficulty of samples in a multidimensional and historically perceptive manner. By introducing a state manager, the system not only focuses on the current loss, but also identifies samples that have been "learned and then forgotten" (based on the number of forgettings) and samples that have been "unlearned for a long time" (based on historical average loss), thereby enabling more intelligent allocation of training resources and overcoming key difficulties.

[0024] (3) The reward mechanism of this invention provides timely and accurate feedback. By comparing the instantaneous performance changes of the student model before and after a single parameter update to generate rewards, it provides efficient and highly relevant guidance signals for the optimization of the mentor strategy, thereby accelerating the convergence speed and stability of the overall training. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a flowchart of the training method for a deep face forgery detection model based on reinforcement learning in an embodiment of the present invention; Figure 2 This is a framework diagram of the deep face forgery detection model training method based on reinforcement learning in this embodiment of the invention. Detailed Implementation

[0027] Example 1 This embodiment provides a training method for a deep face forgery detection model based on reinforcement learning. Please refer to [link to relevant documentation]. Figure 1 ,include: S1: Obtain the training set and construct a combined state vector for the samples in the training set, which includes current state information and historical state information. The current state information is extracted from the samples using the student detection model, and the historical state information is obtained from the pre-built state manager. The historical state information includes at least the exponential moving average loss and the number of historical forgettings.

[0028] Specifically, the current state information includes features, confidence levels, etc. A state manager is pre-built to maintain the historical state information of all samples, which is used to store and manage the historical state of the training set samples.

[0029] In the specific implementation process, a state manager is initialized to maintain the historical state information of N samples (i=1...N) in the training set. The historical state information includes at least the exponential moving average loss and the number of times the sample has been forgotten.

[0030] S1 can be implemented in the following way: S1.1: Obtain the training samples and corresponding labels in the current batch, use the student detection model and its feature extraction function to perform forward propagation, and extract the current state information of the training samples; S1.2: Obtain the historical state information of the training samples in the current batch from the state manager, and construct a combined state vector from the current state information and the historical state information.

[0031] In the specific implementation process, S1.1: Obtain the training sample with index i in the current batch. and corresponding tags Using student detection models ( ) and its feature extraction function ( Perform forward propagation to extract its (training samples) Current status information The current state information includes at least: sample feature vectors extracted from the intermediate layers of the model. The model's predicted probability of the sample And the model's current prediction accuracy for the samples. ,in, ,in , For feature dimension, Category 1 confidence scores extracted from , , and will Convert to one-hot encoding ,in .

[0032] S1.2 includes: S1.2.1: Query the historical state information corresponding to the sample with index i from the pre-built state manager. And perform normalization processing. At least include normalized exponential moving average loss Number of times of historical forgetting with normalization ,in, , , T is the scaling constant. In this embodiment, T is set to 10.

[0033] S1.2.2: Current status information obtained in step S1.1 Compared with the historical status information obtained in step S1.2.1 Concatenate the data according to predetermined dimensions to construct the final combined state vector. : , ,in , Combined state vector Feature dimensions, 1, 1, 1, 2 and respectively with The feature dimensions correspond to each other.

[0034] The state manager updates itself during the iteration process as follows:

[0035] For smoothing coefficients, Let i be the current loss of sample i in the t-th iteration. Let i be the exponential moving average loss of sample i in round t-1. is the exponential moving average loss of sample i in the t-th iteration process, used to characterize the historical learning difficulty trend of the sample during the training process. It filters out the oscillation noise generated by stochastic gradient descent by smoothing the current instantaneous loss and serves as the input of the state observation value for reinforcement learning.

[0036] S2: Input the combined state vector into the mentor agent, and the mentor agent outputs the loss weights based on the combined state vector and the mentor agent's policy.

[0037] Specifically, after the combined state vector is input into the mentor agent, the mentor agent outputs loss weights according to its policy. The loss weights are used to guide the weighted update of the student detection model.

[0038] S3: Based on the loss weights, the original loss of the student detection model is weighted, and the parameters of the student detection model are updated using the weighted loss. Then, the reward signal is calculated based on the performance change of the student detection model before and after the parameter update, and the reward signal is used to update the policy of the mentor agent.

[0039] Please see Figure 2 This is a framework diagram of the deep face forgery detection model training method based on reinforcement learning in this embodiment of the invention.

[0040] Specifically, the reward signal is calculated based on the performance changes of the student detection model before and after parameter updates. Afterwards, the experience Store in the buffer, and update the policy for the mentor agent. For status information, To lose weight, This is a reward signal.

[0041] The reward signal is calculated based on the performance changes of the student detection model before and after parameter updates, including: S3.1: Obtain the output probability of the student detection model before parameter update and the output probability after parameter update; S3.2: Calculate the prediction correctness before the update, the prediction correctness after the update, the confidence level of the true label before the update, and the confidence level of the true label after the update based on the output probability of the student detection model before the parameter update and the output probability after the parameter update. S3.3: Based on the correctness of the prediction before the update and the correctness of the prediction after the update, determine the change in state, and update the reward signal according to the change in state, the confidence of the true label, and the confidence of the true label after the update.

[0042] In the specific implementation process, step S3.1 involves obtaining the student detection model. Output probability before parameter update Output probability after parameter update ,in, , , For the updated model; Step S3.2: According to and Calculate the prediction accuracy before the update separately. The accuracy of the updated predictions Before updating, the actual tags were... confidence level And the updated real tags confidence level ,in, , , , ; Step S3.3: Calculate the reward for each sample based on the state changes in step S3.2. : like and (Error -> Correct) =+1; like and (Correct -> Incorrect) =-1; like and (Correct -> Correct), then = ; like and (Error -> Error), then = ; in A preset confidence level reward scaling hyperparameter is used; in this embodiment, The value is 0.5.

[0043] In one implementation, before acquiring the training set and constructing a combined state vector containing current and historical state information for the samples in the training set, the method further includes pre-training the tutor agent, specifically: Load a student detection model trained with supervised learning as an expert model, and use the expert model to traverse all N training samples, collecting the state vector and corresponding loss value of each training sample. Define an expert action for each sample based on a pre-defined heuristic strategy; The collected state vectors are paired with defined expert actions to form a supervised learning dataset with N data points; The policy network of the mentor agent is trained using a supervised learning dataset through behavior cloning.

[0044] Specifically, before performing full reinforcement learning training, the process also includes a pre-training step for the "mentor" agent, which specifically includes: Pre-training step 1: Load a "student" detection model trained using supervised learning as the "expert model". The model is then used to iterate through all N training samples, collecting the combined state vector for each sample i. (Construction method is the same as step S1) and the corresponding loss value ; Pre-training step 2: Define an "expert action" for each sample i based on a pre-defined heuristic strategy. The preset heuristic strategy is as follows: ,like ; ,like ; in The average loss of the dataset, and These are preset high and low weights; Pre-training step 3: The state vectors collected in pre-training step 1 are... and the expert actions defined in pre-training step 2 Pairing them up creates a supervised learning dataset of N data points. ; Pre-training step 4: Using the dataset from pre-training step 3 The policy network of the "mentor" agent is cloning its behavior. Supervised learning training is performed, and the optimized loss function is the mean squared error loss: In this context, N represents the sample size. For policy networks, For the collected state vectors, Expert actions as defined.

[0045] In one implementation, the policy of the mentor agent is updated using a reward signal, including updating the mentor agent's policy using a proximal policy optimization algorithm, wherein the total loss function used in the policy update process is:

[0046] in, Represents the total loss function. This represents the expectation over time step t. Let c1 and c2 represent the objective function for policy pruning, and c1 and c2 are the corresponding loss coefficients. This represents the mean squared error loss of the value function. The entropy represents the strategy.

[0047] in, This represents the ratio of the old to the new strategies. Indicates the current action. Indicates the current state. For the new strategy, For the old strategy, Let ε be the dominance function and ε be the clipping coefficient. For the cropping operation, The parameters represent the objective function of the strategy pruning;

[0048] in, The value of the current state, For target value.

[0049] To illustrate the effectiveness of the method of this invention, specific experiments are conducted below to compare the results on different fake datasets and different fake models. The experimental results show that the present invention can improve the learning effect of the model on difficult and key samples through the dynamic course guidance of the reinforcement learning tutor. Tables 1 and 2 show the performance improvement of each model after adding the reinforcement learning module. The test is based on the AUC index.

[0050] Table 1. Comparison of results on different forged datasets. "+RL" indicates the inclusion of the reinforcement learning module of this invention.

[0051] Table 2 Comparison of results on different forgery models, "+RL" indicates the addition of the reinforcement learning module of this invention.

[0052] In Tables 1 and 2, the first column on the left mentions Clip (a multimodal pre-trained model), Effort (a model optimized for the CLIP base detector), and prodet (a detection model combining traditional detectors with Clip semantic functions), which are the student models used in this invention. The first row above, FF++, DFDC, UADFV, CDFv2, Uniface, Blendface, E4s, Facedancer, Fsgan, inswap, and simswap, are open-source deepfake detection datasets used to evaluate model performance.

[0053] Example 2 Based on the same inventive concept, this embodiment discloses a training device for a deep face forgery detection model based on reinforcement learning. The deep face forgery detection model includes a student detection model and a tutor agent. The training device includes: The state construction module is used to acquire the training set and construct a combined state vector containing current state information and historical state information for the samples in the training set. The current state information is extracted from the samples using the student detection model, and the historical state information is obtained from the pre-built state manager. The historical state information includes at least the exponential moving average loss and the number of historical forgettings. The reinforcement learning module is used to input the combined state vector into the mentor agent, and the mentor agent outputs loss weights based on the combined state vector and the mentor agent's policy. The reward and update module is used to weight the original loss of the student detection model according to the loss weight, update the parameters of the student detection model using the weighted loss, calculate the reward signal based on the performance change of the student detection model before and after the parameter update, and update the policy of the tutor agent using the reward signal.

[0054] Specifically, the student detection model performs Deepfake detection tasks and provides current state information for samples; the state manager persistently tracks, manages, and provides historical dynamic difficulty information for each sample in the dataset; the state construction module combines information from the student detection model and the state manager to construct a combined state vector input to the mentor agent; the mentor agent receives the combined state vector and decides to output a dynamic loss weight; and the reward and update module handles the performance changes of the student model before and after updates and generates reward signals to update the mentor agent.

[0055] Since the apparatus in Embodiment 2 of this invention is the same apparatus used in the reinforcement learning-based deep face forgery detection model training method in Embodiment 1, those skilled in the art can understand the specific structure and variations of this apparatus based on the method described in Embodiment 1 of this invention, and therefore will not be repeated here. All apparatuses used in the method of Embodiment 1 of this invention fall within the scope of protection of this invention.

[0056] Example 3 Based on the same inventive concept, the present invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the reinforcement learning-based deep face forgery detection model training method of Embodiment 1.

[0057] Since the computer-readable storage medium described in Embodiment 3 of this invention is the same computer-readable storage medium used in implementing the deep face forgery detection model training method based on reinforcement learning in Embodiment 1 of this invention, those skilled in the art can understand the specific structure and variations of this computer-readable storage medium based on the method described in Embodiment 1 of this invention, and therefore will not be repeated here. All computer-readable storage media used in the method of Embodiment 1 of this invention fall within the scope of protection of this invention.

[0058] Example 4 Based on the same inventive concept, the present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in Embodiment 1.

[0059] Since the computer device described in Embodiment 4 of this invention is the same computer device used to implement the reinforcement learning-based deep face forgery detection model training method in Embodiment 1 of this invention, those skilled in the art can understand the specific structure and variations of this computer device based on the method described in Embodiment 1 of this invention, and therefore will not be repeated here. All computer devices used in the method of Embodiment 1 of this invention fall within the scope of protection of this invention.

[0060] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0061] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0062] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention. Clearly, those skilled in the art can make various modifications and variations to the embodiments of the invention without departing from the spirit and scope of the invention. Thus, if these modifications and variations of the embodiments of the invention fall within the scope of the claims of the invention and their equivalents, the invention also intends to include these modifications and variations.

Claims

1. A training method for a deep face forgery detection model based on reinforcement learning, characterized in that, The deep face forgery detection model includes a student detection model and a tutor agent, and the training methods include: Obtain a training set and construct a combined state vector containing current state information and historical state information for the samples in the training set. The current state information is extracted from the samples using the student detection model, and the historical state information is obtained from the pre-built state manager. The historical state information includes at least the exponential moving average loss and the number of historical forgettings. The combined state vector is input into the mentor agent, and the mentor agent outputs the loss weights based on the combined state vector and the mentor agent's policy. Based on the loss weights, the original loss of the student detection model is weighted, and the parameters of the student detection model are updated using the weighted loss. Then, a reward signal is calculated based on the performance change of the student detection model before and after the parameter update, and the reward signal is used to update the policy of the tutor agent.

2. The training method for a deep face forgery detection model based on reinforcement learning as described in claim 1, characterized in that, Obtain training samples and construct a combined state vector for the training samples, including current state information and historical state information, including: Obtain the training samples and corresponding labels in the current batch, and use the student detection model and its feature extraction function for forward propagation to extract the current state information of the training samples; Obtain historical state information of training samples in the current batch from the pre-built state manager, and construct a combined state vector from the current state information and historical state information.

3. The training method for a deep face forgery detection model based on reinforcement learning as described in claim 2, characterized in that, Obtain the current state information of the training samples in the current batch from the state manager, and construct a combined state vector from the current state information and historical state information, including: Based on the index of the training sample in the current batch, query the corresponding current state information from the state manager; The current state information obtained from the query is concatenated with the extracted historical state information according to a predetermined dimension to construct a combined state vector.

4. The training method for a deep face forgery detection model based on reinforcement learning as described in claim 2, characterized in that, The state manager updates in the following way during the iteration process: in, For smoothing coefficients, Let i be the current loss of sample i in the t-th iteration. Let i be the exponential moving average loss of sample i in round t-1. Let $\frac{i}{t}$ be the exponential moving average loss of sample $i$ during the $t$ iterations, used to characterize the historical learning difficulty trend of the sample during the training process.

5. The training method for a deep face forgery detection model based on reinforcement learning as described in claim 1, characterized in that, The reward signal is calculated based on the performance changes of the student detection model before and after parameter updates, including: Obtain the output probability of the student detection model before parameter update and the output probability after parameter update; Calculate the prediction accuracy before the update, the prediction accuracy after the update, the confidence level of the true label before the update, and the confidence level of the true label after the update based on the output probability of the student detection model before and after the parameter update. Based on the accuracy of the prediction before and after the update, the change in state is determined, and the reward signal is updated and calculated based on the change in state, the confidence level of the true label, and the confidence level of the true label after the update.

6. The training method for a deep face forgery detection model based on reinforcement learning as described in claim 1, characterized in that, Before acquiring the training set and constructing a combined state vector containing current and historical state information for the samples in the training set, the method further includes pre-training the tutor agent, specifically: Load a student detection model trained with supervised learning as an expert model, and use the expert model to traverse all N training samples, collecting the state vector and corresponding loss value of each training sample. Define an expert action for each sample based on a pre-defined heuristic strategy; The collected state vectors are paired with defined expert actions to form a supervised learning dataset with N data points; The policy network of the mentor agent is trained using a supervised learning dataset through behavior cloning.

7. The training method for a deep face forgery detection model based on reinforcement learning as described in claim 1, characterized in that, The policy of the mentor agent is updated using the reward signal, including updating the policy of the mentor agent using a proximal policy optimization algorithm, wherein the total loss function used in the policy update process is: in, Represents the total loss function. This represents the expectation over time step t. Let c1 and c2 represent the objective function for policy pruning, and c1 and c2 are the corresponding loss coefficients. This represents the mean squared error loss of the value function. The entropy of the strategy; in, This represents the ratio of the old to the new strategies. Indicates the current action. Indicates the current state. For the new strategy, This is the old strategy. Let ε be the dominance function and ε be the clipping coefficient. For the cropping operation, The parameters represent the objective function of the strategy pruning; in, The value of the current state, For target value.

8. A training method for a deep face forgery detection model based on reinforcement learning, characterized in that, The deep face forgery detection model includes a student detection model and a tutor agent, and the training methods include: The state construction module is used to acquire the training set and construct a combined state vector containing current state information and historical state information for the samples in the training set. The current state information is extracted from the samples using the student detection model, and the historical state information is obtained from the pre-built state manager. The historical state information includes at least the exponential moving average loss and the number of historical forgettings. The reinforcement learning module is used to input the combined state vector into the mentor agent, and the mentor agent outputs loss weights based on the combined state vector and the mentor agent's policy. The reward and update module is used to weight the original loss of the student detection model according to the loss weight, update the parameters of the student detection model using the weighted loss, calculate the reward signal based on the performance change of the student detection model before and after the parameter update, and update the policy of the tutor agent using the reward signal.

9. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the deep face forgery detection model training method based on reinforcement learning as described in any one of claims 1 to 7.

10. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the deep face forgery detection model training method based on reinforcement learning as described in any one of claims 1 to 7.

Images