High-generalization-oriented counterfeited area self-guiding collaborative learning double-flow detection system

Abstract

The invention, which relates to the technical field of image detection, discloses a high-generalization-oriented counterfeited region self-guiding collaborative learning double-flow detection system comprising a data processing module, an attribute tag generation module, a region guiding feature extraction module, a collaborative feature discrimination module and a training test optimization module. According to the method, random image transformation is performed on a standard sample to generate a twin image pair and an initial region mask, an input generator automatically outputs three types of attribute tags including a counterfeit type, a counterfeit region mask and a tampering proportion value, and a dynamic weighting mixing strategy is adopted to synthesize an enhanced training sample according to the counterfeit type. According to the design, diversified training materials and stable supervision signals can be obtained without manual annotation, the problems that manual annotation is high in cost and limited in coverage range are solved, the adaptability to different counterfeit modes can be improved through rich training samples, and the robustness of the system is improved. And a solid foundation is laid for feature extraction and discrimination tasks.

Description

Technical Field

[0001] This invention relates to the field of image detection technology, specifically to a self-guided collaborative learning dual-stream detection system for forged regions with high generalization. Background Technology

[0002] With the rapid development of generative artificial intelligence technology, deepfake technology has expanded from early partial facial replacement to complex scenarios such as controllable facial attribute editing and semantic-driven synthesis. The forged content it generates is becoming increasingly close to real samples in terms of visual appearance and potential distribution. While it has brought innovation to fields such as content generation and virtual interaction, it has also caused serious security and trust risks in areas such as public opinion manipulation, identity theft, and evidence collection, creating an urgent need for digital content security testing.

[0003] However, existing deepfake detection technologies are mainly divided into two categories: one is based on artifact clues to mine low-level features, and the other relies on semantic models for logical reasoning. However, both methods have obvious defects. They rely on a large amount of high-quality manually labeled data, which has high labeling costs and limited coverage. They lack an adaptive mechanism for focusing on forged regions, are easily affected by irrelevant backgrounds, and are difficult to capture subtle tampering traces. Local artifacts and global semantic information are mostly shallowly fused, lacking deep interaction and insufficient integration of discriminative information. They have weak cross-domain and cross-forgery type generalization capabilities, and their performance degrades significantly when faced with forgery techniques outside the training distribution. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a self-guided collaborative learning dual-stream detection system for forged regions with high generalization. This invention generates twin image pairs and initial region masks by performing random image transformations on standard samples. The generator automatically outputs three attribute labels: forgery type, forged region mask, and tampering ratio. Based on the forgery type, a dynamic weighted mixing strategy is used to synthesize enhanced training samples and simultaneously construct multi-task self-supervised learning objectives. This design can obtain diverse training materials and stable supervision signals without manual annotation, which not only solves the problems of high cost and limited coverage of manual annotation, but also improves the adaptability to different forgery modes through rich training samples, laying a solid foundation for feature extraction and discrimination tasks.

[0005] To address the aforementioned technical problems, this invention provides the following technical solution: a self-guided collaborative learning dual-stream detection system for forged regions with high generalization capabilities, comprising:

[0006] Data processing module: integrates face detection and alignment tools, identifies face regions in input videos, extracts key points and determines bounding boxes, crops face regions and scales them uniformly to a preset size, and samples frame images of each video at fixed intervals to form standard samples and training datasets;

[0007] Attribute label generation module: Based on standard samples, it performs random image transformation to generate twin image pairs and initial region masks, and inputs them into the generator to output three types of attribute labels. According to the forgery type, it uses a multi-strategy dynamic mixing mechanism to synthesize enhanced training samples and simultaneously constructs a multi-task self-supervised learning objective.

[0008] Region-guided feature extraction module: Based on enhanced training samples, it adopts a heterogeneous dual-branch architecture, namely a local feature extraction branch and a global semantic extraction branch. It uses the forged region probability map as a spatial prior to extract high-level semantic features of the tampered region, forming a dual-stream complementary feature.

[0009] Collaborative feature discrimination module: Based on dual-stream complementary features, a shared feature space is constructed through channel cascading strategy, and counterfeit type identification, tampering ratio estimation and image-level authenticity determination are carried out in parallel. Feature depth interaction is achieved through multi-task joint optimization.

[0010] Training, testing, and optimization module: The training process is built based on a deep learning framework. The training dataset and augmented training samples are used as training data. The detection model is built based on dual-branch feature extraction and multi-task discrimination logic. Training hyperparameters and adaptive optimizers are set. The overall objective function is designed, consisting of image-level true / false binary classification loss and multi-task self-supervised learning objective. The complete structure and function of the region-guided feature extraction module are retained in both the training and testing phases.

[0011] Furthermore, in the data processing module, the face region in the input video is identified using the DLIB toolkit, 81 facial key points are extracted, and the minimum bounding box is determined; the face region is cropped and uniformly scaled to a preset size of 256×256 to form a standard sample, ensuring the consistency and effectiveness of subsequent feature extraction and detection model training; 32 frames of images are uniformly sampled for each video at fixed time intervals, and all sampled images are integrated to form a training dataset, providing training data support for the detection model.

[0012] Furthermore, in the attribute label generation module, the standard samples undergo random image transformations with perturbations in hue, brightness, saturation, and contrast to generate semantically consistent twin image pairs and corresponding initial region masks, simulating facial changes under different imaging conditions in real-world scenes. The twin image pairs and initial region masks are input into a generator based on the Xception architecture, outputting three types of attribute labels: forgery type, forged region mask, and tampering ratio. A dynamic weighted mixing enhancement strategy is selected based on the forgery type, and the twin image pairs are fused at the region level using a dynamic weighted mixing formula to synthesize enhanced training samples. Finally, a multi-task self-supervised learning objective is constructed by combining the three types of attribute labels, including forgery type discrimination loss, forgery region alignment loss, and tampering ratio regression loss. The dynamic weighted mixing formula is as follows: ,in, To enhance the training samples, The dynamic weighting coefficients are randomly selected from the list [0.25, 0.5, 0.75, 1, 1, 1]. The percentage of tampering output by the generator. The fake region mask output by the generator. For twin image pairs, This is an element-wise multiplication operation.

[0013] Furthermore, in the attribute label generation module, the multi-task self-supervised learning objective is constructed by weighted summation using the multi-task self-supervised learning objective formula. The three types of losses in this multi-task self-supervised learning objective correspond one-to-one with the three types of attribute labels output by the generator. Specifically, the forgery region alignment loss measures the difference between the predicted value of the forged region mask and the actual value of the initial region mask, and is calculated using the forgery region alignment loss formula. The tampering proportional regression loss measures the difference between the predicted value of the tampering proportional value and the corresponding reference value, and is calculated using the tampering proportional regression loss formula. Both the forgery region alignment loss and the forgery type discrimination loss use AM-Softmax loss, and the tampering proportional regression loss uses L1 distance as the difference metric.

[0014] Furthermore, in the attribute tag generation module,

[0015] The formula for the multi-task self-supervised learning objective is: By using a weighted summation method, the contributions of the three types of losses to the training of the detection model are balanced, ensuring the stability of the training process;

[0016] The formula for the alignment loss of forged regions is: It accurately quantifies the difference between the predicted and actual values ​​of the forged area mask, thereby enhancing the detection model's ability to locate forged areas.

[0017] The formula for the tampered proportional regression loss is: This reduces the bias in predicting the tampering ratio and improves the accuracy of the detection model in estimating the tampering ratio.

[0018] in, For multi-task self-supervised learning objectives, To determine the loss for forgery type discrimination, AM-Softmax loss is used, corresponding to the forgery type attribute label. To manipulate the proportional regression loss, the corresponding proportional numerical attribute label is manipulated. To simulate region alignment loss, corresponding to simulate region mask attribute labels, The weight parameters for the three types of losses are determined based on the loss convergence characteristics and detection performance feedback results of the corresponding deepfake detection historical training data, and are 0.05, 0.1, and 0.5, respectively. For the sample size, For the first Predicted values ​​of the fake region mask for each sample. For the first The actual value of the fake region mask for each sample. For the first The predicted value of the tampering rate for each sample. For the first Reference value for the tampering rate of a sample. It is an L1 norm. This represents absolute value operations.

[0019] Furthermore, in the region-guided feature extraction module, the local feature extraction branch uses an Xception network as an encoder to extract and fuse the intermediate layer features from Block 1 to Block 3, obtaining the intermediate layer fused features. These intermediate layer fused features are then subjected to 3×3 convolution and adaptive average pooling to extract local fine-grained features. Simultaneously, the intermediate layer fused features are input into a depthwise separable convolution, whose parameters are kernel size 3, stride 1, and padding 1. A spatial feature map is obtained using the spatial feature map calculation formula, then compressed through a 1×1 convolution, and finally activated by a Sigmoid activation function to pass through the fake region. The domain probability map generation formula generates a fake region probability map; the global semantic extraction branch is constructed based on the CLIP model with ViT-B / 16 or ViT-L / 14 configuration. First, the fake region probability map is mapped by 3×3 convolution. The fused features are obtained by adding the initial RGB convolution features extracted by the CLIP model in the global semantic extraction branch element by element through the fusion feature formula. Then, the features are processed by batch normalization and ReLU activation, and high-level semantic features are extracted through the attention pooling layer formula. The local fine-grained features output by the local feature extraction branch are integrated with the high-level semantic features output by the global semantic extraction branch to form a dual-stream complementary feature and output it.

[0020] Furthermore, in the region-guided feature extraction module,

[0021] The formula for calculating spatial feature maps is: This effectively extracts spatial information from the intermediate layer fusion features, laying the foundation for the subsequent generation of probabilistic maps of forged regions;

[0022] The formula for generating a fake region probability map is: The feature values ​​are mapped to the 0-1 interval to clearly mark the confidence level of suspected forgery areas;

[0023] The formula for fusion features is: This enables the effective fusion of the probability map of the forged region and the initial features of the RGB convolution, focusing on key forged regions;

[0024] The formula for attention pooling layer is: It efficiently aggregates global contextual information and enhances the ability to discriminate high-level semantic features;

[0025] in, For spatial feature maps, This is a two-dimensional convolution operation. This is a feature of intermediate layer fusion. The kernel size is [size]. Step size, For filling, To forge regional probability maps, It is the Sigmoid activation function. This is a single-channel feature map after 1×1 convolution compression. As a feature of fusion, The initial features for RGB convolution are... To fuse features after batch normalization and ReLU activation, These are the query, key, and value projection matrices for the attention pooling layer, respectively. For feature dimension, For high-level semantic features, The Softmax activation function is used. This is the matrix transpose operation.

[0026] Furthermore, in the collaborative feature discrimination module, local fine-grained features and high-level semantic features are concatenated in the channel dimension to form joint features. This concatenation operation does not change the spatial resolution of the feature map, but only superimposes the number of channels of the two types of features in the channel dimension. The number of channels of the joint features is the sum of the number of channels of the local fine-grained features and the number of channels of the high-level semantic features. The joint features are mapped to the corresponding dimensional space through a fully connected layer, and the forgery type is identified by the Softmax activation function. The joint features are output through the fully connected layer and the tampering ratio is estimated by the Sigmoid activation function. The joint features are processed by a two-layer fully connected + batch normalization + ReLU network structure, and then mapped to a two-dimensional space by a fully connected layer and the image-level authenticity determination is completed by the Softmax activation function.

[0027] Furthermore, in the training, testing, and optimization module, the training dataset serves as the basic training sample set, which, together with the augmented training samples, constitutes the training data used to detect the end-to-end parameter optimization process of the model. The training dataset provides initial real face samples and corresponding basic label information, while the augmented training samples supplement diverse types of fake samples. An end-to-end training process is built based on the PyTorch deep learning framework, with the specific training hyperparameters set as follows: batch size 32, total number of iterations 30, and initial learning rate 5×10⁻⁶. -5The adaptive optimizer uses the Adam optimizer, and the overall objective function is constructed using the system's overall objective function formula. During the testing phase, the images to be detected are processed by the data processing module to form standard samples, and then sequentially input into the attribute label generation module, the region-guided feature extraction module, and the collaborative feature discrimination module, directly outputting the forgery type, tampering ratio, and authenticity determination results.

[0028] Furthermore, in the training, testing, and optimization module, the overall objective function of the system is formulated as follows: ,in, Let the overall objective function of the system be... For image-level true / false binary classification loss, The goal is to achieve multi-task self-supervised learning.

[0029] Compared with existing technologies, this self-guided collaborative learning dual-stream detection system for highly generalizable forged regions has the following advantages:

[0030] I. This invention generates twin image pairs and initial region masks by performing random image transformations on standard samples. The generator automatically outputs three attribute labels: forgery type, forgery region mask, and tampering ratio. Based on the forgery type, a dynamic weighted mixing strategy is used to synthesize enhanced training samples and simultaneously construct multi-task self-supervised learning objectives. This design can obtain diverse training materials and stable supervision signals without manual annotation. It not only solves the problems of high cost and limited coverage of manual annotation, but also improves the adaptability to different forgery modes through rich training samples, laying a solid foundation for feature extraction and discrimination tasks.

[0031] Second, this invention extracts features through a heterogeneous dual-branch architecture. The local feature extraction branch generates a probability map of the forged region, while the global semantic extraction branch, guided by this probability map, focuses on the tampered region to extract high-level semantic features. Then, through channel cascading fusion, a shared feature space is constructed, and multi-task discrimination and joint optimization are carried out in parallel. This mechanism realizes deep synergy between local fine-grained features and global semantic features, which not only enhances the accurate perception of the forged region, but also avoids the limitations of a single feature dimension. It can comprehensively capture forgery clues, effectively improve the adaptability to complex forgery scenarios, and ensure the reliability of detection results.

[0032] Other advantages, objectives and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination or study, or may be learned from the practice of the invention. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0034] Figure 1 A flowchart for a self-guided collaborative learning dual-stream detection system for forged regions with high generalization;

[0035] Figure 2 A framework diagram for a self-guided collaborative learning dual-stream detection system for forged regions with high generalization;

[0036] Figure 3 This is a framework diagram of the region-guided feature extraction module in a self-guided collaborative learning dual-stream detection system for forged regions with high generalization. Detailed Implementation

[0037] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.

[0038] Example 1:

[0039] In the scenario of deepfake content detection on social media, social media platforms need to perform deepfake detection on user-uploaded short videos to prevent the spread of false content and misleading the public. First, the platform processes the user-uploaded short videos received by the platform. Using the DLIB toolkit, facial regions in the video are identified frame by frame, accurately extracting 81 key points of the face in each frame and determining the minimum bounding box. The cropped facial regions are then uniformly scaled to a preset size of 256×256 to form standard samples, ensuring consistency in subsequent processing. Simultaneously, 32 frames of images are uniformly sampled from the short video at fixed time intervals, and all sampled images are integrated to form a training dataset, such as... Figure 1 As shown.

[0040] Based on the obtained standard samples, random perturbation transformations of hue, brightness, saturation, and contrast are performed to generate semantically consistent twin image pairs with slight appearance differences. Simultaneously, corresponding initial region masks are generated to simulate the facial presentation effects under different environments. The twin image pairs and initial region masks are input into a generator based on the Xception architecture. The generator outputs three attribute labels: forgery type, forged region mask, and alteration ratio, accurately locating key forgery-related information. Based on the forgery type output by the generator, a dynamic weighted mixing enhancement strategy is selected. The twin image pairs are then fused at the region level using a dynamic weighted mixing formula to synthesize enhanced training samples, such as... Figure 2 As shown, to enrich the diversity of training data, the dynamic weighted mixing formula is: Simultaneously, combining the three types of attribute labels, a multi-task self-supervised learning objective is constructed using the multi-task self-supervised learning objective formula. The multi-task self-supervised learning objective formula is as follows: The loss includes forgery type discrimination loss, forgery region alignment loss, and tampering ratio regression loss. The forgery type discrimination loss uses AM-Softmax loss. The forgery region alignment loss measures the difference between the predicted value of the forged region mask and the actual value of the initial region mask. The forgery region alignment loss is calculated using the following formula: The tampered proportional regression loss measures the difference between the predicted value and the corresponding reference value of the tampered proportional figure. The tampered proportional regression loss formula is used for calculation: Both the forgery region alignment loss and the forgery type discrimination loss use AM-Softmax loss, and the tampering proportion regression loss uses L1 distance as the difference measure.

[0041] Using synthesized enhanced training samples as input, a heterogeneous dual-branch architecture is employed to extract features. The local feature extraction branch uses the Xception network as the encoder, extracting and fusing the intermediate layer features from Block 1 to Block 3 of this network to obtain the intermediate layer fused features. These intermediate layer fused features are then processed by 3×3 convolution and adaptive average pooling to extract fine-grained local features and capture subtle forgery traces on the face. Simultaneously, the intermediate layer fused features are input into a depthwise separable convolution with parameters of kernel size 3, stride 1, and padding 1. The spatial feature map is obtained using the spatial feature map calculation formula: After further compression of the channels by a 1×1 convolution, a probability map of the forged region is generated using the Sigmoid activation function and the formula for generating the probability map of the forged region. This accurately marks suspected forged regions. The formula for generating the probability map of the forged region is as follows: The global semantic extraction branch is constructed based on the ViT-B / 32 configuration of the CLIP model. First, a 3×3 convolution mapping is performed on the probability map of the forged region. Then, the fused features are obtained by adding element-wise the initial RGB convolution features extracted by the CLIP model in this branch to the fused feature formula. The fused feature formula is as follows: After batch normalization and ReLU activation, high-level semantic features are extracted by aggregating global contextual information through an attention pooling layer. These features are then combined with the overall facial structure logic to determine authenticity. The attention pooling layer formula is as follows: Integrating local fine-grained features with high-level semantic features forms a dual-stream complementary feature that takes into account both details and overall information.

[0042] In the channel dimension, complementary features from two streams are concatenated to form joint features. This concatenation operation does not change the spatial resolution of the feature map; it only superimposes the number of channels of the two types of features in the channel dimension, maximizing the retention of effective information. The joint features are mapped to the corresponding dimensional space through a fully connected layer, and the forgery type is identified by the Softmax activation function, clarifying the generation method of fake content. The joint features are output through the fully connected layer, and the tampering ratio is estimated by the Sigmoid activation function, quantifying the proportion of forged regions. A network structure of two fully connected layers + batch normalization + ReLU is used to process the joint features, and then the network is mapped to two-dimensional space through a fully connected layer. The image-level authenticity determination is completed by the Softmax activation function. The three discrimination tasks achieve feature depth interaction through multi-task joint optimization, improving detection accuracy.

[0043] An end-to-end training process was built based on the PyTorch deep learning framework. The training dataset was used as the basic training sample set, which, together with the synthesized augmented training samples, constituted the training data for end-to-end parameter optimization of the detection model. The detection model was constructed based on bi-branch feature extraction and multi-task discrimination logic. The training dataset provided initial real face samples and corresponding basic label information, while the augmented training samples supplemented diverse fake sample types to improve the generalization ability of the detection model. Training hyperparameters were set, specifically batch size 32, total number of iterations 30, and initial learning rate 5×10⁻⁶. -5 The Adam optimizer is used for parameter updates, and a total objective function is designed, consisting of an image-level binary classification loss and a multi-task self-supervised learning objective. The system's total objective function formula guides the continuous optimization of the detection model. The system's total objective function formula is as follows: During the testing phase, after the short videos to be detected are processed to form standard samples, they go through the attribute tag generation, feature extraction, and collaborative discrimination process in sequence, and directly output the forgery type, tampering ratio and authenticity judgment results. The platform intercepts or labels false content based on the results to ensure the safety of the content ecosystem. The entire training and testing process retains the complete structure and function of regional guided feature extraction.

[0044] In summary, in the scenario of short video detection on social media, for user-uploaded short videos, tools are used to identify and standardize facial regions to generate basic materials required for training. Based on standard samples, twin image pairs and initial region masks are generated. After the generator outputs three types of attribute labels, enhanced training samples are synthesized. Simultaneously, a multi-task self-supervised learning objective is constructed. Local fine-grained features and high-level semantic features are extracted separately through a dual-branch architecture, and integrated to form complementary features. A shared space is constructed through channel cascading and multi-task discrimination is carried out. The detection model is optimized in combination with training data, and the detection results are directly output during the testing phase. This effectively prevents the spread of false content and ensures the authenticity and compliance of the platform's content ecosystem.

[0045] Example 2:

[0046] In the scenario of facial verification for remote account opening on online financial platforms, in order to prevent financial fraud such as impersonation and forged facial videos for remote account opening, online financial platforms need to conduct deepfake detection on the facial verification videos submitted by users to ensure the security of account funds. First, the facial verification videos uploaded by users are processed. The DLIB toolkit is used to identify the facial regions in the video frame by frame, accurately extracting 81 key points of the face in each frame and determining the minimum bounding box. The cropped facial regions are then uniformly scaled to a preset size of 256×256 to form standard samples, ensuring that all verification facial images are in a uniform and standardized format. At the same time, 32 frames of images are uniformly sampled from the facial verification video at fixed time intervals, and all sampled images are integrated to form a training dataset.

[0047] Based on standard samples, random perturbation transformations of hue, brightness, saturation, and contrast are performed to generate semantically consistent twin image pairs with slight appearance differences. Simultaneously, a corresponding initial region mask is generated to simulate the facial presentation effects under different shooting lighting, indoor / outdoor environments, and mobile phone / computer shooting devices during user account opening. The twin image pairs and initial region masks are input into a generator based on the Xception architecture. The generator outputs three attribute labels: forgery type, forged region mask, and alteration ratio, accurately capturing potential forgery traces in the account opening video, such as face replacement, photo synthesis, and AI-generated faces. Based on the forgery type output by the generator, a dynamic weighted mixing enhancement strategy is selected. The twin image pairs are fused at the region level using a dynamic weighted mixing formula to synthesize enhanced training samples, covering various common forgery patterns in financial fraud. The dynamic weighted mixing formula is as follows: Simultaneously, combining the three types of attribute labels, a multi-task self-supervised learning objective is constructed using the multi-task self-supervised learning objective formula. The multi-task self-supervised learning objective formula is as follows: The loss includes forgery type discrimination loss, forgery region alignment loss, and tampering ratio regression loss. The forgery type discrimination loss uses AM-Softmax loss. The forgery region alignment loss measures the difference between the predicted value of the forged region mask and the actual value of the initial region mask. The forgery region alignment loss is calculated using the following formula: The tampered proportional regression loss measures the difference between the predicted value and the corresponding reference value of the tampered proportional figure. The tampered proportional regression loss formula is used for calculation: Both the forgery region alignment loss and the forgery type discrimination loss use AM-Softmax loss, and the tampering proportion regression loss uses L1 distance as the difference measure.

[0048] Using enhanced training samples as input, a heterogeneous dual-branch architecture is employed to extract features. The local feature extraction branch uses the Xception network as the encoder, extracting and fusing the intermediate layer features from Blocks 1 to 3 to obtain the intermediate layer fused features. These fused features are then processed with 3×3 convolution and adaptive average pooling to extract fine-grained local features, capturing forgery traces in details such as facial contours, skin texture, and subtle eye movements. Simultaneously, the intermediate layer fused features are input into a depthwise separable convolution with a kernel size of 3, a stride of 1, and padding of 1. The spatial feature map is then calculated using the following formula: After further channel compression via 1×1 convolution, a fake region probability map is generated using the Sigmoid activation function and the fake region probability map generation formula. This accurately marks key regions suspected of being faked. The fake region probability map generation formula is as follows: The global semantic extraction branch is built based on the ViT-L / 16 configuration of the CLIP model. First, a 3×3 convolution mapping is applied to the probability map of the forged region to match its dimension with the initial RGB convolution features extracted by the CLIP model in this branch. Then, the two are added element-wise using a fusion feature formula to obtain the fused features. The fusion feature formula is as follows: After batch normalization and ReLU activation, high-level semantic features are extracted by aggregating global contextual information through an attention pooling layer. The authenticity of the account opening video is then determined by combining logical judgments such as the dynamic coherence of the face and the synchronization between facial expressions and voice commands. The attention pooling layer formula is as follows: Integrating local fine-grained features with high-level semantic features forms a dual-stream complementary feature set, such as... Figure 3 As shown, it takes into account both detailed traces and overall logical information.

[0049] This method concatenates complementary features from two streams along the channel dimension to form joint features. This concatenation operation does not change the spatial resolution of the feature map; it only adds the number of channels of the two types of features along the channel dimension, preserving effective discriminative information. The joint features are mapped to the corresponding dimensional space through a fully connected layer, and the Softmax activation function is used to complete the forgery type identification, clarifying whether it is photo replacement, AI-synthesized face, or other forgery methods. The joint features are output through a fully connected layer, and the Sigmoid activation function is used to estimate the tampering ratio, quantify the proportion of forged areas, and assist in judging the fraud risk level. A network structure of two fully connected layers + batch normalization + ReLU is used to process the joint features, and then the fully connected layer is used to map them to a two-dimensional space. The Softmax activation function is used to complete the image-level authenticity determination. The three discrimination tasks achieve deep feature interaction through multi-task joint optimization, improving the detection accuracy in financial account opening scenarios.

[0050] An end-to-end training process was built based on the PyTorch deep learning framework. The training dataset served as the basic training sample set, which, together with synthetic augmented training samples, constituted the training data for end-to-end parameter optimization of the detection model. The detection model was constructed based on bi-branch feature extraction and multi-task discrimination logic. The training dataset provided initial real account opening face samples and corresponding basic label information, while the augmented training samples supplemented diverse types of forged account opening samples, helping the detection model adapt to various complex forgery scenarios in financial fraud. The batch size was set to 32, the total number of iterations to 30, and the initial learning rate to 5×10⁻⁶. -5 The training hyperparameters are determined, and the Adam optimizer is used for parameter updates. The overall system objective function is designed, consisting of an image-level binary classification loss and a multi-task self-supervised learning objective. The overall system objective function formula guides the continuous optimization and iteration of the model. The overall system objective function formula is: During the testing phase, the facial verification videos of users applying for account opening, after being processed to form standard samples, go through the attribute label generation, feature extraction, and collaborative discrimination processes in sequence. The system directly outputs the forgery type, tampering ratio, and authenticity determination results. Based on the results, the system verifies the legality of the user's account opening identity, prevents financial fraud by forging identities to open accounts, and protects the funds of the platform and users. The entire training and testing process retains the complete structure and function of region-guided feature extraction.

[0051] In summary, in the scenario of remote account opening on online financial platforms, the facial verification video submitted by users is processed using a unified tool to form standard samples and training datasets. Based on the standard samples, relevant image pairs and initial region masks are generated. After the generator outputs attribute labels, enhanced training samples are synthesized. Multi-task self-supervised learning objectives are constructed by combining the labels. Detail features and global semantic features are extracted and integrated through a dual-branch architecture. After fusion to construct the space, multi-task discrimination is completed. The detection model parameters are optimized based on the training data. Accurate detection results are output during the testing phase, effectively preventing identity theft and financial fraud, and ensuring the platform's compliant operation and user fund security.

[0052] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A self-guided collaborative learning dual-stream detection system for forged regions with high generalization, characterized in that: The system includes: Data processing module: integrates face detection and alignment tools, identifies face regions in input videos, extracts key points and determines bounding boxes, crops face regions and scales them uniformly to a preset size, and samples frame images of each video at fixed intervals to form standard samples and training datasets; Attribute label generation module: Based on standard samples, it performs random image transformation to generate twin image pairs and initial region masks, and inputs them into the generator to output three types of attribute labels. According to the forgery type, it uses a multi-strategy dynamic mixing mechanism to synthesize enhanced training samples and simultaneously constructs a multi-task self-supervised learning objective. Region-guided feature extraction module: Based on enhanced training samples, it adopts a heterogeneous dual-branch architecture, namely a local feature extraction branch and a global semantic extraction branch. It uses the forged region probability map as a spatial prior to extract high-level semantic features of the tampered region, forming a dual-stream complementary feature. Collaborative feature discrimination module: Based on dual-stream complementary features, a shared feature space is constructed through channel cascading strategy, and counterfeit type identification, tampering ratio estimation and image-level authenticity determination are carried out in parallel. Feature depth interaction is achieved through multi-task joint optimization. Training, testing, and optimization module: The training process is built based on a deep learning framework. The training dataset and augmented training samples are used as training data. The detection model is built based on dual-branch feature extraction and multi-task discrimination logic. Training hyperparameters and adaptive optimizers are set. The overall objective function is designed, consisting of image-level true / false binary classification loss and multi-task self-supervised learning objective. The complete structure and function of the region-guided feature extraction module are retained in both the training and testing phases.

2. The forgery region self-guided collaborative learning dual-stream detection system for high generalization as described in claim 1, characterized in that, In the data processing module, the face region in the input video is identified using the DLIB toolkit, 81 facial key points are extracted and the minimum bounding box is determined; the face region is cropped and uniformly scaled to a preset size of 256×256 to form a standard sample; 32 frames of images are uniformly sampled for each video at fixed time intervals, and all sampled images are integrated to form a training dataset.

3. The forgery region self-guided collaborative learning dual-stream detection system for high generalization as described in claim 1, characterized in that, In the attribute label generation module, the standard samples are subjected to random image transformations with perturbations in hue, brightness, saturation, and contrast to generate semantically consistent twin image pairs and corresponding initial region masks. The twin image pairs and initial region masks are input into a generator based on the Xception architecture, which outputs three types of attribute labels: forgery type, forgery region mask, and tampering ratio. A dynamic weighted mixing enhancement strategy is selected based on the forgery type. The twin image pairs are fused at the region level through the dynamic weighted mixing formula to synthesize enhanced training samples. The three types of attribute labels are combined to construct a multi-task self-supervised learning objective that includes forgery type discrimination loss, forgery region alignment loss, and tampering ratio regression loss. Its dynamic weighted mixing formula is: ,in, To enhance the training samples, The dynamic weighting coefficients are randomly selected from the list [0.25, 0.5, 0.75, 1, 1, 1]. The percentage of tampering output by the generator. The fake region mask output by the generator. For twin image pairs, This is an element-wise multiplication operation.

4. The forgery region self-guided collaborative learning dual-stream detection system for high generalization as described in claim 3, characterized in that, In the attribute label generation module, the multi-task self-supervised learning objective is constructed by weighted summation of the multi-task self-supervised learning objective formula. The three types of losses in the multi-task self-supervised learning objective correspond one-to-one with the three types of attribute labels output by the generator. Specifically, the forgery region alignment loss measures the difference between the predicted value of the forged region mask and the actual value of the initial region mask, and is calculated using the forgery region alignment loss formula. The tampering proportional regression loss measures the difference between the predicted value of the tampering proportional value and the corresponding reference value, and is calculated using the tampering proportional regression loss formula. Both the forgery region alignment loss and the forgery type discrimination loss use AM-Softmax loss, and the tampering proportional regression loss uses L1 distance as the difference metric.

5. The self-guided collaborative learning dual-stream detection system for forged regions with high generalization as described in claim 4, characterized in that, In the attribute tag generation module The formula for the multi-task self-supervised learning objective is: ; The formula for the alignment loss of forged regions is: ; The formula for the tampered proportional regression loss is: ; in, For multi-task self-supervised learning objectives, To determine the loss for forgery type discrimination, AM-Softmax loss is used, corresponding to the forgery type attribute label. To manipulate the proportional regression loss, the corresponding proportional numerical attribute label is manipulated. To simulate region alignment loss, corresponding to simulate region mask attribute labels, The weight parameters for the three types of losses are determined based on the loss convergence characteristics and detection performance feedback results of the corresponding deepfake detection historical training data, and are 0.05, 0.1, and 0.5, respectively. For the sample size, For the first Predicted values ​​of the fake region mask for each sample. For the first The actual value of the fake region mask for each sample. For the first The predicted value of the tampering rate for each sample. For the first Reference value for the tampering rate of a sample. It is an L1 norm. This represents absolute value operations.

6. The forgery region self-guided collaborative learning dual-stream detection system for high generalization as described in claim 1, characterized in that, In the region-guided feature extraction module, the local feature extraction branch uses an Xception network as the encoder to extract and fuse the intermediate layer features from Block 1 to Block 3, obtaining the intermediate layer fused features. These fused features are then subjected to 3×3 convolution and adaptive average pooling to extract fine-grained local features. Simultaneously, the intermediate layer fused features are input into a depthwise separable convolution with parameters of kernel size 3, stride 1, and padding 1. A spatial feature map is obtained using the spatial feature map calculation formula, then compressed through a 1×1 convolution, and finally activated by a Sigmoid activation function to obtain the probabilities of fake regions. The graph generation formula generates a probability map of forged regions; the global semantic extraction branch is constructed based on the CLIP model with ViT-B / 16 or ViT-L / 14 configuration. First, the probability map of forged regions is mapped by 3×3 convolution. The fused features are obtained by adding the initial RGB convolution features extracted by the CLIP model in the global semantic extraction branch element by element through the fusion feature formula. The fused features are then processed by batch normalization and ReLU activation, and then high-level semantic features are extracted through the attention pooling layer formula. The local fine-grained features output by the local feature extraction branch are integrated with the high-level semantic features output by the global semantic extraction branch to form a dual-stream complementary feature and output it.

7. The forgery region self-guided collaborative learning dual-stream detection system for high generalization as described in claim 6, characterized in that, In the region-guided feature extraction module The formula for calculating spatial feature maps is: ; The formula for generating a fake region probability map is: ; The formula for fusion features is: ; The formula for attention pooling layer is: ; in, For spatial feature maps, This is a two-dimensional convolution operation. This is a feature of intermediate layer fusion. The kernel size is [size]. Step size, For filling, To forge regional probability maps, It is the Sigmoid activation function. This is a single-channel feature map after 1×1 convolution compression. As a feature of fusion, The initial features for RGB convolution are... To fuse features after batch normalization and ReLU activation, These are the query, key, and value projection matrices for the attention pooling layer, respectively. For feature dimension, For high-level semantic features, The Softmax activation function is used. This is the matrix transpose operation.

8. The forgery region self-guided collaborative learning dual-stream detection system for high generalization as described in claim 1, characterized in that, In the collaborative feature discrimination module, local fine-grained features and high-level semantic features are concatenated along the channel dimension to form joint features. This concatenation operation does not change the spatial resolution of the feature map, but only superimposes the number of channels of the two types of features along the channel dimension. The number of channels of the joint features is the sum of the number of channels of the local fine-grained features and the number of channels of the high-level semantic features. The joint features are mapped to the corresponding dimensional space through a fully connected layer, and the forgery type is identified by the Softmax activation function. The joint features are output through the fully connected layer and the tampering ratio is estimated by the Sigmoid activation function. The joint features are processed by a two-layer fully connected + batch normalization + ReLU network structure, and then mapped to a two-dimensional space by a fully connected layer and the image-level authenticity determination is completed by the Softmax activation function.

9. The forgery region self-guided collaborative learning dual-stream detection system for high generalization as described in claim 1, characterized in that, In the training, testing, and optimization module, the training dataset serves as the basic training sample set, which, together with the augmented training samples, constitutes the training data used to detect the end-to-end parameter optimization process of the model. The training dataset provides initial real face samples and corresponding basic label information, while the augmented training samples supplement diverse types of fake samples. An end-to-end training process is built based on the PyTorch deep learning framework, with the following specific training hyperparameters: batch size 32, total number of iterations 30, and initial learning rate 5×10⁻⁶. -5 The adaptive optimizer uses the Adam optimizer, and the overall objective function is constructed using the system's overall objective function formula. During the testing phase, the images to be detected are processed by the data processing module to form standard samples, and then sequentially input into the attribute label generation module, the region-guided feature extraction module, and the collaborative feature discrimination module, directly outputting the forgery type, tampering ratio, and authenticity determination results.

10. The forgery region self-guided collaborative learning dual-stream detection system for high generalization as described in claim 9, characterized in that, In the training, testing, and optimization module, the overall objective function of the system is formulated as follows: ,in, Let the overall objective function of the system be... For image-level true / false binary classification loss, The goal is to achieve multi-task self-supervised learning.

Images