Face-swapped video detection method and system based on overall forgery traces and local detail information extraction

Abstract

The present invention discloses a face-swap video detection method and system based on the extraction of overall forgery traces and local detail information. The method comprises the following steps: video preprocessing, generating a self-fused enhanced image based on cropping faces and facial key points, submitting the compressed image and the original image to a comparative deep supervision framework for comparative learning, intercepting the intermediate features of the backbone network to extract regions with obvious forgery traces, compressing the regions, submitting the compressed region images and the original regions to a comparative deep supervision framework for comparative learning, and using a combined loss function of overall classification loss, overall comparative deep supervision loss, regional classification loss, regional comparative deep supervision loss, proposal constraint loss, and structural modeling loss to supervise the training of the network model. The present invention effectively integrates low-level features and high-level semantic features, as well as global and local features, achieving good detection accuracy across different databases and exhibiting good domain generalization and versatility.

Description

Technical Field

[0001] The present invention relates to the technical field of multimedia information forensics, and in particular to a face-swapped video detection method and system based on extraction of overall forgery traces and local detail information. Background Art

[0002] In the field of face-swapped video detection, most detection algorithms achieve good detection results within a database, but poor cross-database detection. Numerous face-swapped video detection algorithms have been proposed to address this issue, but most of these algorithms improve their generalization performance by encouraging the network to learn globally invariant features, while ignoring the importance of the network's ability to observe details. Furthermore, while some algorithms currently propose using auxiliary supervision detection to locate local tampering at the pixel level, thereby encouraging the model to focus on local areas, these algorithms still require additional masks or bounding boxes for auxiliary supervision training of the network, and obtaining these additional labels is time-consuming and labor-intensive. Furthermore, most current network architectures only provide supervision at the last layer of the neural network. This design not only fails to fully utilize shallow features, but also makes it difficult to effectively optimize intermediate network layers.

[0003] Recently, the introduction of deep supervision has provided a possibility to solve this problem. This method classifies low-level features by adding auxiliary classifiers in the middle layer of the neural network. However, it is worth noting that these auxiliary classifiers require high-level semantic features rather than low-level features, which may lead to certain limitations in information transmission in the network, especially when dealing with complex tasks and capturing abstract features. For the task of face forgery detection, how to reasonably utilize low-level features and high-level semantic features, as well as global features and local features, is still worthy of in-depth exploration. The new network architecture needs to be able to integrate these features more effectively to improve model performance and enhance the generalization ability of different forgery methods. Summary of the Invention

[0004] In order to overcome the defects and shortcomings of the existing technology, the present invention provides a face-swap video detection method and system based on the extraction of overall forgery traces and local detail information. The present invention is based on the enhanced image obtained by self-fusion of multiple masks, target faces and source faces, thereby creating more generalized, more difficult to identify high-quality fake samples to encourage the model to learn more robust feature representations; a contrastive deep supervision framework is established, and a corresponding adaptive mapping head is designed to encourage the network model to learn the enhanced invariance of compressed data, so that the network model still performs well when dealing with detection samples with high compression ratios and has relatively excellent robustness; based on the face authenticity label and the prediction probability matrix, the proposal constraint loss is calculated, and the authenticity label is used alone to extract the areas with obvious forgery traces, thereby enhancing the model's ability to observe details and mine detailed features. The structural modeling loss is calculated to promote the model to perceive contextual information to capture the structural interaction information between local semantics, obtain image patches for local feature extraction, and calculate the regional classification loss to encourage the network model to observe details and mine detailed features, thereby improving the generalization performance of the model. The present invention has excellent detection performance in both within-library and cross-library tests.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] The present invention provides a face-swapped video detection method based on extraction of overall forgery traces and local detail information, comprising the following steps:

[0007] Divide the video to be detected into frames, extract facial key points and crop the face image from the video frame;

[0008] According to the input image label, the image is judged to be true or false. If the image is judged to be true, a diversified mask self-fusion enhancement operation is performed according to the set probability to obtain the input image. If the image is judged to be false, it is directly used as the input image. The diversified mask self-fusion enhancement operation specifically includes: combining different key points to obtain different point sets, calculating the convex hull of each point set, obtaining multiple masks based on the convex hull of each point set, obtaining the target face and the source face, and obtaining the enhanced image based on the self-fusion of the mask, the target face and the source face as the input image;

[0009] Build a backbone network and a compressed feature extraction network for image feature extraction. Based on the backbone network, extract features at different levels of the input image, calculate the true and false prediction probabilities, and calculate the overall classification loss based on the true and false prediction probabilities and true and false labels.

[0010] Compress the input image, extract features at different levels of the compressed image based on the compression feature extraction network, map the features at each level of the uncompressed image to a first feature vector via the corresponding mapping head, and map the features at each level of the compressed image to a second feature vector via the corresponding mapping head, and calculate the overall contrast deep supervision loss between the first feature vector and the second feature vector;

[0011] Truncating the backbone network to obtain the intermediate feature map, calculating the prediction probability matrix, obtaining the receptive field area coordinates of the original image corresponding to the element point with the largest prediction probability value, cropping the original image based on the receptive field area coordinates to obtain an image patch;

[0012] Calculate the proposal constraint loss based on the face authenticity label and the predicted probability matrix, obtain the spatial attention map corresponding to the intermediate feature map, and use the maximum value point in the spatial self-attention map as the extreme point to calculate the structural modeling loss;

[0013] Based on the backbone network, the local features of the image patches at different levels are extracted to obtain the true and false prediction probabilities and calculate the regional classification loss.

[0014] Performing a compression operation on the image patch, extracting local features of different levels on the compressed image patch based on a compression feature extraction network, mapping the features of each level of the uncompressed image patch into a third eigenvector via a corresponding mapping head, mapping the features of each level of the compressed image patch into a fourth eigenvector via a corresponding mapping head, and calculating the regional contrast deep supervision loss between the third eigenvector and the fourth eigenvector;

[0015] The total loss is obtained by combining the overall classification loss, overall contrastive deep supervision loss, proposal constraint loss, structure modeling loss, regional classification loss, and regional contrastive deep supervision loss;

[0016] The backbone network is trained based on the total loss, the binary classification prediction probability is obtained based on the trained backbone network, and the prediction result is output.

[0017] As a preferred technical solution, the enhanced image obtained by self-fusion of the mask, the target face, and the source face is used as the input image, specifically comprising: randomly generating a probability value and setting a probability threshold; if the probability value is determined to be less than or equal to the probability threshold, the original face is subjected to hue conversion, saturation adjustment, brightness conversion, contrast ratio conversion, color conversion, and image sharpening operations in sequence to obtain the target face, while the original face is directly used as the source face;

[0018] If the probability value is greater than the probability threshold, the original face is sequentially subjected to hue transformation, saturation adjustment, brightness transformation, contrast ratio transformation, color transformation, image sharpening, affine transformation, translation, and scaling operations to obtain the source face, and the original face is directly used as the target face;

[0019] The mask is sequentially subjected to affine transformation, elastic transformation, random scaling, Gaussian blur, and normalization to obtain a blurred mask;

[0020] The enhanced image is represented as:

[0021] I blend =MB×I source ×ratio+(1-MB)×I target

[0022] Among them, I blend represents the enhanced image, MB represents the blurred mask, ratio represents the fusion ratio, I source represents the source face, I target Indicates the target face.

[0023] As an optimal technical solution, the backbone network and compressed feature extraction network for image feature extraction are constructed, using EfficientNet-b4 as the backbone network and EfficientNet-b4 as the compressed feature extraction network, and the compressed feature extraction network shares weights with the backbone network.

[0024] As a preferred technical solution, the overall classification loss is calculated based on the true and false prediction probabilities and true and false labels, which is specifically expressed as:

[0025]

[0026] Among them, p i Represents a predicted probability value in a batch B, y i Indicates the value of the true label, p i (y i ) indicates the prediction is label y i probability.

[0027] As a preferred technical solution, the calculation of the overall contrast depth supervision loss between the first eigenvector and the second eigenvector specifically includes:

[0028]

[0029] Among them, z i,j represents the first eigenvector, z i+N,j represents the second eigenvector, j represents different levels, i represents the image number in a batch, N represents the batch size, and M represents the number of mapping heads. Represents the indicator function. If k≠i, the value of the indicator function is 1, otherwise it is 0. τ is the temperature coefficient, weight j Represents the weights of the contrast loss function of different layers;

[0030] And / or, calculating the regional contrast depth supervision loss between the third eigenvector and the fourth eigenvector specifically includes:

[0031]

[0032] Among them, zp i,j represents the third eigenvector, zp i+N,j represents the fourth eigenvector.

[0033] As a preferred technical solution, obtaining the image patch specifically includes:

[0034] According to the face authenticity label, the predicted probability value of the corresponding channel in the predicted probability matrix is ​​obtained, and the coordinates corresponding to the element point with the largest predicted probability value are obtained. Based on the coordinates, the receptive field area of ​​the original image corresponding to the pixel point is obtained in the receptive field area storage dictionary, and the original image is cropped according to the coordinates of the receptive field area of ​​the original image to obtain an image patch.

[0035] As a preferred technical solution, the proposal constraint loss is calculated based on the face authenticity label and the prediction probability matrix, specifically including:

[0036] Downsample the channel dimension of the features of the intermediate feature map to obtain the prediction probability matrix;

[0037] Expand the face authenticity label into a label matrix with the same length and width;

[0038] The proposal constraint loss is expressed as:

[0039]

[0040] Among them, p i Represents the value of a certain element of the predicted probability matrix P, y i is the value of the corresponding position in the label matrix Y, p i (y i ) indicates the prediction is label y i probability.

[0041] As a preferred technical solution, the calculation of structural modeling loss specifically includes:

[0042] The spatial self-attention map generates corresponding polar diameter and polar angle through a polar coordinate predictor;

[0043] Get the coordinates corresponding to the maximum value of the spatial attention map, and obtain the coordinate prediction map based on the coordinates;

[0044] The coordinate prediction graph is passed through the polar diameter predictor to obtain the predicted polar diameter, and the polar diameter loss is calculated, which is specifically expressed as:

[0045]

[0046]

[0047] (x0,y0)=argmax x,y (SA)

[0048] Among them, i∈[1,N],j∈[1,N], N is the size of the spatial attention map, ρ ij Indicates the polar radius of the (i, j) position, SA ij Represents the attention value of position (i, j) in the spatial attention map, ρ′ ij Indicates the predicted polar radius at position (i, j), (x o ,y0) represents the coordinate corresponding to the maximum value of the spatial attention map, L distance represents the pole diameter loss;

[0049] The coordinate prediction graph is passed through the polar angle predictor to obtain the predicted polar angle, and the polar angle loss is calculated, which is specifically expressed as:

[0050]

[0051]

[0052]

[0053] Among them, θ ij represents the polar angle of the position (i, j), represents the predicted polar angle of position (i, j);

[0054] The structural modeling loss is obtained by adding the polar radius loss and the polar angle loss, which is specifically expressed as:

[0055] L sm =L distance +L angle

[0056] Among them, L sm represents the structural modeling loss.

[0057] As a preferred technical solution, the calculation region classification loss is specifically expressed as:

[0058]

[0059] Among them, p i Represents a predicted probability value in a batch B, y i Indicates the value of the true label, p i (y i ) indicates the prediction is label y i probability.

[0060] The present invention also provides a face-swap video detection system based on overall forgery traces and local detail information extraction, comprising: a video preprocessing module, a diversified mask self-fusion enhancement module, a feature extraction network construction module, a feature extraction module, an overall classification loss calculation module, an image compression enhancement module, a feature mapping module, an overall contrast depth supervision loss calculation module, an image patch acquisition module, a region proposal module, a region classification loss calculation module, a region contrast depth supervision loss calculation module, a total loss construction module, a training module, and a prediction module;

[0061] The video preprocessing module is used to divide the video to be detected into frames, extract facial key points and crop facial images from the video frames;

[0062] The diversified mask self-fusion enhancement module is used to perform diversified mask self-fusion enhancement operations on real faces according to set probabilities, specifically including: combining different key points to obtain different point sets, calculating the convex hull of each point set, obtaining multiple masks based on the convex hull of each point set, obtaining a target face and a source face, and using the enhanced image obtained by self-fusion of the masks, target face and source face as the input image;

[0063] The feature extraction network construction module is used to construct a backbone network for image feature extraction and a compression feature extraction network;

[0064] The feature extraction module is used to extract features of different levels from the input image based on the backbone network;

[0065] The overall classification loss calculation module is used to calculate the overall classification loss based on the true and false prediction probabilities and the true and false labels;

[0066] The image compression enhancement module is used to perform compression operation on the input image;

[0067] The feature extraction module is used to extract features of different levels from the compressed image based on the compression feature extraction network;

[0068] The feature mapping module is used to map the features of each level of the uncompressed image into a first feature vector through a corresponding mapping head, and map the features of each level of the compressed image into a second feature vector through a corresponding mapping head;

[0069] The overall contrast depth supervision loss calculation module is used to calculate the overall contrast depth supervision loss between the first eigenvector and the second eigenvector;

[0070] The image patch acquisition module is used to obtain image patches, truncate the backbone network to obtain intermediate feature maps, calculate the prediction probability matrix, obtain the coordinates of the receptive field area of ​​the original image corresponding to the element point with the largest prediction probability value, and crop the original image based on the coordinates to obtain the image patch;

[0071] The region proposal module is used to calculate the proposal constraint loss based on the face authenticity label and the prediction probability matrix, obtain the spatial attention map corresponding to the intermediate feature map, and calculate the structural modeling loss with the maximum point in the spatial self-attention map as the extreme point;

[0072] The feature extraction module is used to extract local features of different levels from image patches based on the backbone network to obtain true or false prediction probabilities;

[0073] The regional classification loss calculation module is used to calculate the regional classification loss based on the true and false prediction probabilities;

[0074] The image compression enhancement module is used to perform compression operations on image patches;

[0075] The feature extraction module is used to extract local features of different levels on the compressed image patch based on the compression feature extraction network;

[0076] The feature mapping module is used to map the features of each level of the uncompressed image patch into a third feature vector via a corresponding mapping head, and map the features of each level of the compressed image patch into a fourth feature vector via a corresponding mapping head;

[0077] The regional contrast depth supervision loss calculation module is used to calculate the regional contrast depth supervision loss between the third eigenvector and the fourth eigenvector;

[0078] The total loss building module is used to combine the overall classification loss, the overall contrast deep supervision loss, the proposal constraint loss, the structure modeling loss, the regional classification loss, and the regional contrast deep supervision loss to obtain the total loss;

[0079] The training module is used to train the backbone network based on the total loss;

[0080] The prediction module is used to obtain the binary classification prediction probability based on the trained backbone network and output the prediction result.

[0081] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0082] (1) The present invention performs diversified mask self-fusion enhancement operations on real faces according to set probabilities, and obtains an enhanced image based on the self-fusion of multiple masks, target faces and source faces. Unlike traditional image enhancement methods, self-fusion enhancement does not simply transform the image, but generates new fake samples, thereby creating more general and more difficult to identify high-quality fake samples to encourage the model to learn more robust feature representations.

[0083] (2) The present invention adopts a contrastive deep supervision framework to encourage the network model to learn the data after JPEG compression and WEBP compression, so that the algorithm model can still perform well when dealing with detection samples with high compression rates.

[0084] (3) The present invention calculates the proposal constraint loss based on the authenticity label of the face and the prediction probability matrix, and extracts the area with obvious forgery traces only with the help of the authenticity label, thereby enhancing the model's ability to observe details and mine detailed features, calculating the structural modeling loss, promoting the model to perceive contextual information, so as to capture the structural interaction information between local semantics, and obtain the image patch with the richest authenticity discrimination information to extract local features, and calculating the regional classification loss, thereby encouraging the network model to observe details and mine detailed features, thereby improving the generalization performance of the model. The present invention has excellent detection performance in both intra-library and cross-library tests. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] Figure 1 Schematic diagram of the process of face-swapped video detection method based on overall forgery traces and local detail information extraction of the present invention;

[0086] Figure 2 Schematic diagram of the process of obtaining an enhanced image based on self-fusion of diversified masks in the present invention;

[0087] Figure 3 This is a schematic diagram of the network structure of the mapping header of the present invention;

[0088] Figure 4 Schematic diagram of the process for calculating the proposal constraint loss and structural modeling loss for the present invention;

[0089] Figure 5 Schematic diagram of the architecture of the face-swap video detection system based on the extraction of overall forgery traces and local detail information of the present invention. DETAILED DESCRIPTION

[0090] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0091] Example 1

[0092] This embodiment is trained on FaceForensics++ (FF++) (C23) and tested on CelebDF, DeepfakeDetection (DFD) (C23, C40), Deepfake Detection Challenge (DFDC), and FF++ (C0, C23, C40). The FaceForensics++ (FF++) database uses the H.264 encoder to synthesize video data with compression rates of C0, C23, and C40 respectively. The database is composed of 1,000 original videos on YouTube, forged videos generated using four tampering techniques (Deepfake, Face2Face, FaceSwap, NeuralTextures), and 1,000 original videos. The DFD database contains 363 real videos and 3,068 face-swapped videos, which are also post-processed using the C0, C23, and C40 methods. DFDC contains 128,154 video clips, and uses 8 forgery methods to generate forged videos. Celeb-DF contains 590 real videos and 5639 fake videos. This example was conducted on an Ubuntu system and was mainly implemented based on the deep learning framework Pytorch 1.9.1, CUDA version 10.2, and cudnn version 7.6.0.

[0093] like Figure 1 As shown, this embodiment provides a face-swapped video detection method based on overall forgery traces and local detail information extraction, including the following steps:

[0094] S1: Divide the video to be detected into frames, use Dlib to extract 81 facial key points, use RetinaFace to extract the face frame set, and crop the face image from the video frame;

[0095] In this example, the FF++ database was divided into training, validation, and test sets in a ratio of 7:2:1. Since a portion of real faces were used to generate self-fused fake faces during the experiment, to ensure a balance between positive and negative samples, the ratio of real video frames to fake frames was maintained at approximately 6:4 during the video segmentation process.

[0096] In this embodiment, Dlib is used to extract 81 key points of the face, and then the minimum bounding rectangle is calculated based on the key point set. Then, RetinaFace is used to extract a set of face frames. The face frame with the largest intersection-union ratio with the minimum bounding rectangle is selected as the final face rectangle. The face rectangle is then enlarged by 1.125 times. Finally, the face image is cropped from the video frame based on the face rectangle and resampled to a resolution of 380×380 as the face image input for the model of the present invention. The interpolation method is bilinear interpolation.

[0097] S2: Determine whether the face is a real face based on the authenticity label. If the face is a real face, perform a diversified mask self-fusion enhancement operation on the real face according to the probability p to obtain the corresponding input data. If the face is determined to be fake, directly use it as the input image. The probability p in this embodiment is preferably True face according to The probability of synthesizing fake faces can ensure that the ratio of real and fake face samples is equal during training. The process of self-fusion enhancement to generate fake faces is as follows: Figure 2 As shown in the figure, the obtained face image and 81 key points of the face are enhanced by self-fusion of diversified masks to generate an enhanced image, which specifically includes:

[0098] S21: According to the facial key points, different key points are combined to obtain different point sets, and the convex hull of each point set is calculated to obtain multiple masks. Subsequently, affine transformation, elastic transformation, random scaling, Gaussian blur, normalization and other operations are performed in sequence to obtain a blurred mask MB;

[0099] In this embodiment, different facial key points are combined according to the facial key points to obtain different point sets, including: 81 facial key point sets, eye and lower face key point sets, chin and mouth key point sets, and eye and nose area key point sets;

[0100] S22: Randomly generate a probability value T. If T≤0.5, the original face is subjected to hue conversion, saturation adjustment, brightness conversion, contrast ratio conversion, color conversion, image sharpening and other operations to obtain the target face I. target , while directly taking the original face as the source face I source ;

[0101] S23: If T>0.5, the original face is subjected to hue transformation, saturation adjustment, brightness transformation, contrast ratio transformation, color transformation, image sharpening, affine transformation, translation, scaling and other operations in sequence to obtain the source face I source , and directly use the original face as the target face I target ;

[0102] S24: Obtain a self-fused face image according to the mask, the target face, and the source face as an enhanced image, which is specifically expressed as:

[0103] I blend =MB×I source ×ratio+(1-MB)×I target

[0104] Wherein, ratio is the fusion ratio. In this embodiment, the value of ratio is randomly selected from [0.25, 0.5, 0.75, 1].

[0105] S3: Send the input data into the backbone network for feature extraction at different levels, and send the features at the end of the network into the global pooling layer and the fully connected layer to obtain the true and false prediction probabilities, and calculate the overall classification loss L global_c;s ;

[0106] In this embodiment, EfficientNet-b4 is used as the backbone network, and features at four different levels are intercepted, with sizes of 32×95×95, 114×24×24, 272×12×12, and 448×12×12, respectively. The overall classification loss calculation formula is specifically expressed as:

[0107]

[0108] Among them, p i Represents a predicted probability value in a batch B, y i Indicates the value of the true label, p i (y i ) indicates the prediction is label y i probability;

[0109] S4: compress the input data to obtain a compressed image;

[0110] In this embodiment, the compression operation includes JPEG compression (the lowest quality factor is 60 and the highest quality factor is 100) and WEBP compression (the lowest quality factor is 70 and the highest quality factor is 100), where the probability of JPEG compression being adopted is 0.7 and the probability of WEBP compression being adopted is 0.3, and either JPEG compression or WEBP compression is randomly selected to perform compression operations on the input image.

[0111] S5: The compressed image is fed into the compression feature extraction network that shares weights with the backbone network to extract features at different levels.

[0112] In this embodiment, EfficientNet-b4 is used as the compressed feature extraction network. The weights of this network are shared with the weights of the backbone network. The compressed feature extraction network also intercepts features at four different levels, whose sizes are 32×95×95, 114×24×24, 272×12×12, and 448×12×12, respectively.

[0113] S6: Send the features of each level corresponding to the uncompressed image and the features of each level corresponding to the compressed image into the mapping head based on the convolutional neural network and map them into feature vectors z i,j and z i+N,j, where j represents different levels, i represents the image number in a batch, and the size of the batch is N. After applying compression enhancement, there are 2N images in total, where the i-th image and its corresponding compression enhanced image are i+N.

[0114] Eigenvector z i,j With z i+N,j have the same dimension, and then calculate the overall contrastive deep supervision loss L between the feature vectors global_contra ;

[0115] In this embodiment, four mapping heads are used to perform mapping operations on features at four different levels, such as Figure 3 As shown in the figure, the mapping head corresponding to the first layer of features contains 3 Basic Module layers and 1 global average pooling layer, which maps the 32×95×95 feature vector to a one-dimensional 448 feature vector; the mapping head corresponding to the second layer of features contains 2 BasicModule layers and 1 global average pooling layer, which maps the 114×24×24 feature vector to a one-dimensional 448 feature vector; the mapping head corresponding to the third layer of features contains 1 Basic Module layer and 1 global average pooling layer, which maps the 272×12×12 feature vector to a one-dimensional 448 feature vector; the mapping head corresponding to the last layer of features contains only 1 global average pooling layer, which maps the 448×12×12 feature vector to a one-dimensional 448 feature vector. The Basic Module layer contains 1 convolution layer with a convolution kernel size of 3, 1 convolution layer with a convolution kernel size of 1, 1 batch normalization layer, 1 ReLu layer, 1 convolution layer with a convolution kernel size of 3, 1 convolution layer with a convolution kernel size of 1, 1 batch normalization layer, and 1 ReLu layer. The overall comparison depth supervision loss calculation formula is specifically expressed as:

[0116]

[0117] Among them, i represents the i-th image in a batch of images, the batch size is N, and there are a total of 2N images including the compressed and enhanced images. If the i-th image of the input is represented by x i , then the corresponding image after compression enhancement operation is x i+N . z i,j is x i The j-th layer mapping features obtained after passing through the backbone network and the mapping head. i+N,j is x i+N The j-th layer mapping features obtained after the compressed feature extraction network and mapping head. is an indicator function. If k≠i, the value of the indicator function is 1, otherwise it is 0. τ is the temperature coefficient. In this embodiment, the value is preferably 0.07. M is the number of mapping heads. In this embodiment, its value is 4. j It represents the weight of the contrast loss function of different layers. The shallow layer has a larger weight than the deep layer. In this embodiment, its values ​​are 0.5, 0.3, 0.1, and 0.1 respectively.

[0118] S7: As Figure 4 As shown, the backbone network is truncated to obtain the intermediate feature map, through which the image patch with the richest true and false discrimination information is extracted, and the proposal constraint loss L is calculated. PCL and structural modeling loss L SM , specifically including:

[0119] The intermediate feature map F is obtained by truncating the backbone network. Its dimension is 56×48×48. The intermediate feature map F is sent to the convolutional neural network layer (Conv Layer 1) to downsample the channel dimension of the feature to obtain a prediction probability matrix P with a dimension of 2×48×48. Conv Layer 1 consists of 1 convolution layer with a convolution kernel size of 3, 1 batch normalization layer, 1 ReLu layer, 1 convolution layer with a convolution kernel size of 3, 1 batch normalization layer, 1 ReLu layer, and 1 convolution layer with a convolution kernel size of 3.

[0120] The face authenticity label is expanded into a label matrix Y with the same length and width. If the input face is real, its corresponding label is 1, and the values ​​of the label matrix are all 1. The proposal constraint loss is obtained through the loss function calculation formula. The specific calculation formula is:

[0121]

[0122] Among them, p i Represents the value of an element in the predicted probability matrix, y i is the value of the corresponding position in the label matrix, p i (y i ) indicates the prediction is label y i probability;

[0123] According to the true or false label 0 or 1, obtain the predicted probability value of the corresponding channel in the predicted probability matrix, and obtain the coordinates (x, y) corresponding to the element point with the largest predicted probability value;

[0124] According to the coordinates, the receptive field area of ​​the original image corresponding to the pixel point is obtained from the receptive field area storage dictionary, and the original image is cropped according to the coordinates of the receptive field area of ​​the original image to obtain an image patch with the richest true and false discrimination information.

[0125] In this embodiment, the 2 in the prediction probability matrix (2×48×48) represents two probability matrices predicted as true and false, respectively. In the subsequent steps, the prediction probability value of the corresponding channel in the prediction probability matrix is ​​obtained according to the true or false label 0 or 1; if the true label is 1, the prediction probability matrix (48×48) of the first channel dimension is obtained, and then the pixel with the largest prediction value in the prediction probability matrix is ​​obtained. The largest prediction value indicates that the pixel point has the greatest contribution to the correctness of the final prediction result, which also indicates that the image patch of the original image corresponding to the pixel point is the most critical for the true or false discrimination of the image, so the original image receptive field area (image patch) corresponding to the pixel point is also the image patch with the richest true or false discrimination information.

[0126] In this embodiment, the specific steps for generating the receptive field area storage dictionary include: obtaining a neural network model consisting of a convolutional neural network layer Conv Layer1 obtained by truncating part of the backbone network and the feature map to the prediction probability matrix, setting the weight value of the convolutional layer to 1, initializing a tensor of size H×W and all element values ​​1, H×W in this embodiment is preferably 380×380, sending the tensor to the neural network model to obtain an h×w feature map, h×w in this embodiment is preferably 48×48, generating an h×w all-zero gradient map for each pixel point (x, y) in the feature map and setting the corresponding coordinate position of the pixel point to 1, backpropagating the gradient map to obtain the gradient map of the input tensor, and calculating the coordinate set index of the non-zero elements in the input tensor gradient map. Subsequently, the coordinate value of the receptive field area corresponding to each pixel point of the original image is calculated, and the coordinate value of the pixel point and the coordinate value of the receptive field area are stored in the receptive field area storage dictionary RF as the key value and the coordinate value of the receptive field area as the value. The specific calculation formula is as follows:

[0127] y0=min(index[0])

[0128] y1=max(index[0]+1)

[0129] x0=min(index[1])

[0130] x1=max(index[1]+1)

[0131] RF[x,y]=[x0,x1,y0,y1]

[0132] Next, the feature map F is fed into a convolutional neural network consisting of a convolutional layer, a batch normalization layer, a ReLu layer, and a Sigmoid activation layer to obtain a single-channel spatial attention map SA with a dimension of 48×48. The spatial attention map SA is fed into the polar coordinate GT predictor to generate the corresponding polar radius ρ and polar angle θ. The specific generation formula in the predictor is:

[0133] (x0,y0)=argmax x,y (SA)

[0134]

[0135]

[0136] Among them, ρ ij ,θ ij The value range is 0-1, ρ ij Represents the polar radius of the (i, j) position, θ ij represents the polar angle of the (i, j) position, and N is the size of the spatial attention map SA;

[0137] Then, the feature map F is sent to the coordinate prediction map generator to obtain the corresponding coordinate prediction map. First, the coordinate (x0, y0) corresponding to the maximum value of the spatial attention map SA is obtained. Then, based on the coordinate, a coordinate prediction map CP with a dimension of 112×48×48 is obtained. The specific generation formula is:

[0138]

[0139] CP=concatenation(A xy ,F)

[0140] Where concatenation represents the concatenation operation.

[0141] Then the coordinate prediction map CP is sent to the diameter predictor composed of convolution layer, batch normalization layer, and ReLu layer to obtain the predicted diameter ρ′, and the diameter loss L is calculated. distance , the specific calculation formula is:

[0142]

[0143] Among them, ρ′ ij Indicates the predicted polar radius at position (i,j).

[0144] Then the coordinate prediction map CP is sent to the polar angle predictor composed of convolution layer, batch normalization layer, and ReLu layer to obtain the predicted polar angle θ′, and the polar angle loss L is calculated. angle , the specific calculation formula is:

[0145]

[0146]

[0147] in, Represents the predicted polar angle of position (i,j).

[0148] Finally, L distance With Langle Add together to obtain the structural modeling loss L sm , specifically expressed as:

[0149] L sm =L distance +L angle

[0150] In this embodiment, through polar coordinate conversion, the maximum value point in the spatial self-attention map SA is used as the pole of polar coordinates, and the positional relationship between each pixel point in the spatial self-attention map and the pole is established. The polar radius and polar angle are calculated. Each pixel point in the spatial attention map corresponds to an image patch of the original image (that is, the local semantics of the image), and the positional relationship between the pixels is established, which promotes the structural interaction information between the local semantics.

[0151] S8: The extracted image patches are fed into the backbone network to extract local features at different levels, and the features at the end of the network are fed into the global pooling layer and the fully connected layer to obtain the true and false prediction probabilities, and the regional classification loss L is calculated. patch_cls The calculation formula of regional classification loss value is specifically expressed as:

[0152]

[0153] Among them, p i Represents a predicted probability value in a batch (B), y i Indicates the value of the true label, p i (y i ) indicates the prediction is label y i probability;

[0154] S9: compressing the extracted image patch to obtain a compressed image patch;

[0155] In this embodiment, the compression operation used in step S9 is the same as the compression operation used for the entire image compression;

[0156] S10: The compressed image patch is fed into a compression feature extraction network that shares weights with the backbone network to extract local features at different levels.

[0157] In this embodiment, the compression feature extraction network used in this step is the same as the compression feature extraction network used for the entire image.

[0158] S11: The features of each level corresponding to the uncompressed image patch and the features of each level corresponding to the compressed image patch are fed into the mapping head based on the convolutional neural network and mapped into feature vectors zp respectively. i,j and zp i+N,j, where j represents different levels, i represents the image number in a batch, and the size of the batch is N. After applying compression enhancement, there are 2N images in total, where the i-th image and its corresponding compression enhancement image are i+N. zp i,j With zp i+N,j have the same dimensions. Then the region contrast depth supervision loss L is calculated patch_contra , the specific steps include:

[0159]

[0160] Among them, i represents the i-th image in a batch of images, the batch size is N, and there are a total of 2N images including the compressed and enhanced images. If the i-th image patch of the input is represented by xp i , then the corresponding image patch after compression operation is xp i+N .zp i,j For XP i The j-th layer mapping feature obtained after passing through the backbone network and the mapping head. zp i+N,j For XP i+N The j-th layer mapping features obtained after the compressed feature extraction network and mapping head. is an indicator function. If k≠i, the value of the indicator function is 1, otherwise it is 0. τ is the temperature coefficient. In this embodiment, the value is preferably 0.07. M is the number of mapping heads. In this embodiment, its value is 4. j It represents the weight of the contrast loss function of different layers. The shallow layer has a larger weight than the deep layer. In this embodiment, its values ​​are 0.5, 0.3, 0.1, and 0.1 respectively.

[0161] S12: The entire network architecture is trained end-to-end. The combined loss constructed by the overall classification loss, overall contrastive deep supervision loss, regional classification loss, regional contrastive deep supervision loss, proposal constraint loss, and structural modeling loss is used as the training loss of the entire network architecture. The overall loss function calculation formula is expressed as:

[0162] L=L global_cls +α×L global_contra +β×(L patch_cls +α×L patch_contra )+γ×L sm +η×L PCL

[0163] Among them, α, β, η, and γ are the weights of the corresponding losses. In this embodiment, the value of α is 0.3, the values ​​of β and η are 0.5, and the value of γ is 0.1.

[0164] The network parameters of the model were iteratively updated based on the overall loss function. SGD optimizer was used with an initial learning rate of 0.001, a first-order smoothing parameter of 0.9, and a second-order smoothing parameter of 0.999. The training period was 100 and the training batch size was 4.

[0165] S13: Send the test set after video preprocessing into the backbone network to obtain the binary classification prediction probability and output the final prediction result.

[0166] In this embodiment, the performance evaluation index of the face-changing video algorithm is evaluated using the area under the ROC curve (AUC).

[0167] In this embodiment, the model trained on FF++(C23) was used to perform tests on FF++(C0), FF++(C23), and FF++(C40). The specific test results are shown in Table 1 below.

[0168] Table 1 Test results on the FF++ database

[0169]

[0170]

[0171] S14: Application effect of the test model on other libraries;

[0172] In this example, AUC is also used for evaluation. The model trained on FF++ (C23) is used to test on CelebDF, DFD (C23, C40), and DFDC. The specific test results are shown in Table 2 below:

[0173] Table 2 Test results from FF++ to other databases

[0174]

[0175] Example 2

[0176] like Figure 5 As shown, this embodiment provides a face-swapped video detection system based on overall forgery traces and local detail information extraction, which is used to implement the face-swapped video detection method based on overall forgery traces and local detail information extraction of the above-mentioned embodiment 1. The system includes: a video preprocessing module, a diversified mask self-fusion enhancement module, a feature extraction network construction module, a feature extraction module, an overall classification loss calculation module, an image compression enhancement module, a feature mapping module, an overall contrast depth supervision loss calculation module, an image patch acquisition module, a region proposal module, a region classification loss calculation module, a region contrast depth supervision loss calculation module, a total loss construction module, a training module, and a prediction module;

[0177] In this embodiment, the video preprocessing module is used to divide the video to be detected into frames, extract facial key points and crop facial images from the video frames;

[0178] In this embodiment, the diversified mask self-fusion enhancement module is used to perform diversified mask self-fusion enhancement operations on real faces according to set probabilities. Specifically, the module includes: combining different key points to obtain different point sets, calculating the convex hull of each point set, obtaining multiple masks based on the convex hull of each point set, obtaining a target face and a source face, and using the enhanced image obtained by self-fusion of the masks, target face, and source face as the input image;

[0179] In this embodiment, the feature extraction network construction module is used to construct a backbone network for image feature extraction and a compression feature extraction network;

[0180] In this embodiment, the feature extraction module is used to extract features of different levels from the input image based on the backbone network;

[0181] In this embodiment, the overall classification loss calculation module is used to calculate the overall classification loss based on the true and false prediction probabilities and the true and false labels;

[0182] In this embodiment, the image compression enhancement module is used to perform compression operations on the input image;

[0183] In this embodiment, the feature extraction module is used to extract features at different levels from the compressed image based on the compression feature extraction network;

[0184] In this embodiment, the feature mapping module is used to map the features of each level of the uncompressed image into a first feature vector via a corresponding mapping head, and to map the features of each level of the compressed image into a second feature vector via a corresponding mapping head;

[0185] In this embodiment, the overall contrast depth supervision loss calculation module is used to calculate the overall contrast depth supervision loss between the first eigenvector and the second eigenvector;

[0186] In this embodiment, the image patch acquisition module is used to obtain image patches, truncate the backbone network to obtain intermediate feature maps, calculate the prediction probability matrix, obtain the coordinates of the receptive field area of ​​the original image corresponding to the element point with the largest prediction probability value, and crop the original image based on the coordinates to obtain the image patch;

[0187] In this embodiment, the region proposal module is used to calculate the proposal constraint loss based on the face authenticity label and the prediction probability matrix, obtain the spatial attention map corresponding to the intermediate feature map, and calculate the structural modeling loss with the maximum point in the spatial self-attention map as the extreme point;

[0188] In this embodiment, the feature extraction module is used to extract local features of different levels from the image patch based on the backbone network to obtain true or false prediction probabilities;

[0189] In this embodiment, the regional classification loss calculation module is used to calculate the regional classification loss based on the true and false prediction probabilities;

[0190] In this embodiment, the image compression enhancement module is used to perform compression operations on image patches;

[0191] In this embodiment, the feature extraction module is used to extract local features at different levels on the compressed image patch based on the compression feature extraction network;

[0192] In this embodiment, the feature mapping module is used to map the features of each level of the uncompressed image patch into a third feature vector via a corresponding mapping head, and to map the features of each level of the compressed image patch into a fourth feature vector via a corresponding mapping head;

[0193] In this embodiment, the regional contrast depth supervision loss calculation module is used to calculate the regional contrast depth supervision loss between the third eigenvector and the fourth eigenvector;

[0194] In this embodiment, the total loss construction module is used to combine the overall classification loss, the overall contrast deep supervision loss, the proposal constraint loss, the structure modeling loss, the regional classification loss, and the regional contrast deep supervision loss to obtain the total loss;

[0195] In this embodiment, the training module is used to train the backbone network based on the total loss;

[0196] In this embodiment, the prediction module is used to obtain the binary classification prediction probability based on the trained backbone network and output the prediction result.

[0197] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A face-swapped video detection method based on overall forgery traces and local detail information extraction, characterized in that: The steps include: Divide the video to be detected into frames, extract facial key points and crop the face image from the video frame; According to the input image label, the image is judged to be true or false. If the image is judged to be true, a diversified mask self-fusion enhancement operation is performed according to the set probability to obtain the input image. If the image is judged to be false, it is directly used as the input image. The diversified mask self-fusion enhancement operation specifically includes: combining different key points to obtain different point sets, calculating the convex hull of each point set, obtaining multiple masks based on the convex hull of each point set, obtaining the target face and the source face, and obtaining the enhanced image based on the self-fusion of the mask, the target face and the source face as the input image; Build a backbone network and a compressed feature extraction network for image feature extraction. Based on the backbone network, extract features at different levels of the input image, calculate the true and false prediction probabilities, and calculate the overall classification loss based on the true and false prediction probabilities and true and false labels. Compress the input image, extract features at different levels of the compressed image based on the compression feature extraction network, map the features at each level of the uncompressed image to a first feature vector via the corresponding mapping head, and map the features at each level of the compressed image to a second feature vector via the corresponding mapping head, and calculate the overall contrast deep supervision loss between the first feature vector and the second feature vector; Truncating the backbone network to obtain the intermediate feature map, calculating the prediction probability matrix, obtaining the receptive field area coordinates of the original image corresponding to the element point with the largest prediction probability value, cropping the original image based on the receptive field area coordinates to obtain an image patch; Calculate the proposal constraint loss based on the face authenticity label and the predicted probability matrix, obtain the spatial attention map corresponding to the intermediate feature map, and use the maximum value point in the spatial self-attention map as the extreme point to calculate the structural modeling loss; Based on the backbone network, the local features of the image patches at different levels are extracted to obtain the true and false prediction probabilities and calculate the regional classification loss. Performing a compression operation on the image patch, extracting local features of different levels on the compressed image patch based on a compression feature extraction network, mapping the features of each level of the uncompressed image patch into a third eigenvector via a corresponding mapping head, mapping the features of each level of the compressed image patch into a fourth eigenvector via a corresponding mapping head, and calculating the regional contrast deep supervision loss between the third eigenvector and the fourth eigenvector; The total loss is obtained by combining the overall classification loss, overall contrastive deep supervision loss, proposal constraint loss, structure modeling loss, regional classification loss, and regional contrastive deep supervision loss; The backbone network is trained based on the total loss, the binary classification prediction probability is obtained based on the trained backbone network, and the prediction result is output.

2. The face-swapped video detection method based on overall forgery traces and local detail information extraction according to claim 1 is characterized in that: The enhanced image obtained by self-fusion of the mask, the target face, and the source face is used as the input image, specifically comprising: randomly generating a probability value and setting a probability threshold; if it is determined that the probability value is less than or equal to the probability threshold, subjecting the original face to hue conversion, saturation adjustment, brightness conversion, contrast ratio conversion, color conversion, and image sharpening operations in sequence to obtain the target face, while directly using the original face as the source face; If the probability value is greater than the probability threshold, the original face is sequentially subjected to hue transformation, saturation adjustment, brightness transformation, contrast ratio transformation, color transformation, image sharpening, affine transformation, translation, and scaling operations to obtain the source face, and the original face is directly used as the target face; The mask is sequentially subjected to affine transformation, elastic transformation, random scaling, Gaussian blur, and normalization to obtain a blurred mask; The enhanced image is represented as: I blend =MB×I source ×ratio+(1-MB)×I target Among them, I blend represents the enhanced image, MB represents the blurred mask, ratio represents the fusion ratio, I source represents the source face, I target Indicates the target face.

3. The face-swapped video detection method based on overall forgery traces and local detail information extraction according to claim 1 is characterized in that: The backbone network and compressed feature extraction network for image feature extraction are constructed, using EfficientNet-b4 as the backbone network and EfficientNet-b4 as the compressed feature extraction network, and the compressed feature extraction network and the backbone network weights are shared.

4. The face-swapped video detection method based on overall forgery traces and local detail information extraction according to claim 1 is characterized in that: The overall classification loss is calculated based on the true and false prediction probabilities and true and false labels, specifically expressed as: Among them, p i Represents a predicted probability value in a batch B, y i Indicates the value of the true label, p i (y i ) indicates the prediction is label y i probability.

5. The face-swapped video detection method based on overall forgery traces and local detail information extraction according to claim 1 is characterized in that: The calculating of the overall contrastive depth supervision loss between the first eigenvector and the second eigenvector specifically includes: Among them, z i,j represents the first eigenvector, z i+N,j represents the second eigenvector, j represents different levels, i represents the image number in a batch, N represents the batch size, and M represents the number of mapping heads. Represents the indicator function. If k≠i, the value of the indicator function is 1, otherwise it is 0. τ is the temperature coefficient, weight j Represents the weights of the contrast loss function of different layers; And / or, calculating the regional contrast depth supervision loss between the third eigenvector and the fourth eigenvector specifically includes: Among them, zp i,j represents the third eigenvector, zp i+N,j represents the fourth eigenvector.

6. The face-swapped video detection method based on overall forgery traces and local detail information extraction according to claim 1 is characterized in that: The obtaining of the image patch specifically includes: According to the face authenticity label, the predicted probability value of the corresponding channel in the predicted probability matrix is ​​obtained, and the coordinates corresponding to the element point with the largest predicted probability value are obtained. Based on the coordinates, the receptive field area of ​​the original image corresponding to the element point is obtained in the receptive field area storage dictionary, and the original image is cropped according to the coordinates of the receptive field area of ​​the original image to obtain an image patch.

7. The face-swapped video detection method based on overall forgery traces and local detail information extraction according to claim 1 is characterized in that: The calculation of proposal constraint loss based on face authenticity labels and prediction probability matrix specifically includes: Downsample the channel dimension of the features of the intermediate feature map to obtain the prediction probability matrix; Expand the face authenticity label into a label matrix with the same length and width; The proposal constraint loss is expressed as: Among them, p i Represents the value of a certain element of the predicted probability matrix P, y i is the value of the corresponding position in the label matrix Y, p i (y i ) indicates the prediction is label y i probability.

8. The face-swapped video detection method based on overall forgery traces and local detail information extraction according to claim 1 is characterized in that: Calculate the structural modeling loss, including: The spatial self-attention map generates corresponding polar diameter and polar angle through a polar coordinate predictor; Get the coordinates corresponding to the maximum value of the spatial attention map, and obtain the coordinate prediction map based on the coordinates; The coordinate prediction graph is passed through the polar diameter predictor to obtain the predicted polar diameter, and the polar diameter loss is calculated, which is specifically expressed as: (x0,y0)=argmax x,y (IN) Among them, i∈[1,N],j∈[1,N], N is the size of the spatial attention map, ρ ij Indicates the polar radius of the (i, j) position, SA ij Represents the attention value of position (i, j) in the spatial attention map, ρ′ ij represents the predicted radius of the (i, j) position, (x0, y0) represents the coordinates corresponding to the maximum value of the spatial attention map, and L distance represents the pole diameter loss; The coordinate prediction graph is passed through the polar angle predictor to obtain the predicted polar angle, and the polar angle loss is calculated, which is specifically expressed as: Among them, θ ij represents the polar angle of the position (i, j), represents the predicted polar angle of position (i, j); The structural modeling loss is obtained by adding the polar radius loss and the polar angle loss, which is specifically expressed as: L sm =L distance +L angle Among them, L sm represents the structural modeling loss.

9. The face-swapped video detection method based on overall forgery traces and local detail information extraction according to claim 1 is characterized in that: The calculation area classification loss is specifically expressed as: Among them, p i Represents a predicted probability value in a batch B, y i Indicates the value of the true label, p i (y i ) indicates the prediction is label y i probability.

10. A face-swapping video detection system based on extraction of overall forgery traces and local detail information, characterized in that: include: Video preprocessing module, diversified mask self-fusion enhancement module, feature extraction network construction module, feature extraction module, overall classification loss calculation module, image compression enhancement module, feature mapping module, overall contrast deep supervision loss calculation module, image patch acquisition module, region proposal module, region classification loss calculation module, region contrast deep supervision loss calculation module, total loss construction module, training module, prediction module; The video preprocessing module is used to divide the video to be detected into frames, extract facial key points and crop facial images from the video frames; The diversified mask self-fusion enhancement module is used to perform diversified mask self-fusion enhancement operations on real faces according to set probabilities, specifically including: combining different key points to obtain different point sets, calculating the convex hull of each point set, obtaining multiple masks based on the convex hull of each point set, obtaining a target face and a source face, and using the enhanced image obtained by self-fusion of the masks, target face and source face as the input image; The feature extraction network construction module is used to construct a backbone network for image feature extraction and a compression feature extraction network; The feature extraction module is used to extract features of different levels from the input image based on the backbone network; The overall classification loss calculation module is used to calculate the overall classification loss based on the true and false prediction probabilities and the true and false labels; The image compression enhancement module is used to perform compression operation on the input image; The feature extraction module is used to extract features of different levels from the compressed image based on the compression feature extraction network; The feature mapping module is used to map the features of each level of the uncompressed image into a first feature vector through a corresponding mapping head, and map the features of each level of the compressed image into a second feature vector through a corresponding mapping head; The overall contrast depth supervision loss calculation module is used to calculate the overall contrast depth supervision loss between the first eigenvector and the second eigenvector; The image patch acquisition module is used to obtain image patches, truncate the backbone network to obtain intermediate feature maps, calculate the prediction probability matrix, obtain the coordinates of the receptive field area of ​​the original image corresponding to the element point with the largest prediction probability value, and crop the original image based on the coordinates to obtain the image patch; The region proposal module is used to calculate the proposal constraint loss based on the face authenticity label and the prediction probability matrix, obtain the spatial attention map corresponding to the intermediate feature map, and calculate the structural modeling loss with the maximum point in the spatial self-attention map as the extreme point; The feature extraction module is used to extract local features of different levels from image patches based on the backbone network to obtain true or false prediction probabilities; The regional classification loss calculation module is used to calculate the regional classification loss based on the true and false prediction probabilities; The image compression enhancement module is used to perform compression operations on image patches; The feature extraction module is used to extract local features of different levels on the compressed image patch based on the compression feature extraction network; The feature mapping module is used to map the features of each level of the uncompressed image patch into a third feature vector via a corresponding mapping head, and map the features of each level of the compressed image patch into a fourth feature vector via a corresponding mapping head; The regional contrast depth supervision loss calculation module is used to calculate the regional contrast depth supervision loss between the third eigenvector and the fourth eigenvector; The total loss building module is used to combine the overall classification loss, the overall contrast deep supervision loss, the proposal constraint loss, the structure modeling loss, the regional classification loss, and the regional contrast deep supervision loss to obtain the total loss; The training module is used to train the backbone network based on the total loss; The prediction module is used to obtain the binary classification prediction probability based on the trained backbone network and output the prediction result.

Images