Tracing method and device for deeply-forged video
By combining language supervision and image self-supervision multi-task model and local feature point matching technology, the problem of deep fake video traceability is solved, and the rapid traceability and clue amplification of videos are achieved, and the negative impact of malicious videos is effectively dealt with.
Patent Information
- Application Number
- CN202310623041.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-30
- Publication Date
- 2024-12-03
AI Technical Summary
The existing technology cannot effectively trace the source and control the spread of deep fake videos, making it difficult to deal with the negative impacts of malicious videos.
A multi-task model combining language supervision and image self-supervision is adopted, and the target model is trained using the global feature matching pre-trained data set to trace the source of deep fake videos, and the clues are expanded through local feature point matching to realize the traceability and rumors of videos.
It has achieved rapid and accurate traceability and amplification of deep fake videos, effectively curbing the spread of harmful videos, and improving the efficiency of rumor refutation.
Smart Images

Figure CN119068376A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of Internet, and more specifically, to a method and device for tracing the source of deep fake videos. Background Art
[0002] With the maturity of facial deep fake technology (which uses deep learning technology to produce audio and audio-visual content to present things that people have never said or done, or to create people who have never existed), a large number of deep fake videos (i.e. deep fake videos) produced by technologies such as Deepfake, FaceSwap, and Face2Face have appeared on the Internet. Among these videos, there are some that specifically target specific people for malicious slander, insults, and false videos. The spread of these videos will have serious negative impacts.
[0003] With regard to the above-mentioned problem of being unable to trace the source of deep fake videos, no effective solution has been proposed so far. Summary of the invention
[0004] The embodiments of the present application provide a method and device for tracing the source of a deep fake video, so as to at least solve the technical problem in the related art that a deep fake video cannot be traced to the source.
[0005] According to one aspect of an embodiment of the present application, a method for tracing the source of a deep fake video is provided, including: obtaining a global feature matching pre-training data set collected from the Internet; using the global feature matching pre-training data set to train a first original model to obtain a first target model, wherein the first original model is a multi-task model that combines language supervision and image self-supervision, and is used to learn feature associations between images and feature associations between images and texts from the global feature matching pre-training data set; and using the first target model to trace the source of the deep fake video.
[0006] Optionally, obtaining a global feature matching pre-training data set collected from the Internet includes: obtaining images and corresponding title pairs collected from the Internet to form a global feature matching pre-training data set.
[0007] Optionally, the first original model is trained using a global feature matching pre-training data set to obtain a first target model, including: obtaining an image in the global feature matching pre-training data set as an original image and performing a homologous transformation on the original image, filling the original image and the image after the homologous transformation of the original image as a positive sample pair, and the original image and the transformed image of the non-original image as a negative sample pair into the first sample library, and inputting the two images of the sample pair in the first sample library into the visual side of the first original model respectively to obtain a feature representation of the image and the image sample pair; obtaining an image in the global feature matching pre-training data set as an original image, and filling the original image and the text corresponding to the original image as a positive sample pair, and the text corresponding to the original image and the non-original image as a negative sample pair into the second sample library, and inputting the image and text of the sample pair in the second sample library into the visual side and text side of the first original model respectively to obtain a feature representation of the image and the text sample pair; using a contrastive learning loss function, performing self-supervised pre-training and weakly supervised pre-training respectively, and updating the first original model through back propagation until the model converges to obtain the first target model.
[0008] Optionally, the method also includes: obtaining a local feature point matching pre-training data set collected from the Internet; using the local feature point matching pre-training data set to train a second original model to obtain a second target model, wherein the second original model is an end-to-end feature point and descriptor extraction network model for learning feature points and descriptors from the local feature point matching pre-training data set; and using the second target model to expand clues of the deep fake video.
[0009] Optionally, a local feature point matching pre-training data set collected from the Internet is obtained, including: taking an image collected from the Internet as an original image and performing a homologous transformation on the original image, and taking each original image and the image after the homologous transformation of the original image as an image pair, thereby constituting a local feature point matching pre-training data set.
[0010] Optionally, the second original model is trained using a local feature point matching pre-training data set to obtain a second target model, including: taking the original image in the local feature point matching pre-training data set and the image after homologous transformation of the original image as a positive sample pair, and taking the original image and the image after homologous transformation of a non-original image as a negative sample pair; inputting the two images in the sample pair into the second original model respectively, extracting local feature points using the local feature point extraction module in the second original model, and sending the extracted local feature points to the local feature matching module of the second original model to obtain a matched score; calculating the loss function using a weakly supervised contrast learning method, and updating the model parameters by back propagation until the model converges to obtain the second target model.
[0011] Optionally, the first original model is a SLIP model, and the second original model is a SuperPoint model.
[0012] According to another aspect of an embodiment of the present application, a tracing device for a deep fake video is also provided, including: an acquisition unit, used to acquire a global feature matching pre-training data set collected from the Internet; a training unit, used to train a first original model using the global feature matching pre-training data set to obtain a first target model, wherein the first original model is a multi-task model combining language supervision and image self-supervision, and is used to learn feature associations between images and feature associations between images and texts from the global feature matching pre-training data set; a tracing unit, used to trace the deep fake video using the first target model.
[0013] Optionally, the acquisition unit is further used to: acquire images and corresponding title pairs collected from the Internet to form a global feature matching pre-training data set.
[0014] Optionally, the training unit is also used to: obtain an image in a global feature matching pre-training data set as an original image and perform a homologous transformation on the original image, fill the original image and the image after the homologous transformation of the original image as a positive sample pair, and the original image and the transformed image of the non-original image as a negative sample pair into the first sample library, and input the two images of the sample pair in the first sample library into the visual side of the first original model respectively to obtain a feature representation of the image and the image sample pair; obtain an image in a global feature matching pre-training data set as an original image, fill the original image and the text corresponding to the original image as a positive sample pair, and the text corresponding to the original image and the non-original image as a negative sample pair into the second sample library, and input the image and text of the sample pair in the second sample library into the visual side and text side of the first original model respectively to obtain a feature representation of the image and the text sample pair; use the loss function of contrastive learning to perform self-supervised pre-training and weakly supervised pre-training respectively, and update the first original model through back propagation until the model converges to obtain the first target model.
[0015] Optionally, the acquisition unit is also used to: acquire a local feature point matching pre-training data set collected from the Internet; the training unit is also used to: use the local feature point matching pre-training data set to train the second original model to obtain a second target model, wherein the second original model is an end-to-end feature point and descriptor extraction network model, used to learn feature points and descriptors from the local feature point matching pre-training data set; the device of the present application may also include: a clue expansion unit, used to use the second target model to expand clues of the deep fake video.
[0016] Optionally, the acquisition unit is further used to: use images collected from the Internet as original images and perform homology transformation on the original images, and use each original image and the image after homology transformation of the original image as an image pair, thereby forming a local feature point matching pre-training data set.
[0017] Optionally, the training unit is also used to: use the original image in the local feature point matching pre-training data set and the image after homologous transformation of the original image as a positive sample pair, and use the original image and the image after homologous transformation of the non-original image as a negative sample pair; input the two images in the sample pair into the second original model respectively, extract the local feature points using the local feature point extraction module in the second original model, and send the extracted local feature points to the local feature matching module of the second original model to obtain the matched score; calculate the loss function using weakly supervised contrast learning, and update the model parameters by back propagation until the model converges to obtain the second target model.
[0018] Optionally, the first original model is a SLIP model, and the second original model is a SuperPoint model.
[0019] According to another aspect of an embodiment of the present application, an electronic device is further provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the above method through the computer program.
[0020] According to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps of any one of the embodiments of the above method.
[0021] By applying the technical solution of the present invention, a global feature matching pre-training data set collected from the Internet is obtained; the first original model is trained using the global feature matching pre-training data set to obtain a first target model, wherein the first original model is a multi-task model that combines language supervision and image self-supervision, and is used to learn the feature associations between images and the feature associations between images and texts from the global feature matching pre-training data set; the first target model is used to trace the source of a deep fake video, thereby solving the technical problem in the related art that the deep fake video cannot be traced.
[0022] In addition to the above-described purposes, features and advantages, the present invention has other purposes, features and advantages. The present invention will be further described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] The accompanying drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the accompanying drawings:
[0024] Figure 1is a schematic diagram of an optional tracing of deep fake face videos according to an embodiment of the present application;
[0025] Figure 2 is a schematic diagram of an optional amplification of deep fake video clues of a face according to an embodiment of the present application;
[0026] Figure 3 is a flowchart of an optional method for tracing the source of a deep fake video according to an embodiment of the present application;
[0027] Figure 4 It is a schematic diagram of tracing the source and amplifying clues of a deep fake face video according to an embodiment of the present application;
[0028] Figure 5 This is a schematic diagram of an optional deep fake video source tracing device according to an embodiment of the present application. DETAILED DESCRIPTION
[0029] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.
[0030] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0031] After analyzing the existing technologies, the inventors realized that with the maturity of deep fake face technology, a large number of deep fake videos produced by Deepfake, FaceSwap, Face2Face and other technologies have appeared on the Internet. Among these videos, there are some that specifically target specific people for malicious slander, insults, and falsehoods. The spread of these videos will have serious negative impacts. Therefore, on the one hand, after obtaining clues of existing deep fake videos, finding and tracing the original video and its variants can effectively play a role in refuting rumors, such as Figure 1 As shown; on the other hand, searching for variant videos of existing deep fake videos after homologous transformation can expand the existing clues, thereby effectively curbing the further spread of harmful deep fake videos, such as Figure 2 shown.
[0032] The above two functions can be expanded based on the video retrieval system based on image matching based on actual needs. The video retrieval method based on image matching can be used to extract several key frames that can represent the video content from a given video. Feature extractors such as HOG and SIFT are used to extract feature information in the key frames to represent the visual information of the video. The matching algorithm is used to calculate the matching degree of the query video and the video in the sample library respectively, and an empirical threshold is set. If the matching score is higher than the threshold, the query sample is considered to match the detected sample. However, features such as HOG and SIFT can only represent the visual information of the video (image), such as corners, line segments, and colors, and lack the expression of high-level semantic information, resulting in some videos related to the video content cannot be reasonably recalled, and videos that are not related to the video content but have similar visual content are misdetected.
[0033] The feature extraction methods mentioned above include HOG and SIFT, while the methods based on deep learning semantics are usually based on convolutional neural networks (CNN). They each have some shortcomings:
[0034] HOG and SIFT methods require manually designed feature extractors, which means that a lot of time and effort is needed to select the best feature extractor, and these algorithms usually fail to capture higher-level semantic information. Deep learning semantic-based methods usually require a large amount of data to train and require a lot of computing resources for training and inference. In addition, since these algorithms are black-box models, it is difficult to explain the decision-making process of the algorithm, which is unacceptable in some applications.
[0035] HOG and SIFT methods are usually sensitive to illumination changes and noise in images, which affects their performance. Methods based on deep learning semantics are usually better able to handle these problems, but they may also have overfitting problems for some unusual situations. In short, although HOG and SIFT methods are still useful in some scenarios, with the continuous development of deep learning technology, methods based on deep learning semantics have become the first choice for many computer vision applications.
[0036] In order to overcome the above problems, according to one aspect of the embodiments of the present application, an embodiment of a method for tracing the source of a deep fake video is provided. The method for tracing the source of a deep fake video in the embodiment of the present application can be executed by a server or a terminal. Among them, the terminal executing the method for tracing the source of a deep fake video in the embodiment of the present application can also be executed by a client installed thereon. Figure 3 is a flowchart of an optional method for tracing the source of a deep fake video according to an embodiment of the present application, such as Figure 3 As shown, the method may include the following steps:
[0037] Step S31, obtaining a global feature matching pre-training data set collected from the Internet. Specifically, images and corresponding title pairs collected from the Internet can be obtained, and a large number of title pairs can be used to form a global feature matching pre-training data set.
[0038] Step S32, using the global feature matching pre-training data set to train the first original model to obtain a first target model, the first original model is a multi-task model combining language supervision and image self-supervision, and is used to learn the feature associations between images and the feature associations between images and texts from the global feature matching pre-training data set.
[0039] The first original model mentioned above can be a SLIP model, which stands for Self-supervision meet Language-Image Pre-training. During pre-training, separate views of each input image are constructed for language supervision and image self-supervision branches, and then fed back through a shared image encoder. During the training process, the image encoder learns to represent the visual input in a semantically meaningful way.
[0040] Optionally, an image in a global feature matching pre-training data set is obtained as an original image and a homologous transformation is performed on the original image. The original image and the image after the homologous transformation of the original image are used as a positive sample pair, and the original image and the transformed image of the non-original image (i.e., other images other than the original image, such as the original image is image A, and the non-original image is image B in the training data set that is different from image A) are used as negative sample pairs to fill in the first sample library, and the two images of the sample pairs in the first sample library are respectively input into the visual side of the first original model to obtain a feature representation of the image and the image sample pair; an image in a global feature matching pre-training data set is obtained as an original image, and the original image and the text corresponding to the original image are used as a positive sample pair, and the text corresponding to the original image are used as a negative sample pair to fill in the second sample library, and the image and text of the sample pairs in the second sample library are respectively input into the visual side and text side of the first original model to obtain a feature representation of the image and the text sample pair; a contrastive learning loss function is used to perform self-supervised pre-training and weakly supervised pre-training, respectively, and the first original model is updated by back propagation until the model converges to obtain the first target model.
[0041] Step S33: Use the first target model to trace the deep fake video.
[0042] Before or after step S31 or during the process thereof, the following steps may be performed to amplify clues of the deep fake video:
[0043] 1) Obtain a local feature point matching pre-training data set collected from the Internet. For example, an image collected from the Internet is used as an original image and the original image is homologously transformed. Each original image and the image after the homologous transformation of the original image are used as an image pair, thereby forming a local feature point matching pre-training data set.
[0044] 2) Using the local feature point matching pre-training data set to train the second original model to obtain a second target model, wherein the second original model is an end-to-end feature point and descriptor extraction network model, such as a SuperPoint model, which is used to learn feature points and descriptors from the local feature point matching pre-training data set.
[0045] For example, the original image in the local feature point matching pre-training data set and the image after homologous transformation of the original image can be used as a positive sample pair, and the original image and the image after homologous transformation of the non-original image can be used as a negative sample pair; the two images in the sample pair are respectively input into the second original model, and the local feature points are extracted by the local feature point extraction module in the second original model, and the extracted local feature points are sent to the local feature matching module of the second original model to obtain the matched score; the loss function is calculated using the weakly supervised contrast learning method, and the model parameters are updated by back propagation until the model converges to obtain the second target model.
[0046] 3) Use the second target model to expand clues in deep fake videos.
[0047] In the process of tracing the source of deep fake videos, it is expected to quickly and accurately find videos related to the high-level semantic content of the video. Therefore, the present application proposes a method of using a large-scale pre-trained image-text multimodal model as a visual feature extractor. The extractor undergoes visual-visual contrast learning and visual-text contrast learning, and the proposed features contain both rich visual intrinsic attributes and high-level semantic attributes. The global semantic matching extracted by the feature extractor can effectively realize the tracing of deep fake videos of faces. In addition, in the process of amplifying clues of deep fake videos, it is expected to input existing deep fake videos and retrieve variant videos that are highly similar to the video from the sample set searched on the Internet. If the above-mentioned global high-level semantic matching method is used for retrieval alone, a large number of videos related to the video content but not homologous videos are retrieved, which is inconsistent with the goal of this task. Therefore, it is crucial to explore a method for high-similarity video matching based on local feature point matching. This type of method can effectively extract local features in videos (images), and use these local feature points to take spatial constraints, and finally achieve matching. Different from the method using face detection and recognition, this method does not require the design of additional face recognition models and has higher model generalization.
[0048] Through the above steps, a global feature matching pre-training data set collected from the Internet is obtained; the first original model is trained using the global feature matching pre-training data set to obtain a first target model, wherein the first original model is a multi-task model that combines language supervision and image self-supervision, and is used to learn the feature associations between images and between images and texts from the global feature matching pre-training data set; the first target model is used to trace the source of the deep fake video, thereby solving the technical problem in the related technology that the deep fake video cannot be traced.
[0049] The present invention can solve two technical problems: for tracing the source of high-similarity deep fake face videos, a SLIP feature global semantic matching method based on the combination of image-text weak supervision and self-supervised pre-training is used; for clue amplification of high-similarity deep fake face videos, a SuperGlue-based local feature point matching method is used.
[0050] like Figure 4 As shown, it is a schematic diagram of high-similarity face deep fake video tracing and clue amplification based on global semantic matching and local feature point matching. The present invention mainly includes 5 key steps: Step 1, data preparation, building a pan-collection video retrieval database for query sample retrieval, building a global feature matching pre-training data set and a local feature point matching pre-training data set for subsequent model training; Step 2, global SLIP feature model pre-training, using the global feature point matching pre-training data provided in step 1 to pre-train the SLIP model; Step 3, local SuperGlue feature model pre-training, using the local feature point matching pre-training data provided in step 1 to pre-train the SuperGlue model; Step 4, video tracing, using the SLIP pre-trained model pre-trained in step 2 to perform global semantic matching to achieve high-similarity face deep fake video tracing; Step 5, video clue amplification, using the SuperGlue pre-trained model pre-trained in step 3 to perform local feature point matching to achieve clue amplification of high-similarity face deep fake videos. The following is a detailed introduction:
[0051] Step 1: Data preparation, the detailed process is as follows:
[0052] Step 1.1: Given a deep fake face video as a query example Q and a large number of widely collected videos as a retrieval database D S ;
[0053] Step 1.2: Collect a large number of images and their caption pairs collected from the Internet to form a global feature matching pre-training dataset D G ;
[0054] Step 1.3: Collect a large number of original images and perform various homologous transformations. Each original image and its transformed image constitute an image pair, which constitutes the local feature point matching pre-training dataset D L .
[0055] Step 2: Pre-training of the global SLIP feature model. The detailed process is as follows:
[0056] Step 2.1: Use the D obtained in step 1.2 G , take the images and perform various homologous transformations, take the original image and the image after homologous transformation as the positive sample pair, and the original image and other non-original image transformations as the negative sample pair, input the above two images into the visual side of the SLIP model respectively, and obtain the feature representation of the image-image sample pair;
[0057] Step 2.2: Use the D obtained in step 1.2 G , take the image and text, take the original image and its corresponding text as the positive sample pair, and the original image and other samples other than the original image as the negative sample pair, input the above image and text into the visual side and text side of the SLIP model respectively, and obtain the feature representation of the image-text sample pair;
[0058] Step 2.3: After obtaining the feature representations of the image-image sample pairs and image-text sample pairs constructed in steps 2.1 and 2.2, use the contrastive learning loss function to perform self-supervised pre-training and weakly supervised pre-training respectively, and update the SLIP model through back propagation until the model converges.
[0059] Step 3: Pre-training of local SuperGlue feature model. The detailed process is as follows:
[0060] Step 3.1: Use the D obtained in step 1.3 L , the original image and the image after homology transformation are taken as positive sample pairs, and the original image and other non-original image transformations are taken as negative sample pairs. The above two images are input into the SuperPoint model respectively to obtain local feature points;
[0061] Step 3.2: Send the above local feature points to the SuperGlue local feature matching module to obtain the matching score;
[0062] Step 3.3: Use weakly supervised contrastive learning to calculate the loss function and back-propagate to update the model parameters.
[0063] Step 4: Tracing the source of deep fake videos with high similarity faces based on global semantic matching. The detailed process is as follows:
[0064] Step 4.1: Use the SLIP model obtained in step 2 as the feature extraction model to retrieve the video database D collected in step 1.1.S Extract features from the video and save them;
[0065] Step 4.2: Use the SLIP model obtained in step 2 as a feature extraction model to extract features for the deep fake face video query sample Q given in step 1.1;
[0066] Step 4.3: Calculate the features of the query example Q in step 4.2 and the video retrieval database D in step 4.1 S The cosine similarity of each video feature in is sorted by size, and the top five videos with the largest similarity are selected as the tracing results.
[0067] Step 5: Amplify the visual clues of the high-similarity face deep fake based on local feature point matching. The detailed process is as follows:
[0068] Step 5.1: Use the SuperGlue model obtained in step 3 as the local feature point matching network, and match the query sample Q in step 1.1 with the retrieval database D S The video in is input into the network;
[0069] Step 5.2: Calculate the number of local feature point matches between the query sample in step 5.1 and all samples in the database, sort them by the number of matches, and select the top 5 videos with the greatest similarity as the clue amplification results.
[0070] Step 2 and step 3 of the present invention are pre-trained on the constructed global feature matching pre-training data set and the local feature point matching data set. Pre-training is adopted. On the one hand, the pre-training data set does not need to be labeled and is very easy to obtain; on the other hand, pre-training can effectively improve the generalization of the basic model.
[0071] In this regard, we trained a global SLIP feature model pre-training without step 2 and a global SLIP feature model pre-training without step 3 for two tasks: high-similarity face deep fake video tracing based on global semantic matching and high-similarity face deep fake visual expansion based on local feature point matching. We took the face deep fake video collected in step 1 as the query sample Q and a large number of widely collected videos as the retrieval database D_S for testing. The test results are shown in Table 1. The evaluation indicators are the accuracy and recall rates commonly used in tasks. It can be seen that the model without pre-training has a significant decline in both indicators. This proves that pre-training helps improve the prediction accuracy of this model.
[0072] Table 1
[0073]
[0074] In the technical solution of the present application, a pre-training method is used to learn relevant prior knowledge as the initialization weight of the model, so that the model can be subsequently trained more effectively on a small-scale target data set; a high-similarity face deep fake video tracing and clue amplification method based on global semantic matching and local feature point matching is designed. The above two methods focus on global semantic matching and local feature matching.
[0075] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.
[0076] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.
[0077] According to another aspect of an embodiment of the present application, a deep fake video tracing device for implementing the above-mentioned deep fake video tracing method is also provided. Figure 5 is a schematic diagram of an optional deep fake video source tracing device according to an embodiment of the present application, such as Figure 5 As shown, the device may include:
[0078] An acquisition unit 51 is used to acquire a global feature matching pre-training data set collected from the Internet;
[0079] A training unit 52 is used to train a first original model using the global feature matching pre-training data set to obtain a first target model, wherein the first original model is a multi-task model combining language supervision and image self-supervision, and is used to learn feature associations between images and feature associations between images and texts from the global feature matching pre-training data set;
[0080] The tracing unit 53 is used to trace the deep fake video by using the first target model.
[0081] Optionally, the acquisition unit is further used to: acquire images and corresponding title pairs collected from the Internet to form the global feature matching pre-training data set.
[0082] Optionally, the training unit is also used to: obtain an image in the global feature matching pre-training data set as an original image and perform a homologous transformation on the original image, fill the original image and the image after the homologous transformation of the original image as a positive sample pair, and the original image and the transformed image of the non-original image as a negative sample pair into the first sample library, and input the two images of the sample pair in the first sample library into the visual side of the first original model respectively to obtain a feature representation of the image and the image sample pair; obtain an image in the global feature matching pre-training data set as an original image, fill the original image and the text corresponding to the original image as a positive sample pair, and the text corresponding to the original image and the non-original image as a negative sample pair into the second sample library, and input the image and text of the sample pair in the second sample library into the visual side and text side of the first original model respectively to obtain a feature representation of the image and text sample pair; use the loss function of contrastive learning to perform self-supervised pre-training and weakly supervised pre-training respectively, and update the first original model through back propagation until the model converges to obtain the first target model.
[0083] Optionally, the acquisition unit is also used to: acquire a local feature point matching pre-training data set collected from the Internet; the training unit is also used to: use the local feature point matching pre-training data set to train a second original model to obtain a second target model, wherein the second original model is an end-to-end feature point and descriptor extraction network model, used to learn feature points and descriptors from the local feature point matching pre-training data set; the device of the present application may also include: a clue expansion unit, used to use the second target model to expand clues of the deep fake video.
[0084] Optionally, the acquisition unit is further used to: use images collected from the Internet as original images and perform homology transformation on the original images, and use each original image and the image after homology transformation of the original image as an image pair, thereby constituting the local feature point matching pre-training data set.
[0085] Optionally, the training unit is also used to: use the original image in the local feature point matching pre-training data set and the image after homologous transformation of the original image as a positive sample pair, and use the original image and the image after homologous transformation of the non-original image as a negative sample pair; input the two images in the sample pair into the second original model respectively, extract the local feature points using the local feature point extraction module in the second original model, and send the extracted local feature points to the local feature matching module of the second original model to obtain the matched score; calculate the loss function using weakly supervised contrast learning, and update the model parameters by back propagation until the model converges to obtain the second target model.
[0086] Optionally, the first original model is a SLIP model, and the second original model is a SuperPoint model.
[0087] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, and this embodiment will not be described in detail here.
[0088] Optionally, in this embodiment, the above-mentioned storage medium may include but is not limited to: a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and other media that can store program codes.
[0089] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.
[0090] If the integrated units in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in the above computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling one or more computer devices (which may be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application.
[0091] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.
[0092] In the several embodiments provided in the present application, it should be understood that the disclosed client can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.
[0093] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0094] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0095] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.
Claims
1. A method for tracing the source of deep fake videos, characterized in that: include: Obtain a global feature matching pre-training dataset collected from the Internet; Using the global feature matching pre-training data set to train a first original model to obtain a first target model, wherein the first original model is a multi-task model combining language supervision and image self-supervision, and is used to learn feature associations between images and feature associations between images and text from the global feature matching pre-training data set; The first target model is used to trace the deep fake video.
2. The method according to claim 1, characterized in that Get the global feature matching pre-training dataset collected from the Internet, including: Images and corresponding title pairs collected from the Internet are obtained to form the global feature matching pre-training dataset.
3. The method according to claim 2, characterized in that The first original model is trained using the global feature matching pre-training data set to obtain a first target model, including: An image in the global feature matching pre-training data set is obtained as an original image and a homology transformation is performed on the original image, the original image and an image after the homology transformation of the original image are used as a positive sample pair, and the original image and an image after the transformation of the non-original image are used as a negative sample pair to be filled into a first sample library, and two images of the sample pair in the first sample library are respectively input into the visual side of the first original model to obtain a feature representation of the image and the image sample pair; An image in the global feature matching pre-training data set is obtained as an original image, the original image and the text corresponding to the original image are used as a positive sample pair, and the original image and the text corresponding to a non-original image are used as a negative sample pair to be filled into a second sample library, and the image and the text of the sample pair in the second sample library are respectively input into the visual side and the text side of the first original model to obtain a feature representation of the image and text sample pair; The loss function of contrastive learning is adopted to perform self-supervised pre-training and weakly supervised pre-training respectively, and the first original model is updated by back propagation until the model converges to obtain the first target model.
4. The method according to any one of claims 1 to 3, characterized in that The method further comprises: Obtain a local feature point matching pre-training dataset collected from the Internet; Using the local feature point matching pre-training data set to train a second original model to obtain a second target model, wherein the second original model is an end-to-end feature point and descriptor extraction network model for learning feature points and descriptors from the local feature point matching pre-training data set; The second target model is used to perform clue expansion on the deep fake video.
5. The method according to claim 4, characterized in that Get the local feature point matching pre-training dataset collected from the Internet, including: An image collected from the Internet is used as an original image and homology transformation is performed on the original image. Each original image and the image after homology transformation of the original image are used as an image pair, thereby forming the local feature point matching pre-training data set.
6. The method according to claim 5, characterized in that The second original model is trained using the local feature point matching pre-training data set to obtain a second target model, including: The original image in the local feature point matching pre-training data set and the image after homology transformation of the original image are used as a positive sample pair, and the original image and the image after homology transformation of the original image are used as a negative sample pair; Inputting the two images in the sample pair into the second original model respectively, extracting local feature points using the local feature point extraction module in the second original model, and sending the extracted local feature points into the local feature matching module of the second original model to obtain a matched score; The loss function is calculated using a weakly supervised contrastive learning method, and the model parameters are updated by back propagation until the model converges to obtain the second target model.
7. The method according to claim 4, characterized in that The first original model is a SLIP model, and the second original model is a SuperPoint model.
8. A device for tracing the source of deep fake videos, characterized in that: include: An acquisition unit, used for acquiring a global feature matching pre-training data set collected from the Internet; A training unit, used to train a first original model using the global feature matching pre-training data set to obtain a first target model, wherein the first original model is a multi-task model combining language supervision and image self-supervision, and is used to learn feature associations between images and feature associations between images and texts from the global feature matching pre-training data set; A tracing unit is used to trace the deep fake video using the first target model.
9. A storage medium, characterized in that: The storage medium includes a stored program, wherein the program executes the method described in any one of claims 1 to 7 when executed.
10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: The processor executes the method described in any one of claims 1 to 7 through the computer program.
