A forged video detection method and system based on pre-trained multimodal model

Abstract

The present invention discloses a method and system for detecting forged videos based on a pre-trained multimodal model, the method comprising: step 1, converting the sound information in the video to be detected into text, and completing the alignment of the subtitles and the video frames; step 2, screening out a set of video frames containing subtitles; step 3, judging the authenticity of each video frame in the set of video frames containing subtitles frame by frame based on the pre-trained multimodal model; step 4, synthesizing the authenticity judgment results of all video frames in the set of video frames containing subtitles to obtain the authenticity of the video to be detected. The present invention uses a pre-trained multimodal model to simultaneously extract the picture and sound information in the video, realize the fusion of visual and auditory information, and improve the accuracy of forged video detection.

Description

Technical Field

[0001] The present invention belongs to the technical field of multimedia information content processing and cyberspace security, and in particular relates to a forged video detection method and system based on a pre-trained multimodal model. Background Art

[0002] There are two main methods at present. One is to detect only based on the image information of the video frame; the other is to send the image information and sound information in the multimodal forged video to the image processing and voice processing modules respectively for detection, without considering the correlation between the image information and the sound content information. Both methods do not make full use of the multimodal information in the video, resulting in low recognition accuracy. Summary of the invention

[0003] Aiming at the problem of low accuracy in detecting and recognizing cross-modal forged videos that simultaneously contain the face and sound of a target person, the present invention proposes a forged video detection method and system based on a pre-trained multimodal model. The pre-trained multimodal model is used to simultaneously extract the image and sound information in the video, thereby realizing the fusion of visual and auditory information and improving the accuracy of forged video detection.

[0004] In order to achieve the above object, the present invention adopts the following technical solutions:

[0005] On the one hand, the present invention provides a forged video detection method based on a pre-trained multimodal model, comprising:

[0006] Step 1: Convert the sound information in the video to be detected into text and align the subtitles with the video frame;

[0007] Step 2: Filter out the video frame set containing subtitles;

[0008] Step 3: Based on the pre-trained multimodal model, the authenticity of each video frame in the set of video frames containing subtitles is judged frame by frame;

[0009] Step 4: The authenticity of the video to be detected is determined by synthesizing the authenticity judgment results of all video frames in the video frame set containing subtitles.

[0010] Furthermore, the step 1 comprises:

[0011] For the video to be detected, first extract the visual information to obtain the video frame sequence; then extract the audio information in the video and use Google's speech-to-text API to transcribe the audio into text; finally, align the video frame sequence and the transcribed text;

[0012] For video frames without speech content, their subtitles are empty and marked as "NULL".

[0013] Furthermore, aligning the video frame sequence and the transcribed text includes:

[0014] For each sentence in the transcribed text, find out the position of its starting frame and ending frame in the original video, and then mark the text of the sentence as subtitles on each video frame between the starting frame and the ending frame as subtitles for these frames.

[0015] Furthermore, the model consists of three layers. The first layer is the embedding layer, which embeds the video frame by superimposing four types of information embedding; the second layer is the feature encoding layer, which extracts visual and auditory information from the video frame based on the pre-trained VL-BERT model to obtain a unified representation feature vector; the third layer is the decision layer, which makes a decision on the authenticity of the video frame based on the multi-layer perceptron MLP.

[0016] Furthermore, the step 3 comprises:

[0017] The text elements and visual elements in the subtitled video frames are converted into sequences, encoded and embedded, and then sent to the pre-trained VL-BERT model to extract multimodal unified features;

[0018] Embed the input sequence. The embedding of the input sequence consists of four types of information: the first type of information is position embedding, the second type of information is fragment embedding, the third type of information is visual feature embedding, and the fourth type of information is Token embedding;

[0019] For each video frame, after converting it into an input sequence, the embedding of the above four types of information is obtained, and the original information embedding of the input sequence is obtained after superposition;

[0020] The above raw information is embedded into the trained VL-BERT model to obtain a unified representation that integrates visual and language information.

[0021] The unified representation is input into a multi-layer perceptron for judgment. The hidden layer of the perceptron has two nodes. The output value of the first node is used as the probability that the video frame is a real video frame. If the output result is greater than 0.5, the video frame is judged to be real, otherwise it is judged to be forged.

[0022] Furthermore, the visual features are composed of visual geometric features and visual appearance features; the visual geometric embedding of each visual element is represented by a four-dimensional vector Where (x LT ,y LT ) and (x RB ,y RB ) represent the coordinates of the upper left corner and the lower right corner, respectively, W and H represent the width and height of the input video frame; the visual appearance features of the visual elements in the input sequence are extracted by the AlexNet detector.

[0023] Furthermore, the VL-BERT model consists of multiple bidirectional Transformer encoders, each Transformer consists of a self-attention layer and a fully connected layer; the model is pre-trained on the Conceptual Captions dataset.

[0024] Furthermore, the step 4 comprises:

[0025] For each video, if the number of video frames judged to be real is greater than the number of video frames judged to be forged, the video is judged to be a real video, otherwise it is judged to be a forged video.

[0026] Another aspect of the present invention provides a forged video detection system based on a pre-trained multimodal model, comprising:

[0027] An information conversion module is used to convert the sound information in the video to be detected into text and complete the alignment of subtitles and video frames;

[0028] A filtering module, used to filter out a set of video frames containing subtitles;

[0029] A first judgment module is used to judge the authenticity of each video frame in a set of video frames containing subtitles frame by frame based on a pre-trained multimodal model;

[0030] The second judgment module is used to comprehensively judge the authenticity of all video frames in the video frame set containing subtitles to determine the authenticity of the video to be detected.

[0031] Furthermore, the information conversion module is specifically used for:

[0032] For the video to be detected, first extract the visual information to obtain the video frame sequence; then extract the audio information in the video and use Google's speech-to-text API to transcribe the audio into text; finally, align the video frame sequence and the transcribed text;

[0033] For video frames without speech content, their subtitles are empty and marked as "NULL".

[0034] Furthermore, aligning the video frame sequence and the transcribed text includes:

[0035] For each sentence in the transcribed text, find out the position of its starting frame and ending frame in the original video, and then mark the text of the sentence as subtitles on each video frame between the starting frame and the ending frame as subtitles for these frames.

[0036] Furthermore, the model consists of three layers. The first layer is the embedding layer, which embeds the video frame by superimposing four types of information embedding; the second layer is the feature encoding layer, which extracts visual and auditory information from the video frame based on the pre-trained VL-BERT model to obtain a unified representation feature vector; the third layer is the decision layer, which makes a decision on the authenticity of the video frame based on the multi-layer perceptron MLP.

[0037] Furthermore, the first determination module is specifically used for:

[0038] The text elements and visual elements in the subtitled video frames are converted into sequences, encoded and embedded, and then sent to the pre-trained VL-BERT model to extract multimodal unified features;

[0039] Embed the input sequence. The embedding of the input sequence consists of four types of information: the first type of information is position embedding, the second type of information is fragment embedding, the third type of information is visual feature embedding, and the fourth type of information is Token embedding;

[0040] For each video frame, after converting it into an input sequence, the embedding of the above four types of information is obtained, and the original information embedding of the input sequence is obtained after superposition;

[0041] The above raw information is embedded into the trained VL-BERT model to obtain a unified representation that integrates visual and language information.

[0042] The unified representation is input into a multi-layer perceptron for judgment. The hidden layer of the perceptron has two nodes. The output value of the first node is used as the probability that the video frame is a real video frame. If the output result is greater than 0.5, the video frame is judged to be real, otherwise it is judged to be forged.

[0043] Furthermore, the visual features are composed of visual geometric features and visual appearance features; the visual geometric embedding of each visual element is represented by a four-dimensional vector Where (x LT ,y LT ) and (x RB ,y RB ) represent the coordinates of the upper left corner and the lower right corner, respectively, W and H represent the width and height of the input video frame; the visual appearance features of the visual elements in the input sequence are extracted by the AlexNet detector.

[0044] Furthermore, the VL-BERT model consists of multiple bidirectional Transformer encoders, each Transformer consists of a self-attention layer and a fully connected layer; the model is pre-trained on the Conceptual Captions dataset.

[0045] Furthermore, the second judgment module is specifically used for:

[0046] For each video, if the number of video frames judged to be real is greater than the number of video frames judged to be forged, the video is judged to be a real video, otherwise it is judged to be a forged video.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] Aiming at the problem of low accuracy in detecting and recognizing cross-modal forged videos that simultaneously contain the face and sound of a target person, the present invention proposes a forged video detection method and system based on a pre-trained multimodal model. The pre-trained multimodal model is used to simultaneously extract the image and sound information in the video, thereby realizing the fusion of visual and auditory information and improving the accuracy of forged video detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 This is a flow chart of a forged video detection method based on a pre-trained multimodal model according to an embodiment of the present invention;

[0050] Figure 2 This is a flowchart of forged video frame detection based on VLBERT-MLP in an embodiment of the present invention;

[0051] Figure 3 This is a schematic diagram of the network structure of AlexNet according to an embodiment of the present invention;

[0052] Figure 4 This is a schematic diagram of the structure of the Transformer according to an embodiment of the present invention;

[0053] Figure 5 The figure is a schematic diagram of the architecture of a forged video detection system based on a pre-trained multimodal model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0054] The present invention will be further explained below with reference to the accompanying drawings and specific embodiments:

[0055] like Figure 1 As shown, a forged video detection method based on a pre-trained multimodal model of the present invention comprises:

[0056] Step 1: Convert the sound information in the video to be detected into text and align the subtitles with the video frame;

[0057] Step 2: Filter out the video frame set containing subtitles;

[0058] Step 3: Based on the pre-trained multimodal model, the authenticity of each video frame in the set of video frames containing subtitles is judged frame by frame;

[0059] Step 4: The authenticity of the video to be detected is determined by synthesizing the authenticity judgment results of all video frames in the video frame set containing subtitles.

[0060] The present invention integrates the image information and sound information in the video for the detection of forged videos. Since the vision-language model is used to fuse these two types of information in this patent, the sound information must first be converted into text and the alignment of the text with the video frame must be completed. The specific process is as follows.

[0061] For the video to be detected, first extract visual information to obtain a video frame sequence; then extract audio information from the video and use Google's speech-to-text API to transcribe the audio into text; finally, align the video frame sequence and the transcribed text. The specific process is to find the position of the start frame and the end frame of each sentence in the transcribed text in the original video, and then mark the text of this sentence as a subtitle on each video frame between the start frame and the end frame as the subtitle of these frames. For video frames without speech content, their subtitles are empty and marked as "NULL". For example, there is a video with a total of 20 frames, and the characters in it say two sentences: "we sing birthday part, we have fun." If "we sing birthday part" starts from the 3rd frame and ends at the 6th frame; "we have fun." starts from the 10th frame and ends at the 11th frame, then the subtitle annotation results for these frames are as follows:

[0062] Table 1. Results of processing the sound information in a video and matching it with the video frame

[0063]

[0064] Next, take the video frames whose subtitles are not NULL, and follow Figure 1 The processing flow shown performs authenticity detection frame by frame. For the results of the sound and video frame matching in Table 1, it is necessary to perform authenticity detection on the 3rd, 4th, 5th, 6th, 10th, 11th, and 12th frames one by one.

[0065] Figure 1 The authenticity detection of each video frame is performed as follows Figure 2 The model (VLBERT-MLP) shown in the figure is mainly composed of three layers. The first layer is the embedding layer, which embeds the video frame by superimposing four types of information embedding; the second layer is the feature encoding layer, which extracts visual and auditory information from the video frame based on the trained pre-trained model VL-BERT to obtain a unified feature vector; the third layer is the decision layer, which makes a decision on the authenticity of the video frame based on the multilayer perceptron (MLP). Each layer is introduced below.

[0066] We take the processing of the third frame in Table 1 as an example to illustrate. Figure 2 As shown in the figure, the text elements and visual elements in the video frame with subtitles are converted into sequences, encoded and embedded, and then sent to VL-BERT to extract multimodal unified features. VL-BERT can only process sequences of fixed length, so here we limit the length of the input sequence to 30. For video frames where the total number of text elements, visual elements, and other identification elements is greater than 30, we intercept the subtitle text information and only take the first part of the content to ensure that the total length of the sequence is 30. Specifically, the total number of visual elements and text elements in the input sequence cannot exceed 27, excluding the three required identification tokens [CLS], [SEP], and . If the number of faces in a video frame is N, then only the first (27-N) words in the subtitle text can be taken as text elements. If the sum of the number of words and faces in the subtitle text is less than 27, the sequence is padded with 0 to 30.

[0067] We have previously introduced how to convert video frames into input sequences. Now we will introduce how to embed the input sequences. The embedding of the input sequence consists of four types of information. The first type of information is position embedding, which is used to add a learnable sequence position feature to each input element to represent its order in the input sequence. The sequence position feature of each word is related to its position in the subtitle sentence. The position of each word in the sentence is arranged in sequence according to the relationship between the previous and the next words. Since there is no natural order between the input visual elements, any arrangement of them in the input sequence should give the same result, so the sequence position features of the visual elements are the same.

[0068] The second type of information is fragment embeddings, which are used to distinguish input elements from different sources, such as Figure 2 As shown, A represents information from subtitles, and B represents information from images.

[0069] The third type of information is visual feature embedding, such as Figure 2 As shown in the figure on the right, for the face area extracted from the video frame and the entire video frame area, visual geometric features and visual appearance features are extracted respectively, and then the two are connected in series to form visual features. The face extraction process can be completed using the RetinaFace model. Visual geometric embedding is used to represent the geometric position of each input visual element in the video frame. The visual geometric embedding of each visual element is represented by a four-dimensional vector. Where (x LT ,y LT ) and (x RB ,y RB) represent the coordinates of the upper left corner and the lower right corner, respectively, and W and H represent the width and height of the input video frame. The visual appearance features of the visual elements in the input sequence are extracted by the AlexNet detector; the corresponding visual appearance features of the non-visual elements in the input sequence are the features extracted from the entire input video frame. The network structure of AlexNet is as follows Figure 3 As shown in Figure 1, the parameters of the network are trained using the training data of the ImageNet dataset.

[0070] Figure 3 In the above code, fc stands for fully connected layer, pool stands for pooling layer, and conv stands for convolution layer. The first parameter after the convolution layer stands for the number of convolution kernels, and the second parameter stands for the step size of the convolution. Figure 3 As shown in the figure, the number of convolution kernels in the first convolution layer is 96 and the stride is 4; the number of convolution kernels in the second convolution layer is 256.

[0071] The fourth type of information is token embedding, where each word in the subtitle text gets its own embedding, and a special token is assigned to each special element. A [CLS] tag is added at the beginning of the sequence to identify the learnable classification result; for visual elements, a special [IMG] tag is assigned to each element; in addition, a [SEP] tag is used to separate the visual information and the subtitle information.

[0072] For each video frame, after converting it into an input sequence, the above four types of information are embedded, and after superposition, the original information embedding of the input sequence is obtained; then the above original information embedding is input into the pre-trained VL-BERT model to output a feature vector containing both visual and language representations, so as to extract all information features in the video frame. VL-BERT consists of multiple bidirectional Transformer encoders, each of which consists of a self-attention layer and a fully connected layer. The structure of the Transformer is as follows: Figure 4 shown.

[0073] The specific process of extracting features from the input sequence using the Transformer-based multi-head attention network is as follows:

[0074] z' l =MHA(LN(z l-1 ))+(z l-1 ),l=1...L (1)

[0075] z l =MLP(LN(z' l ))+(z' l ),l=1...L (2)

[0076]

[0077] MHA(·) in formulas (1)-(3) represents Figure 4 The multi-head attention mechanism in LN(·) represents Figure 4 The layer normalization in MLP(·) means Figure 4 The multi-layer perceptron in Represents z l The data of the first dimension. Among them, LN(·) is used to normalize one or several dimensions in the input. Assume that the dimension data to be normalized for the input X is x={x1,x2,…x n}, the calculation formula is:

[0078]

[0079] Where E(x) is the mean of x, Var[x] is the variance of x, ε is a very small value added to prevent the denominator from being 0, and is generally taken as 1e-05.

[0080] The calculation process of MHA(·) is as follows:

[0081] Q=W Q ·X (5)

[0082] K=W K ·X (6)

[0083] V=W V ·X (7)

[0084]

[0085] Among them, W Q , W K , W V is the learnable parameter matrix, d is K T The dimension of Q, assuming that K represents the number of output categories of the neural network, v is the output vector, and v j is the value of the jth output category in v, i represents the category that needs to be calculated, the calculation result is between 0 and 1, and the sum of the softmax values ​​of all categories is 1. The calculation formula of softmax(·) is:

[0086]

[0087] The calculation process of MLP(·) is as follows:

[0088] We use a multilayer perceptron with only 2 hidden layers. Assume that the dimension of the input vector x is M, the first hidden layer has H1 nodes (the value of H1 is the same as the dimension D of the input image block embedding), and the second hidden layer has H2 nodes. The value of H2 is the output dimension of the multilayer perceptron. The calculation formula for the output of the first hidden layer node is:

[0089]

[0090] The calculation formula for the output of the second hidden layer node is:

[0091]

[0092] are the learnable weights of the first and second layers respectively. g(·) represents the activation function, and the ReLU function is used here, and the calculation formula is:

[0093]

[0094] In the present invention, VL-BERT is composed of the above three Transformer structures. The model is first pre-trained on the ConceptualCaptions dataset, which contains about 3.3 million pictures with caption annotations. During the training process, the text in the title in the dataset needs to be processed as follows. The words in the text of all titles in the dataset are statistically analyzed to build a text encoding dictionary. The specific process is to count all English words in the titles of the training dataset, extract the 1000 words with the highest frequency, and perform one-hot encoding on these 1000 words to form an encoding dictionary.

[0095] After the original information of each video frame is embedded into the above VL-BERT model, a unified representation integrating visual and language information is obtained, which is then input into Figure 2 The decision is made in the multi-layer perceptron shown at the top layer. Fake video detection is essentially a binary classification problem, so the hidden layer of this perceptron has two nodes. The output value of the first node is used as the probability that the video frame is a real video frame; the output value of the second node is used as the probability that the video frame is a fake video frame.

[0096] right Figure 2 The training process of the detection model (VLBERT-MLP) for detecting authenticity of video frames is as follows:

[0097] Training phase: Collect 100 real videos with sound and faces and 100 deepfake-generated videos with sound and faces. For each real video, use Google's speech-to-text API to transcribe the audio into text. Finally, align the video frame sequence and the transcribed text. For details, see Figure 1 The following text description. After the above processing for each real video, extract the video frames whose subtitles are not NULL as real valid frames. The real valid frames in all real videos form a positive sample set with subtitles Where N p Represents the sum of real valid frames in all real videos.

[0098] Similarly, for each deep fake video, the same alignment operation of video frames and subtitles is performed, and then the video frames whose subtitles are not NULL are extracted as fake valid frames. The fake valid frames in all fake videos are combined into a negative sample set with subtitles. Where N n Represents the sum of forged valid frames in all deep fake videos.

[0099] Enter P and N into Figure 1 The network shown in Figure 1 is trained in which the label of each sample in the positive sample set is 1 and the label of each sample in the negative sample set is 0.

[0100] In the detection phase, for the video to be detected, Google's speech-to-text API is used to transcribe the audio into text; then, the video frame sequence and the transcribed text are aligned. For video frames without speech content, their subtitles are empty and marked as "NULL". For video frames whose subtitles are not marked as "NULL", such as Figure 1 The video is sent to the trained model for detection. If the output result is greater than 0.5, it is judged to be real, otherwise it is judged to be forged. For each video, if the number of video frames judged to be real is greater than the number of video frames judged to be forged, the video is judged to be a real video, otherwise it is judged to be a forged video.

[0101] The following is a detailed description of the detection process of the first video frame (i.e., the third frame of the video) whose subtitle is not NULL in Table 1. The number of text elements in this frame is 4 (4 words), the number of visual elements is 2 (2 faces), plus the identification tokens [CLS], [SEP], and , the total sequence length is 9. The sequence of length 9 is first encoded and embedded, and then padded to a sequence of length 30.

[0102] The embedding of the four types of information maps the corresponding elements to a 9×1004-dimensional space. Position embedding uses a learnable encoding method, so it is initialized to a 9×1004-dimensional matrix of all zeros. Fragment embedding uses one-hot encoding. For elements encoded as "A" from text, they are embedded into {1,0,0,....,0}, except that the first value is 1 and there are 1003 zeros behind it; for elements encoded as "B" from images, they are embedded into {0,1,0,0,....,0}, except that the second value is 1, the first value and the remaining 1002 values ​​are all 0. In visual embedding, such as Figure 2 As shown, the visual elements at the 1st, 2nd, 3rd, 4th, 5th, 6th and 9th positions of the sequence are the entire video frame, and the visual elements at the 7th and 8th positions are two faces detected from the video. Each visual element is fed into Figure 3 The process of extracting visual embeddings in the AlnexNet network shown is as follows:

[0103] (1) For the input visual information, it is first reshaped into a size of 227×227×3, and then sent to the first convolution layer. This layer has 96 convolution kernels, each of which is 11×11×3 in size and has a step size of 4. Since (227-11) / 4+1=55, the output after convolution is 55×55×96. These output values ​​are then filtered using Relu, and the data dimension after Relu is also 55×55×96. Then, it is sent to the Max pool, which uses a 3X3 pooling kernel with a step size of 2. The data dimension after the Max pool is 27×27×96. Finally, LRN (Local Response Normalization) is used for normalization to obtain an output with a dimension of 27×27×96.

[0104] (2) The above 27×27×96 output is used as the input of the second convolutional layer. The second convolutional layer has 256 convolution kernels. The size of each convolution kernel is 5×5 and the stride is 1. The dimension after convolution is 27×27×256. Then, these output values ​​are filtered by Relu. The data dimension after Relu is also 27×27×256. Maxpool uses a 3×3 pooling kernel with a stride of 2. The data dimension after Max pool is 13×13×256. Finally, LRN (Local Response Normalization) is used for normalization to obtain an output with a dimension of 13×13×256.

[0105] (3) The above 13×13×256 output is used as the input of the third convolutional layer. The third convolutional layer has 384 convolution kernels. The size of each convolution kernel is 3×3, the stride is 1, and the dimension after convolution is 13×13×384. Then these output values ​​are filtered by Relu. The data dimension after Relu is also 13×13×384 as the output of this layer.

[0106] (4) The above 13×13×384 output is used as the input of the fourth convolutional layer. The fourth convolutional layer has 384 convolution kernels. The size of each convolution kernel is 3×3, the stride is 1, and the dimension after convolution is 13×13×384. Then these output values ​​are filtered by Relu. The data dimension after Relu is also 13×13×384 as the output of this layer.

[0107] (5) The above 13×13×384 output is used as the input of the fifth convolutional layer. The fifth convolutional layer has 256 convolution kernels. The size of each convolution kernel is 3×3, the stride is 1, and the dimension after convolution is 13×13×256. Then these output values ​​are filtered by Relu. The data dimension after Relu is also 13×13×256. Maxpool uses a 3×3 pooling kernel with a stride of 2. The data dimension after Max pool is 6×6×256. Finally, LRN (Local Response Normalization) is used for normalization to obtain an output with a dimension of 6×6×256.

[0108] (6) The above 6×6×256 output is sent to the fully connected layer of the sixth layer, linearly mapped into a 4096×1 dimensional output, and then through ReLU and Drop out, outputting a 4096×1 dimensional result;

[0109] (7) The above 4096×1 output is sent to the fully connected layer of the 7th layer, linearly mapped to a 4096×1 output, and then through ReLU and Drop out, outputting a 4096×1 result;

[0110] (8) The above 4096×1 output is sent to the 8th fully connected layer and linearly mapped into a 1000×1 dimensional output.

[0111] The above 1000×1-dimensional visual features and 4×1-dimensional visual geometric embedding of each visual element are concatenated to obtain a 1004×1-dimensional visual embedding, where the visual geometric embedding of the face is represented by a 4-dimensional vector Where (x LT ,y LT ) and (x RB ,y RB) represent the coordinates of the upper left corner and lower right corner of the face area, W and H represent the width and height of the input image. The geometric embeddings of other visual elements are all {0,1,1,0}. Figure 2 The size of the entire video frame is 272×272, where the coordinates of the upper left corner and lower right corner of the first red face region are {49,190} and {114,149}, respectively, and the coordinates of the upper left corner and lower right corner of the second purple face region are {133,217} and {177,163}, respectively. The geometric embeddings of the visual elements at the 1st, 2nd, 3rd, 4th, 5th, 6th and 9th positions of the sequence are {0,1,1,0}, and those at the 7th and 8th positions are {0.18,0.699,0.419,0.548} and {0.489,0.798,0.651,0.599}, respectively.

[0112] Token embedding uses one-hot encoding, where [CLS], [SEP], [IMG], and are encoded into 1004-dimensional vectors c [CLS] ={1,0,...,0}(1 followed by 1003 0s), c [SEP] ={0,1,0,....,0}(1 followed by 1002 0s), c [IMG] ={0,0,1,0,....,0}(1 followed by 1001 0s) and c [END] ={0,0,0,1,0,....,0}(1 followed by 1000 0s). For other tokens (words), the first 4 dimensions are all 0 during the embedding process, and encoding is performed from the 5th dimension to the 1004th dimension. For each word, a query is performed in the encoding dictionary built based on the training corpus. If it can be found, the 1000-dimensional encoding is taken out and 4 0s are added in front to form the token embedding here. Figure 2 In the code dictionary, the three words "we", "sing" and "part" can be found and their codes are c we ={0,...,0,1,0,...,0}(1 has 40 zeros before it and 959 zeros after it), c sing ={0,...,0,1,0,...,0}(1 has 111 0s before it and 888 0s after it), c part ={0,...,0,1,0,...,0}c sing ={0,...,0,1,0,...,0} (1 has 519 zeros in front and 480 zeros in the back). Combining the encoding results of the previous four special tokens, the encoding of these three words is expanded to 1004 dimensions, which are (1 has 44 zeros before it and 959 zeros after it), (1 has 115 zeros before it and 888 zeros after it) and (There are 523 zeros before 1 and 480 zeros after it.) The word "birthday" is not in the encoding dictionary, so it is encoded as a 1004-dimensional all-zero vector. (1004 zeros).

[0113] After adding the embeddings of the above four types of information, we get the original information embedding of 9×1004 dimensions. As mentioned above, it needs to be expanded to 30*1004 dimensions. The specific method is to fill the 10th to 30th rows with all 0s.

[0114] After the encoding is embedded, the above 30×1004 dimension is input into VL-BERT for feature extraction. As shown above, VL-BERT consists of 3 Figure 4 The structure is shown in Figure 1. The Transformer consists of 6 heads of the multi-head attention mechanism. The multi-layer perceptron in the Transformer has 1 hidden layer, which is used to convert the 30×1004-dimensional input into 30×200 dimensions.

[0115] Finally, the first row of the 30×200 dimension output of VL-BERT is input to Figure 2 The top multi-layer perceptron for discrimination makes the decision. This perceptron has one hidden layer, which is used to convert the 1×200-dimensional input into 1×2 dimensions. Figure 2 For example, in the third video frame, the output of the multi-layer perceptron is {0.7, 0.15}. Since 0.7>0.15, the video frame is judged to be a real video frame.

[0116] According to the above process, the 3rd, 4th, 5th, 6th, 10th and 11th frames of the video are detected in turn and the results are shown in Table 2:

[0117] Table 2. Detection results of the 3rd, 4th, 5th, 6th, 10th, and 11th video frames

[0118]

[0119] Based on the detection results of all frames in Table 2, it is determined that the video is a forged video.

[0120] It is worth noting that, in this embodiment, the language spoken in the video is described as English, and this method is also applicable to the case where the language spoken in the video is Chinese. When the language spoken in the video is Chinese, when performing Token embedding, each element of the text portion sequence is each word in the subtitle of this video frame.

[0121] Based on the above embodiments, Figure 5As shown, the present invention also proposes a forged video detection system based on a pre-trained multimodal model, comprising:

[0122] An information conversion module is used to convert the sound information in the video to be detected into text and complete the alignment of subtitles and video frames;

[0123] A filtering module, used to filter out a set of video frames containing subtitles;

[0124] A first judgment module is used to judge the authenticity of each video frame in a set of video frames containing subtitles frame by frame based on a pre-trained multimodal model;

[0125] The second judgment module is used to comprehensively judge the authenticity of all video frames in the video frame set containing subtitles to determine the authenticity of the video to be detected.

[0126] Furthermore, the information conversion module is specifically used for:

[0127] For the video to be detected, first extract the visual information to obtain the video frame sequence; then extract the audio information in the video and use Google's speech-to-text API to transcribe the audio into text; finally, align the video frame sequence and the transcribed text;

[0128] For video frames without speech content, their subtitles are empty and marked as "NULL".

[0129] Furthermore, aligning the video frame sequence and the transcribed text includes:

[0130] For each sentence in the transcribed text, find out the position of its starting frame and ending frame in the original video, and then mark the text of the sentence as subtitles on each video frame between the starting frame and the ending frame as subtitles for these frames.

[0131] Furthermore, the model consists of three layers. The first layer is the embedding layer, which embeds the video frame by superimposing four types of information embedding; the second layer is the feature encoding layer, which extracts visual and auditory information from the video frame based on the pre-trained VL-BERT model to obtain a unified representation feature vector; the third layer is the decision layer, which makes a decision on the authenticity of the video frame based on the multi-layer perceptron MLP.

[0132] Furthermore, the first determination module is specifically used for:

[0133] The text elements and visual elements in the subtitled video frames are converted into sequences, encoded and embedded, and then sent to the pre-trained VL-BERT model to extract multimodal unified features;

[0134] Embed the input sequence. The embedding of the input sequence consists of four types of information: the first type of information is position embedding, the second type of information is fragment embedding, the third type of information is visual feature embedding, and the fourth type of information is Token embedding;

[0135] For each video frame, after converting it into an input sequence, the embedding of the above four types of information is obtained, and the original information embedding of the input sequence is obtained after superposition;

[0136] The above raw information is embedded into the trained VL-BERT model to obtain a unified representation that integrates visual and language information.

[0137] The unified representation is input into a multi-layer perceptron for judgment. The hidden layer of the perceptron has two nodes. The output value of the first node is used as the probability that the video frame is a real video frame. If the output result is greater than 0.5, the video frame is judged to be real, otherwise it is judged to be forged.

[0138] Furthermore, the visual features are composed of visual geometric features and visual appearance features; the visual geometric embedding of each visual element is represented by a four-dimensional vector Where (x LT ,y LT ) and (x RB ,y RB ) represent the coordinates of the upper left corner and the lower right corner, respectively, W and H represent the width and height of the input video frame; the visual appearance features of the visual elements in the input sequence are extracted by the AlexNet detector.

[0139] Furthermore, the VL-BERT model consists of multiple bidirectional Transformer encoders, each Transformer consists of a self-attention layer and a fully connected layer; the model is pre-trained on the Conceptual Captions dataset.

[0140] Furthermore, the second judgment module is specifically used for:

[0141] For each video, if the number of video frames judged to be real is greater than the number of video frames judged to be forged, the video is judged to be a real video, otherwise it is judged to be a forged video.

[0142] In summary, the present invention aims to solve the problem of low accuracy in detecting and recognizing cross-modal forged videos that contain both the face and sound of a target person. A forged video detection method and system based on a pre-trained multimodal model are proposed. The pre-trained multimodal model is used to simultaneously extract the image and sound information in the video, thereby realizing the fusion of visual and auditory information and improving the accuracy of forged video detection.

[0143] The above is only a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A forged video detection method based on a pre-trained multimodal model, characterized in that: include: Step 1: Convert the sound information in the video to be detected into text and align the subtitles with the video frame; Step 2: Filter out the video frame set containing subtitles; Step 3: Based on the pre-trained multimodal model, the authenticity of each video frame in the set of video frames containing subtitles is judged frame by frame; including: The text elements and visual elements in the subtitled video frames are converted into sequences, encoded and embedded, and then sent to the pre-trained VL-BERT model to extract multimodal unified features; Embed the input sequence. The embedding of the input sequence consists of four types of information: the first type of information is position embedding, the second type of information is fragment embedding, the third type of information is visual feature embedding, and the fourth type of information is Token embedding; For each video frame, after converting it into an input sequence, the embedding of the above four types of information is obtained, and the original information embedding of the input sequence is obtained after superposition; The above raw information is embedded into the trained VL-BERT model to obtain a unified representation that integrates visual and language information. The unified representation is input into a multi-layer perceptron for judgment. The hidden layer of the perceptron has two nodes. The output value of the first node is used as the probability that the video frame is a real video frame. If the output result is greater than 0.5, the video frame is judged to be real, otherwise it is judged to be forged. The visual features are composed of visual geometric features and visual appearance features; the visual geometric embedding of each visual element is represented by a four-dimensional vector Where (x LT ,y LT ) and (x RB ,y RB ) represent the coordinates of the upper left corner and the lower right corner, respectively, W and H represent the width and height of the input video frame; the visual appearance features of the visual elements in the input sequence are extracted by the AlexNet detector; The VL-BERT model consists of multiple bidirectional Transformer encoders, each of which consists of a self-attention layer and a fully connected layer; the model is pre-trained on the Conceptual Captions dataset; Step 4: The authenticity of the video to be detected is determined by synthesizing the authenticity judgment results of all video frames in the video frame set containing subtitles.

2. According to claim 1, a forged video detection method based on a pre-trained multimodal model is characterized in that: The step 1 comprises: For the video to be detected, first extract the visual information to obtain the video frame sequence; then extract the audio information in the video and use Google's speech-to-text API to transcribe the audio into text; finally, align the video frame sequence and the transcribed text; For video frames without speech content, their subtitles are empty and marked as "NULL".

3. The method for detecting fake videos based on a pre-trained multimodal model according to claim 2, characterized in that: The aligning of the video frame sequence and the transcribed text comprises: For each sentence in the transcribed text, find out the position of its starting frame and ending frame in the original video, and then mark the text of the sentence as subtitles on each video frame between the starting frame and the ending frame as subtitles for these frames.

4. The method for detecting forged videos based on a pre-trained multimodal model according to claim 1, characterized in that: The model consists of three layers. The first layer is an embedding layer, which embeds the video frame by superimposing four types of information embeddings. The second layer is the feature encoding layer, which extracts visual and auditory information from the video frames based on the pre-trained VL-BERT model to obtain a unified representation feature vector; the third layer is the decision layer, which judges the authenticity of the video frames based on the multi-layer perceptron MLP.

5. The method for detecting forged videos based on a pre-trained multimodal model according to claim 1, characterized in that: The step 4 comprises: For each video, if the number of video frames judged to be real is greater than the number of video frames judged to be forged, the video is judged to be a real video, otherwise it is judged to be a forged video.

6. A forged video detection system based on a pre-trained multimodal model, characterized in that: include: An information conversion module is used to convert the sound information in the video to be detected into text and complete the alignment of subtitles and video frames; A filtering module, used to filter out a set of video frames containing subtitles; The first judgment module is used to judge the authenticity of each video frame in the set of video frames containing subtitles frame by frame based on the pre-trained multimodal model; specifically used for: The text elements and visual elements in the subtitled video frames are converted into sequences, encoded and embedded, and then sent to the pre-trained VL-BERT model to extract multimodal unified features; Embed the input sequence. The embedding of the input sequence consists of four types of information: the first type of information is position embedding, the second type of information is fragment embedding, the third type of information is visual feature embedding, and the fourth type of information is Token embedding; For each video frame, after converting it into an input sequence, the embedding of the above four types of information is obtained, and the original information embedding of the input sequence is obtained after superposition; The above raw information is embedded into the trained VL-BERT model to obtain a unified representation that integrates visual and language information. The unified representation is input into a multi-layer perceptron for judgment. The hidden layer of the perceptron has two nodes. The output value of the first node is used as the probability that the video frame is a real video frame. If the output result is greater than 0.5, the video frame is judged to be real, otherwise it is judged to be forged. The visual features are composed of visual geometric features and visual appearance features; the visual geometric embedding of each visual element is represented by a four-dimensional vector Where (x LT ,y LT ) and (x RB ,y RB ) represent the coordinates of the upper left corner and the lower right corner, respectively, W and H represent the width and height of the input video frame; the visual appearance features of the visual elements in the input sequence are extracted by the AlexNet detector; The VL-BERT model consists of multiple bidirectional Transformer encoders, each of which consists of a self-attention layer and a fully connected layer; the model is pre-trained on the Conceptual Captions dataset; The second judgment module is used to comprehensively judge the authenticity of all video frames in the video frame set containing subtitles to determine the authenticity of the video to be detected.

Images