Video detection methods, devices, and computer equipment based on multimodal information fusion

Abstract

This application relates to a video detection method, apparatus, and computer device that integrates multimodal information fusion. It identifies important information such as scenes, public figures, locations, entities, and keywords appearing in a video to form visual text, and extracts visual text features, video text features, visual features, and audio features. First, it fuses the visual text features and video text features to obtain new text features, thereby strengthening the text information and initially achieving cross-modal fusion of visual and text information. Then, it fuses this new text feature with the extracted visual and frequency features to further achieve cross-modal fusion of visual, audio, and text information. In summary, the video detection method using progressive cross-modal information fusion in this solution can fully utilize the correlation between modalities and leverage multiple modal information, thereby improving the detection accuracy of fake videos.

Description

Technical Field

[0001] This application relates to the field of fake video detection technology, and in particular to a video detection method, apparatus and computer equipment based on multimodal information fusion. Background Technology

[0002] Most existing methods for video detection rely solely on the visual information of the video as a single modality, or detect the visual, text, and audio information separately. These methods fail to consider the correlation between modalities and do not fully utilize the multimodal information of the video, resulting in poor video recognition accuracy. Summary of the Invention

[0003] Therefore, it is necessary to provide a video detection method, apparatus, and computer equipment based on multimodal information fusion that can improve the accuracy of video recognition, addressing the aforementioned technical problems.

[0004] A video detection method based on multimodal information fusion, the method comprising:

[0005] Obtain the video to be detected; the video to be detected includes visual information, text information, and audio information;

[0006] Identify scenes, people, locations, entities, and keywords in visual information and assemble them into visual text;

[0007] Visual features and visual text features in visual information, video text features in text information, and audio features in audio text are extracted using a pre-trained feature extraction model.

[0008] By fusing visual text features and video text features, a fused text feature is obtained;

[0009] Multimodal fusion features are obtained by fusing visual features, audio features, and fused text features;

[0010] The authenticity of the video to be detected is determined based on the multimodal fusion features.

[0011] A video detection device based on multimodal information fusion, the device comprising:

[0012] The video acquisition module is used to acquire the video to be detected; the video to be detected includes visual information, text information and audio information;

[0013] The visual text acquisition module is used to identify scenes, people, locations, entities, and keywords in visual information and assemble them into visual text.

[0014] The feature extraction module is used to extract visual features and visual text features from visual information, video text features from text information, and audio features from audio text using a pre-trained feature extraction model.

[0015] The text fusion module is used to fuse visual text features and video text features to obtain fused text features.

[0016] The multimodal information fusion module is used to fuse visual features, audio features, and fused text features to obtain multimodal fused features;

[0017] The video detection module is used to determine whether a video to be detected is real or fake based on multimodal fusion features.

[0018] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program performing the following steps:

[0019] Obtain the video to be detected; the video to be detected includes visual information, text information, and audio information;

[0020] Identify scenes, people, locations, entities, and keywords in visual information and assemble them into visual text;

[0021] Visual features and visual text features in visual information, video text features in text information, and audio features in audio text are extracted using a pre-trained feature extraction model.

[0022] By fusing visual text features and video text features, a fused text feature is obtained;

[0023] Multimodal fusion features are obtained by fusing visual features, audio features, and fused text features;

[0024] The authenticity of the video to be detected is determined based on the multimodal fusion features.

[0025] The aforementioned multimodal information fusion video detection method, apparatus, and computer equipment identify important information such as scenes, public figures, locations, entities, and keywords appearing in the video to form visual text. It also extracts visual text features, video text features, and visual and audio features. First, it fuses the visual text features and video text features to obtain new text features, thereby strengthening the text information and initially achieving cross-modal fusion of visual and text information. Then, it fuses this new text feature with the extracted visual and audio features to further achieve cross-modal fusion of visual, audio, and text information. In summary, the video detection method using progressive cross-modal information fusion in this scheme can fully utilize the correlation between modalities and leverage multiple modal information, thereby improving the detection accuracy of fake videos. Attached Figure Description

[0026] Figure 1 This is a flowchart illustrating a video detection method based on multimodal information fusion in one embodiment.

[0027] Figure 2 This is a schematic diagram of a video detection method based on multimodal information fusion in one embodiment;

[0028] Figure 3 This is a schematic diagram of the VIT model structure in one embodiment;

[0029] Figure 4 This is a schematic diagram of the text information fusion process in one embodiment;

[0030] Figure 5 This is a schematic diagram of multimodal fusion feature generation in one embodiment;

[0031] Figure 6 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. Detailed Implementation

[0032] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0033] In one embodiment, such as Figure 1 As shown, a video detection method based on multimodal information fusion is provided, including the following steps:

[0034] Step 102: Obtain the video to be tested.

[0035] The videos to be tested can be news videos, videos of public figures, etc. Taking news videos as an example, with the advancement of technology, people have become increasingly convenient in receiving news information, especially with the popularity of short video platforms, making news videos one of the main channels for people to obtain news. However, there are also some news video accounts that publish fake or fabricated news videos, which need to be tested before they can be published to prevent people from receiving false news information.

[0036] The video to be tested includes visual information, text information, and audio information. Visual information refers to the video frame information, excluding subtitles; text information refers to the video's title, description, and heading information, excluding subtitles; audio information does not include the corresponding text.

[0037] Step 104: Identify the scene, people, location, entity and keywords in the visual information and form visual text.

[0038] Taking news videos as an example, the media content analysis API of Baidu Smart Cloud Platform can be used to analyze the video to be detected. From the returned results, the structured tag information of the identified scenes, public figures, locations, entities, and keywords can be selected and concatenated into visual text W. v ={w v0 ,w v1 ,...,w vn}, where w vi Let i∈{1,2,...,n} represent the i-th visual text word.

[0039] Using the short video of Messi entering the World Cup final on Douyin (TikTok) as an example, the results obtained using Baidu AI Cloud Media Content Analysis API are as follows:

[0040] Scene: Sports - Football

[0041] Public figures: Messi, Aguero

[0042] Place:

[0043] Entity: Football

[0044] Keywords: Argentina, Croatia, Messi, Caloni, Aguero, World Cup Final

[0045] If some fields are not recognized (such as location), an empty character can be used to represent them.

[0046] Step 106: Extract visual features and visual text features from visual information, video text features from text information, and audio features from audio text using a pre-trained feature extraction model.

[0047] Step 108: Fuse visual text features and video text features to obtain fused text features.

[0048] Step 110: Combine visual features, audio features, and combined text features to obtain multimodal fusion features.

[0049] Step 112: Determine whether the video to be detected is real or fake based on the multimodal fusion features.

[0050] like Figure 2 The diagram illustrates a video detection method based on multimodal information fusion.

[0051] The aforementioned multimodal information fusion video detection method identifies important information such as scenes, public figures, locations, entities, and keywords appearing in the video to form visual text. It then extracts visual text features, video text features, and visual and audio features. First, it fuses the visual and video text features to obtain new text features, thereby strengthening the text information and initially achieving cross-modal fusion of visual and text information. Next, it fuses this new text feature with the extracted visual and audio features to further achieve cross-modal fusion of visual, audio, and text information. In summary, the progressive cross-modal information fusion video detection method in this scheme can fully utilize the correlation between modalities and leverage multiple modal information, thereby improving the detection accuracy of fake videos.

[0052] In one embodiment, a pre-trained feature extraction model extracts visual features and visual text features from visual information, video text features from text information, and audio features from audio text, including:

[0053] The video to be detected is uniformly sampled to obtain a video frame sequence. The visual encoder of the XCLIP model is used to extract the features of each video frame in the video frame sequence to obtain visual features.

[0054] The title, description, and tag information of the video to be detected are concatenated into video text, and the video text features and visual text features are extracted by the visual encoder of the CLIP model.

[0055] The audio of the video to be detected is extracted and saved as a WAV file. The audio features of the WAV file are then extracted using the Wav2Vec2 model.

[0056] In one embodiment, visual text features and video text features are fused to obtain fused text features, including:

[0057] The semantic information of video text features and visual text features is fused using a cross-attention mechanism to obtain cross-video text features and cross-visual text features; the cross-video text features and cross-visual text features are then concatenated to obtain fused text features.

[0058]

[0059] Among them, T t ' represents cross-video text features, T v 'Indicates cross-visual text features.' This indicates that the feature matrices are concatenated horizontally.

[0060] The network model corresponding to the cross-attention mechanism is as follows: Figure 4As shown. Formula (1) represents querying video text information using visual text features, that is, fusing video text information onto visual text, and Formula (2) represents fusing visual text information onto video text:

[0061]

[0062]

[0063] Q, K, and V represent the query, key, and value, respectively.

[0064] In one embodiment, visual features, audio features, and fused text features are fused to obtain multimodal fused features, including:

[0065] Visual features, audio features, and fused text features are connected to obtain a feature sequence; the feature sequence is then input into the encoder of the transformer model to obtain multimodal fusion features that integrate multimodal information.

[0066] By connecting features from multiple modalities and using a transformer structure to obtain fused multimodal information, the correlation between visual features, audio features, and fused text features is learned.

[0067] In one embodiment, judging the authenticity of the video to be detected based on the multimodal fusion features further includes: projecting the multimodal fusion features onto the authenticity space using a fully connected network with softmax activation to obtain the probability that the video is real or fake; if the probability that the video is real is greater than a preset value, it is judged as real, otherwise it is fake.

[0068] A specific embodiment is provided here to illustrate this solution in detail:

[0069] A video sequence is composed of 32 uniformly sampled frames using FFmpeg, with each frame's dimensions adjusted to [224, 224]. The XCLIP model's visual encoder utilizes the VIT model to encode visual features. A schematic diagram of the VIT model structure is shown below. Figure 3 As shown, VIT divides the image into multiple patches, using a special [CLS] token and multiple patches as input to the TransformerEncoder to encode the image. The feature representation of the [CLS] token can be selected to represent the features of the entire frame, resulting in frame sequence features, i.e., visual features V = {v1, v2, ..., v...}. 32 The feature dimension is 768.

[0070] Concatenate the video's title, description, and tags into a video text W. t ={w t0 ,w t1 ,...,w tn}, where w tj Let j∈{1,2,...,m} represent the j-th word of the video text. Let the video text w... t and visual text w v Text features are extracted using a CLIP model text encoder. For ease of training, the length of the obtained text features is fixed: video text features are fixed at 128 words, and visual text features are fixed at 64 words. If the original feature length is longer than the fixed length, truncation is performed, taking the first 128 words; if the original feature length is shorter than the fixed length, it is padded with zeros to obtain the visual text feature T. t and video text features T v The feature dimension is 512.

[0071] The audio was extracted using ffmpeg at a sampling rate of 16000 and saved as a wav file. The audio feature vector was extracted using the Wav2Vec2 model. The audio information from each frame was then fused using an average pooling layer to obtain the audio feature vector A of the video, with a feature dimension of 768.

[0072] Audio feature A and visual feature V are passed through their respective fully connected networks, with an input dimension of 768 and an output dimension of 512. The feature vectors are projected into a 512-dimensional space to obtain new visual feature V' and new audio feature A'.

[0073] The input feature vector f is obtained by concatenating the feature vectors. input =concat(T,V',A'), adds position vector embedding and modality vector embedding, and inputs them into the transformer's encoder, such as Figure 5 As shown, the fusion of multimodal information is achieved through an attention mechanism, resulting in the fused feature embedding f. m .

[0074] The fused feature embeddings are projected onto the real and pseudo-spaces through a fully connected layer and a softmax layer, as shown in the following formula:

[0075] (p0,p1)=softmax(Wf m +b)

[0076] The probability p0 of a video being true and the probability p1 of a video being false are obtained. If the probability of a video being true is greater than 0.5, it is judged as true; otherwise, it is false. Binary cross-entropy is chosen as the objective function for training, as shown in formula (3) below:

[0077] L=-[ylogp0+(1-y)logp1] (3)

[0078] Where y∈{0,1} represents the true or false label of the training data, where 0 indicates that the training data is true and 1 indicates that the training data is false.

[0079] It should be understood that, although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0080] In one embodiment, a video detection device based on multimodal information fusion is provided, wherein:

[0081] The video acquisition module is used to acquire the video to be detected; the video to be detected includes visual information, text information and audio information;

[0082] The visual text acquisition module is used to identify scenes, people, locations, entities, and keywords in visual information and assemble them into visual text.

[0083] The feature extraction module is used to extract visual features and visual text features from visual information, video text features from text information, and audio features from audio text using a pre-trained feature extraction model.

[0084] The text fusion module is used to fuse visual text features and video text features to obtain fused text features.

[0085] The multimodal information fusion module is used to fuse visual features, audio features, and fused text features to obtain multimodal fused features;

[0086] The video detection module is used to determine whether a video to be detected is real or fake based on multimodal fusion features.

[0087] Specific limitations regarding the video detection device based on multimodal information fusion can be found in the limitations of the video detection method based on multimodal information fusion described above, and will not be repeated here. Each module in the aforementioned video detection device based on multimodal information fusion can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0088] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 6 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a video detection method based on multimodal information fusion. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0089] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0090] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the method described above.

[0091] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above.

[0092] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0093] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0094] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A video detection method based on multimodal information fusion, characterized in that, The method includes: Acquire the video to be detected; the video to be detected includes visual information, text information, and audio information; Identify scenes, people, locations, entities, and keywords in the visual information and assemble them into visual text; Visual features and visual text features in the visual information, video text features in the text information, and audio features in the audio information are extracted using a pre-trained feature extraction model. By fusing the visual text features and the video text features, a fused text feature is obtained; The visual features, audio features, and fused text features are fused to obtain multimodal fused features; The authenticity of the video to be detected is determined based on the multimodal fusion features. The fused text features, obtained by fusing the visual text features and the video text features, include: The semantic information of video text features and visual text features is fused through a cross-attention mechanism to obtain cross-video text features and cross-visual text features; By concatenating cross-video text features and cross-visual text features, a fused text feature is obtained: in, Represents cross-video text features. Indicates cross-visual text features. This indicates that the feature matrices are concatenated horizontally; The multimodal fusion feature obtained by fusing the visual features, the audio features, and the fused text features includes: By connecting the visual features, the audio features, and the fused text features, a feature sequence is obtained; The feature sequence is input into the encoder of the transformer model to obtain multimodal fusion features that integrate multimodal information.

2. The method according to claim 1, characterized in that, The process of identifying scenes, people, locations, entities, and keywords in the visual information and assembling them into visual text includes: The video to be detected is analyzed using a media content analysis API. The structured tag information of the scenes, people, places, entities and keywords identified in the returned analysis results is selected and spliced ​​into visual text.

3. The method according to claim 1, characterized in that, The step of extracting visual features and visual text features from the visual information, video text features from the text information, and audio features from the audio information using a pre-trained feature extraction model includes: The video to be detected is uniformly sampled to obtain a video frame sequence. The visual encoder of the XCLIP model is used to extract the features of each video frame in the video frame sequence to obtain visual features. The title, description, and tag information of the video to be detected are concatenated into video text, and the video text features and visual text features are extracted by the visual encoder of the CLIP model. The audio of the video to be detected is extracted and saved as a WAV file. The audio features of the WAV file are then extracted using the Wav2Vec2 model.

4. The method according to claim 3, characterized in that, Uniform sampling of the video to be tested includes: using ffmpeg to perform uniform sampling of the video to be tested; Extracting the audio from the video to be tested includes using ffmpeg to extract the audio from the video to be tested.

5. The method according to claim 1, wherein Judging the authenticity of the video to be detected based on the multimodal fusion features also includes: The multimodal fusion features are projected onto the real / fake space using a fully connected network with softmax activation to obtain the probability of a video being real or fake. If the probability of a video being real is greater than a preset value, it is judged as real; otherwise, it is judged as fake.

6. A video detection device based on multimodal information fusion, characterized in that, The device includes: The video acquisition module is used to acquire the video to be detected; the video to be detected includes visual information, text information and audio information; The visual text acquisition module is used to identify scenes, people, locations, entities, and keywords in the visual information and assemble them into visual text. The feature extraction module is used to extract visual features and visual text features from the visual information, video text features from the text information, and audio features from the audio information using a pre-trained feature extraction model. A text fusion module is used to fuse the visual text features and the video text features to obtain fused text features; A multimodal information fusion module is used to fuse the visual features, the audio features, and the fused text features to obtain multimodal fused features; The video detection module is used to determine the authenticity of the video to be detected based on the multimodal fusion features. Specifically, the text fusion module is used for: The semantic information of video text features and visual text features is fused through a cross-attention mechanism to obtain cross-video text features and cross-visual text features; By concatenating cross-video text features and cross-visual text features, a fused text feature is obtained: in, Represents cross-video text features. Indicates cross-visual text features. This indicates that the feature matrices are concatenated horizontally; The multimodal information fusion module is specifically used for: By connecting the visual features, the audio features, and the fused text features, a feature sequence is obtained; The feature sequence is input into the encoder of the transformer model to obtain multimodal fusion features that integrate multimodal information.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images