A short video information detection method, device, equipment and storage medium

Abstract

The present invention discloses a short video information detection method, device, equipment and storage medium, which relates to the technical field of cyberspace security information detection, and includes the following steps: obtaining multimodal information in a target short video, extracting multiple modal features corresponding to each modal information; aligning the title feature with other modal features respectively, and obtaining multiple key features most relevant to the title; modeling the consistency between the multiple key features most relevant to the title respectively, and obtaining a first fusion feature representation and a second fusion feature representation; splicing the first fusion feature representation, the second fusion feature representation and the title feature to obtain a multimodal fusion feature; classifying the multimodal fusion feature, and obtaining the detection result of the multimodal information in the target short video. Based on feature alignment and semantic consistency modeling, this method better mines and integrates the key information in the short video, and greatly improves the accuracy of the information detection result.

Description

Technical Field

[0001] The present invention relates to the technical field of cyberspace security information detection, and in particular to a short video information detection method, device, equipment and storage medium. Background Art

[0002] In recent years, with the evolution of society and the development of the Internet, social networks have become a platform that "connects everything". Their influence has also increased significantly, promoting the widespread and rapid dissemination of massive information resources. However, the rapid development of social networks will also lead to an increase in harmful information in the network environment. False information is one of them, which misleads people by means of fabrication, exaggeration and distortion of facts. The problem of false information in social platforms such as short videos has become very serious. How to quickly and effectively automatically detect false information has become an important problem that needs to be solved and a research hotspot.

[0003] Most existing information detection methods are based on information such as text, images, and social context, and use deep learning models to extract features for detection. For example, the EANN (Event adversarial neural networks) model proposed by Wang et al. uses an auxiliary adversarial network to learn the invariant features of events based on splicing features to improve the accuracy of detection. The disadvantage of this type of splicing method is that it cannot capture the relationship between text and images, and cannot fully understand and utilize multimodal information. Wu et al. simulated people's reading habits and used a common attention mechanism to fuse the extracted text features, image spatial domain and frequency domain features to enhance information understanding between modalities.

[0004] Information detection technology based on images and text has become increasingly mature, and short video information detection has become a new challenge. However, most of the work on video-oriented information detection is still in the stage of manual feature extraction. The first video work used video comments and metadata features to build an SVM (Support Vector Machine) classifier for detection. Palod et al. introduced LSTM (Long Short-Term Memory) to model video comments and spliced ​​them with manually extracted features. Subsequently, Choi et al. proposed to use topic modeling to estimate the position difference between titles / descriptions and comments, and built an adversarial neural network to improve the generalization ability of the model. Considering that videos contain multiple modalities, Hou et al. added original acoustic features to provide more information. Some studies effectively fused video frames and audio features and used subtitles to guide visual representation learning. Qi et al. extracted text, audio, keyframes, video clips and social context features, and used the attention mechanism to locally and globally fuse multimodal features for information detection on short video platforms. Based on this work, they proposed an auxiliary task to improve the performance of the original detector using real videos and false videos of the same event. Although these methods have promoted the progress of short video false information video detection research, there are still some challenges. First, short videos have high information density, including various modes such as text, images or audio, and false information may exist in one or more modes; second, with the popularization of artificial intelligence technology and video editing tools, users can freely create and edit videos, which brings a lot of noise (such as text boxes, virtual backgrounds, animation filters and background music); in addition, some short videos use fabricated facts, deliberate sensationalism and rehashing old news to attract attention, making the authenticity of short video content more difficult to discern. Most video work fails to focus on key information in a large amount of redundant information, but simply fuses features.

[0005] Through the above analysis, the existing short video information detection method only relies on partial modal information, and only performs simple fusion after feature extraction of partial modal information. At the same time, short videos are also full of a large amount of redundant information and noise. The existing method is unable to process a large amount of redundant information, resulting in low accuracy of the final detection result. Summary of the invention

[0006] Based on the defects of the above-mentioned prior art, the present invention provides a short video information detection method, device, equipment and storage medium, which solves the problem that the existing short video information detection method only relies on partial modal information, performs simple fusion after feature extraction of partial modal information, and is unable to process a large amount of redundant information, resulting in low accuracy of the final detection result.

[0007] The present invention adopts the following technical solutions:

[0008] In a first aspect, the present invention provides a short video information detection method, comprising the following steps:

[0009] Acquire multimodal information in the target short video, and extract multiple modal features corresponding to each modal information, wherein the multiple modal features include title features, video subtitle features, video frame features, video spatiotemporal features, and audio frame features;

[0010] Aligning the title feature with other modal features respectively to obtain multiple key features most relevant to the title; the multiple key features include video frame key features, video subtitle key features, video spatiotemporal key features and audio frame key features;

[0011] Model the consistency between the key features of the video frame and the key features of the audio frame, as well as the key features of the video subtitle and the key features of the video spatiotemporal information, respectively, to obtain a first fusion feature representation of the video frame and the audio frame, and a second fusion feature representation of the video subtitle and the video spatiotemporal information; splice the first fusion feature representation, the second fusion feature representation and the title feature to obtain a multimodal fusion feature;

[0012] The multimodal fusion features are classified to obtain the detection results of the multimodal information in the target short video.

[0013] Preferably, the multimodal information includes text information, a video frame sequence and an audio frame sequence, and the text information includes a title and a video subtitle;

[0014] The step of extracting multiple modal features corresponding to each modal information comprises the following steps:

[0015] Input the title and video subtitles into the BERT pre-trained model to extract title features and video subtitle features ;

[0016] Input the video frame sequence into the VGG19 model to extract video frame features ;

[0017] Input the audio frame sequence into the VGGish model to extract audio frame features ;

[0018] Input the video frame sequence as a video clip into the C3D model to extract the video spatiotemporal features .

[0019] Preferably, the step of aligning the title feature with other modal features to obtain a plurality of key features most relevant to the title comprises the following steps:

[0020] Map the title feature to the query matrix input of the cross-modal attention, map other modal features to the key matrix and value matrix input of the cross-modal attention, and obtain the corresponding multiple cross-modal attention values;

[0021] Multiple cross-modal attention values ​​are projected into the output space through the weight matrix to obtain multiple key features most relevant to the title.

[0022] Preferably, the obtaining of the corresponding multiple cross-modal attention values, the cross-modal attention values ​​are specifically as follows:

[0023] ;

[0024] In the formula, To scale the dot product attention mechanism, is the query matrix corresponding to the title feature, for , , or The corresponding key matrix, for , , or The corresponding value matrix, is the transpose of the key matrix, is the scaling factor;

[0025] The multiple cross-modal attention values ​​are projected into the output space through the weight matrix, as shown below:

[0026] ;

[0027] In the formula, is a video frame key feature, a video subtitle key feature, a video spatiotemporal key feature, or an audio frame key feature, is the LayerNorm function, is the weight matrix.

[0028] Preferably, the consistency between the key features of the video frame and the key features of the audio frame and the key features of the video subtitle and the key features of the video spatiotemporal information is modeled separately to obtain a first fused feature representation of the video frame and the audio frame and a second fused feature representation of the video subtitle and the video spatiotemporal information, specifically comprising the following steps:

[0029] The semantic consistency between the key features of the audio frame and the key features of the video frame is modeled through two independent bidirectional gated recurrent unit Bi-GRU models, and the context-related global representation is used to initialize the Bi-GRU to obtain the first fusion feature, as shown below:

[0030] ;

[0031] ;

[0032] ;

[0033] ;

[0034] In the formula, The initial hidden vector input to the Bi-GRU model, is the key feature of the audio frame, is the key feature of the video frame, is the enhanced audio frame feature representation, is the enhanced video frame feature representation, is the first fusion feature representation;

[0035] The semantic consistency between the key features of video subtitles and the key features of video spatiotemporal features is modeled through two independent Bi-GRU models, and the context-related global representation is used to initialize the Bi-GRU to obtain the second fusion feature, as shown below:

[0036] ;

[0037] ;

[0038] ;

[0039] ;

[0040] In the formula, The initial hidden vector input to the Bi-GRU model, is the key feature of video subtitles, is the key spatiotemporal feature of the video, For the enhanced video subtitle feature representation, is the enhanced video spatiotemporal feature representation, It is the second fusion feature representation.

[0041] Preferably, the first fusion feature representation, the second fusion feature representation and the title feature are concatenated to obtain a multimodal fusion feature, and the multimodal fusion feature is specifically as follows:

[0042] ;

[0043] In the formula, is the Transformer encoder layer, is the multimodal fusion feature. The title feature.

[0044] Preferably, the training process of the multimodal fusion feature is optimized by cross entropy loss, and the cross entropy loss is specifically as follows:

[0045] ;

[0046] In the formula, is the cross entropy loss, Predict labels for samples, is the true label of the sample;

[0047] The feature alignment process is optimized by contrast loss, which is as follows:

[0048] ;

[0049] In the formula, is the total contrast loss, and They are the contrastive losses for title-video frame, title-audio frame, title-video spatiotemporal, and title-video subtitle pairs;

[0050] The loss function is the weighted sum of the cross entropy loss and the total contrast loss. By minimizing the loss function, the parameters of the Bi-GRU model are updated. The loss function is as follows:

[0051] ;

[0052] In the formula, is the loss function, is a hyperparameter that controls the balance between the two loss functions.

[0053] In a second aspect, the present invention provides a short video information detection device, comprising:

[0054] An extraction module is used to obtain multimodal information in a target short video and extract multiple modal features corresponding to each modal information, wherein the multiple modal features include title features, video subtitle features, video frame features, video spatiotemporal features, and audio frame features;

[0055] An alignment module is used to align the title feature with other modal features to obtain multiple key features most relevant to the title; the multiple key features include video frame key features, video subtitle key features, video spatiotemporal key features and audio frame key features;

[0056] A fusion module is used to respectively model the consistency between the key features of the video frame and the key features of the audio frame, and the key features of the video subtitle and the key features of the video spatiotemporal information, to obtain a first fusion feature representation of the video frame and the audio frame, and a second fusion feature representation of the video subtitle and the video spatiotemporal information; the first fusion feature representation, the second fusion feature representation and the title feature are spliced ​​to obtain a multimodal fusion feature;

[0057] The classification module is used to classify the multimodal fusion features and obtain the detection results of the multimodal information in the target short video.

[0058] In a third aspect, the present invention provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned short video information detection method when executing the program.

[0059] In a fourth aspect, the present invention provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program implements the above-mentioned short video information detection method when executed by a processor.

[0060] Compared with the prior art, at least one of the above technical solutions adopted by the present invention can achieve the following beneficial effects:

[0061] The present invention first obtains as much multimodal information as possible, then extracts features from it, and aligns the title features with the features of other modalities respectively, thereby aligning the features of each modality in a joint latent feature space to obtain multiple key features most relevant to the title. The present invention reduces the impact of redundant information and noise in short videos by aligning the key features in each modality of the video by title. Then, the consistency between the multiple key features most relevant to the title is modeled respectively, and the first fusion feature representation, the second fusion feature representation and the title feature are spliced ​​to obtain multimodal fusion features. The present invention models the semantic consistency within and between modalities to integrate key information, thereby integrating key information to obtain a short video fusion feature representation. Finally, the multimodal fusion features are classified to obtain detection results, which greatly improves the accuracy of the information detection results. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0063] Figure 1 A flowchart of a short video information detection method of the present invention;

[0064] Figure 2 A flowchart of a short video information detection method of the present invention;

[0065] Figure 3 It is a feature alignment visualization analysis diagram provided by the present invention;

[0066] in, Figure 3 (a): baseline model feature alignment visualization, Figure 3 (b): Feature alignment visualization of TMACM. DETAILED DESCRIPTION

[0067] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0068] 1. Explanatory Examples In order to enable those skilled in the art to fully understand how to implement the present invention, this section provides an illustrative example that expands and describes the technical solution of the claims.

[0069] Figure 1 A flowchart of a short video information detection method provided by an embodiment of the present invention, Figure 2 A flowchart of a short video information detection method provided by an embodiment of the present invention. Figure 1 and Figure 2 The short video information detection method based on feature alignment and semantic consistency modeling provided by an embodiment of the present invention is described in detail, and specifically includes the following steps:

[0070] S1: Input the real short video information dataset on the social platform V .

[0071] Furthermore, the short video information dataset in S1 is represented as follows:

[0072] Specifically, given a set of short video datasets on social platforms , for each short video , Represents the multimodal information contained in the video. Represents text information, such as titles and video subtitles; represents a sequence of equally spaced sampled video frames; is a sequence of consecutive audio frames.

[0073] S2: Extract the multimodal information and its features contained in the short video.

[0074] The multimodal information includes text information, video frame sequences, and continuous audio frame sequences.

[0075] Furthermore, the multimodal feature extraction process in S2 is as follows:

[0076] S21: Text feature extraction, the text information used includes titles and video subtitles. First, they are represented as and ,in, is the maximum length of the text. Secondly, the BERT (Bidirectional Encoder Representations from Transformers) pre-trained model (text encoder) is used to extract the above text features and scale them to dimension, get title features and video subtitle features .

[0077] S22: Image feature extraction, sampling from the original video to obtain a video frame sequence, and using the VGG19 model (image encoder) to extract video frame features ,in is the average number of frames of all short video samples, Represents the set of real numbers.

[0078] S23: Audio frame feature extraction, separate audio from video, and extract audio frame features using the VGGish model (audio encoder) ,in Indicates the average number of audio frames.

[0079] S24: Extract spatiotemporal features of the video, and use the video frame sequence sampled in S22 as a video clip Input into the C3D (Convolutional 3D) model (video encoder) to extract video spatiotemporal features .

[0080] S3: Align the title feature with other modal features respectively to obtain the key features of multimodal information, namely the joint feature space representation.

[0081] The feature fusion of this embodiment includes feature alignment and semantic consistency modeling.

[0082] Furthermore, the feature alignment operation in S3 is processed as follows:

[0083] S31: Use cross-modal attention mechanism to align features between titles and video frames. Since the features of each modality are extracted by independent pre-trained models, their feature distribution is not consistent. Therefore, it is difficult to model the interaction between different modalities without alignment. The cross-modal attention mechanism aims to adjust the video frame tags according to the title to obtain the frame-level key feature representation that is most relevant to the title. , thereby aligning the features of each modality in a joint latent feature space.

[0084] Furthermore, the cross-modal attention mechanism used in feature alignment in S3 is expressed as follows:

[0085] ;

[0086] In the formula, To scale the dot product attention mechanism, is the query matrix corresponding to the title feature, for , , or The corresponding key matrix, for , , or The corresponding value matrix, is the transpose of the key matrix, is the scaling factor.

[0087] Multiple cross-modal attention values ​​are projected into the output space through the weight matrix:

[0088] ;

[0089] In the formula, is a video frame key feature, a video subtitle key feature, a video spatiotemporal key feature, or an audio frame key feature, is the LayerNorm function, is the weight matrix.

[0090] S32: Similarly, according to the steps recorded in S31, the title is aligned with the audio frame, the video subtitle and the video spatiotemporal features respectively, and the key features of each modality are obtained in the same feature space. , and .

[0091] S311: Map the title features and video frame features to the corresponding input matrices respectively. The title features correspond to the query matrix input of cross-modal attention, and the video frame features correspond to the key and value matrix input:

[0092] ;

[0093] ;

[0094] ;

[0095] in, and correspond to the query, key, and value matrices of the cross-modal attention mechanism, respectively. is the transpose of the title feature, is the video frame feature, is the weight matrix of the learnable parameters, Represents the LayerNorm function.

[0096] S312: The cross-modal attention mechanism aims to perform weighted tag fusion in the frame dimension based on the title query to select the frames most similar to the title as the key information of that modality.

[0097] Specifically, the title query is labeled with the most relevant regions via a scaled dot-product attention mechanism. and The similarity between After obtaining the cross-modal attention value, the weight matrix Project to the output space to get the video key frame features :

[0098] ;

[0099] ;

[0100] In the formula, To scale the dot product attention mechanism, is the query matrix corresponding to the title feature, is the key matrix corresponding to the video frame features, is the value matrix corresponding to the video frame features, is the transpose of the key matrix, is the scaling factor, is the weight matrix.

[0101] S4: The Bi-GRU model (Bidirectional Gated Recurrent Unit) is used to model the semantic consistency of key features within and between modalities to obtain the final fusion features.

[0102] Furthermore, the semantic consistency modeling process in S4 is as follows:

[0103] S41: Use Bi-GRU models to capture the correlations within and between modalities to integrate multimodal information, such as temporal continuity between frames within a modality and shared semantics between different modalities. Specifically, two independent Bi-GRU models are used to extract the key features of audio frames. and video frame key features The semantic consistency between them is modeled, and the context-dependent global representation is used to initialize the Bi-GRU to ensure that the model integrates the semantic features within and between modalities at each time step, thereby obtaining the first fusion feature , the formula is as follows:

[0104] ;

[0105] ;

[0106] ;

[0107] ;

[0108] in, The initial hidden vector input to the Bi-GRU model contains semantic information from both audio and video frames. and The feature representation of the enhanced audio and video frames includes both consistent semantic features from different modalities and features of continuous time segments within the modality; It is the fusion feature representation.

[0109] S42: Similarly, semantic consistency modeling is performed on the video subtitles and spatiotemporal features to obtain the second fusion feature , as shown below:

[0110] ;

[0111] ;

[0112] ;

[0113] ;

[0114] In the formula, The initial hidden vector input to the Bi-GRU model, is the key feature of video subtitles, is the key spatiotemporal feature of the video, For the enhanced video subtitle feature representation, is the enhanced video spatiotemporal feature representation, It is the second fusion feature representation.

[0115] S43: Combine the above features with the initial title features Concatenate and input into the Transformer encoder layer to obtain the final multimodal fusion features :

[0116] ;

[0117] in, Represents a Transformer encoder layer.

[0118] Furthermore, the Bi-GRU model (multimodal feature fuser) used in semantic consistency modeling in S4 includes:

[0119] The Bi-GRU model can effectively capture the long-range dependency information in the sequence data, thereby obtaining the temporal dependency in the short video modality (audio frame and video frame, etc.). At the same time, the context-dependent global representation is used to initialize the Bi-GRU, which can model the semantic relevance of the input modalities to effectively fuse multimodal information. For each time step The input feature vector , the Bi-GRU model updates the hidden state vector and the output vector as follows:

[0120] ;

[0121] ;

[0122] ;

[0123] ;

[0124] ;

[0125] in, represents the update gate, Represents the reset gate, Indicates that at time step The candidate hidden state vector of is the updated hidden state vector, and is the weight matrix of the learnable parameters. is the sigmoid function, is the tanh function, It is Hadamard.

[0126] At the initial moment, the Bi-GRU model accepts the input feature vector and the initial hidden vector .

[0127] S5: Calculate the model loss function, including cross entropy loss and contrast loss.

[0128] Furthermore, the model loss function construction process in S5 is as follows:

[0129] S51: Calculate the cross entropy loss. After obtaining the fusion features, use the fully connected layer (classifier) ​​with a softmax function to obtain the probability (classification) of the short video being predicted as true or false. The category with a high probability is: ， in and is the parameter to be learned. The cross entropy loss is defined as follows:

[0130] ;

[0131] in, and are the predicted labels and true labels of the samples respectively.

[0132] S52: Calculate contrast loss. Use contrast loss to train the feature alignment part in S3 to obtain a more robust key feature representation. Consider the existence of title and key feature embedding pairs, where matching title-key feature pairs are considered positive examples and other pairs in the batch are considered negative examples. This loss is optimized in a symmetrical way: from title to key feature loss and from key features to title loss , and the sum of these two losses is used as the contrast loss between title and video frame :

[0133] ;

[0134] ;

[0135] ;

[0136] ;

[0137] in, It is i The title is embedded with the j The cosine similarity matrix between the key feature embeddings (feature similarity matrix), is the batch size, τ is the learnable temperature coefficient. and are the contrastive losses for title-video frame, title-audio frame, title-video spatiotemporal, and title-video subtitle pairs, respectively. is the total contrast loss.

[0138] S53: Calculate the final loss function value. The final loss function is the weighted sum of the cross entropy loss and the total contrast loss:

[0139] ;

[0140] in, is the loss function, is a hyperparameter that controls the balance between the two loss functions.

[0141] S6: Back propagation and optimization: Update the parameters of the Bi-GRU model based on minimizing the loss function value obtained in S5.

[0142] S7: After the model training is completed, the results of short video detection are output.

[0143] 2. Evidence of the effects of the embodiments The embodiments of the present invention have achieved some positive effects during the development or use process, and indeed have great advantages over the prior art, which are described below in conjunction with data and charts from the test process.

[0144] The present invention uses the repository FakeSV for testing. FakeSV is currently the largest short video misinformation dataset in China, including 5538 videos from Douyin and Kuaishou and their related information. The detailed statistical data are shown in Table 1, including 1827 misinformation videos, 1827 real information, and 1884 rumor-debunked videos. In addition, the repository also includes short video multimodal content information and social context information, such as video titles, keywords, tags, likes, favorites, user information, and comment information, and some fields are shown in Table 2.

[0145] Table 1 Detailed statistical data

[0146]

[0147] Table 2 Partial fields

[0148]

[0149] Table 3 is a comparison of the accuracy, precision, recall and F1 value of the method TMACM (Text-Conditioned Multimodal Alignment and Consistency Modeling networks) of the present invention and other false information detection methods. Table 3 shows the detailed results of the performance indicators of the detection of false information using the method provided by the present invention on a real data set and other ten methods. As shown in Table 3, the unimodal method uses a pre-trained model to extract the corresponding unimodal features (text, image, audio and video) for detection, the multimodal methods include EANN (Event adversarial neural networks), FANVM (Using topic modeling and adversarial neural networks for fake news video detection), SV-FEND (Short video fake news detection) and SV-FEND (-S), the large model method uses GPT3.5 turbo and GPT4.0 vision interfaces to judge the authenticity of the information, and TMACM is the method proposed by the present invention. It can be seen that the multimodal model performs better than the unimodal model, which shows that adding modal features usually improves model performance. Among the four modalities, text has the greatest impact on the detection results, followed by video, image and audio, which also verifies the rationality of the method of aligning multimodal key features conditionally based on the title proposed in the present invention. When using the same modality, the method of the present invention is better than SV-FEND(-S), and even with the addition of social context information, the performance of the method of the present invention remains optimal. In addition, a large model interface was used to detect false information, and the results were poor. The reason may be that the large model cannot learn false information and can only use the original features for judgment. Overall, compared with all baselines, the short video information detection method proposed in the present invention performs best in four evaluation indicators.

[0150] Table 3 Comparative experimental results of the present invention on the FakeSV dataset

[0151]

[0152] Table 4 shows the results of ablation experiments of TMACM on the dataset. By removing individual modules and observing the impact on model performance, their contributions are evaluated. Among them, w / o TMFA means removing the multimodal feature alignment module, w / o VSCM means video semantic consistency modeling module, and w / o InfoNCE means removing contrast loss. It can be seen that removing the above three modules leads to a decrease in model accuracy, which shows the importance of each module in TMACM. The TMFA module helps to align multimodal features and mine key information. The VSCM module helps to integrate key information between different modalities and form a more robust fusion feature representation. Contrastive loss helps to guide the TMFA module to focus on the multimodal regional feature aspects that are most relevant to the text. In summary, each component of the TMACM method plays a key role in ensuring optimal performance.

[0153] Table 4 Ablation experiments of various modules of the present invention

[0154]

[0155] Figure 3 This figure shows the visualization of the title-video frame alignment in the TMFA module. The samples are from a small batch of data in the FakeSV test dataset. Each element in the figure represents the similarity between the corresponding title and video frame features, and the diagonal represents the paired title and video frame. The similarity value is obtained by calculating the cosine distance between the title and the video frame. Figure 3 In (a), in the baseline model, there is no obvious mapping relationship between the title and the video frame pair. In contrast, Figure 3 In (b), in the TMACM method, the colors of the diagonal elements are obviously darker than the off-diagonal elements, which indicates that good alignment is achieved between the title and the video frame through TMACM.

[0156] In summary, the present invention provides a short video information detection method based on feature alignment and semantic consistency modeling. Since short video titles usually summarize the main points of the video, the method of the present invention aligns the video title with multimodal features to extract key information that is more relevant to the title in each modality, thereby reducing the impact of noise; at the same time, contrast loss is used to guide the alignment module to generate robust key feature representations. In order to integrate key information from different modalities, the method of the present invention strengthens the semantic consistency and continuity of intra-modal and inter-modal features, thereby obtaining a fused feature representation of short videos, further improving the detection accuracy.

[0157] Based on the same concept, the present invention also provides a short video information detection device, including an extraction module, an alignment module, a fusion module and a classification module.

[0158] The extraction module is used to obtain multimodal information in the target short video and extract multiple modal features corresponding to each modal information. The multiple modal features include title features, video subtitle features, video frame features, video spatiotemporal features and audio frame features.

[0159] The alignment module is used to align the title features with other modal features respectively to obtain multiple key features most relevant to the title; the multiple key features include video frame key features, video subtitle key features, video spatiotemporal key features and audio frame key features.

[0160] The fusion module is used to model the consistency between the key features of video frames and audio frames and the key features of video subtitles and video spatiotemporal key features, respectively, to obtain a first fused feature representation of video frames and audio frames and a second fused feature representation of video subtitles and video spatiotemporal information; the first fused feature representation, the second fused feature representation and the title feature are spliced ​​to obtain a multimodal fusion feature.

[0161] The classification module is used to classify the multimodal fusion features and obtain the detection results of the multimodal information in the target short video.

[0162] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the above-mentioned short video information detection method is implemented when the processor executes the program.

[0163] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned short video information detection method is implemented.

[0164] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0165] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A short video information detection method, characterized in that: The following steps are involved: Acquire multimodal information in the target short video, and extract multiple modal features corresponding to each modal information, wherein the multiple modal features include title features, video subtitle features, video frame features, video spatiotemporal features, and audio frame features; Aligning the title feature with other modal features respectively to obtain multiple key features most relevant to the title; the multiple key features include video frame key features, video subtitle key features, video spatiotemporal key features and audio frame key features; Model the consistency between the key features of the video frame and the key features of the audio frame, as well as the key features of the video subtitle and the key features of the video spatiotemporal information, respectively, to obtain a first fusion feature representation of the video frame and the audio frame, and a second fusion feature representation of the video subtitle and the video spatiotemporal information; splice the first fusion feature representation, the second fusion feature representation and the title feature to obtain a multimodal fusion feature; Classify the multimodal fusion features to obtain the detection results of the multimodal information in the target short video; The multimodal information includes text information, a video frame sequence and an audio frame sequence, and the text information includes a title and a video subtitle; The step of extracting multiple modal features corresponding to each modal information comprises the following steps: Input the title and video subtitles into the BERT pre-trained model to extract the title feature R T and video subtitle feature R C ; Input the video frame sequence into the VGG19 model and extract the video frame features R F ; Input the audio frame sequence into the VGGish model to extract the audio frame features R A ; The video frame sequence is input into the C3D model as a video clip to extract the video spatiotemporal features R V ; The method of respectively modeling the consistency between the key features of the video frame and the key features of the audio frame and the key features of the video subtitle and the key features of the video spatiotemporal information to obtain a first fusion feature representation of the video frame and the audio frame and a second fusion feature representation of the video subtitle and the video spatiotemporal information specifically includes the following steps: The semantic consistency between the key features of the audio frame and the key features of the video frame is modeled through two independent Bi-GRU models, and the context-related global representation is used to initialize the Bi-GRU to obtain the first fusion feature, as shown below: R AF|T =Bi-GRU(R A|T ,H AF ); R FA|T =Bi-GRU(R F|T ,H AF ); In the formula, H AF The initial hidden vector R is the input of the Bi-GRU model. A|T is the key feature of the audio frame, R F|T is the key feature of the video frame, R AF|T is the enhanced audio frame feature representation, R FA|T is the enhanced video frame feature representation, R (A,F)|T is the first fusion feature representation; The semantic consistency between the key features of video subtitles and the key features of video spatiotemporal features is modeled through two independent Bi-GRU models, and the context-related global representation is used to initialize the Bi-GRU to obtain the second fusion feature, as shown below: R CV|T =Bi-GRU(R C|T ,H CV ); R VC|T =Bi-GRU(R V|T ,H CV ); In the formula, H CV The initial hidden vector R is the input of the Bi-GRU model. C|T is the key feature of video subtitles, R V|T is the key spatiotemporal features of the video, R CV|T is the enhanced video caption feature representation, R VC|T is the enhanced video spatiotemporal feature representation, R (C,V)|T It is the second fusion feature representation.

2. A short video information detection method as claimed in claim 1, characterized in that: The step of aligning the title feature with other modal features to obtain a plurality of key features most relevant to the title includes the following steps: Map the title feature to the query matrix input of the cross-modal attention, map other modal features to the key matrix and value matrix input of the cross-modal attention, and obtain the corresponding multiple cross-modal attention values; Multiple cross-modal attention values ​​are projected into the output space through the weight matrix to obtain multiple key features most relevant to the title.

3. A short video information detection method as claimed in claim 2, characterized in that: The corresponding multiple cross-modal attention values ​​are obtained, and the cross-modal attention values ​​are specifically as follows: Where Attn(·) is the scaled dot product attention mechanism, Q T is the query matrix corresponding to the title feature, K I For R C , R F , R A or R V The corresponding bond matrix, V I For R C , R F , R A or R V The corresponding value matrix, is the transpose of the key matrix, D p is the scaling factor; The multiple cross-modal attention values ​​are projected into the output space through the weight matrix, as shown below: R I|T =LN(Attn(Q T ,K I ,V I )W O ); In the formula, R I|T is the key feature of video frame, video subtitle, video spatiotemporal or audio frame, LN(·) is the LayerNorm function, W O is the weight matrix.

4. A short video information detection method as claimed in claim 1, characterized in that: The first fusion feature representation, the second fusion feature representation and the title feature are concatenated to obtain a multimodal fusion feature, and the multimodal fusion feature is specifically as follows: Where Trm(·) is the Transformer encoder layer, R m is the multimodal fusion feature, R T The title feature.

5. A short video information detection method as claimed in claim 1, characterized in that: Also includes: The training process of multimodal fusion features is optimized through cross entropy loss. The cross entropy loss is as follows: In the formula, is the cross entropy loss, is the sample prediction label, and y is the sample true label; The feature alignment process is optimized by contrast loss, which is as follows: In the formula, is the total contrast loss, and They are the contrastive losses for title-video frame, title-audio frame, title-video spatiotemporal, and title-video subtitle pairs; The loss function is the weighted sum of the cross entropy loss and the total contrast loss. By minimizing the loss function, the parameters of the Bi-GRU model are updated. The loss function is as follows: In the formula, is the loss function, and λ∈(0,1) is a hyperparameter that controls the balance between the two loss functions.

6. A short video information detection device, characterized in that: include: An extraction module is used to obtain multimodal information in a target short video and extract multiple modal features corresponding to each modal information, wherein the multiple modal features include title features, video subtitle features, video frame features, video spatiotemporal features, and audio frame features; An alignment module is used to align the title feature with other modal features to obtain multiple key features most relevant to the title; the multiple key features include video frame key features, video subtitle key features, video spatiotemporal key features and audio frame key features; A fusion module is used to respectively model the consistency between the key features of the video frame and the key features of the audio frame, and the key features of the video subtitle and the key features of the video spatiotemporal information, to obtain a first fusion feature representation of the video frame and the audio frame, and a second fusion feature representation of the video subtitle and the video spatiotemporal information; the first fusion feature representation, the second fusion feature representation and the title feature are spliced ​​to obtain a multimodal fusion feature; The classification module is used to classify the multimodal fusion features and obtain the detection results of the multimodal information in the target short video; The method of respectively modeling the consistency between the key features of the video frame and the key features of the audio frame and the key features of the video subtitle and the key features of the video spatiotemporal information to obtain a first fusion feature representation of the video frame and the audio frame and a second fusion feature representation of the video subtitle and the video spatiotemporal information specifically includes the following steps: The semantic consistency between the key features of the audio frame and the key features of the video frame is modeled through two independent Bi-GRU models, and the context-related global representation is used to initialize the Bi-GRU to obtain the first fusion feature, as shown below: R AF|T =Bi-GRU(R A|T ,H AF ); R FA|T =Bi-GRU(R F|T ,H AF ); In the formula, H AF The initial hidden vector R is the input of the Bi-GRU model. A|T is the key feature of the audio frame, R F|T is the key feature of the video frame, R AF|T is the enhanced audio frame feature representation, R FA|T is the enhanced video frame feature representation, R (A,F)|T is the first fusion feature representation; The semantic consistency between the key features of video subtitles and the key features of video spatiotemporal features is modeled through two independent Bi-GRU models, and the context-related global representation is used to initialize the Bi-GRU to obtain the second fusion feature, as shown below: R CV|T =Bi-GRU(R C|T ,H CV ); R VC|T =Bi-GRU(R V|T ,H CV ); In the formula, H CV The initial hidden vector R is the input of the Bi-GRU model. C|T is the key feature of video subtitles, R V|T is the key spatiotemporal features of the video, R CV|T is the enhanced video caption feature representation, R VC|T is the enhanced video spatiotemporal feature representation, R (C,V)|T It is the second fusion feature representation.

7. A computer device, characterized in that: It includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the short video information detection method described in any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the short video information detection method described in any one of claims 1 to 5 is implemented.

Images