Misleading short video detection method based on social relation

Abstract

The invention discloses a misleading short video detection method based on a social relation, and belongs to the technical field of artificial intelligence and social networks, and the method comprises the following steps: S1, obtaining short video modal features, and carrying out the feature fusion; s2, establishing strong and weak connection relations between videos; s3, establishing a potential relation graph for the strong connection relation and the weak connection relation; s4, extracting attribute graph representation features in the potential relation graph through a network; s5, data enhancement is carried out on the short video modal features and the attribute graph representation features through pruning; s6, performing intra-modal and inter-modal comparative learning, and judging whether convergence occurs or not; and S7, obtaining a classification result of the misleading short video. In this way, the method and the device are suitable for short videos with social context attributes and event attributes. The scheme is compared with other methods for misleading short video detection tasks on the real world social network, and the result shows that the scheme is superior to other comparison methods and has high accuracy.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and social network technology, and more specifically to a method for understanding misleading short videos based on social relationships. Background Art

[0002] The emergence of social networks has not only brought convenience to communication and knowledge acquisition, but also caused the flooding of misleading information, which has had a significant impact on politics, economy, culture and health. In recent years, with the rise of short video platforms, the carriers of information exist not only in simple text, images, and audio, but also in the interaction between social channels and users. The difficulty of detecting misleading information is increasing. Therefore, it is an urgent problem to design a misleading short video detection method based on social relations suitable for social media to reduce negative impacts.

[0003] Existing misleading short video detection methods can be divided into two categories, those based on semantic features and those based on social relations. For methods based on semantic features, existing methods mainly rely on the extraction, fusion, and enhancement of data from different modalities. In order to avoid simple feature fusion engineering, some methods aggregate information through fact checking, reading interests, external knowledge, etc. For methods based on social relations, unlike semantic feature methods, social relations are mainly represented between different news through graphs. Recent methods mainly construct propagation relationship graphs through comment and forwarding mechanisms. Graph structure mining is performed through graph neural network-based methods to complete node classification or graph classification tasks. Furthermore, related work focuses on the learning and mining of attribute graphs, integrating graphs with modal information, and performing classification tasks. These methods have been proven to achieve good results in multiple datasets and have great application value.

[0004] However, there are certain challenges in the above work. As for social relationships, due to the different recommendation mechanisms, the social relationships of short video data are more complex, and it is difficult to obtain the dissemination relationship of the video like Twitter and Weibo. Therefore, it is difficult for traditional information cascade mining tasks to obtain complete short video relationships. Furthermore, the more complex network structure provides challenges for mining potential relationships of short videos. It is difficult for traditional graphs to mine the global information of nodes, which limits the learning of potential relationships and styles of videos. In terms of features, the above methods are overly dependent on external features, and the features of the video itself are only extracted and spliced ​​using pre-trained models, without aligning the relationships between modalities. In addition, due to the particularity of the misleading short video detection task, there is not enough labeled data for experimental analysis. Summary of the invention

[0005] An embodiment of the present invention provides a misleading short video detection method based on social relations, so as to solve the problems raised in the above background technology.

[0006] The embodiment of the present invention provides a misleading short video detection method based on social relations, comprising the following steps:

[0007] S1: Obtain short video modality features and perform feature fusion;

[0008] S2: Establish strong and weak connections between videos;

[0009] S3: Establish a potential relationship graph for strong and weak connections;

[0010] S4: Extracting attribute graph representation features in the latent relationship graph through the network;

[0011] S5: Pruning is used to perform data enhancement on short video modality features and attribute graph representation features respectively;

[0012] S6: Perform intra-modal and inter-modal comparative learning and determine whether convergence has occurred;

[0013] Yes, the attribute graph represents the input to the multi-layer perceptron classifier;

[0014] No, update the network weights and repeat steps S4-S6;

[0015] S7: Obtain classification results for misleading short videos.

[0016] Furthermore, the specific operations for establishing strong and weak connection relationships between videos are as follows:

[0017] Strong connections between videos are established through event relationships;

[0018] The weak connection relationship between videos is established by calculating the similarity value through semantic relevance.

[0019] Furthermore, the strong connections between videos include events in the same event, which constitute a subgraph of the complete graph. The strong connections establish a set named ε s .

[0020] Furthermore, the set of corresponding edges in the weak connection relationship between videos is recorded as ε w , ε w and ε s The calculation method is the same, ε w The calculation is as follows:

[0021]

[0022] Among them, sim(v i ,v j ) is the distance between video nodes i and j; x i , xj are the attributes of different video nodes, k is the iterator, and D is the feature dimension;

[0023]

[0024] Among them, K is a trainable hyperparameter,

[0025] Furthermore, the potential relationship graph of strong connection and weak connection is calculated as follows:

[0026] ε=ε s ∪ε w

[0027] ε s is the set of corresponding edges in the strong connection relationship between videos, and ε is the potential relationship graph between strong connection relationships and weak connection relationships.

[0028] Furthermore, the network used to extract the latent attribute graph representation features through the network is a graph neural network.

[0029] Furthermore, pruning is used to perform data enhancement on the short video modal features and the latent attribute graph representation features, and the contrastive learning loss functions of intra-modality and inter-modality are enhanced. The specific calculation is as follows:

[0030]

[0031] in, are the features that have not been pruned and pruned, respectively. τ is a variable hyperparameter. L tra , L ter are the intra-modal and inter-modal contrastive learning loss functions, respectively.

[0032] Furthermore, comparative learning within and between modes is performed to determine whether convergence has occurred, including:

[0033] L CL =λL tra +(1-λ)L ter

[0034] Among them, L CL is the loss function of the contrastive learning, and λ is a hyperparameter.

[0035] Furthermore, the attribute graph representation is input into the multi-layer perceptron classifier to obtain the classification results of misleading short videos. The loss function used is the Focal loss function, including:

[0036]

[0037] Among them, α and γ are updateable hyperparameters, C is the short video classification category, and y is the actual label. To predict the label for classification, is the number of graph nodes (videos), i and c are iterators.

[0038] Compared with the prior art, the beneficial effects are as follows:

[0039] The present invention develops a multimodal misleading short video detection method that can mine potential relationship graphs, and considers the relationship between the modality and between modalities at the same time, solving the problem of unlabeled video expansion. Specifically, a potential feature relationship graph is established to mine the potential relationship between different videos. That is, a strong correlation graph is established between different videos of the same event; in order to obtain the relationship between short videos of different events, a weak correlation relationship is established according to the semantic correlation between different videos. The two together constitute the potential relationship graph between short video data, wherein the strong correlation graph between the same event is a subgraph of the entire potential feature relationship graph. The present invention proposes a multimodal graph contrast learning method for obtaining the difference between videos of different categories. First, data enhancement is performed on the same modality to generate two views, and contrast learning is performed within the modality. Then, contrast learning is performed between different modalities. Thus, the contrast loss within the modality and between the modalities are respectively obtained, which are used for fine-tuning the video semantic feature extraction module. Finally, the video semantic feature extraction module after fine-tuning the test data is used to establish a new potential relationship graph structure and perform a node classification task. The present invention is suitable for short videos with social context attributes and event attributes. The scheme of the present invention is compared with other methods in the misleading short video detection task on real-world social networks. The results show that the scheme is superior to other comparison methods and has higher accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A flowchart of a misleading short video detection method based on social relations provided in an embodiment of the present invention;

[0041] Figure 2 A detailed illustration of a misleading short video detection method based on social relations provided by an embodiment of the present invention;

[0042] Figure 3 The experimental results of the misleading short video detection method based on social relations provided by the embodiment of the present invention are compared with the results of other existing methods;

[0043] Figure 4 Ablation experiment results of the misleading short video detection method based on social relations provided by an embodiment of the present invention;

[0044] Figure 5The experimental results of the contrastive learning method mentioned in the misleading short video detection method based on social relations provided in the embodiment of the present invention;

[0045] Figure 6 The misleading short video detection effect under different hyperparameters provided by the embodiment of the present invention;

[0046] Figure 7 The present invention provides an embodiment of the present invention which provides a misleading short video detection effect under the condition of imbalanced labels. DETAILED DESCRIPTION

[0047] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] The embodiment of the present invention provides

[0049] A misleading short video detection method based on social relations comprises the following steps:

[0050] S1: Obtain short video modality features and perform feature fusion;

[0051] S2: Establish strong and weak connections between videos;

[0052] S3: Establish a potential relationship graph for strong and weak connections;

[0053] S4: Extracting attribute graph representation features in the latent relationship graph through the network;

[0054] S5: Pruning is used to perform data enhancement on short video modality features and attribute graph representation features respectively;

[0055] S6: Perform intra-modal and inter-modal comparative learning and determine whether convergence has occurred;

[0056] Yes, the attribute graph represents the input to the multi-layer perceptron classifier;

[0057] No, update the network weights and repeat steps S4-S6;

[0058] S7: Obtain classification results for misleading short videos.

[0059] The specific operations for establishing strong and weak connection relationships between videos are as follows:

[0060] Strong connections between videos are established through event relationships;

[0061] The weak connection relationship between videos is established by calculating the similarity value through semantic relevance.

[0062] The strong connection between videos includes events in the same event, which constitute a subgraph of the complete graph. The strong connection establishes a set named ε s .

[0063] The set of corresponding edges in the weak connection relationship between videos is recorded as ε w , ε w and ε s The calculation method is the same, ε w The calculation is as follows:

[0064]

[0065] Among them, sim(v i ,v j ) is the distance between video nodes i and j; x i , x j are the attributes of different video nodes, k is the iterator, and D is the feature dimension;

[0066]

[0067] Among them, K is a trainable hyperparameter,

[0068] Furthermore, the potential relationship graph of strong connection and weak connection is calculated as follows:

[0069] ε=ε s ∪ε w

[0070] ε s is the set of corresponding edges in the strong connection relationship between videos, and ε is the potential relationship graph between strong connection relationships and weak connection relationships.

[0071] The network used to extract latent attribute graph representation features through the network is a graph neural network.

[0072] Pruning is used to perform data enhancement on the short video modal features and the latent attribute graph representation features, and the contrastive learning loss functions within and between modalities are enhanced. The specific calculation is as follows:

[0073]

[0074] in, are the features that have not been pruned and pruned, respectively. τ is a variable hyperparameter. L tra , L ter are the intra-modal and inter-modal contrastive learning loss functions, respectively.

[0075] Furthermore, comparative learning within and between modes is performed to determine whether convergence has occurred, including:

[0076] L CL =λL tra +(1-λ)L ter

[0077] Among them, L CL is the loss function of the contrastive learning, and λ is a hyperparameter.

[0078] The attribute graph representation is input into the multi-layer perceptron classifier to obtain the classification results of misleading short videos. The loss function used is the Focal loss function, including:

[0079]

[0080] Among them, α and γ are updateable hyperparameters, C is the short video classification category, and y is the actual label. To predict the label for classification, is the number of graph nodes (videos), i and c are iterators.

[0081] Figure 2 The figure shows a detailed diagram of the present invention. Specifically, the implementation process of the present invention can be divided into four modules: (1) multimodal feature extraction and aggregation, (2) latent graph construction, (3) pruning and contrastive learning, and (4) training and fine-tuning.

[0082] (1) The module is divided into a multimodal feature extraction module and a multimodal aggregation module.

[0083] 1..1 The multimodal feature extraction module mainly extracts text, OCR, audio, visual information, metadata and other features from short videos. The extraction methods for text, OCR, metadata, etc. include but are not limited to BERT, the extraction methods for audio include but are not limited to VGGish, and the extraction methods for visual information include but are not limited to VGG and C3D;

[0084] 1..2 The multimodal aggregation module mainly uses collaborative attention to aggregate features such as text, OCR, audio, and visual information according to their relevance, and aggregates all features into an integrated vector using Transformer and multi-layer perceptron; in addition, the parameters of pre-trained models such as BERT, VGGish, VGG, and C3D are not updated, but the parameters of network structures such as collaborative attention, Transformer, and multi-layer perceptron are updated.

[0085] The modal features used by the (2) module are the features extracted after the multimodal features in (1) are aggregated. A potential relationship graph is established based on the relationship between the above features and the events contained in the data.

[0086] (3) The module is divided into a pruning module and a contrastive learning module. (A) The pruning module performs data enhancement by pruning the parameters with the smallest amplitude, and inputs the data into the subsequent contrastive learning module; (B) The contrastive learning module includes intra-modality contrastive learning and inter-modality contrastive learning. This method is an unsupervised learning.

[0087] (4) The input of the module is the video features and image features obtained after the parameters of the previous neural network are updated, and the final output is the category of whether the video is misleading.

[0088] Figure 3 The experimental results of the present invention (MM-GCL) are compared with the results of other existing methods. Acc.. is the accuracy index, Prec.. is the precision index, Rec.. is the recall index, and F1 is the harmonic mean. The calculation formula is: The results show that the indicators of the present invention (MM-GCL) on the dataset FakeSV are greatly improved compared with the existing methods.

[0089] Figure 4 The ablation experiment results of the present invention (MM-GCL) are shown, with the purpose of proving that the steps of the present invention can improve the detection effect of misleading news. w / o PRG means that the potential relationship graph is not established, and PRG includes weak connection relationships and strong connection relationships; w / o Weak Link means that the weak connection relationship described in S2 is not established, and w / o Strong Link means that the strong connection relationship described in S2 is not established. The experimental results show that the experimental indicators will decrease if any component is removed, and the experimental results of the present invention cannot be achieved.

[0090] Figure 5 The experimental results of contrastive learning of the present invention (MM-GCL) are shown to prove the effectiveness of contrastive learning. w / o Intra-Modal is to remove the intra-modal contrastive learning described in S6, and w / o Inter-Modal is to remove the inter-modal contrastive learning described in S6. The comparative experimental results show that both intra-modal contrastive learning and inter-modal contrastive learning can improve the experimental results of the present invention.

[0091] Figure 6 The results of the hyperparameter analysis of the present invention are shown to demonstrate the robustness of the method of the present invention. a) The weak connection relationship described in S2 is mainly analyzed by the K hyperparameter described in S2; b) The attribute graph representation feature in the potential relationship graph extracted through the network described in S4 is represented by D; c) The changes in different values ​​of the hyperparameter λ within and between modalities described in S6; d) The strength of pruning described in S5 is represented by e.

[0092] Figure 7The effect of the present invention is shown in the case of unbalanced training data. The negative-positive sample ratio is The value of , in the figure, we can see how the indicator changes when the parameter changes.

[0093] In summary, the misleading short video detection algorithm involved in the present invention is a self-supervised learning algorithm based on graph neural network and contrastive learning, which can obtain the neighbor relationship between different nodes. However, the disadvantage is that it is difficult for existing short video platforms to obtain the propagation relationship between videos, so as to accurately judge misleading short videos. Therefore, the present invention proposes a new potential graph representation established through the event relationship and similarity relationship of videos in social networks, which enables a learnable relationship between short videos, and aggregates the relationship between short videos through graph neural networks. In addition, in order to capture the relationship between video modality and graph structure, the present invention proposes contrastive learning within and between modalities, and uses the self-supervised learning method to compare and enhance the vector representation after video fusion and the vector representation after graph structure representation learning, and uses the pruning method to enhance the data itself. Finally, the method of the present invention is applied to the detection of misleading short videos on social network platforms, and the misleading short videos that may exist in short videos on social network platforms are successfully detected, which provides a scientific basis for short video platform managers to formulate misleading video management plans.

[0094] Disclosed above are only a few specific embodiments of the present invention. Those skilled in the art may make various changes and modifications to the embodiments of the present invention without departing from the spirit and scope of the present invention. However, the embodiments of the present invention are not limited thereto. Any changes that can be conceived by those skilled in the art should fall within the scope of protection of the present invention.

Claims

1. A misleading short video detection method based on social relations, characterized in that: The following steps are involved: S1: Obtain short video modality features and perform feature fusion; S2: Establish strong and weak connections between videos; S3: Establish a potential relationship graph for strong and weak connections; S4: Extracting attribute graph representation features in the latent relationship graph through the network; S5: Pruning is used to perform data enhancement on short video modality features and attribute graph representation features respectively; S6: Perform intra-modal and inter-modal comparative learning and determine whether convergence has occurred; Yes, the attribute graph represents the input to the multi-layer perceptron classifier; No, update the network weights and repeat steps S4-S6; S7: Obtain classification results for misleading short videos.

2. The misleading short video detection method based on social relations as claimed in claim 1, characterized in that: The specific operations for establishing strong and weak connection relationships between videos are as follows: Strong connections between videos are established through event relationships; The weak connection relationship between videos is established by calculating the similarity value through semantic relevance.

3. The misleading short video detection method based on social relations as claimed in claim 2, characterized in that: The strong connection between videos includes events in the same event, which constitute a subgraph of the complete graph. The strong connection establishes a set named ε s .

4. The misleading short video detection method based on social relations as claimed in claim 1, characterized in that: The set of corresponding edges in the weak connection relationship between videos is recorded as ε w , ε w and ε s The calculation method is the same, ε w The calculation is as follows: Among them, sim(v i ,v j ) is the distance between video nodes i and j; x i , x j are the attributes of different video nodes, k is the iterator, and D is the feature dimension; Among them, K is a trainable hyperparameter, 5. The misleading short video detection method based on social relations as claimed in claim 4, characterized in that: The potential relationship graph calculation for strong and weak connections is as follows: e=e s ∪e w ε s is the set of corresponding edges in the strong connection relationship between videos, and ε is the potential relationship graph between strong connection relationships and weak connection relationships.

6. The misleading short video detection method based on social relations as claimed in claim 5, characterized in that: The network used to extract latent attribute graph representation features through the network is a graph neural network.

7. The misleading short video detection method based on social relations as claimed in claim 6, characterized in that: Pruning is used to perform data enhancement on the short video modal features and the latent attribute graph representation features, and the contrastive learning loss functions within and between modalities are enhanced. The specific calculation is as follows: in, are the features that have not been pruned and pruned, respectively. τ is a variable hyperparameter. L tra , L ter are the intra-modal and inter-modal contrastive learning loss functions, respectively.

8. The misleading short video detection method based on social relations as claimed in claim 7, characterized in that: Perform intra-modal and inter-modal comparative learning and determine whether convergence has occurred, including: THE CL =λL tra +(1-λ)L ter Among them, L CL is the loss function of the contrastive learning, and λ is a hyperparameter.

9. The misleading short video detection method based on social relations as claimed in claim 1, characterized in that: The attribute graph representation is input into the multi-layer perceptron classifier to obtain the classification results of misleading short videos. The loss function used is the Focal loss function, including: Among them, α and γ are updateable hyperparameters, C is the short video classification category, and y is the actual label. To predict the label for classification, is the number of graph nodes (videos), i and c are iterators.

Images