A misleading short video detection method based on social relations

Abstract

The present invention discloses a method for detecting misleading short videos based on social relationships, which belongs to the field of artificial intelligence and social network technology, and includes the following steps: S1: obtaining short video modal features and performing feature fusion; S2: establishing strong and weak connection relationships between videos; S3: establishing a potential relationship graph for strong connection relationships and weak connection relationships; S4: extracting attribute graph representation features in the potential relationship graph through the network; S5: using pruning to perform data enhancement on the short video modal features and attribute graph representation features respectively; S6: performing intra-modal and inter-modal comparative learning, and judging whether convergence has occurred; S7: obtaining the classification results of misleading short videos. In the above manner, the present invention is applicable to short videos with social context attributes and event attributes. The scheme of the present invention is compared with other methods in the task of misleading short video detection on real-world social networks. The results show that the scheme is superior to other comparison methods and has higher accuracy.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and social network technology, and more specifically to a method for understanding misleading short videos based on social relationships. Background Art

[0002] The emergence of social networks has not only facilitated communication and knowledge acquisition, but has also led to a proliferation of misleading information, significantly impacting politics, the economy, culture, and health. In recent years, with the rise of short video platforms, information carriers exist not only in simple text, images, and audio, but also in social channels and user interactions, making the detection of misleading information increasingly difficult. Therefore, designing a social relationship-based misleading short video detection method for social media to mitigate its negative impact is an urgent problem.

[0003] Existing methods for detecting misleading short videos can be divided into two main categories: semantic feature-based and social relationship-based. For semantic feature-based methods, existing methods mainly rely on the extraction, fusion, and enhancement of data from different modalities. To avoid simple feature fusion engineering, some methods aggregate information through fact-checking, reading interests, external knowledge, and other aspects. For social relationship-based methods, unlike semantic feature-based methods, social relationships are mainly represented between different news items through graphs. Recent methods have mainly constructed communication relationship graphs through comment and forwarding mechanisms. Graph neural network-based methods are used to mine graph structures and complete node classification or graph classification tasks. Furthermore, related work focuses on the learning and mining of attribute graphs, fusing graphs with modal information to perform classification tasks. These methods have been proven to achieve good results in multiple datasets and have great application value.

[0004] However, there are certain challenges in the above work. As for social relationships, due to the different recommendation mechanisms, the social relationships of short video data are more complex, and it is difficult to obtain the dissemination relationship of the video like Twitter and Weibo. Therefore, it is difficult for traditional information cascade mining tasks to obtain complete short video relationships. Furthermore, the more complex network structure poses a challenge to the mining of potential relationships in short videos. Traditional graphs find it difficult to mine the global information of nodes, which limits the learning of potential relationships and styles of videos. In terms of features, the above methods are overly dependent on external features, and the features of the video itself are only extracted and spliced ​​using pre-trained models, without aligning the relationships between modalities. In addition, due to the particularity of the misleading short video detection task, there is not enough labeled data for experimental analysis. Summary of the Invention

[0005] An embodiment of the present invention provides a misleading short video detection method based on social relationships, which is used to solve the problems raised in the above background technology.

[0006] An embodiment of the present invention provides a method for detecting misleading short videos based on social relationships, comprising the following steps:

[0007] S1: Obtain short video modal features and perform feature fusion;

[0008] S2: Establish strong and weak connection relationships between videos;

[0009] S3: Establish a potential relationship graph for strong and weak connections;

[0010] S4: Extracting attribute graph representation features from the latent relationship graph through the network;

[0011] S5: Data enhancement is performed using pruning on short video modality features and attribute graph representation features respectively;

[0012] S6: Perform intra-modal and inter-modal comparative learning and determine whether convergence has occurred;

[0013] Yes, the attribute graph represents the input to the multilayer perceptron classifier;

[0014] If no, update the network weights and repeat steps S4-S6;

[0015] S7: Obtain classification results for misleading short videos.

[0016] Furthermore, the specific operations for establishing strong and weak connection relationships between videos are as follows:

[0017] Strong connections between videos are established through event relationships;

[0018] The weak connection relationship between videos is established by calculating the similarity value based on semantic relevance.

[0019] Furthermore, the strong connections between videos including events in the same event constitute a subgraph of the complete graph, and the strong connections establish a set named ε s .

[0020] Furthermore, the set of corresponding edges in the weak connection relationship between videos is recorded as ε w , ε w and ε s The calculation method is the same, ε w The calculation is as follows:

[0021]

[0022] Among them, sim(v i ,v j ) is the distance between video nodes i and j; x i , xj are the attributes of different video nodes, k is the iterator, and D is the feature dimension;

[0023]

[0024] Among them, K is a trainable hyperparameter,

[0025] Furthermore, a potential relationship graph is established for strong and weak connections and the calculation is as follows:

[0026] ε=ε s ∪ε w

[0027] ε s is the set of edges corresponding to the strong connection relationship between videos, and ε is the potential relationship graph between strong connection relationships and weak connection relationships.

[0028] Furthermore, the network used to extract the latent attribute graph representation features through the network is a graph neural network.

[0029] Furthermore, we use pruning to perform data enhancement on the short video modality features and the latent attribute graph representation features, and enhance the intra-modality and inter-modality contrast learning loss functions, which are calculated as follows:

[0030]

[0031] in, are the features that have not been pruned and pruned, respectively, τ is a variable hyperparameter, L tra 、L ter are the intra-modal and inter-modal contrastive learning loss functions, respectively.

[0032] Furthermore, we conduct intra-modal and inter-modal comparative learning and determine whether convergence has occurred, including:

[0033] L CL =λL tra +(1-λ)L ter

[0034] Among them, L CL is the loss function of the contrastive learning, and λ is a hyperparameter.

[0035] Furthermore, the attribute graph representation is input into the multi-layer perceptron classifier to obtain the classification results of misleading short videos. The loss function used is the Focal loss function, which includes:

[0036]

[0037] Among them, α and γ are updateable hyperparameters, C is the short video classification category, and y is the actual label. For classification prediction labels, is the number of graph nodes (videos), i and c are iterators.

[0038] Compared with the prior art, the beneficial effects are as follows:

[0039] This paper develops a multimodal misleading short video detection method that mines latent relationship graphs and simultaneously considers intra-modal and inter-modal relationships, addressing the problem of unlabeled video expansion. Specifically, a latent feature relationship graph is established to mine potential relationships between different videos. Specifically, a strong correlation graph is created between different videos of the same event. To capture the relationships between short videos of different events, weak correlations are established based on the semantic relevance between the videos. Together, these two constitute the latent relationship graph between short video data, where the strong correlation graph between the same event is a subgraph of the entire latent feature relationship graph. This paper proposes a multimodal graph contrastive learning method for extracting differences between videos of different categories. First, data augmentation is performed on the same modality to generate two views, and intra-modal contrastive learning is performed. Then, inter-modal contrastive learning is performed on different modalities. This yields intra-modal and inter-modal contrastive losses, which are used to fine-tune the video semantic feature extraction module. Finally, the fine-tuned video semantic feature extraction module is used on the test data to construct a new latent relationship graph structure and perform node classification. This method is applicable to short videos with both social contextual and event attributes. The solution of the present invention is compared with other methods in the task of misleading short video detection on real-world social networks. The results show that the solution is superior to other comparison methods and has higher accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A flowchart of a method for detecting misleading short videos based on social relationships provided by an embodiment of the present invention;

[0041] Figure 2 A detailed diagram of a misleading short video detection method based on social relationships provided by an embodiment of the present invention;

[0042] Figure 3 The experimental results of the misleading short video detection method based on social relationships provided by an embodiment of the present invention are compared with the results of other existing methods;

[0043] Figure 4 Ablation experiment results of the misleading short video detection method based on social relationships provided by an embodiment of the present invention;

[0044] Figure 5Experimental results of the contrastive learning method mentioned in the misleading short video detection method based on social relationships provided by an embodiment of the present invention;

[0045] Figure 6 The misleading short video detection effect under different hyperparameters provided by the embodiment of the present invention;

[0046] Figure 7 This is the misleading short video detection effect under label imbalance provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0048] The embodiment of the present invention provides

[0049] A method for detecting misleading short videos based on social relationships, comprising the following steps:

[0050] S1: Obtain short video modal features and perform feature fusion;

[0051] S2: Establish strong and weak connection relationships between videos;

[0052] S3: Establish a potential relationship graph for strong and weak connections;

[0053] S4: Extracting attribute graph representation features from the latent relationship graph through the network;

[0054] S5: Data enhancement is performed using pruning on short video modality features and attribute graph representation features respectively;

[0055] S6: Perform intra-modal and inter-modal comparative learning and determine whether convergence has occurred;

[0056] Yes, the attribute graph represents the input to the multilayer perceptron classifier;

[0057] If no, update the network weights and repeat steps S4-S6;

[0058] S7: Obtain classification results for misleading short videos.

[0059] The specific operations for establishing strong and weak connection relationships between videos are as follows:

[0060] Strong connections between videos are established through event relationships;

[0061] The weak connection relationship between videos is established by calculating the similarity value based on semantic relevance.

[0062] The strong connection relationship between videos includes events in the same event, which constitute a subgraph of the complete graph. The strong connection relationship establishes a set named ε s .

[0063] The set of corresponding edges in the weak connection relationship between videos is recorded as ε w , ε w and ε s The calculation method is the same, ε w The calculation is as follows:

[0064]

[0065] Among them, sim(v i ,v j ) is the distance between video nodes i and j; x i , x j are the attributes of different video nodes, k is the iterator, and D is the feature dimension;

[0066]

[0067] Among them, K is a trainable hyperparameter,

[0068] Furthermore, a potential relationship graph is established for strong and weak connections and the calculation is as follows:

[0069] ε=ε s ∪ε w

[0070] ε s is the set of edges corresponding to the strong connection relationship between videos, and ε is the potential relationship graph between strong connection relationships and weak connection relationships.

[0071] The network used to extract potential attribute graph representation features through the network is a graph neural network.

[0072] Pruning is used to perform data enhancement on the short video modality features and the latent attribute graph representation features. The enhanced intra-modality and inter-modality contrast learning loss functions are obtained. The specific calculation is as follows:

[0073]

[0074] in, are the features that have not been pruned and pruned, respectively, τ is a variable hyperparameter, L tra 、L ter are the intra-modal and inter-modal contrastive learning loss functions, respectively.

[0075] Furthermore, we conduct intra-modal and inter-modal comparative learning and determine whether convergence has occurred, including:

[0076] L CL =λL tra +(1-λ)L ter

[0077] Among them, L CL is the loss function of the contrastive learning, and λ is a hyperparameter.

[0078] The attribute graph representation is input into the multi-layer perceptron classifier to obtain the classification results of misleading short videos. The loss function used is the Focal loss function, which includes:

[0079]

[0080] Among them, α and γ are updateable hyperparameters, C is the short video classification category, and y is the actual label. For classification prediction labels, is the number of graph nodes (videos), i and c are iterators.

[0081] Figure 2 The figure shows a detailed diagram of the present invention. Specifically, the implementation process of the present invention can be divided into four modules: (1) multimodal feature extraction and aggregation, (2) latent graph construction, (3) pruning and contrastive learning, and (4) training and fine-tuning.

[0082] (1) The module is divided into a multimodal feature extraction module and a multimodal aggregation module.

[0083] 1..1 The multimodal feature extraction module mainly extracts text, OCR, audio, visual information, metadata, and other features from short videos. Methods for extracting text, OCR, and metadata include but are not limited to BERT, methods for extracting audio include but are not limited to VGGish, and methods for extracting visual information include but are not limited to VGG and C3D.

[0084] 1..2 The multimodal aggregation module mainly uses collaborative attention to aggregate features such as text, OCR, audio, and visual information according to their relevance, and aggregates all features into an integrated vector using Transformer and multi-layer perceptron; in addition, the parameters of pre-trained models such as BERT, VGGish, VGG, and C3D are not updated, but the parameters of network structures such as collaborative attention, Transformer, and multi-layer perceptron are updated.

[0085] (2) The modal features used by the module are the features extracted after the aggregation of the multimodal features in (1). Based on the relationship between the above features and the events contained in the data, a potential relationship graph is established.

[0086] (3) The module is divided into a pruning module and a contrastive learning module. (A) The pruning module performs data augmentation by pruning the parameters with the smallest amplitude, and then inputs the data into the subsequent contrastive learning module. (B) The contrastive learning module includes intra-modality contrastive learning and inter-modality contrastive learning. This method is an unsupervised learning method.

[0087] (4) The input of the module is the video features and image features obtained after the parameters of the previous neural network are updated, and the final output is the category of whether the video is misleading.

[0088] Figure 3 The experimental results of the present invention (MM-GCL) are compared with those of other existing methods. Acc.. is the accuracy index, Prec.. is the precision index, Rec.. is the recall index, and F1 is the harmonic mean. The calculation formula is: The results show that the performance of the proposed method (MM-GCL) on the FakeSV dataset is significantly improved compared with existing methods.

[0089] Figure 4 The results of ablation experiments using the MM-GCL method of the present invention are presented to demonstrate that the steps of the present invention can improve the detection of misleading news. w / o PRG indicates that a potential relationship graph (PRG) is not established; PRG includes weak and strong connections. w / o Weak Link indicates that the weak connection described in S2 is not established, and w / o Strong Link indicates that the strong connection described in S2 is not established. The experimental results show that removing any component will cause the experimental indicators to decline, failing to achieve the experimental results of the present invention.

[0090] Figure 5 This paper presents experimental results of contrastive learning (MM-GCL) using the present invention to demonstrate its effectiveness. The results of the experiments with intra-modal learning (without intra-modal learning) and inter-modal learning (without inter-modal learning) demonstrate that both intra-modal and inter-modal contrastive learning improve the experimental results of the present invention.

[0091] Figure 6 The results of the hyperparameter analysis presented in this paper demonstrate the robustness of the proposed method. a) The weak connections described in S2 are analyzed primarily for the K hyperparameter; b) The attribute graph representation of the latent relationship graph extracted through the network described in S4 is represented by D; c) The variation in the intra-modal and inter-modal hyperparameter λ for different values ​​described in S6; d) The pruning strength described in S5 is represented by e.

[0092] Figure 7This demonstrates the effectiveness of the present invention when training data is unbalanced, with a negative-positive sample ratio of The figure shows the change of the indicator when the parameter changes.

[0093] In summary, the misleading short video detection algorithm involved in the present invention is a self-supervised learning algorithm based on graph neural networks and contrastive learning, which can obtain the neighbor relationship between different nodes. However, the disadvantage is that it is difficult for existing short video platforms to obtain the propagation relationship between videos, so as to make accurate judgments on misleading short videos. Therefore, the present invention proposes a new method of establishing a latent graph representation through the event relationship and similarity relationship of videos in social networks, which enables a learnable relationship between short videos, and aggregates the relationship between short videos through graph neural networks. In addition, in order to capture the relationship between video modality and graph structure, the present invention proposes intra-modal and inter-modal contrastive learning, and uses the self-supervised learning method to compare and enhance the vector representation after video fusion and the vector representation after graph structure representation learning, and uses the pruning method to perform data enhancement on the data itself. Finally, the method of the present invention is applied to the detection of misleading short videos on social network platforms, and successfully detects misleading short videos that may exist in short videos on social network platforms, which provides a scientific basis for short video platform managers to formulate misleading video management plans.

[0094] The above disclosure is only a few specific embodiments of the present invention. Those skilled in the art can make various changes and modifications to the embodiments of the present invention without departing from the spirit and scope of the present invention. However, the embodiments of the present invention are not limited to these. Any changes that can be conceived by those skilled in the art should fall within the scope of protection of the present invention.

Claims

1. A method for detecting misleading short videos based on social relationships, characterized in that: The following steps are involved: S1: Obtain short video modal features and perform feature fusion; S2: Establish strong and weak connection relationships between videos; The specific operations for establishing strong and weak connection relationships between videos are as follows: strong connection relationships between videos are established through event relationships; weak connection relationships between videos are established by calculating similarity values ​​through semantic relevance; S3: Establishing a potential relationship graph for the strong connection relationships and the weak connection relationships, wherein all nodes in the potential relationship graph are videos; S4: Extracting attribute graph representation features from the latent relationship graph through the network; S5: Data enhancement is performed using pruning on short video modality features and attribute graph representation features respectively; S6: Perform intra-modal and inter-modal comparative learning and determine whether convergence has occurred; Yes, the attribute graph represents the input to the multilayer perceptron classifier; If no, update the network weights and repeat steps S4-S6; S7: Obtain classification results for misleading short videos.

2. The misleading short video detection method based on social relationships according to claim 1, characterized in that: The strong connection relationship between videos includes events in the same event, which constitute a subgraph of the complete graph. The strong connection relationship establishes a set named ε s .

3. The misleading short video detection method based on social relationships according to claim 1, characterized in that: The set of corresponding edges in the weak connection relationship between videos is recorded as ε w , ε w and ε s The calculation method is the same, ε w The calculation is as follows: Among them, sim(v i ,v j ) is the distance between video nodes i and j; x i , x j are the attributes of different video nodes, k is the iterator, and D is the feature dimension; Among them, K is a trainable hyperparameter, 4. The misleading short video detection method based on social relationships according to claim 3, characterized in that: The potential relationship graph calculation for strong and weak connection relationships is as follows: e=e s ∪e w ε s is the set of edges corresponding to the strong connection relationship between videos, and ε is the potential relationship graph between strong connection relationships and weak connection relationships.

5. The misleading short video detection method based on social relationships according to claim 4, characterized in that: The network used to extract potential attribute graph representation features through the network is a graph neural network.

6. The misleading short video detection method based on social relationships according to claim 5, characterized in that: Pruning is used to perform data enhancement on the short video modality features and the latent attribute graph representation features. The enhanced intra-modality and inter-modality contrast learning loss functions are obtained. The specific calculation is as follows: in, are the features without pruning and pruned respectively, τ is the variable hyperparameter, L tra 、L ter are the intra-modal and inter-modal contrastive learning loss functions, respectively.

7. The misleading short video detection method based on social relationships according to claim 6, characterized in that: Perform intra-modal and inter-modal comparative learning and determine whether convergence has occurred, including: THE CL =λL tra +(1-λ)L ter Among them, L CL is the loss function of the contrastive learning, and λ is a hyperparameter.

8. The misleading short video detection method based on social relationships according to claim 1, characterized in that: The attribute graph representation is input into the multi-layer perceptron classifier to obtain the classification results of misleading short videos. The loss function used is the Focal loss function, which includes: Among them, α and γ are updateable hyperparameters, C is the short video classification category, and y is the actual label. For classification prediction labels, is the number of graph nodes (videos), i and c are iterators.

Images