A short video information detection method based on event aggregation

Abstract

The present invention discloses a method for detecting short video information based on event aggregation, which relates to the technical field of cyberspace security information detection. The method comprises: obtaining all short videos and their multimodal information under a target event, extracting event, video, and multimodal information features; constructing an event graph containing event, video, and modal nodes, and establishing corresponding edges based on the inclusion relationship and similarity between nodes; using a heterogeneous graph neural network based on a double-layer attention mechanism to perform graph encoding on the event graph to obtain feature representations of event nodes and video nodes; combining event-level detection results to guide video-level detection and obtain short video detection results. This method effectively mines the complementary and mutually exclusive relationships between videos under the same event through event graph reasoning, significantly optimizes the feature representation of video nodes, and greatly improves the accuracy of short video information detection.

Description

Technical Field

[0001] The present application relates to the technical field of cyberspace security information detection, and in particular to a method for detecting short video information based on event aggregation. Background Art

[0002] With the development of society and advancements in network technology, social networks have become a vital medium for connecting users and transmitting information, significantly improving the efficiency of information resource acquisition and promoting the widespread dissemination and sharing of information content. However, the rapid development of social networks has also led to a more complex online information environment. The large-scale spread of harmful information, including false information, on social networks has become a new global problem. Especially on emerging social platforms such as short videos, how to efficiently and automatically identify false information has become a core issue and a research hotspot that urgently needs to be addressed.

[0003] In the existing technology, some studies have effectively integrated video frames and audio features, used subtitles to guide visual representation learning, and used attention mechanisms to locally and globally fuse multimodal features for information detection on short video platforms; based on these works, some researchers have proposed using news videos and rumor-busting videos of the same event to improve the performance of the original detector; in addition, there are also studies that capture useful clues for detection from the perspectives of modal tampering, video creation, and feature debiasing.

[0004] However, short videos often grow explosively around hot events. Most existing detection methods lack analysis of the potential relationships between news items and find it difficult to fully capture the characteristics of false information. As a result, the accuracy of short video detection is insufficient to support the demand for short video detection in the face of data explosions. Summary of the Invention

[0005] Based on this, it is necessary to provide a short video information detection method for event aggregation to address the above technical problems.

[0006] The present invention adopts the following technical solutions:

[0007] The present invention provides a method for detecting short video information based on event aggregation, comprising:

[0008] Obtain all short videos under the target event and the event description information, video title and multimodal information corresponding to each short video;

[0009] Extract event node features, video node features, and multimodal node features based on event description information, video title, and multimodal information;

[0010] Determine event-video edges, video-video edges, video-modality edges, and modality-modality edges based on the affiliation and similarity among event node features, video node features, and multimodal node features; and construct a heterogeneous event graph based on event node features, video node features, multimodal node features, event-video edges, video-video edges, video-modality edges, and modality-modality edges;

[0011] The heterogeneous event graph is input into the trained heterogeneous graph neural network model based on the two-layer attention mechanism for graph encoding, and the feature representation of event nodes and video nodes is output;

[0012] Based on the feature representation of event nodes, event-level detection is performed on event nodes to obtain event-level detection results; based on the feature representation of video nodes and event-level detection results, video-level detection is performed on video nodes to obtain detection results of target short videos.

[0013] Preferably, the multimodal information includes: video subtitle information, video frame information, video clip information and audio information in the video.

[0014] Preferably, event node features, video node features, and multimodal node features are extracted based on the event description information, video title, and multimodal information, specifically including:

[0015] Input event description information into the BERT pre-trained model to extract event node features;

[0016] Input the video title into the BERT pre-trained model to extract video node features;

[0017] Input the video subtitle information into the CLIP pre-trained model to extract text modality features;

[0018] Input the video frame information sequence into the CLIP pre-training model to extract the video frame modality features;

[0019] The video frame information sequence is used as video clip information and input into the TimeSformer model to extract the video clip modal features;

[0020] Input the audio information sequence in the video into the AST pre-training model to extract audio modal features;

[0021] The text modal features, video frame modal features, video clip modal features and audio modal features are spliced ​​together to obtain the modal node features.

[0022] Preferably, the subordinate relationship among the event node features, the video node features, and the multimodal node features includes: each event node feature includes multiple video node features, and each video node feature includes a multimodal node feature; based on the subordinate relationship and similarity among the event node features, the video node features, and the multimodal node features, determining the event-video edge, video-video edge, video-modality edge, and modality-modality edge specifically includes:

[0023] Establish event-video edges based on the subordinate relationship between each event node feature and its corresponding video node feature;

[0024] Establish video-modality edges based on the affiliation between each video node feature and its corresponding multimodal node;

[0025] Calculate the cosine similarity between the features of any two video nodes under the same event node. If the cosine similarity between the embedding representations of the two nodes is greater than a preset threshold, then construct a video-video edge between the two nodes;

[0026] The cosine similarity between any two modal node features under the same video node feature is calculated. If the cosine similarity between the embedding representations of the two nodes is greater than a preset threshold, a modal-modal edge is constructed between the two nodes.

[0027] Preferably, a heterogeneous event graph is constructed based on event node features, video node features, multimodal node features, event-video edges, video-video edges, video-modality edges, and modality-modality edges, specifically including:

[0028] According to event node characteristics , video node features and modal node features As well as event-video edges, video-video edges, video-modality edges, and modality-modality edges, all short videos under the target event are modeled as an event graph ;in, Represents a collection of node features, , is the set of event-video edges, video-video edges, video-modality edges, and modality-modality edges;

[0029] Construct an adjacency matrix including self-connections Sum degree matrix ;

[0030] According to the adjacency matrix including self-connection Sum degree matrix , and obtain the symmetric normalized adjacency matrix ;

[0031] A symmetric and normalized adjacency matrix is ​​introduced into the initial event graph to obtain a heterogeneous event graph.

[0032] Preferably, the heterogeneous event graph is input into a trained heterogeneous graph neural network model based on a two-layer attention mechanism for graph encoding, and the feature representation of event nodes and video nodes is output, specifically including:

[0033] Different types of node features in the event graph Perform heterogeneous graph convolution based on the double-layer attention mechanism to obtain node aggregation features. The formula is:

[0034] ;

[0035] Where, Indicates the l The node aggregation features of the layer, represents a nonlinear activation function; represents the attention weight matrix; Indicates that the corresponding type node is in The hidden layer feature representation of the layer, initial ; Represents the transformation matrix, which is used to project the feature spaces of different nodes into the common feature space;

[0036] Different types of node features in the event graph After L layers of heterogeneous graph convolution based on the double-layer attention mechanism, the node aggregation features are obtained ;

[0037] The node aggregation features are classified through a fully connected layer with a Softmax activation function to obtain the prediction result of the event node. The formula is:

[0038] ;

[0039] ;

[0040] Where, and Represent the probability distribution of event node and video node prediction respectively, and They represent the aggregated feature representation of event nodes and the aggregated feature representation of video nodes respectively.

[0041] Preferably, based on the feature representation of the video node and the event-level detection result, the video node is subjected to video-level detection to obtain the detection result of the target short video, specifically including:

[0042] Prediction probability distribution based on event nodes The labels of the feature representations of the video nodes are iteratively propagated using the symmetric and standardized adjacency matrix of the heterogeneous event graph to obtain the predicted probability distribution of the video nodes under the guidance of the event detection results. , the formula is:

[0043] ;

[0044] ;

[0045] ;

[0046] ;

[0047] Where, To initialize the total node label prediction probability matrix, A matrix containing all zeros is the initial label of the video node; is a symmetric adjacency matrix, Indicates the adjacency relationship between event nodes and video nodes. Indicates the adjacency relationship between the video node and the event node. Represents the adjacency relationship between video nodes; Indicates the After the label matrix is ​​updated, execute After the label propagation iteration, the final predicted probability distribution of the video node under the guidance of the event is obtained .

[0048] Preferably, the training loss of the heterogeneous graph neural network model based on the dual-layer attention mechanism includes HGAT prediction loss and LPA prediction loss;

[0049] The formula for the HGAT prediction loss is:

[0050] ;

[0051] Where, is the true label of the video node, is the predicted probability distribution of video nodes;

[0052] The formula for the LPA prediction loss is:

[0053] ;

[0054] Where, The predicted probability distribution of video nodes under the guidance of event detection results;

[0055] Training loss of heterogeneous graph neural network model based on two-layer attention mechanism The calculation formula is:

[0056] ;

[0057] Where, , which is a hyperparameter that controls the balance between the two loss functions.

[0058] The present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned short video information detection method for event aggregation.

[0059] The present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the program, the above-mentioned event-aggregated short video information detection method is implemented.

[0060] At least one of the above technical solutions adopted by the present invention can achieve the following beneficial effects:

[0061] Considering that short video content on social platforms often shows explosive growth around specific events, there is a problem of insufficient short video detection accuracy. The present invention provides an event-aggregated short video information detection method, which first obtains event, video and multimodal information data; then constructs a heterogeneous event graph containing event, video and modal nodes; subsequently, uses a heterogeneous graph neural network based on a double-layer attention mechanism to encode the event graph and obtain embedded representations of event nodes and video nodes; and guides video-level detection through event-level detection results.

[0062] The above technical solutions of the present invention make full use of the aggregation capabilities of the event level to analyze and process a large amount of heterogeneous short video data under the same event, construct a heterogeneous event graph containing events, videos and multimodal information, integrate complex information structures and establish multi-dimensional correlation relationships between nodes, realize a comprehensive analysis of the potential relationships between news, comprehensively capture the characteristics of false information, significantly improve the accuracy of video information detection, and are more suitable for short video social platform environments with massive data explosions. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0064] Figure 1 A schematic diagram of a flow chart of a method for detecting short video information based on event aggregation provided by the present invention;

[0065] Figure 2 A flowchart of a method for detecting short video information based on event aggregation provided by the present invention;

[0066] Figure 3 A diagram of a computer device for implementing a short video information detection method for event aggregation provided by the present invention. DETAILED DESCRIPTION

[0067] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments of the present invention and corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in the specification, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0068] The following describes in detail the technical solutions provided by various embodiments of the present application in conjunction with the accompanying drawings.

[0069] Figure 1 The following is a flow chart of a method for detecting short video information based on event aggregation in the present invention, which specifically includes the following steps:

[0070] S101: Obtain all short videos under the target event and the event description information, video title and multimodal information corresponding to each short video.

[0071] Optionally, the multimodal information includes: video subtitle information, video frame information, video clip information and audio information in the video.

[0072] Specifically, Python's computer vision library (OpenCV) is used to extract a fixed number of video frames, and the video frame sequence is used as a video clip; the text recognition tool is used to extract the corresponding video subtitle information from the video frames; and the Python-based multimedia processing library (MoviePy) is used to obtain the audio sequence corresponding to the video.

[0073] Each short video can obtain the multimodal information contained therein, namely subtitles, video frames, video clips, and audio.

[0074] S102: Extracting event node features, video node features, and multimodal node features based on event description information, video title, and multimodal information.

[0075] Optionally, the event description information, video title and multimodal information are respectively input into the specified pre-trained model to extract event node features, video node features and multimodal node features, specifically including: inputting the event description information into the BERT pre-trained model to extract event node features; inputting the video title into the BERT pre-trained model to extract video node features; inputting the video subtitle information into the CLIP pre-trained model to extract text modal features; inputting the video frame information sequence into the CLIP pre-trained model to extract video frame modal features; inputting the video frame information sequence as video clip information into the TimeSformer model to extract video clip modal features; inputting the audio information sequence in the video into the AST pre-trained model to extract audio modal features; and splicing the text modal features, video frame modal features, video clip modal features and audio modal features to obtain modal node features.

[0076] Specifically, event node features correspond to the extracted event description information features, video nodes correspond to the text features of video titles, and multimodal node features correspond to the subtitles, video frames, video clips, and audio features contained in short videos.

[0077] S103: Determine event-video edges, video-video edges, video-modal edges, and modal-modal edges based on the subordinate relationships and similarities among event node features, video node features, and multimodal node features; and construct a heterogeneous event graph based on event node features, video node features, multimodal node features, event-video edges, video-video edges, video-modal edges, and modal-modal edges.

[0078] Optionally, the subordinate relationship between event node features, video node features and multimodal node features includes: each event node feature includes multiple video node features, and each video node feature includes multimodal node features; based on the subordinate relationship and similarity between event node features, video node features and multimodal node features, event-video edges, video-video edges, video-modal edges and modal-modal edges are determined, specifically including: establishing an event-video edge based on the subordinate relationship between each event node feature and its corresponding video node feature; establishing a video-modal edge based on the subordinate relationship between each video node feature and its corresponding multimodal node; calculating the cosine similarity between any two video node features under the same event node, and if the cosine similarity between the embedded representations between the two nodes is greater than a preset threshold, constructing a video-video edge between the two nodes; calculating the cosine similarity between any two modal node features under the same video node feature, and if the cosine similarity between the embedded representations between the two nodes is greater than a preset threshold, constructing a modal-modal edge between the two nodes.

[0079] Specifically, taking a news event as an example, the event contains five related short videos, and these five videos are subordinate to the target event. The target event contains five related short videos. Through the subordinate relationship between the target event and the short videos it contains, an event-video edge can be established; there is a subordinate relationship between the short video and the modality, that is, each of the five short videos contains four obtainable modal information. Through the short video and the modal information it contains, a video-modal edge is established.

[0080] Optionally, an event graph is constructed based on event-video edges, video-video edges, video-modality edges, and modality-modality edges, specifically including: , video node features and modal node features As well as event-video edges, video-video edges, video-modality edges, and modality-modality edges, all short videos under the target event are modeled as an event graph ; Model all short videos under the target event as the initial event graph ;in, Represents a collection of node features, , is a set of event-video edges, video-video edges, video-modality edges, and modality-modality edges; construct an adjacency matrix including self-connections Sum degree matrix ; According to the adjacency matrix including self-connection Sum degree matrix , and obtain the symmetric normalized adjacency matrix ; A symmetric and standardized adjacency matrix is ​​introduced into the initial event graph to obtain a heterogeneous event graph.

[0081] S104: Input the heterogeneous event graph into the trained heterogeneous graph neural network model based on the double-layer attention mechanism for graph encoding, and output the feature representation of event nodes and video nodes.

[0082] Optionally, the event graph is input into a trained heterogeneous graph neural network model based on a two-layer attention mechanism, and the feature representations of event nodes and video nodes are output, specifically including:

[0083] Different types of node features in the event graph Perform heterogeneous graph convolution based on the double-layer attention mechanism to obtain node aggregation features. The formula is:

[0084] ;

[0085] Where, Indicates the l The node aggregation features of the layer, represents a nonlinear activation function; represents the attention weight matrix; Indicates that the corresponding type node is in The hidden layer feature representation of the layer, initial ; Represents the transformation matrix, which is used to project the feature spaces of different nodes into the common feature space;

[0086] Different types of node features in the event graph After L layers of heterogeneous graph convolution based on the double-layer attention mechanism, the node aggregation features are obtained ;

[0087] The node aggregation features are classified through a fully connected layer with a Softmax activation function to obtain the prediction result of the event node. The formula is:

[0088] ;

[0089] ;

[0090] Where, and Represent the probability distribution of event node and video node prediction respectively, and They represent the aggregated feature representation of event nodes and the aggregated feature representation of video nodes respectively.

[0091] Optionally, the two-layer attention mechanism includes: a type-level attention mechanism and a node-level attention mechanism; the type-level attention mechanism calculates the importance of different types of neighboring nodes of the input node to obtain the type-level attention weight, and the formula is:

[0092] ;

[0093] in,

[0094] ;

[0095] Where, is the node feature, is the sum of the features of the corresponding type of neighbor nodes, Represents a splicing operation, is the parameter vector of the attention mechanism of the corresponding type, through the nonlinear activation function Perform nonlinear transformation and finally use the Softmax function to standardize different types of neighbor nodes;

[0096] Optionally, the node-level attention mechanism is implemented by giving Type-specific nodes Its type is Neighboring nodes Calculate neighboring nodes The node-level attention score is normalized to obtain the node-level attention weight, which is:

[0097] ;

[0098] ;

[0099] Where, For neighboring nodes The node-level attention score of represents a nonlinear activation function, represents the parameter vector in the attention mechanism, is the node-level attention weight, is the number of neighbor nodes.

[0100] Specifically, the node-level attention mechanism aims to capture the importance of different neighboring nodes of the same type and reduce the weights of noisy nodes.

[0101] S105: Perform event-level detection on the event node based on the feature representation of the event node to obtain an event-level detection result; perform video-level detection on the video node based on the feature representation of the video node and the event-level detection result to obtain a detection result of the target short video.

[0102] Optionally, based on the feature representation of the video node and the event-level detection results, the video node is detected at the video level to obtain the detection results of the target short video, specifically including: the predicted probability distribution based on the event node , iteratively propagates the labels of the feature representation of the video nodes, and obtains the predicted probability distribution of the video nodes under the guidance of the event detection results , the formula is:

[0103] ;

[0104] ;

[0105] ;

[0106] ;

[0107] Where, To initialize the total node label prediction probability matrix, is the initial label of the video node (all zeros); is a symmetric adjacency matrix, Indicates the adjacency relationship between event nodes and video nodes. Indicates the adjacency relationship between the video node and the event node. Represents the adjacency relationship between video nodes; Indicates the After the label matrix is ​​updated, execute After the label propagation iteration, the final predicted probability distribution of the video node under the guidance of the event is obtained .

[0108] Specifically, event-level detection refers to obtaining the predicted probability distribution of event nodes based on the feature representation of event nodes; then, a label propagation mechanism is adopted to introduce the predicted results of event nodes as prior information, and combined with the connection relationship between event nodes and video nodes in the event graph, it is iteratively propagated to the video nodes, thereby obtaining the predicted probability distribution of video nodes under event guidance; finally, the predicted probability distribution of video nodes under event guidance and the predicted probability distribution of video nodes obtained by the event graph neural network are combined to train the parameters of the heterogeneous graph neural network model, and finally the video-level detection results, that is, the detection results of video nodes, are obtained.

[0109] Optionally, the training loss of the heterogeneous graph neural network model based on the dual-layer attention mechanism includes HGAT prediction loss and LPA prediction loss;

[0110] The formula for HGAT prediction loss is:

[0111] ;

[0112] Where, is the true label of the video node, is the predicted probability distribution of video nodes;

[0113] The formula for LPA prediction loss is:

[0114] ;

[0115] Where, The predicted probability distribution of video nodes under the guidance of event detection results;

[0116] Training loss of heterogeneous graph neural network model based on two-layer attention mechanism The calculation formula is:

[0117] ;

[0118] Where, , which is a hyperparameter that controls the balance between the two loss functions.

[0119] The embodiments of the present invention have achieved some positive results during the development or use process and indeed have great advantages over the existing technology. The following content describes them in conjunction with the data and charts of the test process.

[0120] This paper uses the FakeSV and FakeTT repositories for testing. Detailed statistics are shown in Table 1. The FakeSV dataset is China's largest publicly available dataset for detecting fake news videos, containing short news videos from Douyin and Kuaishou. The FakeTT dataset contains short English news videos from TikTok. Each dataset provides news videos and related information, such as title, publisher profile, and event. Some fields are shown in Table 2.

[0121] Table 1 Detailed statistical data

[0122]

[0123] Some fields in Table 2

[0124]

[0125] Table 3 compares the accuracy, precision, recall, and F1 score of the proposed method (EGR) with other disinformation detection methods. Tables 3 and 4 show detailed results of various disinformation detection performance metrics using the proposed method on real-world datasets compared with ten other methods. As shown in the table, the unimodal methods use pre-trained models to extract corresponding unimodal features (text, image, audio, and video); the multimodal methods include HCFC, FANVM, SV-FEND, and FakingRecipe; the large-model method uses the GPT-4o interface to determine the authenticity of multimodal information; and EGR is the proposed method. It can be seen that the proposed method outperforms the other methods on both real-world datasets, demonstrating that videos of the same event can provide auxiliary information for detection. Furthermore, the detection accuracy using the large-model interface is lower because the large model can only use raw features for judgment and cannot be trained for specific tasks. Overall, the proposed short video information detection method performs best on all four evaluation metrics compared to all baselines.

[0126] Table 3 Comparative experimental results of the present invention on the FakeSV dataset

[0127]

[0128] Table 4 Comparative experimental results of the present invention on the FakeTT dataset

[0129]

[0130] In summary, the present invention provides a short video information detection method based on event aggregation. Taking into account that short video content in social platforms often shows explosive growth around specific events, the present invention makes full use of the aggregation ability of the event level to analyze and process a large amount of heterogeneous short video data under the same event, construct a heterogeneous event graph containing events, videos and multimodal information, integrate complex information structures and establish multidimensional correlation relationships between nodes. On this basis, the present invention introduces a heterogeneous graph neural network and a label propagation algorithm based on a double-layer attention mechanism to learn the interaction between different nodes in the graph to optimize the video node representation and thereby improve the accuracy of information detection. The method of the present invention not only improves the quality of multimodal feature fusion, but also improves the robustness and generalization performance of the detection method in large-scale, unstructured data scenarios, embodies strong big data processing and analysis capabilities, and has good industrial-grade application promotion potential.

[0131] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of the present invention.

Claims

1. A short video information detection method for event aggregation, characterized in that: include: Obtain all short videos under the target event and the event description information, video title and multimodal information corresponding to each short video; Extract event node features, video node features, and multimodal node features based on event description information, video title, and multimodal information; Determine event-video edges, video-video edges, video-modality edges, and modality-modality edges based on the affiliation and similarity among event node features, video node features, and multimodal node features; and construct a heterogeneous event graph based on event node features, video node features, multimodal node features, event-video edges, video-video edges, video-modality edges, and modality-modality edges; The heterogeneous event graph is input into the trained heterogeneous graph neural network model based on the two-layer attention mechanism for graph encoding, and the feature representation of event nodes and video nodes is output; Based on the feature representation of event nodes, event-level detection is performed on event nodes to obtain event-level detection results; based on the feature representation of video nodes and event-level detection results, video-level detection is performed on video nodes to obtain detection results of target short videos.

2. The method for detecting short video information based on event aggregation according to claim 1, wherein: The multimodal information includes: video subtitle information, video frame information, video clip information and audio information in the video.

3. The method for detecting short video information of event aggregation according to claim 2, characterized in that: The extracting of event node features, video node features, and multimodal node features based on event description information, video title, and multimodal information specifically includes: Input event description information into the BERT pre-trained model to extract event node features; Input the video title into the BERT pre-trained model to extract video node features; Input the video subtitle information into the CLIP pre-trained model to extract text modality features; Input the video frame information sequence into the CLIP pre-training model to extract the video frame modality features; The video frame information sequence is used as video clip information and input into the TimeSformer model to extract the video clip modal features; Input the audio information sequence in the video into the AST pre-training model to extract audio modal features; The text modal features, video frame modal features, video clip modal features and audio modal features are spliced ​​together to obtain the modal node features.

4. The method for detecting short video information of event aggregation according to claim 1, wherein: The subordinate relationship among the event node features, the video node features, and the multimodal node features includes: each event node feature includes multiple video node features, and each video node feature includes a multimodal node feature; and determining the event-video edge, video-video edge, video-modality edge, and modality-modality edge based on the subordinate relationship and similarity among the event node features, the video node features, and the multimodal node features, specifically includes: Establish event-video edges based on the subordinate relationship between each event node feature and its corresponding video node feature; Establish video-modality edges based on the affiliation between each video node feature and its corresponding multimodal node; Calculate the cosine similarity between the features of any two video nodes under the same event node. If the cosine similarity between the embedding representations of the two nodes is greater than a preset threshold, then construct a video-video edge between the two nodes; The cosine similarity between any two modal node features under the same video node feature is calculated. If the cosine similarity between the embedding representations of the two nodes is greater than a preset threshold, a modal-modal edge is constructed between the two nodes.

5. The method for detecting short video information of event aggregation according to claim 1, wherein: The method of constructing a heterogeneous event graph based on event node features, video node features, multimodal node features, event-video edges, video-video edges, video-modality edges, and modality-modality edges specifically includes: According to event node characteristics , video node features and modal node features As well as event-video edges, video-video edges, video-modality edges, and modality-modality edges, all short videos under the target event are modeled as an event graph ;in, represents a collection of node features, , is the set of event-video edges, video-video edges, video-modality edges, and modality-modality edges; Construct an adjacency matrix including self-connections Sum degree matrix ; According to the adjacency matrix including self-connection Sum degree matrix , and obtain the symmetric normalized adjacency matrix ; A symmetric and normalized adjacency matrix is ​​introduced into the initial event graph to obtain a heterogeneous event graph.

6. The method for detecting short video information of event aggregation according to claim 1, wherein: The heterogeneous event graph is input into the trained heterogeneous graph neural network model based on the double-layer attention mechanism for graph encoding, and the feature representation of event nodes and video nodes is output, specifically including: Different types of node features in the event graph Perform heterogeneous graph convolution based on the double-layer attention mechanism to obtain node aggregation features. The formula is: ； Where, Indicates the l The node aggregation features of the layer, represents a nonlinear activation function; represents the attention weight matrix; Indicates that the corresponding type node is in The hidden layer feature representation of the layer, initial ; Represents the transformation matrix, which is used to project the feature spaces of different nodes into the common feature space; Different types of node features in the event graph After L layers of heterogeneous graph convolution based on the double-layer attention mechanism, the node aggregation features are obtained ; The node aggregation features are classified through a fully connected layer with a Softmax activation function to obtain the prediction result of the event node. The formula is: ； ； Where, and Represent the probability distribution of event node and video node prediction respectively, and They represent the aggregated feature representation of event nodes and the aggregated feature representation of video nodes respectively.

7. The method for detecting short video information of event aggregation according to claim 1, wherein: The video-level detection is performed on the video node based on the feature representation of the video node and the event-level detection result to obtain the detection result of the target short video, specifically including: Prediction probability distribution based on event nodes The labels of the feature representations of the video nodes are iteratively propagated using the symmetric and standardized adjacency matrix of the heterogeneous event graph to obtain the predicted probability distribution of the video nodes under the guidance of the event detection results. , the formula is: ； ； ； ； Where, To initialize the total node label prediction probability matrix, A matrix containing all zeros is the initial label of the video node; is a symmetric adjacency matrix, Indicates the adjacency relationship between event nodes and video nodes. Indicates the adjacency relationship between the video node and the event node. Represents the adjacency relationship between video nodes; Indicates the After the label matrix is ​​updated, execute After the label propagation iteration, the final predicted probability distribution of the video node under the guidance of the event is obtained .

8. The method for detecting short video information of event aggregation according to claim 1, wherein: The training loss of the heterogeneous graph neural network model based on the dual-layer attention mechanism includes HGAT prediction loss and LPA prediction loss; The formula for the HGAT prediction loss is: ； Where, is the true label of the video node, is the predicted probability distribution of video nodes; The formula for the LPA prediction loss is: ； Where, The predicted probability distribution of video nodes under the guidance of event detection results; Training loss of heterogeneous graph neural network model based on two-layer attention mechanism The calculation formula is: ； Where, , which is a hyperparameter that controls the balance between the two loss functions.

Images