Malicious information propagation prediction method based on multi-modal cross attention mechanism

Abstract

The application belongs to the field of internet application, and relates to a malicious information propagation prediction method based on a multi-modal cross attention mechanism. The method comprises the following steps: obtaining malicious information of a social network; performing feature extraction based on user propagation structure, user propagation content and user propagation behavior according to the obtained malicious information, and constructing a malicious information propagation feature space; adopting a multi-modal cross attention mechanism to perform optimized processing on the features of the user propagation structure, the user propagation content and the user propagation behavior, and splicing and fusing the optimized features; performing sparse representation on the fused features to generate a malicious information propagation feature space sparse coding matrix; performing time slicing processing on the life cycle of the malicious information, inputting the malicious information propagation feature space sparse coding matrix and a global network user relationship network adjacency matrix into a malicious information propagation prediction model constructed based on a graph convolutional neural network, and predicting the malicious information propagation behavior of unknown user nodes.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of Internet application, and relates to user propagation analysis, in particular to a malicious information propagation prediction method based on a multi-modal cross attention mechanism. BACKGROUND

[0002] In recent years, various information dissemination channels have emerged, and due to the lack of supervision mechanism, malicious information has expanded its influence through the network, causing serious interference to social order and political ecology. The spread of malicious information has become more and more intense, and has become a hot and difficult problem that is universally concerned in the security application of virtual network space. Therefore, the current research on malicious information detection, propagation and control is of great significance to the security and governance of virtual network space, and needs to be further carried out.

[0003] Domestic and foreign scholars have conducted extensive research on the spread of malicious information on the network. Establishing a malicious information propagation model not only can describe the process of malicious information propagation, but also can play a crucial role in accurately predicting the results of malicious information propagation. The prediction of malicious information propagation can be mainly divided into three aspects: the popularity prediction of malicious information, the prediction of user forwarding behavior and the prediction of malicious information propagation path. The popularity prediction of malicious information mainly focuses on the macroscopic propagation trend of malicious information. The prediction of user forwarding behavior mainly starts from the individual level of users, and predicts the propagation behavior of users through the historical propagation records and user characteristics of users. The prediction of malicious information propagation path describes the propagation of malicious information from a more fine-grained level.

[0004] Although the research on malicious information propagation has achieved remarkable results, there are still some challenges, which are as follows:

[0005] 1. The space of malicious information propagation has diversity and complexity, and many influencing factors need to be considered, which brings difficulties to feature extraction and effective representation.

[0006] 2. The processing method of the extracted features is relatively single, and the optimization, fusion and high-dimensional data are not comprehensively considered, which easily leads to low network model performance.

[0007] 3. Malicious information propagation will dynamically evolve over time, and how to effectively analyze it is particularly important. At the same time, non-Euclidean data does not have translational invariance, and cannot use convolution kernels to extract the same structural information, so traditional convolutional neural networks are difficult to process this kind of data. SUMMARY

[0008] In view of the above challenges, the present application proposes a malicious information propagation prediction method based on a multi-modal cross attention mechanism, which not only considers the multi-modal feature optimization fusion problem, but also can deeply explore the internal law of malicious information propagation and perceive the propagation situation of malicious information. The present application is a malicious information propagation prediction method based on a multi-modal cross attention mechanism. First, based on malicious information, feature extraction is performed from three aspects of user propagation content, user propagation structure and user propagation behavior. Considering that the user propagation content has rich semantic information, the user propagation structure has rich propagation structure information, and the user propagation behavior has rich user detail information, in order to further capture important information of the interaction between the user propagation content, the user propagation structure and the user propagation behavior, a multi-modal cross attention mechanism is used to let the three interact with each other, so as to obtain more accurate feature vector representations of the user propagation content, the user propagation structure and the user propagation behavior respectively, and then the three vectors are spliced to obtain a fused vector representation. Then the fused features are represented by a full view representation strategy based on sparse representation, and finally, a graph convolution network is used to predict whether the user nodes in the network forward the malicious information, so as to realize the malicious information propagation prediction. The steps of the prediction method are as shown in Figure 1

[0009] The present application is a malicious information propagation prediction method based on a multi-modal cross attention mechanism, which comprises the following steps:

[0010] Obtain the malicious information of the social network, including the whole network user relationship network, the information basic attribute and the user self attribute;

[0011] According to the obtained malicious information, perform feature extraction based on the user propagation structure, the user propagation content and the user propagation behavior, and construct a malicious information propagation feature space;

[0012] Optimize the features of the user propagation structure, the user propagation content and the user propagation behavior by using a multi-modal cross attention mechanism, and splice and fuse the optimized features;

[0013] Perform sparse representation on the spliced and fused features to generate a malicious information propagation feature space sparse coding matrix;

[0014] Perform time slicing processing on the life cycle of the malicious information, input the malicious information propagation feature space sparse coding matrix and the whole network user relationship network adjacency matrix into a malicious information propagation prediction model constructed based on a graph convolution neural network, and predict the malicious information propagation behavior of unknown user nodes.

[0015] The present application has the following advantages: ​

[0016] This invention optimizes multiple features based on a multimodal cross-attention mechanism, further capturing crucial information about the interactions between user propagation structure, content, and behavior. This results in superior, more discriminative features and implicitly enhances the similarity of intra-class samples and the difference between inter-class samples, thereby improving the model's learning ability. Considering the advantage of sparse representation in capturing the most essential features, this invention uses a sparse representation algorithm to simplify the malicious information propagation feature space, thus alleviating the problem of training difficulties caused by the high-dimensional malicious information propagation space. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of a malicious information propagation prediction model based on a multimodal cross-attention mechanism in an embodiment of the present invention.

[0018] Figure 2 This is a schematic diagram of a multimodal cross-attention mechanism in an embodiment of the present invention.

[0019] Figure 3 This is a schematic diagram of sparse representation in an embodiment of the present invention.

[0020] Figure 4 This is a schematic diagram of the GCN model used in an embodiment of the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] like Figure 1 The diagram shown illustrates the overall framework of this invention. It indicates that the input to this invention is the entire network of user relationships under malicious information, the basic attributes of the information, and the user's own attributes. The output after passing through the prediction model is a prediction of whether a user will participate in spreading the malicious information under the current circumstances. An embodiment of this invention provides a method for predicting the spread of malicious information based on a multimodal cross-attention mechanism. The method includes:

[0023] The malicious information of a social network is acquired, including a whole network user relationship network, information basic attributes and user self attributes; according to the acquired malicious information, feature extraction is carried out based on user propagation structure, user propagation content and user propagation behavior, and a malicious information propagation feature space is constructed; a multi-modal cross attention mechanism is used to optimize the features of the user propagation structure, the user propagation content and the user propagation behavior, and the optimized features are spliced and fused; the spliced and fused features are sparsely represented, and a malicious information propagation feature space sparse coding matrix is generated; the life cycle of the malicious information is time-sliced, and the malicious information propagation feature space sparse coding matrix and the whole network user relationship network adjacency matrix are input into a malicious information propagation prediction model constructed based on a graph convolutional neural network, and the malicious information propagation behavior of an unknown user node is predicted.

[0024] As shown in Figure 1 The malicious information propagation prediction method based on the multi-modal cross attention mechanism includes the following three steps:

[0025] S1: online data acquisition. The data can be acquired from a public data website or by using a mature online social network public API. Here, the data acquired is related information of malicious information in its life cycle based on user propagation structure, user propagation content and user propagation behavior, including a whole network user relationship network, information basic attributes and user self attributes. Specifically, the information based on user propagation structure includes a malicious information propagation participant user relationship network; the information based on user propagation content includes information basic attributes; and the information based on user propagation behavior includes user self attributes.

[0026] S2: extract relevant attributes. According to the acquired data, feature extraction is carried out based on user propagation structure, user propagation content and user propagation behavior, and a malicious information propagation feature space is established based on the three features.

[0027] S3: model establishment. The application first proposes a multi-modal cross attention mechanism, optimizes features based on user propagation content, user propagation structure and user propagation behavior, then fuses the optimized features, and then sparsely represents them, to realize low-rank vectorization of the feature space; finally, the life cycle of the malicious information is time-sliced, a malicious information propagation prediction model is constructed based on a graph convolutional neural network, and the propagation behavior of an unknown user node is predicted based on a known user node.

[0028] The above step S1 acquires a data source. The following three steps are mainly included.

[0029] S11: acquire original data. The original data can be obtained through a public data website or an online social network public API.

[0030] S12: Simple data cleaning. The original data obtained is usually unstructured and cannot be directly used for data analysis. Through simple data cleaning, most of the unstructured data can be structured, so that abnormal values or null values no longer appear, reducing the inconvenience to subsequent calculations.

[0031] S13: Data storage. The data after simple data cleaning needs to be stored using a database, and the data is further standardized through a table structure, and the database can greatly improve the data retrieval efficiency and the mapping of inter-table relationships.

[0032] The above step S2 extracts relevant attributes. It is mainly divided into the following three steps.

[0033] Exploring the process of malicious information propagation is essentially extracting relevant features from the user level and modeling and analyzing these features. The present application aims at the dynamics of user information propagation in malicious information, based on three aspects of user propagation structure, user propagation content, and user propagation behavior, to mine important features of user malicious information propagation, as follows:

[0034] S21: The whole network user set U t The relationship network of all network users

[0035] Wherein U t represents the whole network user set in the time period t, represents the network composed of the whole network users in the time period t. represents the edge set of the whole network user U t in the time period t.

[0036] S22: User propagation structure

[0037] S221: Malicious information propagation participating user M t The relationship network of malicious information propagation participating users

[0038] Wherein M t represents all participating users of malicious information propagation in the time period t, represents the relationship network composed of the malicious information propagation participating users in the time period t. represents the edge set of the malicious information propagation participating users in the time period t.

[0039] S23: User propagation content

[0040] S231: Information basic attribute IP = {(u i , Ip) | u i ∈ U t}

[0041] where Ip represents the malicious information propagation space in the user u i The basic attributes of the propagated information include the special content of the information InfoSpec(u i ) and the information propagation influence InfoAt(u i ). Therefore, Ip can be represented as shown in the formula:

[0042] Ip = [InfoSpec(u i ), InfoAt(u i )] (1)

[0043] The special content of the information refers to whether the information carries URL, picture, video and other content. When the information carries URL, picture, video, InfoSpec(u i ) = 1, otherwise InfoSpec(u i ) = 0. The present application defines that the user with more than 50,000 fans is a popular user, and the information propagation influence is related to the number of popular users participating, that is, the more popular users participate in the propagation of information, the greater the information propagation influence. Therefore, the information propagation influence is defined as the formula:

[0044] InfoAt(u i ) = η * Num(V) (2)

[0045] where η is the weight, and Num(V) is the number of popular users participating in the information propagation.

[0046] S24: User propagation behavior

[0047] S241: User's own attributes UP = {(u i , Up) | u i ∈ U t}

[0048] where Up represents the own attributes of each user u i in the malicious information propagation space, including the number of fans of the user Fans(u i ), the activity of the user Act(u i ), the information perception of the user Pre(u i ) and the historical information forwarding rate of the user Rate(u i ). Therefore, Up can be represented as shown in the formula:

[0049] Up = [Fans(u i ), Act(u i ), Pre(u i ), Rate(u i )] (3)

[0050] wherein the historical information forwarding rate Rate(u i ) of the user can be defined as the formula:

[0051]

[0052] wherein transNum(u i ) refers to the total number of historical forwarding information of the user, and allNum(u i ) refers to the number of all information contacted by the user.

[0053] The above step S3 establishes a model. It is mainly divided into the following three steps.

[0054] The model is mainly divided into three stages: multi-feature fusion based on a multi-modal cross attention mechanism, sparse representation, and a malicious information propagation model based on GCN. In the first stage, considering that the user propagation content has rich semantic information, the user propagation structure has rich propagation structure information, and the user propagation behavior has rich user detail information, a multi-modal cross attention mechanism is proposed for multi-feature optimization and fusion processing. In the second stage, considering the advantage of sparse representation in capturing the most essential features of things, the sparse representation is performed on the fused malicious information space features to realize the low-rank vectorization of the feature space. In the third stage, after time slicing and discretization processing of the life cycle of the malicious information, a propagation model is constructed by combining the GCN network to predict and analyze the propagation trend of the malicious information.

[0055] S31: Multi-feature optimization and fusion based on a multi-modal cross attention mechanism

[0056] The application proposes a multi-modal cross attention mechanism for multi-feature optimization and fusion processing. The specific process can be described as Figure 2 The attention mechanism focuses on important information, so that the user propagation content, user propagation structure and user propagation behavior can be effectively optimized and fused, that is, the three interact with each other to obtain more accurate and more discriminative feature representation, and then the feature representation is fused. The formula used is as follows:

[0057]

[0058]

[0059] wherein Rec represents the feature vector representation of the recipient, represents the t-th dimensional attention weight from the actor to the recipient, T represents the number of feature dimensions of the recipient, Act t represents the t-th dimensional vector representation of the actor, Tanh represents a nonlinear activation function, W represents an intermediate matrix, and b represents an offset, all of which are randomly initialized and updated during training.

[0060] It can be understood that in the embodiments of the present application, the user propagation structure, the user propagation content and the user propagation behavior will be sequentially taken as the receiver and the actor to realize cross processing, that is, one feature is selected as the receiver from the user propagation structure, the user propagation content and the user propagation behavior, one feature is selected as the actor from the remaining two features, the actor is used to add attention to the receiver, and the feature of the receiver is optimized; the optimal feature of the receiver is selected from the features of the receiver after the same receiver is added attention by different actors.

[0061] Specifically, the user propagation structure is used to add attention to the user propagation content and the user propagation behavior; the user propagation content is used to add attention to the user propagation structure and the user propagation behavior; the user propagation behavior is used to add attention to the user propagation structure and the user propagation content; the user propagation structure vector, the two user propagation content vectors and the two user propagation behavior vectors after adding attention are obtained, and the single-variable feature selection method is used to test the features to be selected, the features are selected according to the correlation, part of the features are discarded according to the scores, and the better features are reserved, and then one user propagation structure vector, one user propagation content vector and one user propagation behavior vector are selected.

[0062] Taking the attention mechanism of the user propagation content acting on the user propagation structure as an example, the above multi-modal cross attention mechanism can include the following steps:

[0063] (1) The attention mechanism of the user propagation content acting on the user propagation structure: the user propagation content information is used to add attention to the corresponding user propagation structure, so that the information in the user propagation structure information closely related to the user propagation content information is given greater attention, and a new S Gk is obtained. The calculation formula of the attention mechanism is as follows:

[0064]

[0065]

[0066] (2) The attention mechanism of the user propagation content acting on the user propagation behavior, the attention mechanism of the user propagation structure acting on the user propagation content, the attention mechanism of the user propagation structure acting on the user propagation behavior, the attention mechanism of the user propagation behavior acting on the user propagation content, and the attention mechanism of the user propagation behavior acting on the user propagation structure are the same as the above C Ik acting on S Gk . Finally, a new H Uk , C Ik , H Uk ', CIk Gk

[0067] (3) from C Ik and C Ik Gk and S Gk Uk and H Uk IGU .

[0068]

[0069] S32: Sparse representation

[0070] The present application uses sparse representation to achieve the purpose of dimension reduction by simplifying the expression of malicious information propagation feature space. The specific process can be described as shown in the figure. The malicious information propagation feature space is represented by matrix Y, wherein the rows and columns of Y represent the attributes of user nodes and user nodes respectively. The target of sparse representation is to decompose the malicious information propagation feature space matrix Y into a malicious information propagation feature space dictionary matrix D and a sparse coding matrix X of the malicious information propagation feature space, that is, Y=D*X, wherein D is composed of normalized column vectors. The objective function can be represented as: Figure 3

[0071]

[0072] wherein X i is the sparse coefficient of the sample, ||0||0 is the l0norm (the number of non-zero elements in the l0norm value vector), and T0 is the sparse limit. The Lagrange multiplier method can be used to convert it into an unconstrained optimization problem, as shown in the formula:

[0073]

[0074] wherein λ is a parameter greater than 0.

[0075] ​​​​​​There are two variables to be optimized in formula (11): malicious information propagation feature space D and malicious information propagation feature space sparse coding matrix X. To solve this problem, the K-SVD algorithm for generating the K-means clustering process is adopted. K-SVD updates the dictionary by combining the update of sparse representation through an iterative strategy, thereby accelerating convergence, and seeks the best sparse representation of each signal of the training set under strict sparse constraints. Obviously, the objective function of K-SVD is a convex function for D or X alone, but not for D and X together, so optimization can be completed by alternating solution, and the specific algorithm idea is as follows: 1) randomly select K malicious information propagation users from the malicious information propagation feature space matrix Y to initialize the malicious information propagation dictionary matrix D, and simultaneously initialize the malicious information propagation feature space sparse coding matrix X as a 0 matrix. 2) Fix the dictionary, and use the orthogonal matching pursuit algorithm (OMP) to obtain the sparse coding of the malicious information propagation space. 3) Update the non-zero elements in the dictionary and the corresponding coding matrix using the SVD algorithm. Steps 2) and 3) are iterated in turn until convergence, and finally the sparse representation X of the malicious information propagation feature space is obtained.

[0076] S33: Malicious information propagation model based on GCN

[0077] The present application aims to predict the forwarding situation of users not participating in the propagation in the malicious information propagation space, including not forwarding malicious information and forwarding malicious information, so as to correspond to the binary classification task, and use the idea of classification to solve the target problem of the present application. The specific process can be described as Figure 4 The input of the model of the present application is as follows: 1) malicious information propagation user propagation feature matrix X=NxF, wherein N represents the number of user nodes in the malicious information propagation space, and F represents the feature dimension of the user. Specifically, the feature refers to the user propagation feature after being optimized and fused by the multi-modal cross attention mechanism and being processed by sparse representation. 2) The full network user relationship adjacency matrix A reflects the implicit connection relationship between any two users in the malicious information propagation space.

[0078] The user propagation feature matrix and the adjacency matrix are input into a double-layer graph convolutional neural network with a Dropout intermediate layer, and the convolution result is mapped into the probability of different prediction categories by means of the SoftMax function. The specific formula is represented as formula:

[0079]

[0080] Wherein, Wi is the weight matrix corresponding to the i-th layer network,

[0081] Because it is considered in the present application that the behavior of the potential user in the malicious information propagation space has two kinds, which can be regarded as a binary classification problem, the output of the model can be represented as C(ui ) = P(m, a | u i ), as shown in the formula:

[0082]

[0083] If C(u i ) = 0, it means that the potential user u i will not participate in malicious information dissemination at the next moment; if C(u i ) = 1, it means that the potential user u i will participate in malicious information dissemination at the next moment.

[0084] Although the embodiments of the present application have been shown and described, it can be understood by those skilled in the art that various changes, modifications, replacements and variations can be made to the embodiments without departing from the principles and spirits of the present application, and the scope of the present application is defined by the appended claims and their equivalents.

Claims

1. A method for predicting malicious information propagation based on a multi-modal cross-attention mechanism, characterized in that, The method comprises: Obtaining malicious information of a social network, including a global user relationship network, basic information attributes, and user self attributes; According to the obtained malicious information, feature extraction is performed based on user propagation structure, user propagation content, and user propagation behavior to construct a malicious information propagation feature space; A multi-modal cross-attention mechanism is used to optimize the features of the user propagation structure, user propagation content, and user propagation behavior, and the optimized features are spliced and fused; The multi-modal cross-attention mechanism used to optimize and fuse the features of the user propagation structure, user propagation content, and user propagation behavior includes sequentially selecting one feature as a receiver from the user propagation structure, user propagation content, and user propagation behavior, and sequentially selecting one feature as an actor from the remaining two features, and using the actor to add attention to the receiver to optimize the feature of the receiver; the optimal feature of the receiver is selected from the features of the receiver after adding attention from different actors to the same receiver; The multi-modal cross-attention mechanism used to optimize the features of the user propagation structure, user propagation content, and user propagation behavior specifically includes using the user propagation structure to add attention to the user propagation content and the user propagation behavior respectively; using the user propagation content to add attention to the user propagation structure and the user propagation behavior respectively; using the user propagation behavior to add attention to the user propagation structure and the user propagation content respectively; obtaining the user propagation structure vector, two user propagation content vectors, and two user propagation behavior vectors after adding attention, and using a univariate feature selection method to test the features to be selected, select the features according to the correlation, discard part of the features according to the scores, and retain better features, and then select a better user propagation structure vector, a user propagation content vector, and a user propagation behavior vector from them; The spliced and fused features are sparsely represented to generate a malicious information propagation feature space sparse coding matrix; The life cycle of the malicious information is time-sliced, the malicious information propagation feature space sparse coding matrix and the global user relationship network adjacency matrix are input into a malicious information propagation prediction model constructed based on a graph convolutional neural network, and the malicious information propagation behavior of an unknown user node is predicted.

2. The method of claim 1, wherein, The calculation formula used by the multi-modal cross-attention mechanism includes: wherein, denotes a feature vector representation of the recipient, denotes a t-th dimension attention weight from the actor to the recipient, denotes a number of feature dimensions of the recipient, denotes a t-th dimension vector representation of the actor, denotes a non-linear activation function, denotes an intermediate matrix, denotes an offset.

3. The method of claim 1, wherein, The sparsely represented spliced and fused features generate a malicious information propagation feature space sparse coding matrix, which includes decomposing the malicious information propagation feature space matrix Y into a malicious information propagation feature space dictionary matrix D and a malicious information propagation feature space sparse coding matrix X; using K-SVD to update the dictionary through an iterative strategy combined with sparse representation, and completing optimization through alternating solution to finally obtain the malicious information propagation feature space sparse coding matrix X.

4. The method of claim 3, wherein, The optimization through alternating solution includes: 1) randomly select K malicious information propagation users from the malicious information propagation feature space matrix Y to initialize the malicious information propagation dictionary matrix D, and initialize the malicious information propagation feature space sparse coding matrix X as a 0 matrix; 2) fix the dictionary, and use the orthogonal matching pursuit algorithm (OMP) to obtain the sparse coding of the malicious information propagation space; 3) use the SVD algorithm to decompose the non-zero elements in the updated dictionary and the corresponding coding matrix; 4) iteratively perform steps 2) and 3) until convergence, and finally obtain the sparse representation X of the malicious information propagation feature space.

Images