Influence maximization method and system based on deep reinforcement learning and graph neural network

Abstract

The invention provides an influence maximization method and system based on deep reinforcement learning and a graph neural network, and belongs to the field of influence maximization in social network analysis. The method aims to optimize selection of seed nodes through a reinforcement learning technology so as to realize maximum propagation of information in a network, and the graph neural network can accurately capture an interaction relationship between nodes in a graph structure by virtue of a message transmission mechanism of the graph neural network, which is highly matched with an information propagation process in a cascade propagation model. Therefore, in the influence maximization model based on the deep reinforcement learning and the graph neural network, the graph neural network is combined with the deep reinforcement learning, the embedded representations of the nodes are learned by using the graph neural network, and the embedded representations are used as the input of the deep Q network so as to fully capture the complex structure and node relationship in the network. In addition, a multi-head attention mechanism is introduced to aggregate information of each round, expression of key features is enhanced, and information redundancy is effectively reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of influence maximization in social network analysis, and in particular to a method and system for maximizing influence based on deep reinforcement learning and graph neural network. Background Art

[0002] In the study of social networks, influence maximization (IM) aims to identify a set of initial nodes (seed nodes) in the network so that the information propagated through these nodes can influence as many other nodes as possible. The study of this problem is of great significance to many practical application scenarios. For example, in advertising promotion, by finding the most influential user group, product publicity and promotion can be carried out more effectively. Influence maximization was proposed by Domingos and Richardson. [1] First proposed, it is an important research problem focusing on the influence propagation process in social networks. However, solving the influence maximization problem is not easy as it has been proven to be an NP-hard problem. [2] This means that finding the optimal solution within a limited time is very difficult, especially when facing large-scale, dynamic and complex social networks.

[0003] In the past few decades, many representative traditional algorithms [3]-[5] Significant results have been achieved in the field of influence maximization. Although the most advanced traditional algorithms [6]-[8] are able to find approximate solutions with at least a certain probability, but they still suffer from the challenges of high computational cost and difficulty in scaling to large network datasets.

[0004] In recent years, with the rapid development of artificial intelligence technology, deep learning technology has been widely used in various fields. In the field of influence maximization, some scholars have combined graph embedding and graph neural network to construct regression tasks. [9]

[10] , by calculating the node scores in the network to select important nodes. However, such methods often ignore the synergy between nodes, so their propagation ability may not be ideal.

[0005] To overcome these limitations, recent research has begun to explore the use of reinforcement learning techniques to solve the influence maximization problem.

[11] -

[13] These algorithms based on deep reinforcement learning have significantly improved computational efficiency compared to traditional algorithms, and can also be extended to large network data sets. Among them, some excellent algorithms such as ToupleGDD

[13] Achieved close to IMM on some data sets [8] However, these algorithms still face challenges in generalization ability, especially when dealing with many real network data sets, and there is still a large gap compared with algorithms such as IMM.

[0006] Existing patents, such as a node feature-aware temporal social network influence maximization method and system (application number: CN202310843355.6), involve the technical field of temporal social network influence maximization and the technical field of social network identification, evaluation and processing. The purpose is to identify nodes with the highest information dissemination potential in social networks by considering the characteristics reflected by the historical interaction behavior of nodes, so as to more accurately characterize node behavior and evaluate its influence in information dissemination. A dynamic independent cascade (DIC) model and an ISVoteRank algorithm are proposed, in which the DIC model combines node characteristics and considers the changes in activation time, activation probability and activation frequency during information dissemination. Before voting, the ISVoteRank algorithm considers the personalized voting ability of nodes by embedding the interaction attributes and self-similarity of nodes into their initial voting ability. During the voting process, weights are assigned according to the interaction strength between nodes, allowing different degrees of personalized voting, and then reducing their voting ability accordingly. After the voting is completed, the top k nodes with the highest voting scores are selected as seed nodes. This strategy avoids the inefficiency of iterative selection of seed nodes in traditional voting algorithms, which is conducive to faster identification of high-quality seed nodes in temporal social networks. But again, there are still scalability challenges when dealing with large impact networks. Summary of the invention

[0007] The technical problems to be solved by the present invention are:

[0008] In order to solve the problems of high computational cost of existing influence maximization algorithms, difficulty in adapting to large-scale social networks, and limited generalization ability of algorithms based on deep reinforcement learning.

[0009] The present invention adopts the following technical solutions to solve the above technical problems:

[0010] The present invention provides an influence maximization method based on deep reinforcement learning and graph neural network, comprising the following steps:

[0011] S100, using graph neural networks and bidirectional neighborhood aggregation strategies to capture the structure and influence characteristics of nodes in social network graphs, and learning the initial embedding of nodes by pre-training the model on a regression task based on influence prediction, mapping each node in the graph into a vector representation;

[0012] S200, using deep reinforcement learning and graph neural network to build a Q network, taking the node vector obtained in step S100 as the input of the Q network, using the state-binding bidirectional neighborhood aggregation strategy in the Q network to learn the node vector representation under different states, introducing a multi-head attention mechanism to aggregate each round of information, and using the Q function to calculate the Q value of each node;

[0013] S300, performing model training on the Q network constructed in step S200 and updating parameters, introducing the online network and the target network based on the experience playback mechanism, and obtaining an optimized Q network;

[0014] S400, using the Q network optimized in step S300 to provide an optimal strategy by optimizing parameters, after calculating the Q values ​​of all nodes, select the first k nodes with the largest Q values ​​as the seed set, and then select the node that contributes the most to the task.

[0015] Further, in step S100, an encoder based on influence prediction is constructed to learn the initial vector of the node, which is used to predict the average influence range of the node after multiple rounds of propagation under the IC model, including:

[0016] S110, generating node labels, using a random graph model to generate a training graph set, using an independent cascade model to calculate the individual influence range of each node, and then performing normalization processing to map its value range to between 0 and 1, so that each node has a label value corresponding to its influence range, and the label value is used as a learning target for the regression task;

[0017] For any training graph G train The label value calculation formula for the nodes in is as follows:

[0018] ds=IC(G train ) (1)

[0019]

[0020] Among them, ds represents any training graph G train The propagation range of all nodes is a vector composed of the propagation ranges of all nodes. The propagation range of the node is obtained by the independent cascade model through the Monte Carlo model; min(·) and max(·) represent the minimum and maximum values ​​in the vector respectively; ds v Indicates the propagation range of node v; INF v represents the result obtained after normalization by ds;

[0021] S120, design a node encoder, wherein the initial input of the node encoder is a 128-dimensional all-1 vector, and a two-way neighborhood aggregation strategy is introduced to update the vector representation of the node to achieve two-way consideration of the node neighborhood information, including:

[0022] Define a convolutional layer to aggregate the information of the node's incoming neighborhood and outgoing neighborhood in parallel for T rounds, where the incoming neighborhood is the neighborhood connected by the edge pointing to the node, and the outgoing neighborhood is the neighborhood connected by the edge starting from the node;

[0023] For node v, at the i+1th layer, the convolutional layer update calculation formulas for its incoming and outgoing neighborhood information are as follows:

[0024]

[0025] Among them, N in (v) N out (v) represent the incoming neighbor and outgoing neighbor sets of node v respectively; p uv Represents edge e uv The probability of successful information dissemination on represents the learnable parameter matrix; d represents the dimension of the input vector, LeakyReLU is used as the activation function, and the negative slope is set to 0.2; They represent the vectors obtained by aggregating the incoming and outgoing neighborhoods of node u at the i-th layer, They represent the vectors obtained by aggregating the incoming and outgoing neighborhoods of the i+1th layer of node v respectively;

[0026] After the aggregation is completed, the in-neighborhood and out-neighborhood information are concatenated to obtain the embedded representation x of node v v ,Right now:

[0027]

[0028] in, Represents a learnable parameter matrix; [] represents a connection operation; fc1 represents a fully connected layer;

[0029] After obtaining the embedded representation of the node, the predicted value of the node's influence is calculated through the fully connected layer Right now:

[0030]

[0031] Among them, fc2 represents the fully connected layer;

[0032] S130, training the model, iteratively training the training set generated in step S110 n rounds, each round divides the training set into N e The prediction value of the propagation capacity of each computing node is calculated, and then the loss is calculated through the loss function, the parameters are updated using the optimizer, and finally the trained parameters are returned.

[0033] Use the average error as the loss function, that is:

[0034]

[0035] Among them, B vRepresents a batch of nodes of a graph set, and then updates the encoder parameters W through the back-propagation algorithm and the gradient descent optimizer to minimize the error of the regression task, that is:

[0036]

[0037] W=W+ΔW (9)

[0038] Among them, α represents the learning rate and ΔW is the update gradient.

[0039] Furthermore, in step S200, a convolutional layer combining states is defined, and an R-layer convolutional layer is used to integrate node information and state information through bidirectional domain aggregation, and the vector output by the node encoder is divided into two parts: and

[0040] For the convolutional layer of the combined state of the i+1th layer into the neighborhood The calculation formula is as follows:

[0041]

[0042] Among them, S t (u) represents the state of node u. When S t When (u) = 1, it means that node u has been selected into the seed set. t When (u) = 0, it means that node u has not been selected into the seed set; and represents the learnable parameter matrix, represents the learnable bias term;

[0043] For the convolutional layer with the combined state of the i+1th out-of-neighborhood, the calculation formula is:

[0044]

[0045] in, and represents the learnable parameter matrix, represents the learnable bias term;

[0046] The results of each convolution layer are retained for subsequent processing.

[0047] Furthermore, in step S200, a multi-head attention mechanism is introduced when aggregating different convolution information, and for each head, an attention score coefficient is calculated using additive attention;

[0048] For each head entering the neighborhood, the attention score coefficient is calculated based on the following logic:

[0049] The convolution result of each layer Where i represents the number of layers, as input;

[0050] Through the three shared linear layers w q 、w k 、w s and a nonlinear activation function tanh to calculate the score,

[0051]

[0052] Among them, w q 、w k 、w s Respectively represent the linear transformation of query and key and the calculation score, query and key are node vectors of each layer; a ij represents the score between the node vector of the i-th layer and the node vector of the j-th layer;

[0053] After the score is normalized by softmax, the corresponding attention weight is obtained.

[0054]

[0055] Use attention weights to weight the features from different layers and get the aggregation results of each head.

[0056] Among them, the j-th layer aggregation result is:

[0057]

[0058] For the outbound neighborhood, the outbound neighborhood aggregation result of each head is obtained as follows:

[0059] After obtaining the aggregation results of all heads, all nodes formed by splicing them are represented as and The attention results of multiple heads at the i-th layer are concatenated as follows:

[0060]

[0061] Among them, W o represents the learnable parameter matrix, H represents the number of attention heads;

[0062] Finally, the aggregation results of all layers are aggregated again to obtain the final representation of all nodes as and

[0063]

[0064] Among them, fc3 and fc4 represent fully connected layers. and It is the weight when performing weighted summation on each layer.

[0065] Furthermore, in step S200, the attention result is added to the result of each convolution layer to obtain the vector of the bidirectional neighborhood of the node:

[0066]

[0067] The node vectors of bidirectional neighborhood aggregation are concatenated to obtain the final embedded representation of the node. The calculation formula is as follows:

[0068]

[0069] in, represents the learnable parameter matrix, represents the learnable bias term;

[0070] Based on the obtained embedding representations of all nodes, calculate the node v∈V / S t Relative to the current seed set S t The marginal gain estimation function, namely the Q function, is defined as follows:

[0071]

[0072] in, and represents the learnable parameter matrix; represents the learnable bias term; are the final embedding representations of nodes v, u, and w respectively; Θ represents all parameters.

[0073] Further, in step S300, the Q network is used to iteratively train E Q Rounds, each round of iteration randomly selects a training graph from the training graph and initializes the seed set to an empty set; then k seed nodes are selected k times, and the ε-greedy strategy is used to select the seed node each time. ε is initially set to 1.0, and gradually decreases to 0.05 and maintains it as the iteration increases; when a node is selected, the quadruple is placed in the experience buffer pool; then a set of samples N are randomly sampled from it b Perform parameter updates and update the target network every 100 iterations.

[0074] Furthermore, in step S300, during the model training process of this step, the loss function takes the average error as the optimization target, and the specific calculation formula is as follows:

[0075]

[0076] Among them, yi =r i +γmax v Q(v;S t+1 ; Θ'), Θ' represents all parameters in the target network; r i represents the reward obtained when adding node v to the seed set, and its value is calculated through Monte Carlo simulation; γ∈[0,1] represents the discount factor, which determines the importance of future rewards; |N b |Indicates the number of samples taken in the experience replay buffer pool.

[0077] The present invention provides an influence maximization system based on deep reinforcement learning and graph neural networks. The system has a program module corresponding to the above steps, and executes the steps in the above-mentioned influence maximization method based on deep reinforcement learning and graph neural networks during runtime.

[0078] The present invention provides a computer-readable storage medium, which stores a computer program. The computer program is configured to implement the steps of an influence maximization method based on deep reinforcement learning and graph neural network when called by a processor.

[0079] Compared with the prior art, the present invention has the following beneficial effects:

[0080] This paper proposes a new influence maximization model that combines graph neural networks (GNNs) and deep reinforcement learning (DRL) technology, aiming to optimize the selection of seed nodes through reinforcement learning technology to achieve maximum information dissemination in the network.

[14] , which can accurately capture the interaction relationship between nodes in the graph structure, which is highly consistent with the information propagation process in the cascade propagation model. Therefore, in the influence maximization model based on deep reinforcement learning and graph neural network (BiGDN), GNNs are combined with DRL, and GNNs are used to learn the embedded representation of nodes. These embedded representations will be used as the deep Q network (DQN)

[15] The input of the proposed method can fully capture the complex structure and node relationships in the network. In addition, the present invention also introduces a multi-head attention mechanism to aggregate the information of each round, which not only enhances the expression of key features but also effectively reduces information redundancy.

[0081] The present invention contributes as follows:

[0082] This paper proposes a new influence maximization model BiGDN, which combines GNNs and DRL techniques to solve the IM problem more accurately;

[0083] The present invention designs a bidirectional neighborhood aggregation strategy to capture the structural and influence characteristics of nodes in NE. In addition, the present invention pre-trains NE on an influence prediction-based regression task to learn initial node embeddings, which are used as inputs to QN and adjusted during the training process of BiGDN to optimize performance.

[0084] The present invention proposes a combination of state convolutional layers and multi-head attention mechanism in QN to more effectively learn node representations in QN;

[0085] Through the pre-trained model, BiGDN can be applied to different network datasets, showing good generalization ability and operation efficiency;

[0086] Experiments on a large number of real social network datasets show that BiGDN is close to IMM and outperforms other reinforcement learning based influence maximization algorithms on most datasets. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] Figure 1 A framework diagram of an influence maximization method based on deep reinforcement learning and graph neural network in an embodiment of the present invention;

[0088] Figure 2 This is a flow chart of a method for maximizing influence based on deep reinforcement learning and graph neural network in an embodiment of the present invention;

[0089] Figure 3 The figure is a comparison chart of the propagation range of seed sets generated by the BiGDN algorithm, the IMM algorithm, the S2V_DQN algorithm and the ToupleGDD algorithm on nine data sets in an embodiment of the present invention, wherein the red line represents the BiGDN algorithm, the blue line represents the IMM algorithm, the orange line represents the S2V_DQN algorithm, and the green line represents the ToupleGDD algorithm;

[0090] Figure 4 This is a comparison chart of the running time of the BiGDN algorithm, IMM algorithm, S2V_DQN algorithm and ToupleGDD algorithm on nine data sets in an embodiment of the present invention, wherein the red line represents the BiGDN algorithm, the blue line represents the IMM algorithm, the orange line represents the S2V_DQN algorithm, and the green line represents the ToupleGDD algorithm. DETAILED DESCRIPTION

[0091] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0092] 1.1 The greatest influence

[0093] Kempe et al. [2]The influence maximization problem is formally proposed and defined as a discrete optimization problem based on a specific influence propagation model. The core of the problem is to find a set of optimal "seed nodes" so that these nodes can trigger the maximum range of influence propagation under limited resource constraints, thereby promoting the sales of products or services. To this end, Kempe et al. proposed an algorithm based on a greedy strategy, which can approximate the theoretical optimal solution with an approximation ratio of 1-1 / e-ε.

[0094] Subsequently, in order to further improve the efficiency of the algorithm, a series of efficient approximate algorithms and heuristic algorithms have been proposed. Among them, Borg et al. [6] The research results of are particularly significant. They proposed an influence maximization algorithm with significant efficiency improvement. The algorithm significantly reduces the computational time required for Monte Carlo simulation by introducing RIS (Reverse Influence Sampling) technology. However, although this method has significantly improved efficiency, in practice, in order to obtain better estimation results, it still requires relatively large computational overhead and consumes a lot of memory to save the reverse reachable set.

[0095] TIM [7] and IMM [8] Algorithms such as have further consolidated the progress in this field. These algorithms not only maintain the same approximation guarantees as the Borg algorithm, but also further reduce the time complexity by adopting different optimization strategies. In particular, the IMM algorithm has achieved remarkable results in reducing unnecessary computational costs, and is therefore widely regarded as one of the most advanced traditional algorithms. However, despite the excellent performance of these algorithms in small and medium-sized networks, they still face scalability challenges when dealing with large influence networks. As the size of the network increases, the computational efficiency and performance of these algorithms will be severely affected, so further research and innovation are needed to solve this problem.

[0096] 1.2 Solving IM Problems Using Graph Neural Networks and Graph Embedding

[0097] Graph embedding technology is an advanced data representation method that aims to convert nodes and edges in graph structured data (such as social networks, etc.) into dense representations in low-dimensional vector space. This type of technology, such as DeepWalk

[16] Node2Vec

[17] and Struct2Vec

[18] , by capturing the structural information in the graph and the relationship between nodes, it provides more effective input features for downstream tasks (such as node classification, link prediction, and graph classification).

[0098] In recent IM research, many scholars have begun to adopt GNNs and graph embedding techniques to seek more effective solutions. Among them, Sonia et al.

[19] proposed a DeepWalk-based influence maximization algorithm (DWIM), which uses DeepWalk to generate low-dimensional embedding representations for nodes in social networks. The core idea of ​​the DWIM algorithm is to calculate the similarity between nodes through the node embeddings learned by DeepWalk. When the similarity between two nodes is high, their ability to spread information in the network is also similar, so the probability of being activated to further influence other nodes will also increase. Based on this heuristic strategy, DWIM attempts to identify key nodes in the network, that is, the set of nodes that can maximize the range of information dissemination. Similar to DWIM, Li et al.

[20] The GAEIM method is proposed, which uses Node2Vec to learn feature representations of nodes and combines it with the Graph Attention Model (GAM) to learn more complex graph topologies. Although both algorithms have achieved some success, they lack a rigorous theoretical foundation as heuristic algorithms. Therefore, its performance varies in different network structures and propagation models, limiting its generality.

[0099] Kumar et al. [9] By integrating the ideas of graph embedding and graph neural network, a new influence maximization method SGNN is innovatively proposed. This method transforms the influence maximization problem in complex networks into a pseudo regression problem. By calculating the potential influence of each node under the susceptible-infected-recovered (SIR) and independent cascade (IC) information diffusion models, the labels required for training are generated for the GNN-based regressor. In SGNN, the Struct2Vec algorithm is used to generate the initial embedding vectors of the nodes, which are then passed as input to the GNN regressor to identify key nodes in the network.

[0100] Subsequently, Kumar et al.

[10] The GLSTM method was further proposed, which used three different centrality measures (degree centrality, closeness centrality, and k-shell centrality) as the initial feature representation of the nodes in the initial embedding stage. These features were then input into a graph-based long short-term memory (GLSTM) regressor to more accurately calculate the influence score of the node. This improvement significantly improved the model's performance in identifying key nodes.

[0101] However, although SGNN and GLSTM have shown strong capabilities in dealing with the problem of maximizing influence, they face the challenge of time-consuming calculation of the initial embedding of nodes on large-scale graph data. In addition, since these methods rely on regression tasks to train the models, the identified key nodes do not always produce ideal synergistic effects in the actual propagation process, affecting the final propagation range.

[0102] 1.3 Using reinforcement learning to solve IM problems

[0103] When solving the IM problem, regression tasks are used to identify key nodes, which do not always produce good synergy in the actual propagation process. This is because the IM problem is essentially a combinatorial optimization problem, which requires careful consideration of the complex interactions between nodes and the combined effects of these interactions.

[0104] In recent years, DRL has gradually emerged and has shown great potential in solving combinatorial optimization problems.

[25] Therefore, more and more studies have begun to model the IM problem as a reinforcement learning task and solve it with the help of DRL methods. Among them, the method combining GNNs and DRL is particularly eye-catching because they can simultaneously capture the node features and structural information in the network.

[0105] For example, Khalil et al.

[11] The proposed S2V-DQN model uses GNN to learn the embedding representation of nodes and makes decisions through DQN. Subsequently, Li et al.

[12] S2V-DQN was further improved and the PIANO model was proposed to better adapt to subgraph training. However, the generalization ability of these methods is still limited when dealing with non-homogeneous networks with different topological characteristics.

[0106] To overcome these challenges, Chen et al.

[13] The ToupleGDD model is proposed. By training on randomly generated graphs, ToupleGDD can be applied to different graph structures and has a certain generalization ability. ToupleGDD learns node embeddings through three coupled GNNs to capture the complex relationship of cascade effects and network topology. Although the performance of ToupleGDD is close to that of the traditional IMM algorithm on some data sets, the information redundancy and lack of generalization ability caused by coupling multiple GNNs are still problems that need to be solved.

[0107] 2.1 Related concepts

[0108] IM is to find a group of nodes in a social network to maximize the scope of dissemination. In order to give a definition of the problem with the greatest influence, the relevant concepts are first defined here.

[0109] Definition 2.1 (Social Network). Given a social network G = (V, E), where V represents the set of users in the social network and E represents the set of social relationships between users. uv ∈[0,1] is the edge e connecting node u to node v uv The success probability of information propagation on represents the probability of successful information propagation from u to v.

[0110] Since the present invention uses an independent cascade propagation model in the experiment, the independent cascade model is introduced here.

[21] .

[0111] Definition 2.2 (Independent Cascade Model). Given a social network G = (V, E). The independent cascade model first activates the seed set Then, at each discrete time step, each activated node tries to uv Deactivate its inactivated neighbor nodes. In this process, each activated node can only try to activate its neighbor once. Regardless of whether the neighbor node is activated, it will only remain activated in the following time steps without performing activation behavior. The propagation ends when no new nodes are activated.

[0112] 2.2 Problem Definition

[0113] Definition 2.3 (Maximizing influence). Given a social network G = (V, E), a propagation model and a budget k. Find a seed set S of k nodes that maximizes the influence spread, that is:

[0114]

[0115] Among them, S * represents the seed set consisting of k selected nodes, |S| represents the number of nodes in the node set S, and σ(S) represents the number of nodes expected to be activated in the node set S. After adding a node to the node set, the increment of the propagation range compared to the propagation range of the original node set is called the marginal gain, that is:

[0116] σ(v;S)=σ(S∪{v})-σ(S) (2)

[0117] The key to defining the IM problem as a reinforcement learning problem is how to design a Q function Make As close as possible to σ(v,S), that is, design a Q function to estimate the marginal gain σ(v;S) of any node v∈V\S after being added to the seed set under the current seed set S. DQN in Deep Reinforcement Learning

[19] Given, use a neural network to estimate the marginal gain of node v. The problem definition of learning-based IM is as follows:

[0118] Definition 2.4 (IM problem based on learning). (1) In the learning phase, given a set of training networks G = {G1, G2, …G n}, a propagation model, a function Train a set of parameters Θ such that as close as possible to σ(v; S); (2) In the reasoning stage, given a target social network G, a set of parameters Θ and a budget k, solve the IM problem about the budget k under a given propagation model.

[0119] Specific implementation plan 1: Combine Figure 1 and Figure 2 As shown, the present invention provides an influence maximization method based on deep reinforcement learning and graph neural network, comprising the following steps:

[0120] Combination Figure 1 As shown in the figure, for the input graph, NE is first used to generate an embedding representation for the node, then QN is used to calculate a Q value for each node, and finally the seed node is selected through the ε-greedy strategy. The nodes in the social network graph can be regarded as individual users. By identifying key users, information can be more widely disseminated, such as for advertising marketing;

[0121] Combination Figure 2 As shown in Figure 2, the structure of BiGDN is described, including the node encoder and Q network.

[0122] Among them, NE adopts a multi-round aggregation mechanism, which not only considers the bidirectional neighborhood information of the node, but also performs weighted processing on this information to splice a bidirectional feature vector containing rich contextual information. After this series of operations, each node in the graph will eventually be mapped into a 128-dimensional vector representation, which accurately captures the key information of the node's position, connection, and attributes in the graph;

[0123] QN uses the node vector generated by the node encoder. It first accumulates the results of bidirectional domain aggregation of multiple rounds of combined states, and introduces a multi-head attention mechanism to weight the results of multiple rounds of aggregation, thereby obtaining a more comprehensive and detailed node vector representation. On this basis, the Q network further weights the bidirectional features of the spliced ​​nodes to enhance the expressive power of the vector representation. Finally, through the Q function, that is, the Q network calculates the Q value of each node, which provides powerful guidance for the intelligent agent to make the best strategy selection under different states.

[0124] 3.1 Node Encoder

[0125] Graph embedding techniques, such as DeepWalk, Node2Vec, and Struct2Vec, have achieved remarkable results in capturing structural information within graphs, but they mainly rely on node sequences and skip-gram models for training, which limits the generalization ability of node vectors between different graphs. For the influence maximization task, we need a vector representation that can accurately reflect the actual influence of nodes in the network.

[0126] Therefore, the present invention learns the initial vector of the node by constructing an encoder based on influence prediction. This method is based on a regression task, the core goal of which is to predict the average influence range of the node after multiple rounds of propagation under the IC model. As a regressor, the encoder can learn the influence characteristics of the node in the network and generate the corresponding node vector accordingly.

[0127] This influence prediction-based encoder design makes the generated node vector more in line with the requirements of the influence maximization task. It not only captures the structural information of the node in the graph, but also reflects the potential influence of the node in the propagation process, thereby improving the generalization ability of the node vector between different graphs.

[0128] 3.1.1 Label Generation

[0129] As the core of the regression task, we need a set of continuous and meaningful label values ​​to guide the learning process. Therefore, we adopt erdos_renyi

[22] , watts_strogatz

[23] and barabasi_albert

[24] We used random graph models to generate the training graph set. These models were chosen based on their ability to simulate key properties of real social networks, including randomness, small-world properties, and scale-free properties. Next, we used the IC model to calculate the individual influence of each node. By calculating the potential influence of each node, we can quantify the importance or influence of the node in the network.

[0130] In order to convert the calculated influence range into a label value suitable for the regression task, it is normalized and its value range is mapped to between 0 and 1. In this way, each node has a label value corresponding to its influence range, and these label values ​​will serve as the learning objectives of the regression task.

[0131] For any training graph G train The label value calculation formula for the nodes in is as follows:

[0132] ds=IC(G train ) (3)

[0133]

[0134] Among them, ds is a vector composed of the propagation ranges of all nodes in any training graph. The propagation range of the node is obtained by using the IC model through the Monte Carlo model. min(·) and max(·) represent the minimum and maximum values ​​in the vector, respectively. ds v Indicates the propagation range of node v, INF v It is the result obtained after normalization by ds.

[0135] In this way, a label value corresponding to its influence range in the network is generated for each node, which will be used as learning targets in subsequent regression tasks to guide the training process of the node encoder.

[0136] 3.1.2 Node Encoder Design

[0137] In the present invention, the initial input of the node encoder uses a 128-dimensional all-1 vector. In order to more accurately capture the topological structure characteristics of the node, a two-way neighborhood aggregation strategy is introduced to update the vector representation of the node, thereby achieving a two-way consideration of the node neighborhood information.

[0138] To achieve this, a convolutional layer is defined ( Figure 2 The conv layer in the node is used to aggregate the neighborhood information of the node. Since the bidirectional neighborhood aggregation method is adopted, in the regression model, the information of the node's in-neighborhood (i.e., the neighbors connected to the edges pointing to the node) and out-neighborhood (i.e., the neighbors connected to the edges starting from the node) will be aggregated in parallel for T rounds, and the present invention sets T = 3. Specifically, for node v, in the i+1th layer, the conv layer update calculation formulas for its in-neighborhood and out-neighborhood information are as follows:

[0139]

[0140]

[0141] Among them, N in (v) N out (v) represent the incoming and outgoing neighbor sets of node v, respectively, and p uv Represents edge e uv The probability of successful information dissemination on is a learnable parameter matrix, d represents the dimension of the input vector, LeakyReLU is used as the activation function, and the negative slope is set to 0.2. The vectors obtained by aggregating the incoming and outgoing neighborhoods of node u at the i-th layer, They represent the vectors obtained by aggregating the incoming and outgoing neighborhoods of the i+1th layer of node v respectively.

[0142] After the aggregation is completed, the in-neighborhood and out-neighborhood information is concatenated to obtain the embedded representation of node v, that is:

[0143]

[0144] in, is a learnable parameter matrix, [] represents a connection operation, and fc1 is a fully connected layer;

[0145] After obtaining the node vector, the predicted value of the node's influence is calculated through the fully connected layer Right now:

[0146]

[0147] Among them, fc2 is the fully connected layer.

[0148] 3.1.3 Model Training

[0149] The mean square error (MSE) is used as the loss function, that is:

[0150]

[0151] Among them, B v Represents a batch of nodes of a graph set, and then updates the encoder parameters W through the back-propagation algorithm and the gradient descent optimizer to minimize the error of the regression task, that is:

[0152]

[0153] W=W+ΔW (11)

[0154] Among them, α represents the learning rate and ΔW is the update gradient.

[0155] The training process of the node encoder NE is as follows. First, the generated training set is iteratively trained E n rounds, each round divides the training set into N e For each copy, NodeEncoder is used to calculate the predicted value of the node's propagation capability, then the loss is calculated, the Adam optimizer is used to update the parameters, and finally the trained parameters are returned.

[0156]

[0157] After pre-training is completed, the parameters of the encoder are saved, and during the reinforcement learning training process, the parameters of the encoder are dynamically adjusted based on the actual influence of the node.

[0158] 3.2Q Network

[0159] The design of the Q network is the core of the present invention. The IM problem is constructed as an RL problem. The components of the Q network are as follows:

[0160] 3.2.1 Reinforcement Learning Formula

[0161] For the IM problem, we can naturally describe it as a reinforcement learning problem and define these reinforcement learning components as follows:

[0162] state: state S t is a |V|-dimensional vector, where V is the set of all nodes in the graph. For each node v in the graph, if node v has been selected as a seed node (that is, has been added to the seed set), then the element S in the corresponding state vector t (v) = 1; otherwise 0. The state vector fully describes the current seed set. When there is no ambiguity, use S t It also represents that in state S t The seed set at that time.

[0163] action: an action A t It means to select a node from the set of nodes that have not been selected as seed nodes and add it to the seed set. In each step, the agent needs to choose an action based on the current state.

[0164] reward: In the case of IM problem, the reward is r t+1 (S t ; A t ) is defined as t Next, select a new node v to join the seed set S t+1 The marginal gain after t+1 (S t ; A t = v) = σ(S t+1 )-σ(S t ).

[0165] Transition: When a node is selected to join the seed set S t After S t (v) Changes from 0 to 1, and obtains a new state S t+1 .

[0166] Policy: According to the current state S t Determine the action A that the agent should take t In the IM problem, the goal of the strategy is to maximize the cumulative reward, that is, to find a seed set under which the number of nodes that can be influenced by influence propagation is the largest.

[0167] 3.2.2 Convolutional Layers with Combined States

[0168] After processing by the node encoder, we obtain a node vector representation that reflects the node's propagation ability and topological structure information. Next, we define a state-combined convolutional layer and use R layers (we set R = 3) of convolutional layers, which integrate node information and state information through bidirectional field aggregation. This method not only considers the direct neighbors of the node (that is, the case where the node activates its neighbors), but also considers the case where the node is activated by its neighbors. Due to the use of bidirectional aggregation, we divide the vector output by the node encoder into two parts: and

[0169] For the convolutional layer of the combined state of the i+1th layer into the neighborhood The calculation formula is as follows:

[0170]

[0171] Among them, S t (u) represents the state of node u. When S t When (u) = 1, it means that node u has been selected into the seed set. t When (u)=0, it means that node u has not been selected into the seed set. and is the learnable parameter matrix, is a learnable bias term.

[0172] Similarly, for the convolutional layer of the combined state of the i+1th layer out of the neighborhood, the calculation formula is:

[0173]

[0174] in, and is the learnable parameter matrix, is a learnable bias term.

[0175] In order to reduce information redundancy and focus on the relationship between nodes, the information of the nodes themselves is not directly incorporated into the design of the convolution layer. Instead, the strategy of saving the results of each convolution layer is adopted for subsequent processing.

[0176] 3.2.3 Multi-Head Attention

[0177] In order to effectively capture the key features of nodes in the graph and distinguish different nodes, while solving the over-smoothing problem that may be caused by graph convolutional networks, a multi-head attention mechanism is introduced to aggregate information from different convolutional layers. The core idea of ​​this mechanism is to focus on different aspects through multiple independent "heads" and combine their results to form a richer node representation.

[27] .

[0178] Specifically, after obtaining the results of each convolution layer, a multi-head attention mechanism is used to further refine and fuse this information. For each "head", additive attention is used

[28] To calculate the attention score coefficient, this is because additive attention allows more flexibility to capture complex relationships between different layers rather than just simple similarities.

[0179] For each head in the in-neighborhood (the set of nodes connected by the edges pointing to the node), the attention score coefficient is calculated based on the following logic:

[0180] ·Convolution results of each layer ( where i represents the number of layers) as input.

[0181] Through the shared three linear layers w q 、w k 、w s and a non-linear activation function tanh to compute a score.

[0182]

[0183] Among them, w q 、w k 、w s Respectively represent the linear transformation of query and key and the calculation score, where query and key are node vectors of each layer, a ij Represents the score between the node vector of the i-th layer and the node vector of the j-th layer.

[0184] After the scores are normalized by softmax, the corresponding attention weights are obtained.

[0185]

[0186] Next, use these attention weights to weight the features from different layers to get the aggregation results of each "head" The j-th layer aggregation result is:

[0187]

[0188] Similarly, perform the same operation on the outbound neighborhood (the set of nodes connected by the edges starting from the node) to obtain the outbound neighborhood aggregation result of each "head"

[0189] After obtaining the aggregation results of all "heads", all nodes formed by splicing them are represented as and This concatenation operation allows multiple independent attention results from different "heads" to be considered simultaneously, resulting in a more comprehensive and rich node representation, where the attention results of multiple heads at layer i are concatenated as follows:

[0190]

[0191] Among them, W o is the learnable parameter matrix and H is the number of attention heads.

[0192] Finally, in order to obtain a unified node representation, the aggregation results of all layers are aggregated again to obtain the final node representation and

[0193]

[0194]

[0195] Among them, fc3 and fc4 represent fully connected layers. and It is the weight when performing weighted summation on each layer.

[0196] 3.2.4Q value calculation

[0197] After attention, add the attention result and the result of each layer of convolution to get the vector of the node's bidirectional neighborhood:

[0198]

[0199] The node vectors of bidirectional neighborhood aggregation are concatenated to obtain the final embedded representation of the node. The calculation formula is as follows:

[0200]

[0201] in, is the learnable parameter matrix, is a learnable bias term.

[0202] Based on the obtained embedding representations of all nodes, calculate the node v∈V / S t Relative to the current seed set S t The marginal gain estimation function (Q function) is defined as follows:

[0203]

[0204] in, and is the learnable parameter matrix, is a learnable bias term, are the final embedding representations of nodes v, u, and w respectively. All parameters in BiDGN are represented by Θ.

[0205] 3.3 Training with DDQN

[0206] Using DDQN

[26] Perform model training and update parameters. The DDQN algorithm is a DQN

[15] The improvement can effectively solve the problem of overestimation of Q value in DQN. By introducing a dual network structure, namely the online network and the target network, DDQN can more accurately evaluate the value of actions and update the strategy accordingly. In addition, the use of the experience replay mechanism further enhances the stability of the algorithm. By learning by randomly sampling from previous experience, the correlation between data can be prevented from interfering with the learning process. These improvements make DDQN more stable and outstanding in performance, especially when dealing with complex tasks, its advantages are more obvious.

[0207] The experience replay mechanism allows the data (i.e., experience) generated by each interaction with the environment to be stored in a replay buffer. During the training process, the latest data is no longer used directly, but a batch of experiences are randomly sampled from the replay buffer as training samples. The advantage of this is that it makes the training samples approximately meet the independent and identically distributed assumption, and improves the efficiency of sample use, because each sample has the opportunity to be used for training multiple times.

[0208] The target network is another key component. The target value is not calculated directly by the current network, but by a relatively lagging target network. This means that the target network is not updated at every step, but synchronized with the current network once every certain number of steps. This design reduces the fluctuation of the target value during training and makes the training process more stable. Denote all the parameters in the target network by Θ'.

[0209] The loss function of DDQN uses MSE as the optimization target. The specific calculation formula is as follows:

[0210]

[0211] Among them, y i =r i +γmax v Q(v;S t+1 ; Θ'), Θ' is all the parameters in the target network, r i is the reward obtained when adding node v to the seed set, and its value is calculated through Monte Carlo simulation. γ∈[0,1] is the discount factor, which determines the importance of future rewards. The present invention sets γ to 0.99. |N b| is the number of samples taken in the experience replay buffer pool, set to 16.

[0212]

[0213] The above content shows the training process of BiGDN. A total of E Q In each round of iteration, first randomly select a training graph from the training graph and initialize the seed set to an empty set. Then perform k times to select k seed nodes, and use the ε-greedy strategy to select seed nodes each time. ε is initially set to 1.0, and gradually decreases to 0.05 and maintains it as the iterations increase. In this way, the model can be fully explored in the early stage. As the training progresses, the parameters of the model gradually become better. At this time, the model is mainly used to select nodes. After selecting a node, the quadruple is then placed in the experience buffer pool. When a set of samples N is randomly sampled from it again b Perform parameter updates and update the target network every 100 iterations.

[0214] 3.4 Seed Selection

[0215] During the training phase, the model learns how to make optimal decisions by optimizing parameters. These parameters imply the optimal strategy for selecting seeds. When selecting seeds from the seed set, in theory, one can refer to the greedy strategy used during training, that is, always select the node with the largest Q value as the seed. However, in practice, this method of stepwise selection and state update is not directly applied because when a seed node is selected and the network embedding and Q value are recalculated, the order of the nodes relative to their Q value usually does not change significantly.

[12] .

[0216] In order to efficiently construct the seed set, a more direct method is adopted: once the Q values ​​of all nodes are calculated, the first k nodes with the largest Q values ​​are directly selected as the seed set. This method not only simplifies the selection process, but also has been verified in subsequent experiments that it can effectively select the nodes that contribute the most to the task.

[0217] Therefore, after calculating the q value of each node, we can directly select the first k nodes with the largest q values. The following describes the seed set selection process. Given a graph G and a budget k, we use the trained parameters to calculate the Q value of each node and select the first k nodes with the largest Q values ​​to add to the seed set.

[0218]

[0219] Specific implementation method 2: The present invention provides an influence maximization system based on deep reinforcement learning and graph neural network. The system has a program module corresponding to the above steps, and executes the steps in the above-mentioned influence maximization method based on deep reinforcement learning and graph neural network during operation.

[0220] The other combinations and connection relationships of this embodiment are the same as those of the first embodiment.

[0221] Specific implementation method three: The present invention provides a computer-readable storage medium, which stores a computer program, and the computer program is configured to implement the steps of an influence maximization method based on deep reinforcement learning and graph neural network when called by a processor.

[0222] The other combinations and connection relationships of this embodiment are the same as those of the first embodiment.

[0223] 4 Experiments

[0224] The performance of the BiGDN model proposed in this paper on different data sets is verified through experiments. This experiment compares BiGDN with the most advanced traditional algorithm IMM and two reinforcement-based IM algorithms: S2V_DQN and ToupleGDD.

[0225] All experiments were completed on a computer equipped with an Intel Xeon CPU (2.40GHz, 12 cores), 80GB RAM, RTX 3080Ti GPU (12GB video memory), and running on the Ubuntu 20.04 operating system. Through these experiments, the performance and practicality of the BiGDN model were fully evaluated.

[0226] 4.1 Experimental Setup

[0227] Datasets. In the experiment, 9 real-world datasets were used. The specific information of these datasets is shown in Table 1. It is worth noting that in order to uniformly process and adapt to the above four models, undirected graphs are processed, that is, each undirected edge is split into two directed edges in opposite directions, so that all datasets are converted into directed graphs. This step ensures the consistency of the data and enables the above four models to be more effectively applied to various graph structured data.

[0228] Table 1 Dataset

[0229]

[0230] Diffusion model. BiGDN can be applied to different diffusion models, such as IC and LT, by simply modifying the calculation method of rewards. Here, the IC model is selected for testing and set This method is widely used. In order to fairly calculate the expected impact of the four methods, we first independently record the seed set of each algorithm, and then perform 10,000 simulation propagations based on the selected seeds. Finally, we take the average result of the 10,000 simulations.

[0231] Training data generation. To pre-train NE, ER, SW, and BA models are used to generate training graphs. Specifically, the probability of edge connection is set to 0.15 in ER, the nearest neighbor is set to 4 in SW, the probability of reconnection of each edge is set to 0.15, and the number of edges connected by new nodes is set to 3 in BA. 50 graphs are generated independently in these three models, with nodes ranging from 20 to 200. These graphs form a set of training graphs for model pre-training. To train BiGDN, the same training dataset as ToupleGDD is used to make the results more comparable.

[0232] Parameter settings. Select the Adam optimizer, which is usually more effective in handling high-dimensional data and gradient noise. Use the default parameter settings of the Adam optimizer, which is usually able to adaptively adjust the learning rate. Set the batch size to 16, which means that in each iteration, the model will randomly sample 16 samples from the experience buffer pool for training. Set the node embedding dimension to 64. When training in reinforcement learning, set the budget k=5, that is, the termination state is reached after selecting 5 nodes, and set the step size n_steps=1. For IMM, the parameter setting ∈=0.1.

[0233] 4.2 Experimental Results

[0234] 4.2.1 Ablation Experiment

[0235] Table 2 Performance comparison of BiGDN under different settings

[0236]

[0237]

[0238] When a seed node is selected and the network embedding and Q-values ​​are recalculated, the ranking of nodes relative to their Q-values ​​tends to remain relatively stable. To verify this conclusion and explore its impact on the seed node selection strategy, a comparative experiment was conducted.

[0239] By comparing the propagation range of the seed set selected under different seed selection strategies, (1) directly select the k nodes with the highest Q value as the seed set (One-time); (2) adopt a greedy strategy to iteratively select the nodes with the highest Q value and add them to the seed set one by one (Iterative). Under each seed selection strategy, the performance of the model with and without pre-training the node encoder is compared, corresponding to √ and × in the PT column, respectively. The experimental results are shown in Table 2.

[0240] According to the data in Table 2, the following conclusions can be drawn:

[0241] (1) In terms of propagation range, for most data sets, the one-time strategy and the iterative strategy show very similar performance in terms of propagation range. This result shows that when selecting the seed set once, the key nodes in the network can be covered quite accurately, so the subsequent iterative selection does not bring significant improvement. This finding also shows that the parameters obtained through training already contain a seed selection strategy close to the optimal one. However, it is worth noting that on small network data sets such as moreno and netscience, the iterative strategy shows a more obvious improvement in propagation range compared to the one-time strategy. This is because in smaller networks, the iterative strategy can more effectively utilize the connection relationship between nodes and more accurately identify nodes with higher propagation capabilities through iteration. In contrast, on medium and large data sets, due to the large number of nodes and the large radius of the graph, the one-time strategy and the iterative strategy are both limited to the range of their R-hop neighborhood when obtaining node information, so the difference in propagation range between the two is not significant. Comparing the three models, it is found that the pre-trained BiGDN model has the widest influence on the propagation range of the node encoder, while the non-pre-trained model has a reduced propagation range. This finding highlights the effectiveness of node encoder pre-training and shows that the pre-training process can significantly improve the model's ability to propagate information and capture graph structure information.

[0242] (2) In terms of propagation time efficiency, the time required for the One-time strategy is almost the same when selecting seed sets of different sizes. This is mainly due to the high efficiency of the torch deep learning framework in processing topk selection, which makes the time difference in the range of 10 to 50 nodes negligible. In contrast, due to its iterative nature, the time required for the Iterative strategy increases significantly as the size of the seed set increases, which is consistent with the expectations of the present invention.

[0243] For most data sets, the One-time strategy has shown sufficient effectiveness. It can select relatively important nodes in a single operation, thereby achieving a wide range of propagation. More importantly, the efficiency of the One-time strategy is more in line with the requirements of the present invention for time efficiency. Therefore, taking all factors into consideration, the One-time strategy is a more appropriate choice.

[0244] 4.2.2 Scope of dissemination

[0245] The performance of IMM, S2V_DQN, ToupleGDD, and BiGDN in generating seed set propagation range is compared. The optimal parameter values ​​of each algorithm are set according to the recommendations in their original papers. Nine real-world datasets are selected and the propagation range of seed sets of sizes 10, 20, 30, 40, and 50 are tested on these datasets. The results are shown in the figure below. Figure 3 shown.

[0246] The experimental results show that BiGDN can achieve a similar propagation effect as IMM on each data set, which fully proves the effectiveness of the BiGDN algorithm. At the same time, BiGDN's performance is generally better than other reinforcement learning-based algorithms.

[0247] However, we also observed that S2V_DQN performed well on some datasets, surpassing ToupleGDD on some datasets such as GitHub and YouTube, but not on many other datasets, which shows that the algorithm has limitations in generalization ability. Similarly, although ToupleGDD showed good performance on all datasets, its generalization ability still needs to be improved compared with BiGDN, especially on some datasets, the gap between the two is more significant.

[0248] In summary, BiGDN has demonstrated strong competitiveness and stability in terms of the scope of seed set dissemination.

[0249] Although the present invention is disclosed as above, the protection scope of the present invention is not limited thereto. Those skilled in the art may make various changes and modifications without departing from the spirit and scope of the present invention, and these changes and modifications will fall within the protection scope of the present invention.

[0250] References

[0251] [1]P.Domingos and M.Richardson,“Mining the network value ofcustomers,”in Proceedings of the Seventh ACM SIGKDDInternational Conferenceon Knowledge Discovery and Data Mining,2001,pp.57–66.

[0252] [2]D.Kempe,J.Kleinberg,and′Eva.Tardos,“Maximizing the spread ofinfluence through a social network,”in Proceedings ofthe Ninth ACMSIGKDDInternational Conference on Knowledge Discovery andData Mining,2003,pp.137–146.

[0253] [3]A.Goyal,W.Lu,and L.V.Lakshmanan,“Celf++:optimizing the greedyalgorithm for influence maximization in social networks,”in Proceedings ofthe 20th International Conference Companion on WorldWide Web,2011,pp.47–48.

[0254] [4]L.Cui,H.Hu,S.Yu,Q.Yan,Z.Ming,Z.Wen,and N.Lu,“Ddse:A novelevolutionary algorithmbased on degree-descending search strategy forinfluence maximization in social networks,”Journal ofNetwork and ComputerApplications,vol.103,pp.119–130,2018.

[0255] [5]J.Bae and S.Kim,“Identifying and ranking influential spreaders incomplex networks byneighborhood coreness,”Physica A:Statistical Mechanics andits Applications,vol.395,pp.549–559,2014.

[0256] [6]C.Borgs,M.Brautbar,J.Chayes,and B.Lucier,“Maximizing socialinfluence in nearly optimaltime,”in Proceedings ofthe Twenty-Fifth AnnualACM-SIAM Symposium on Discrete Algorithms,2014,pp.946–957.

[0257] [7]Y.Tang,X.Xiao,and Y.Shi,“Influence maximization:Near-optimal timecomplexity meets practicalefficiency,”in Proceedings of the 2014 ACM SIGMODInternational Conference on Management ofData,2014,pp.75–86.

[0258] [8]Y.Tang,Y.Shi,and X.Xiao,“Influence maximization in near-lineartime:A martingale approach,”in Proceedings ofthe 2015 ACM SIGMODInternational Conference on Management ofData,2015,pp.1539–1554.

[0259] [9]S.Kumar,A.Mallik,A.Khetarpal,and B.Panda,“Influence maximizationin social networks usinggraph embedding and graph neural network,”InformationSciences,vol.607,pp.1617–1636,2022.

[0260]

[10] S.Kumar,A.Mallik,and B.Panda,“Influence maximization in socialnetworks using transferlearning via graph-based lstm,”Expert Systems withApplications,vol.212,pp.1–16,2023.

[0261]

[11] H.Dai,E.B.Khalil,Y.Zhang,B.Dilkina,and L.Song,“Learningcombinatorial optimizationalgorithms over graphs,”Advances in NeuralInformation Processing Systems,vol.30,2017.

[0262]

[12] H.Li,M.Xu,S.S.Bhowmick,J.S.Rayhan,C.Sun,and J.Cui,“Piano:Influence maximizationmeets deep reinforcement learning,”IEEE Transactions onComputational Social Systems,2020.

[0263]

[13] T.Chen,S.Yan,J.Guo,Member,W.Wu,and S.Member,“Tuplegdd:A fine-designed solution ofinfluence maximization by deep reinforcement learning,”IEEE Transactions on Computational SocialSystems,2023.

[0264]

[14] J.Gilmer,S.S.Schoenholz,P.F.Riley,O.Vinyals,and G.E.Dahl,“Neuralmessage passing forquantum chemistry,”in Proceedings ofthe 34th InternationalConference on Machine Learning,vol.70,2017,pp.1263–1272.

[0265]

[15] V.Mnih,K.Kavukcuoglu,D.Silver,A.A.Rusu,J.Veness,M.G.Bellemare,A.Graves,M.Riedmiller,A.K.Fidjeland,G.Ostrovski,S.Petersen,C.Beattie,A.Sadik,I.Antonoglou,H.King,D.Kumaran,D.Wierstra,S.Legg,and D.Hassabis,“Human-levelcontrol through deep reinforcementlearning,”NATURE,vol.518,no.7540,pp.529–533,2015.

[0266]

[16] B.Perozzi,R.Al-Rfou,and S.Skiena,“Deepwalk:online learning ofsocial representations,”inProceedings of the 20th ACM SIGKDD InternationalConference on Knowledge Discovery and DataMining,2014,pp.701–710.

[0267]

[17] A.Grover and J.Leskovec,“node2vec:Scalable feature learning fornetworks,”in Proceedings ofthe 22nd ACM SIGKDD International Conference onKnowledge Discovery and Data Mining,2016,pp.855–864.

[0268]

[18] L.F.Ribeiro,P.H.Saverese,and D.R.Figueiredo,“struc2vec:Learningnode representations fromstructural identity,”in Proceedings of the 23rd ACMSIGKDD International Conference on KnowledgeDiscovery and Data Mining,2017,pp.385–394.

[0269]

[19] Sonia,S.Kapil,and B.Monika,“Deepwalk based influence maximization(dwim):Influencemaximization using deep learning,”Intelligent Automation&SoftComputing,vol.35,no.1,pp.1087–1101,2023.

[0270]

[20] Y.Li,P.Li,L.Han,Z.-Y.Jiang,Z.Wang,and Z.Wu,“Maximizing theinfluence ofsocial networksbased on graph attention networks,”20233rd Asia-Pacific Conference on Communications Technologyand Computer Science(ACCTCS),pp.396–403,2023.

[0271]

[21] J.Goldenberg,B.Libai,and E.Muller,“Talk of the network:A complexsystems look at theunderlying process ofword-of-mouth,”Marketing Letters,vol.12,no.3,pp.211–223,2001.

[0272]

[22] P.Erd¨os and A.R′enyi,“On random graphs i,”PublicationesMathematicae Debrecen,vol.6,p.290,1959.

[0273]

[23] D.J.Watts and S.H.Strogatz,“Collective dynamics of’small-world’networks,”Nature,vol.393,no.6684,pp.440–442,1998.

[0274]

[24] A.-L.Barab′asi and R.Albert,“Emergence of scaling in randomnetworks,”Science,vol.286,no.5439,pp.509–512,1999.

[0275]

[25] S.Richard S and B.Andrew G,Reinforcement Learning:anIntroduction.MIT Press,2018.

[0276]

[26] H.Van Hasselt,A.Guez,and D.Silver,“Deep reinforcement learningwith double q-learning,”inProceedings ofthe AAAI Conference on ArtificialIntelligence,vol.30,no.1,2016,pp.2094–2100。

Claims

1. A method for maximizing influence based on deep reinforcement learning and graph neural network, characterized in that: The following steps are involved: S100, using graph neural networks and bidirectional neighborhood aggregation strategies to capture the structure and influence characteristics of nodes in social network graphs, and learning the initial embedding of nodes by pre-training the model on a regression task based on influence prediction, mapping each node in the graph into a vector representation; S200, using deep reinforcement learning and graph neural network to build a Q network, taking the node vector obtained in step S100 as the input of the Q network, using the state-binding bidirectional neighborhood aggregation strategy in the Q network to learn the node vector representation under different states, introducing a multi-head attention mechanism to aggregate each round of information, and using the Q function to calculate the Q value of each node; S300, performing model training on the Q network constructed in step S200 and updating parameters, introducing the online network and the target network based on the experience playback mechanism, and obtaining an optimized Q network; S400, using the Q network optimized in step S300 to provide an optimal strategy by optimizing parameters, after calculating the Q values ​​of all nodes, select the first k nodes with the largest Q values ​​as the seed set, and then select the node that contributes the most to the task.

2. The method for maximizing influence based on deep reinforcement learning and graph neural network according to claim 1, characterized in that: In step S100, an encoder based on influence prediction is constructed to learn the initial vector of the node, which is used to predict the average influence range of the node after multiple rounds of propagation under the IC model, including: S110, generating node labels, using a random graph model to generate a training graph set, using an independent cascade model to calculate the individual influence range of each node, and then performing normalization processing to map its value range to between 0 and 1, so that each node has a label value corresponding to its influence range, and the label value is used as a learning target for the regression task; For any training graph G train The label value calculation formula for the nodes in is as follows: ds=IC(G train ) (1) Among them, ds represents any training graph G train The propagation range of all nodes is a vector composed of the propagation ranges of all nodes. The propagation range of the node is obtained by the independent cascade model through the Monte Carlo model; min(·) and max(·) represent the minimum and maximum values ​​in the vector respectively; ds v Indicates the propagation range of node v; INF v represents the result obtained after normalization by ds; S120, design a node encoder, wherein the initial input of the node encoder is a 128-dimensional all-1 vector, and a two-way neighborhood aggregation strategy is introduced to update the vector representation of the node to achieve two-way consideration of the node neighborhood information, including: Define a convolutional layer to aggregate the information of the node's incoming neighborhood and outgoing neighborhood in parallel for T rounds, where the incoming neighborhood is the neighborhood connected by the edge pointing to the node, and the outgoing neighborhood is the neighborhood connected by the edge starting from the node; For node v, at the i+1th layer, the convolutional layer update calculation formulas for its incoming and outgoing neighborhood information are as follows: Among them, N in (v) N out (v) represent the incoming neighbor and outgoing neighbor sets of node v respectively; p uv Represents edge e uv The probability of successful information dissemination on represents the learnable parameter matrix; d represents the dimension of the input vector, LeakyReLU is used as the activation function, and the negative slope is set to 0.2; They represent the vectors obtained by aggregating the incoming and outgoing neighborhoods of node u at the i-th layer, They represent the vectors obtained by aggregating the incoming and outgoing neighborhoods of the i+1th layer of node v respectively; After the aggregation is completed, the in-neighborhood and out-neighborhood information are concatenated to obtain the embedded representation x of node v v ,Right now: in, Represents a learnable parameter matrix; [] represents a connection operation; fc1 represents a fully connected layer; After obtaining the embedded representation of the node, the predicted value of the node's influence is calculated through the fully connected layer Right now: Among them, fc2 represents the fully connected layer; S130, training the model, iteratively training the training set generated in step S110 n rounds, each round divides the training set into N e The prediction value of the propagation capacity of each computing node is calculated, and then the loss is calculated through the loss function, the parameters are updated using the optimizer, and finally the trained parameters are returned. Use the average error as the loss function, that is: Among them, B v Represents a batch of nodes of a graph set, and then updates the encoder parameters W through the back-propagation algorithm and the gradient descent optimizer to minimize the error of the regression task, that is: W=W+ΔW (9) Among them, α represents the learning rate and ΔW is the update gradient.

3. The method for maximizing influence based on deep reinforcement learning and graph neural network according to claim 2, characterized in that: In step S200, a convolutional layer combining states is defined, and an R-layer convolutional layer is used to integrate node information and state information through bidirectional domain aggregation, and the vector output by the node encoder is divided into two parts: and For the convolutional layer of the i+1th layer into the neighborhood, the combined state The calculation formula is as follows: Among them, S t (u) represents the state of node u. When S t When (u) = 1, it means that node u has been selected into the seed set. t When (u) = 0, it means that node u has not been selected into the seed set; and represents the learnable parameter matrix, represents the learnable bias term; For the convolutional layer of the combined state of the i+1th layer out of the neighborhood, the calculation formula is: in, and represents the learnable parameter matrix, represents the learnable bias term; The results of each convolution layer are retained for subsequent processing.

4. The method for maximizing influence based on deep reinforcement learning and graph neural network according to claim 3, characterized in that: In step S200, a multi-head attention mechanism is introduced when aggregating different convolution information, and for each head, an attention score coefficient is calculated using additive attention; For each head entering the neighborhood, the attention score coefficient is calculated based on the following logic: The convolution result of each layer Where i represents the number of layers, as input; Through the three shared linear layers w q 、w k 、w s and a nonlinear activation function tanh to calculate the score, Among them, w q 、w k 、w s Respectively represent the linear transformation of query and key and the calculation score, query and key are node vectors of each layer; a ij represents the score between the i-th layer node vector and the j-th layer node vector; After the score is normalized by softmax, the corresponding attention weight is obtained. Use attention weights to weight the features from different layers and get the aggregation results of each head. Among them, the j-th layer aggregation result is: For the outbound neighborhood, the outbound neighborhood aggregation result of each head is obtained as follows: After obtaining the aggregation results of all heads, all nodes formed by splicing them are represented as and The attention results of multiple heads at the i-th layer are concatenated as follows: Among them, W o represents the learnable parameter matrix, H represents the number of attention heads; Finally, the aggregation results of all layers are aggregated again to obtain the final representation of all nodes as and Among them, fc3 and fc4 represent fully connected layers. and It is the weight when performing weighted summation on each layer.

5. The method for maximizing influence based on deep reinforcement learning and graph neural network according to claim 4, characterized in that: In step S200, the attention result and the result of each layer of convolution are added to obtain the vector of the bidirectional neighborhood of the node: The node vectors of bidirectional neighborhood aggregation are concatenated to obtain the final embedded representation of the node. The calculation formula is as follows: in, represents the learnable parameter matrix, represents the learnable bias term; Based on the obtained embedding representations of all nodes, calculate the node v∈V / S t Relative to the current seed set S t The marginal gain estimation function, namely the Q function, is defined as follows: in, and represents the learnable parameter matrix; represents the learnable bias term; are the final embedding representations of nodes v, u, and w respectively; Θ represents all parameters.

6. The method for maximizing influence based on deep reinforcement learning and graph neural network according to claim 5, characterized in that: In step S300, the Q network is used to iteratively train E Q Rounds, each round of iteration randomly selects a training graph from the training graph and initializes the seed set to an empty set; then k seed nodes are selected k times, and the ε-greedy strategy is used to select the seed node each time. ε is initially set to 1.0, and gradually decreases to 0.05 and maintains it as the iteration increases; when a node is selected, the quadruple is placed in the experience buffer pool; then a set of samples N are randomly sampled from it b Perform parameter updates and update the target network every 100 iterations.

7. The method for maximizing influence based on deep reinforcement learning and graph neural network according to claim 5, characterized in that: In step S300, during the model training process of this step, the loss function uses the average error as the optimization target, and the specific calculation formula is as follows: Among them, y i =r i +γmax v Q(v;S t+1 ; Θ'), Θ' represents all parameters in the target network; r i represents the reward obtained when adding node v to the seed set, and its value is calculated through Monte Carlo simulation; γ∈[0,1] represents the discount factor, which determines the importance of future rewards; |N b |Indicates the number of samples taken in the experience replay buffer pool.

8. An influence maximization system based on deep reinforcement learning and graph neural network, characterized by: The system has a program module corresponding to the steps of any one of claims 1 to 7 above, and executes the steps in the above-mentioned influence maximization method based on deep reinforcement learning and graph neural network when running.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which is configured to implement the steps of the influence maximization method based on deep reinforcement learning and graph neural network described in any one of claims 1 to 7 when called by a processor.

Images