Method for evaluating and analyzing vulnerability of shipping network based on reinforcement learning

Abstract

The invention discloses a reinforcement learning-based shipping network vulnerability evaluation and analysis method, which comprises the following steps: S1, obtaining port route information, ship frequency information and port economic information, S2, constructing a shipping container transportation network based on the port route information, S3, constructing a reinforcement learning model based on the constructed shipping network G = (V, E), and S4, carrying out evaluation and analysis on the vulnerability of the shipping network based on the reinforcement learning model. Training the learning model; a network topology structure is dynamically adjusted by utilizing reinforcement learning so as to adapt to continuously changing attack modes and network states. The method not only can reflect changes of the network under different attack strategies in real time, but also can simulate complex dynamic interaction in an actual environment. The dynamic updating capability can effectively cope with the defects of static analysis in a traditional method; and an attack strategy is automatically optimized by utilizing reinforcement learning, so that key nodes and paths can be identified more accurately, and the evaluation precision of the network vulnerability is improved.

Description

Technical Field

[0001] The present invention relates to the field of shipping technology, and in particular to a shipping network vulnerability assessment and analysis method based on reinforcement learning. Background Art

[0004] The vulnerability of the shipping network is mainly reflected in the fact that the failure of key nodes (ports) or key paths (routes) in the network has a significant impact on the transportation function of the entire network. Due to the high complexity and interdependence of the shipping network, the failure of a node or path may trigger a series of chain reactions, resulting in a decrease in the transportation efficiency of the entire network, and even in extreme cases, causing the interruption of the global supply chain. This vulnerability will not only affect the timely delivery of goods and economic benefits, but may also have a profound impact on regional economic and social stability.

[0005] Therefore, studying the changes in the vulnerability of the shipping network under different attack strategy scenarios and analyzing its potential influencing factors can provide a scientific basis for the subsequent proposal of effective security countermeasures. This has important theoretical significance and practical value for continuously ensuring the stable operation of the network and promoting the connectivity of facilities and smooth trade among countries.

[0006] Most of the current shipping network vulnerability assessment and analysis methods are based on static analysis, focusing on the impact of removing a limited number of nodes on the initial network topology, while ignoring the dynamic reconstruction and adaptation process of the network after the removal of nodes or edges. Vulnerability assessment based on traditional methods lacks intelligent optimization and cannot automatically adjust strategies to cope with changing attack scenarios. Summary of the invention

[0007] The purpose of the present invention is to provide a shipping network vulnerability assessment and analysis method based on reinforcement learning to solve the problems raised in the above background technology.

[0008] To achieve the above object, the present invention provides the following technical solution: a shipping network vulnerability assessment and analysis method based on reinforcement learning, characterized in that it includes the following steps:

[0009] S1, obtain port route information, ship frequency information and port economic information;

[0010] S2, constructs a shipping container transportation network based on port route information, using the Kosaraju algorithm, and uses depth-first search (DFS) on the transportation network twice according to the closeness of the connection between ports to identify the strongly connected areas of the transportation network;

[0011] S3, build a reinforcement learning model based on the built shipping network G = (V, E). The learning model is used to simulate attacker behavior and evaluate the vulnerability changes of the shipping network under different attack strategies, where V is a node set representing ports; E is an edge set representing the routes between ports; each edge e∈E has a weight W;

[0012] S4, trains the learning model. During the training process, the attack strategy is adaptively adjusted according to the current state of the shipping network. After the node fails, the attacker re-evaluates the connectivity of the remaining transportation network based on the returned results and dynamically adjusts the next attack target.

[0013] Furthermore, in S3, the state s of the reinforcement learning model is represented as the adjacency matrix A of the network; the state s also includes a weight matrix W, that is, the weight w of each edge (i, j) ij , expressed as: w ij =αF ij +βC ij ;w ij is the edge weight from port i to port j; F ij is the frequency of ships from port i to port j; C ij is the total volume of container transport from port i to port j; α and β are weight coefficients;

[0014] An important node evaluation index, namely PageRank value, is introduced in state s. The calculation formula of PageRank value PR(v) is:

[0015] d is the damping factor, which indicates the probability of random jumps; N is the total number of nodes in the network; M(v) is the set of all nodes pointing to node v, that is, the set of all nodes u linked to v; L(u) is the out-degree of node u, that is, the number of other nodes linked to node u. Both PR(v) and PR(u) are assigned an initial value of 1 / N.

[0016] Furthermore, in S3, a reward function r is constructed;

[0017]

[0018] C(G) represents the connectivity of the current network G; APL(G) represents the average path length of the current network; C0 and APL0 represent the network connectivity and average path length in the initial state, respectively; ΔC represents the change in connectivity, defined as C(G)-C0;

[0019] ΔAPL represents the change in average path length, defined as APL(G)-APL0; I critical is an indicator variable. If the attack affects a critical node or edge, I critical=1, otherwise 0; α, β and δ are weight coefficients.

[0020] Furthermore, in S3, the DQN algorithm is used to approximate the Q value function Q(s,a,θ) through a deep neural network, that is, the expected return of performing action a in the state, and the Q value function is updated. The experience replay pool D is used to store the attacker's past experience (s,a,r,s′), and the Q value function is updated through random small batch experience. The Q value objective function is: y=r+γmaxa′Q ′(s′,a′,θ′) ; r is the reward, γ is the discount factor, the parameter θ′ is regularly updated from the parameters of the Q-value function, Q′ is the objective function, and the Q-value function is updated using the mean square error loss function: is the expectation, y is the Q-value objective function, Q(s,a;θ) is the current Q-value function, s is the state, a is the action, and θ is the parameter of the Q-value function.

[0021] Compared with the prior art, the present invention has the following beneficial effects:

[0022] (1) The present invention uses reinforcement learning to dynamically adjust the network topology to adapt to the ever-changing attack patterns and network status. This method can not only reflect the changes of the network under different attack strategies in real time, but also simulate the complex dynamic interactions in the actual environment. This dynamic update capability can effectively address the shortcomings of static analysis in traditional methods and enhance the practical application value of network security assessment.

[0023] (2) The present invention uses reinforcement learning to automatically optimize attack strategies, which can more accurately identify key nodes and paths and improve the accuracy of network vulnerability assessment. Compared with traditional methods, the present invention can intelligently adapt and optimize strategies in complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] Example:

[0027] See also Figure 1 ,The present invention provides a technical solution: a shipping network vulnerability assessment and analysis method based on reinforcement learning, the specific steps are as follows;

[0028] S1. Obtain port route information, ship frequency information and port economic information. In this step, route information includes departure port, destination port, port of call, time and fleet information; ship frequency information includes the number of ships traveling back and forth on the route in a certain year or multiple years; port economic information covers the container throughput of the port. The above data can be obtained from authoritative websites in the preset area.

[0029] S2. In this step, we will build a shipping container transportation network based on port route information and divide it into subgroups. First, each port is abstracted as a network node, the connection path between ports is used as a network edge, and the edge weight is determined by linearly combining the ship frequency and container carrying capacity. Then, a complete shipping container transportation network is constructed based on the network nodes, connecting edges and their weights. Using the Kosaraju algorithm, the transportation network is searched twice using depth-first search (DFS) according to the closeness of the connection between ports.

[0030] S3. In this step, a reinforcement learning model will be designed and implemented based on the constructed shipping network G = (V, E) to simulate attacker behavior and evaluate the vulnerability changes of the network under different attack strategies.

[0031] 1. Network Representation

[0032] First, the shipping network is constructed using a graph, that is, G = (V, E), where V is a set of nodes, representing ports; E is a set of edges, representing routes between ports; each edge e∈E has a weight W, which is determined by a linear combination of ship frequency and container carrying capacity. The weight reflects the strength or importance of the route.

[0033] 2. Status indication

[0034] In this model, the representation of the state is crucial, as it directly affects the attacker's understanding and decision-making ability of the shipping network. In order to enhance the performance of the model, not only the basic structure of the network is considered in the state representation, but also important node evaluation indicators are introduced. These indicators can help attackers more accurately evaluate the key nodes and edges in the network, thereby optimizing the attack strategy.

[0035] First, the state s of the reinforcement learning model is represented as the adjacency matrix A of the network, that is:

[0036]

[0037] In addition, the state also includes the weight matrix W, that is, the weight w of each edge (i, j) ij , expressed as:

[0038] w ij =αF ij +βC ij(1)

[0039] w ij is the edge weight from port i to port j. ij is the frequency of ships from port i to port j. ij is the total volume of container transport from port i to port j. α and β are weight coefficients.

[0040] Finally, in order to enhance the comprehensiveness of the state representation, an important node evaluation index, namely the PageRank value, is introduced into the state. It is an indicator to measure the importance of a node. It not only considers the number of connections of the node, but also the quality of the connected nodes. Nodes with high PageRank usually have greater influence in the network. The calculation formula of the PageRank value PR(v) is:

[0041]

[0042] d is the damping factor, usually set to 0.15, which indicates the probability of random jumps. N is the total number of nodes in the network. M(v) is the set of all nodes pointing to node v, that is, the set of all nodes u linked to v. L(u) is the out-degree of node u, that is, the number of other nodes linked to node u. Both PR(v) and PR(u) are assigned an initial value of 1 / N. Through continuous iteration, until the change in PageRank value is less than the set threshold, the stable PageRank value of all nodes is finally obtained.

[0043] 3. Action definition

[0044] In the reinforcement learning environment, action a is an operation on the network, with selective target attacks, which can be defined as the following two types: the first is to attack nodes, select and remove a node (port), thereby destroying all inbound and outbound routes of the node. The second is to attack edges, select and remove an edge (route), thereby weakening the connectivity of the network.

[0045] 4. Reward Function Design

[0046] Defining the reward function r requires considering the target. Whenever an attacker attacks a node or edge, if the connectivity of the network is significantly reduced, the attacker will receive a positive reward, that is, the degree of network damage is maximized through the attack strategy. The indicators of network collapse are the degree of network connectivity reduction and the average path length of nodes in the network. The specific calculation method is as follows:

[0047]

[0048] C(G) represents the connectivity of the current network G. APL(G) represents the average path length of the current network. C0 and APL0 represent the network connectivity and average path length in the initial state, respectively. ΔC represents the change in connectivity, defined as C(G)-C0. ΔAPL represents the change in average path length, defined as APL(G)-APL0. critical is an indicator variable. If the attack affects a critical node or edge, I critical =1, otherwise 0. α, β, and δ are weight coefficients that determine the impact of connectivity changes and average path length changes on the overall reward. Usually, these two coefficients are set to negative values, meaning that the model should avoid connectivity decreases or path length increases. δ is used to measure the importance of critical node or edge failure. It is set to a positive value, indicating that the model should receive a reward if the attack successfully destroys a critical node or edge.

[0049] 5. Deep Q Network (DQN) Algorithm Selection

[0050] The DQN algorithm approximates the Q-value function Q(s,a,θ) through a deep neural network, which is the expected return of performing action a in the state.

[0051] The Q-value function is updated by using the experience replay pool D to store the attacker's past experience (s, a, r, s′), and the Q-value function is updated through random mini-batch experience. The Q-value objective function is:

[0052] y=r+γmaxa′Q ′(s′,a′,θ′) (4)

[0053] Where r is the reward, γ is the discount factor, the parameter θ′ is updated periodically from the parameter of the Q value function, and Q′ is the objective function. The Q value function is updated using the mean square error loss function:

[0054]

[0055] in, is the expectation, y is the Q-value objective function, Q(s,a;θ) is the current Q-value function, s is the state, a is the action, and θ is the parameter of the Q-value function.

[0056] S4, model training. In this step, training is carried out based on the model built in the above steps. During the training process, the attack strategy is adaptively adjusted according to the current state of the network. For example, after a node fails, the algorithm will re-evaluate the connectivity of the remaining transportation network and dynamically adjust the next attack target. This adaptive strategy enables the algorithm to maintain efficient attack capabilities when facing a changing network structure.

[0057] Initialize the experience replay pool D and Q-value function parameters θ.

[0058] In the current state s, select action a according to the ε-greedy strategy. Execute action a and get reward r and the next state s′.

[0059] Store the experience (s, a, r, s′) into the experience replay pool D.

[0060] Randomly extract a small batch of experience from the experience replay pool D, calculate the loss function and update the Q value function parameter θ. Periodically update the target Q value function Q′.

[0061] S5. Vulnerability assessment and analysis. According to the final training results, the model showed high accuracy in simulating attacker behavior and was able to effectively assess the potential impact of the removal of each node and path on the overall network connectivity. Through dynamic optimization strategies, the model identified and marked some key nodes and paths. Once these key areas are attacked, the connectivity and transportation efficiency of the network may be significantly weakened. Preliminary evaluation results show that: First, the model identified and marked about 20% of the nodes and paths as key areas, which are critical to the overall connectivity of the network. Second, once these key nodes or paths are removed, the overall connectivity of the network drops significantly by 78%, and has a significant impact on transportation efficiency. This result provides important data support for the vulnerability assessment of the shipping network, and helps to formulate more effective protection strategies to improve the overall security of the network.

[0062] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A shipping network vulnerability assessment and analysis method based on reinforcement learning, characterized in that: The following steps are involved: S1, obtain port route information, ship frequency information and port economic information; S2, constructs a shipping container transportation network based on port route information, using the Kosaraju algorithm, and uses a depth-first search twice on the transportation network according to the closeness of the connection between ports to identify the strongly connected areas of the transportation network; S3, build a reinforcement learning model based on the built shipping network G = (V, E). The learning model is used to simulate attacker behavior and evaluate the vulnerability changes of the shipping network under different attack strategies, where V is a node set representing ports; E is a set of edges, representing routes between ports; each edge e∈E has a weight W; S4, trains the learning model. During the training process, the attack strategy is adaptively adjusted according to the current state of the shipping network. After the node fails, the attacker re-evaluates the connectivity of the remaining transportation network based on the returned results and dynamically adjusts the next attack target.

2. The shipping network vulnerability assessment and analysis method based on reinforcement learning according to claim 1 is characterized by: In S1, route information includes departure port, destination port, port of call, time and fleet information; ship frequency information includes the number of ships traveling back and forth on the route in a certain year or multiple years; port economic information covers the container throughput of the port.

3. The shipping network vulnerability assessment and analysis method based on reinforcement learning according to claim 1 is characterized by: In S2, each port is abstracted as a network node, and the connection paths between ports are used as network edges. The weight of the edge is determined by linearly combining the ship frequency and container carrying capacity. Then, a complete shipping container transportation network is constructed based on the network nodes, connecting edges and their weights.

4. The shipping network vulnerability assessment and analysis method based on reinforcement learning according to claim 1 is characterized by: In S3, the state s of the reinforcement learning model is represented as the adjacency matrix A of the network; the state s also includes the weight matrix W, that is, the weight w of each edge (i, j) ij , expressed as: w ij =αF ij +βC ij ;w ij is the edge weight from port i to port j; F ij is the frequency of ships from port i to port j; C ij is the total volume of container transport from port i to port j; α and β are weight coefficients; Introduce an important node evaluation indicator in state s: PageRank value.

5. The method for evaluating and analyzing shipping network vulnerability based on reinforcement learning according to claim 4 is characterized in that: The calculation formula of PageRank value PR(v) is: d is the damping factor, which represents the probability of random jump; N is the total number of nodes in the network; M(v) is the set of all nodes pointing to node v, that is, the set of all nodes u linked to v; L(u) is the out-degree of node u, that is, the number of other nodes linked to node u; PR(v) and PR(u) are both assigned an initial value of 1 / N.

6. The shipping network vulnerability assessment and analysis method based on reinforcement learning according to claim 1 is characterized by: In S3, construct the reward function r; C(G) represents the connectivity of the current network G; APL(G) represents the average path length of the current network; C0 and APL0 represent the network connectivity and average path length in the initial state, respectively; ΔC represents the change in connectivity, defined as C(G)-C0; ΔAPL represents the change in average path length, defined as APL(G)-APL0; I critical is an indicator variable. If the attack affects a critical node or edge, then I critical =1, otherwise 0; α, β and δ are weight coefficients.

7. The shipping network vulnerability assessment and analysis method based on reinforcement learning according to claim 1 is characterized by: In S3, the DQN algorithm is used to approximate the Q-value function Q(s,a,θ) through a deep neural network, that is, the expected return of performing action a in the state, the Q-value function is updated, and the experience replay pool D is used to store the attacker's past experience (s,a,r,s′), and the Q-value function is updated through random small batch experience; the Q-value objective function is: y=r+γmaxa′Q ′(s′,a′,θ′) ; r is the reward, γ is the discount factor, the parameter θ′ is regularly updated from the parameters of the Q-value function, Q′ is the objective function, and the Q-value function is updated using the mean square error loss function: is the expectation, y is the Q-value objective function, Q(s,a;θ) is the current Q-value function, s is the state, a is the action, and θ is the parameter of the Q-value function.

Images