A shipping network vulnerability evaluation analysis method based on reinforcement learning

Abstract

The application discloses a kind of based on reinforcement learning's shipping network vulnerability assessment analysis method, comprising the following steps: S1, obtain port route information, ship frequency information and port economic information, S2, based on port route information constructs shipping network, S3, based on the shipping network G=(V,E) that has been constructed constructs reinforcement learning model, S4, carries out training to reinforcement learning model;Dynamic adjustment network topology using reinforcement learning to adapt to changing attack patterns and network state.This method not only can reflect the change of network under different attack strategies in real time, but also can simulate complex dynamic interaction in actual environment.This dynamic updating capability can effectively deal with the shortcomings of static analysis in traditional methods;Using reinforcement learning to automatically optimize attack strategies can more accurately identify critical nodes and paths, improve the accuracy of network vulnerability assessment.

Description

Technical Field

[0001] This invention relates to the field of shipping technology, specifically to a method for assessing and analyzing the vulnerability of shipping networks based on reinforcement learning. Background Technology

[0002] With the accelerating pace of global economic integration, shipping networks play a vital role in the global supply chain, handling nearly 90% of international trade freight volume and becoming a crucial engine for global economic development. Deepening globalization has transformed shipping networks from mere links in cargo transportation into vital bridges connecting the economies of nations worldwide. However, with the continuous increase in global ports and shipping routes, coupled with the complex and volatile global political and economic environment, the stability of shipping networks faces unprecedented challenges.

[0003] The vulnerability of shipping networks is primarily manifested in the significant impact of the failure of critical nodes (ports) or critical routes (shipping routes) on the overall transportation function of the network. Due to the high complexity and interdependence of shipping networks, the failure of a single node or route can trigger a series of chain reactions, leading to a decline in the transportation efficiency of the entire network, and even, in extreme cases, disruption of the global supply chain. This vulnerability not only affects the timely delivery of goods and economic efficiency but can also have profound implications for regional economic and social stability.

[0004] Therefore, studying the vulnerability changes of shipping networks under different attack strategies and analyzing their potential influencing factors provides a scientific basis for proposing effective security countermeasures. This is of great theoretical and practical value for ensuring the stable operation of the network and promoting infrastructure connectivity and trade between countries.

[0005] Current methods for assessing and analyzing vulnerability in shipping networks are mostly based on static analysis, focusing primarily on the impact of removing a limited number of nodes on the initial network topology, while neglecting the dynamic reconstruction and adaptation process of the network after node or edge removal. Vulnerability assessment based on traditional methods lacks intelligent optimization and cannot automatically adjust strategies to cope with constantly changing attack scenarios. Summary of the Invention

[0006] The purpose of this invention is to provide a method for assessing and analyzing the vulnerability of shipping networks based on reinforcement learning, so as to solve the problems mentioned in the background art.

[0007] To achieve the above objectives, the present invention provides the following technical solution: a method for assessing and analyzing the vulnerability of shipping networks based on reinforcement learning, characterized by comprising the following steps:

[0008] S1, obtain port route information, vessel frequency information, and port economic information;

[0009] S2. Based on port route information, a shipping network is constructed. Using the Kosaraju algorithm, the transport network is subjected to two depth-first search (DFS) operations according to the degree of connection between ports, thereby identifying the strongly connected regions of the transport network.

[0010] S3. Based on the existing shipping network G=(V,E), a reinforcement learning model is constructed. The reinforcement learning model is used to simulate attacker behavior and evaluate the vulnerability changes of the shipping network under different attack strategies. Here, V is the set of nodes, representing ports; E is the set of edges, representing shipping routes between ports; each edge e∈E has a weight W.

[0011] S4 trains the reinforcement learning model. During the training process, the attack strategy is adaptively adjusted according to the current state of the shipping network. After a node fails, the attacker re-evaluates the connectivity of the remaining transportation network based on the returned results and dynamically adjusts the next attack target.

[0012] Furthermore, in S3, the state s of the reinforcement learning model is represented by the adjacency matrix A of the network; the state s also includes the weight matrix W, that is, the weight w of each edge (i,j). ij , represented as:

[0013] w ij =α1F ij +β1C ij ;

[0014] w ij F is the edge weight from port i to port j; ij C is the frequency of ships traveling from port i to port j; ij α1 represents the total container traffic from port i to port j; β1 and α2 are weighting coefficients.

[0015] In state s, an important node evaluation metric, namely the PageRank value, is introduced. The formula for calculating the PageRank value PR(v) is as follows:

[0016] PR(v) is the PageRank value of node v, d is the damping factor representing the probability of a random jump; N is the total number of nodes in the network; M(v) is the set of all nodes pointing to node v, i.e., the set of all nodes u linked to v; L(u) is the out-degree of node u, i.e., the number of other nodes linked to by node u. Both PR(v) and PR(u) are initialized to 1 / N.

[0017] Furthermore, in S3, a reward function r is constructed;

[0018] C(G) represents the connectivity of the current network G; APL(G) represents the average path length of the current network; C0 and APL0 represent the network connectivity and average path length in the initial state, respectively; ΔC represents the change in connectivity, defined as C(G)-C0;

[0019] ΔAPL represents the change in average path length, defined as APL(G) - APL0; critical For indicator variables, I is used if the attack affects critical nodes or edges. critical =1, otherwise 0; α, β and δ are weighting coefficients.

[0020] Furthermore, in S3, the DQN algorithm is used to approximate the Q-value function Q(s,a; θ) through a deep neural network, which represents the expected reward of performing action a in a given state. The Q-value function is updated by storing the attacker's past experiences (s,a,r,s′) in an experience replay pool D, and updating the Q-value function through random mini-batch experiences. The objective function for the Q-value is: y = r + γmax a′ Q′(s′,a′;θ′); r is the reward, γ is the discount factor, θ′ is the parameter updated periodically from the Q-value function, Q′ is the objective function, and the Q-value function is updated using the mean squared error loss function: L(θ) is the loss function. Let y be the desired state, y be the objective function of Q, Q(s,a;θ) be the current Q function, s be the state, a be the action, and θ be the parameters of the Q function.

[0021] Compared with the prior art, the beneficial effects of the present invention are:

[0022] (1) This invention utilizes reinforcement learning to dynamically adjust the network topology to adapt to constantly changing attack patterns and network states. This method can not only reflect network changes under different attack strategies in real time, but also simulate complex dynamic interactions in real-world environments. This dynamic update capability effectively addresses the shortcomings of static analysis in traditional methods, enhancing the practical application value of network security assessment.

[0023] (2) This invention utilizes reinforcement learning to automatically optimize attack strategies, enabling more accurate identification of key nodes and paths, and improving the accuracy of network vulnerability assessment. Compared to traditional methods, this invention can intelligently adapt to and optimize strategies in complex environments. Attached Figure Description

[0024] Figure 1 This is a flowchart of the present invention. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] Example:

[0027] Please see Figure 1 The present invention provides a technical solution: a method for assessing and analyzing the vulnerability of shipping networks based on reinforcement learning, the specific steps of which are as follows;

[0028] S1. Obtain port route information, vessel frequency information, and port economic information. In this step, route information includes departure port, destination port, port of call, time, and fleet information; vessel frequency information includes the number of vessels making round trips on the route within a given year or multiple years; and port economic information covers the port's container throughput. This data can be obtained from authoritative websites within the preset region.

[0029] S2. In this step, we will construct a shipping container transport network (hereinafter referred to as the shipping network) based on port route information and divide it into subgroups. First, we abstract each port as a network node, and the connection paths between ports as network edges. We determine the edge weights by linearly combining ship frequency and container carrying capacity. Next, based on the network nodes, connecting edges, and their weights, we construct the complete shipping network. Using the Kosaraju algorithm, we perform a depth-first search (DFS) twice on the transport network according to the closeness of the connections between ports.

[0030] S3. In this step, a reinforcement learning model will be designed and implemented based on the existing shipping network G=(V,E) to simulate attacker behavior and evaluate the network’s vulnerability changes under different attack strategies.

[0031] 1. Network representation

[0032] First, a shipping network is constructed using a graph representation, namely G = (V, E), where V is the set of nodes representing ports; E is the set of edges representing shipping routes between ports; each edge e ∈ E has a weight W, which is determined by a linear combination of ship frequency and container volume, and the weight reflects the strength or importance of the shipping route.

[0033] 2. State representation

[0034] In this model, the representation of the state is crucial, directly impacting an attacker's understanding and decision-making capabilities regarding the shipping network. To enhance model performance, the state representation not only considers the basic network structure but also introduces evaluation metrics for key nodes. These metrics help attackers more accurately assess critical nodes and edges in the network, thereby optimizing their attack strategies.

[0035] First, the state s of the reinforcement learning model is represented by the adjacency matrix A of the network, that is:

[0036]

[0037] In addition, the state also includes a weight matrix W, that is, the weight w of each edge (i,j). ij , represented as:

[0038] w ij =α1F ij +β1C ij (1)

[0039] w ij F is the edge weight from port i to port j. ij C represents the frequency of ships traveling from port i to port j. ij This represents the total container traffic from port i to port j. α1 and β1 are the weighting coefficients.

[0040] Finally, to enhance the comprehensiveness of the state representation, a PageRank value, an important node evaluation metric, is introduced into the state. This metric measures the importance of a node, considering not only the number of connections but also the quality of those connections. Nodes with high PageRank typically have significant influence in the network. The formula for calculating the PageRank value PR(v) is:

[0041]

[0042] PR(v) is the PageRank value of node v, d is the damping factor, usually set to 0.15, representing the probability of a random hop. N is the total number of nodes in the network. M(v) is the set of all nodes pointing to node v, i.e., the set of all nodes u linked to v. L(u) is the out-degree of node u, i.e., the number of other nodes linked to by node u. Both PR(v) and PR(u) are initially assigned the value 1 / N. Through continuous iteration, until the change in the PageRank value is less than the set threshold, the stable PageRank value of all nodes is finally obtained.

[0043] 3. Action Definition

[0044] In a reinforcement learning environment, action 'a' is an operation on the network, a selective target attack, which can be defined in two ways: First, attacking a node, selecting and removing a node (port), thereby disrupting all inbound and outbound routes for that node. Second, attacking an edge, selecting and removing an edge (route), thereby weakening the network's connectivity.

[0045] 4. Reward Function Design

[0046] Defining the reward function *r* requires considering the objective: whenever an attacker attacks a node or edge, if it significantly reduces network connectivity, the attacker receives a positive reward. This means maximizing network damage through the attack strategy. The indicators for network collapse are the degree of network connectivity degradation and the average path length of nodes in the network. The specific calculation method is as follows:

[0047]

[0048] C(G) represents the current connectivity of network G. APL(G) represents the current average path length of the network. C0 and APL0 represent the network connectivity and average path length in the initial state, respectively. ΔC represents the change in connectivity, defined as C(G) - C0. ΔAPL represents the change in average path length, defined as APL(G) - APL0. critical For indicator variables, I is used if the attack affects critical nodes or edges. critical =1 otherwise 0. α, β, and δ are weighting coefficients that determine the impact of changes in connectivity and average path length on the overall reward. Typically, these two coefficients are set to negative values, meaning the model should avoid decreasing connectivity or increasing path length. δ measures the importance of critical node or edge failure. A positive value indicates that the model should receive a reward if an attack successfully destroys a critical node or edge.

[0049] 5. Selection of Deep Q-Network (DQN) Algorithm

[0050] The DQN algorithm approximates the Q-value function Q(s,a;θ) through a deep neural network, which is the expected reward of performing action a in a given state.

[0051] The Q-value function is updated by using an experience replay pool D to store the attacker's past experiences (s, a, r, s′), and updating the Q-value function through random mini-batch experiences. The objective function for the Q-value is:

[0052] y = r + γmax a′ Q′(s′,a′;θ′) (4)

[0053] Where r is the reward, γ is the discount factor, θ′ is the parameter updated periodically from the Q-value function, Q′ is the objective function, s′ is the new environmental state after the action is executed, and a′ is the available action in the new state. The Q-value function is updated using the mean squared error loss function:

[0054]

[0055] Where L(θ) is the loss function, Let y be the expectation, y be the Q-value objective function, Q(s,a;θ) be the current Q-value function, s be the state, a be the action, and θ be the parameters of the Q-value function.

[0056] S4. Model Training. In this step, the model built in the previous steps is trained. During training, the attack strategy is adaptively adjusted based on the current state of the network. For example, after a node fails, the algorithm reassesses the connectivity of the remaining transportation network and dynamically adjusts the next attack target. This adaptive strategy enables the algorithm to maintain efficient attack capabilities when facing changing network structures.

[0057] Initialize the experience replay pool D and the Q-value function parameter θ.

[0058] In the current state s, choose action a according to the ε-greedy policy. Execute action a to obtain reward r and the next state s′.

[0059] Store the experience (s,a,r,s′) in the experience replay pool D.

[0060] Randomly sample mini-batch experiences from the experience replay pool D, calculate the loss function, and update the Q-value function parameters θ. Periodically update the target Q-value function Q′.

[0061] S5. Vulnerability Assessment and Analysis. Based on the final training results, the model demonstrated high accuracy in simulating attacker behavior, effectively assessing the potential impact of removing each node and path on overall network connectivity. Through a dynamic optimization strategy, the model identified and labeled some critical nodes and paths. Attacks on these critical areas could significantly weaken network connectivity and transportation efficiency. Preliminary assessment results show that: First, the model identified and labeled approximately 20% of nodes and paths as critical areas, which are crucial to the overall network connectivity. Second, once these critical nodes or paths were removed, the overall network connectivity decreased significantly, by up to 78%, and had a significant impact on transportation efficiency. This result provides important data support for vulnerability assessment of shipping networks, helping to develop more effective protection strategies to improve the overall security of the network.

[0062] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for assessing and analyzing the vulnerability of shipping networks based on reinforcement learning, characterized in that, Includes the following steps: S1, obtain port route information, vessel frequency information, and port economic information; S2, a shipping network is constructed based on port route information. The Kosaraju algorithm is used to perform a depth-first search twice on the transportation network according to the closeness of the connection between ports, thereby identifying the strongly connected regions of the transportation network. S3. Based on the existing shipping network G=(V,E), a reinforcement learning model is constructed. The reinforcement learning model is used to simulate attacker behavior and evaluate the vulnerability changes of the shipping network under different attack strategies, where V is a set of nodes, representing ports. E is a set of edges representing shipping routes between ports; each edge e∈E has a weight W. The state s of the reinforcement learning model is represented by the adjacency matrix A of the shipping network; the state s also includes the weight matrix W, i.e., the weight w of each edge (i,j). ij , represented as: w ij =α1F ij +β1C ij w ij F is the edge weight from port i to port j; ij C is the frequency of ships traveling from port i to port j; ij α1 and β1 are the total container transport volume from port i to port j; α1 and β1 are weighting coefficients, and an important node evaluation index, PageRank value, is introduced in state s. Construct the reward function r; C(G) represents the connectivity of the current network G; APL(G) represents the average path length of the current network; C0 and APL0 represent the network connectivity and average path length in the initial state s, respectively; ΔC represents the change in connectivity, defined as C(G) - C0; ΔAPL represents the change in average path length, defined as APL(G) - APL0; critical For indicator variables, I is used if the attack affects critical nodes or edges. critical =1, otherwise 0; α, β and δ are weighting coefficients; The DQN algorithm is used to approximate the Q-value function Q(s,a; θ) through a deep neural network, which represents the expected reward of performing action a in state s. The Q-value function is updated by storing the attacker's past experiences (s,a,r,s′) in an experience replay pool D, and updating the Q-value function through random mini-batch experiences. The objective function for the Q-value is: y = r + γmax. a′ Q′(s′,a′;θ′); r is the reward, γ is the discount factor, θ′ is the parameter updated periodically from the Q-value function, Q′ is the objective function, s′ is the new environmental state after the action is performed, and a′ is the available actions in the new state; the Q-value function is updated using the mean squared error loss function. L(θ) is the loss function. Let y be the desired value, Q(s,a;θ) be the current Q-value function, s be the state, a be the action, and θ be the parameters of the Q-value function. S4 trains the reinforcement learning model. During the training process, the attack strategy is adaptively adjusted according to the current state of the shipping network. After a node fails, the attacker re-evaluates the connectivity of the remaining transportation network based on the returned results and dynamically adjusts the next attack target.

2. The method for assessing and analyzing the vulnerability of shipping networks based on reinforcement learning according to claim 1, characterized in that: In S1, route information includes departure port, destination port, port of call, time and fleet information; vessel frequency information includes the number of vessels traveling on the route within a certain year or multiple years; port economic information covers the port's container throughput.

3. The method for assessing and analyzing the vulnerability of shipping networks based on reinforcement learning according to claim 1, characterized in that: In S2, each port is abstracted as a network node, the connection path between ports is taken as a network edge, and the weight of the edge is determined by linearly combining the ship frequency and container carrying capacity. Then, based on the network nodes, the connecting edges and their weights, a complete shipping network is constructed.

4. The method for assessing and analyzing the vulnerability of shipping networks based on reinforcement learning according to claim 1, characterized in that: The formula for calculating the PageRank value PR(v) is: PR(v) is the PageRank value of node v, d is the damping factor, representing the probability of a random jump; N is the total number of nodes in the network; M(v) is the set of all nodes pointing to node v, that is, the set of all nodes u linked to v; L(u) is the out-degree of node u, that is, the number of other nodes linked to by node u; PR(v) and PR(u) are both assigned an initial value of 1 / N.

Images