A Multi-modal Transportation Route Optimization Method Based on Deep Reinforcement Learning

Abstract

The present invention discloses a multimodal transport path optimization method based on deep reinforcement learning, belonging to the field of logistics management, which includes the following steps: S1. Form multiple feasible paths from the starting point to the ending point as training samples, and calculate the transportation time, freight cost, carbon emissions, accident incidence rate, and maximum load index of each sub-path as the quality evaluation indicators corresponding to the feasible paths; S2. Construct a deep reinforcement learning agent based on the state space, action space, and reward function; S3. The agent selects samples using the Sum Tree priority sampling mechanism to generate the optimal path; S4. During the multimodal transport process, monitor the quality evaluation indicators of the path, and dynamically generate the optimal path according to the real-time path state data. The present invention can adapt to dynamic factors such as real-time traffic changes, reduce manual intervention, has a high response efficiency to emergencies, can reduce the occurrence probability of accidents, and reduce carbon emissions and the emissions of other pollutants.

Description

Technical Field

[0001] The present invention belongs to the field of logistics management, and specifically relates to a multi-modal transport path optimization method based on deep reinforcement learning. Background Art

[0002] In a globalized economic system, effective logistics management is crucial for enterprise competitiveness.

[0003] With the increasing complexity of the global supply chain, multi-modal transport has become the core mode of modern logistics due to its flexibility and cost advantages. However, the existing technologies have the following problems: limitations of static models: traditional optimization methods (such as linear programming and heuristic algorithms) rely on fixed parameters and are difficult to adapt to dynamic factors such as real-time traffic changes; insufficient multi-objective trade-off: existing solutions mostly focus on a single objective (such as cost or time) and lack coordinated optimization of comprehensive indicators such as carbon emissions and transportation risks; low response efficiency: path adjustment in case of emergencies relies on manual intervention and cannot achieve automated dynamic replanning; insufficient data utilization: the integrated application of historical transport data and real-time information is insufficient, resulting in lagged decision-making. Summary of the Invention

[0004] The purpose of the present invention is to provide a multi-modal transport path optimization method based on deep reinforcement learning to solve the problems existing in the existing multi-modal transport methods, such as difficulty in adapting to dynamic factors such as real-time traffic changes, insufficient multi-objective trade-off, and low response efficiency to emergencies.

[0005] To achieve the above purpose, the technical solution of the present invention is as follows:

[0006] The present invention relates to a multi-modal transport path optimization method based on deep reinforcement learning, which includes the following steps:

[0007] S1. Build a multi-modal transport path network topology model, abstract the path network into a graph structure, and form multiple feasible paths from the starting point to the ending point as training samples, and calculate the transport time, freight cost, carbon emissions, accident incidence rate, and maximum load index of each sub-path as the quality evaluation index of the corresponding feasible path;

[0008] S2. Build a deep reinforcement learning agent based on the state space, action space, and reward function;

[0009] S3. The agent uses the Sum Tree priority sampling mechanism combined with the uniform sampling + reweighting sampling mechanism to select samples from the training samples and generate the optimal path;

[0010] S4. During the multi-modal transport process, monitor the quality evaluation index of the path. If the quality evaluation index exceeds the threshold, dynamically generate the optimal path according to the real-time path status data.

[0011] Preferably, the index of the transportation time of each path in S1 is obtained by calculating with the following formula:

[0012] ,

[0013] where, d represents the transportation time; Psd represents the feasible path from the starting point s to the end point d ; eij represents the sub-path between adjacent path nodes i and j in the feasible path;

[0014] The index of the freight cost of each path is obtained by calculating with the following formula:

[0015] ,

[0016] where, b represents the freight cost;

[0017] The index of the carbon emission of each path is obtained by calculating with the following formula:

[0018] ,

[0019] where, l represents the carbon emission index;

[0020] The index of the accident rate of each path is obtained by calculating with the following formula:

[0021] ,

[0022] where, m represents the accident rate;

[0023] The index of the maximum load of each path is obtained by calculating with the following formula:

[0024] ,

[0025] where, n represents the maximum load.

[0026] Preferably, the specific steps for constructing a deep reinforcement learning agent for S2 based on the state space, action space, and reward function are as follows:

[0027] S2.1. Establish the state space of the agent. The state space includes two items of information, namely service request information and feasible path status matrix, which are represented by a vector. The vector representation is:

[0028] ,

[0029] Among them, is a vector representing the state at the current moment t . D is the business request information, including the starting point and the ending point. TM is the feasible path status matrix corresponding to the business request information. The feasible path status matrix includes freight cost indicators, transportation time indicators, carbon emission indicators, accident incidence indicators, and maximum load indicators;

[0030] The described feasible path status matrix is expressed as:

[0031] ,

[0032] where k is the number of feasible paths;

[0033] S2.2. Establish the action space of the agent. The action space is used to represent the set of feasible paths, and it makes corresponding path decisions through ε - greed The policy. The decision-making method is as follows:

[0034] ,

[0035] where is the path optimization strategy made under the current path network state , a represents the action, x is a random number within the range of [0, 1], ε is a variable that controls the random exploration probability with an initial value of 1, θ is the path network parameter of the common part; β and α are the unique parameters of the value function and the advantage function respectively, α is the learning rate;

[0036] S2.3. Establish the reward function of the agent. The reward function is expressed as:

[0037] ,

[0038] where R is the reward function, , , , and represent the weights corresponding to the freight cost indicator, transportation time indicator, carbon emission indicator, accident incidence indicator, and maximum load indicator respectively. The sum of the weights is 1;

[0039] S2.4. Define the method for the agent to select the optimal path. The specific method is: using the network state , the path optimization strategy and the reward Iteratively update the action value function, aiming to maximize the expected reward value, and select the optimal path policy for each path set state under , the expression of the action value function is:

[0040] ,

[0041] where Q represents the expected reward value, is the immediate reward value for selecting action in state , represents the maximum reward value for state to select each action, and γ is the discount factor, reflecting the importance of future rewards.

[0042] Preferably, the elements in the feasible path state matrix are normalized by the Min-Max method, and the normalization formula is:

[0043] ,

[0044] where, x i is an element in the feasible path state matrix, corresponding to a certain transportation time index, freight cost index, carbon emission index, accident incidence index, and maximum load index in the matrix, is the minimum value in the corresponding item index, is the maximum value in the corresponding item index, is the index after normalization processing.

[0045] Preferably, the variable controlling the random exploration probability gradually decreases with the increase of the iteration number, and the update formula is:

[0046] ,

[0047] where, is the initial value of ε at the beginning of the iteration; represents the minimum value of ε; is the decay factor.

[0048] Preferably, the S2 constructs a deep reinforcement learning agent based on the Dueling DQN algorithm, and the calculation of the expected reward value Q is divided into a value function and an advantage function, and its expression is:

[0049] ,

[0050] where s represents the state, a represents the action, θ is the path network parameter of the common part; β andα They are the unique parameters of the value function and the advantage function respectively. V is the value function, representing the value inherent in state s. A is the advantage function, representing the outstanding value of the action selected in state s a is an index variable that traverses all possible actions.

[0051] Preferably, the specific way for the agent to select samples from the training samples by using the Sum Tree priority sampling mechanism and the uniform sampling + reweighting mechanism is as follows:

[0052] S3.1. The agent uses the Sum Tree priority sampling mechanism to select samples: Calculate the priority of each sample, and select a set number of samples in descending order of priority. The calculation formula for the priority is:

[0053]

[0054] where i is the sample number, and each sample corresponds to a feasible path. p 1 is the priority. is the temporal difference error of the Sum Tree priority sampling mechanism;

[0055] The calculation formula for the said temporal difference error is:

[0056]

[0057] where is the reward value generated by the target neural network;

[0058] S3.2. The agent uses the uniform sampling + reweighting mechanism to select samples from the training samples: All feasible path samples are stored in a circular queue. Each time during training, a set number of samples are randomly selected uniformly. The probability of each sample being selected is:

[0059] [[ID=The probability of each sample being selected is:

[0060] where N is the total capacity of the sample pool. i is the sample number. p 2 is the probability of the sample being selected;

[0061] Then, for each selected sample, calculate its temporal difference error. The calculation formula is:

[0062]

[0063] where is the temporal difference error of the uniform sampling + reweighting mechanism. is the path reward estimate output by the target neural network.​​​​I is the current network estimated value, is the discount factor, is the sample i corresponding state, is the index variable traversing all sample states, is an index variable traversing all possible actions;

[0064] Then, re - weight the loss function. During the back - propagation process, dynamically adjust the loss weight according to the sample temporal difference error. The loss function is defined as:

[0065] ,

[0066] where, φ is the loss weight, M is the number of batch samples, is the smoothing constant, used to prevent zero - error samples from having no weight, is the priority strength coefficient, w represents the weight, j represents an index label traversing all values;

[0067] S3.3. Combine the samples obtained in S3.1 and S3.2 to form a sample set.

[0068] Preferably, in S4, by calculating the real - time reward value and comparing it with the set reward value threshold, when the real - time reward value is less than the reward value threshold, it is determined that the quality evaluation index exceeds the threshold.

[0069] Preferably, the expression for updating the optimal path in S4 is:

[0070] ,

[0071] where, represents function, p is the optimal path before update, P is the set of feasible paths, is the optimal path after update.

[0072] Preferably, the feasible paths in S1 include the following constraints:

[0073] Time - window constraint: The total path time does not exceed the preset threshold ;

[0074] Load constraint: The cargo weight ;

[0075] Risk constraint: The accident rate ,

[0076] Among them, h is the cargo load threshold, indicating the accident incidence threshold.

[0077] Adopting the technical solution provided by the present invention, compared with the prior art, it has the following beneficial effects:

[0078] 1. The multimodal transport path optimization method based on deep reinforcement learning involved in the present invention constructs a deep reinforcement learning agent based on the state space, action space and reward function, trains the feasible path samples with the deep reinforcement learning agent to find the optimal path, and monitors the quality evaluation index of this path during the intermodal transport process. If the quality evaluation index exceeds the threshold, the optimal path is dynamically updated according to the real-time path state data, which can adapt to dynamic factors such as real-time traffic changes, reduce manual intervention, and has a high response efficiency to emergencies.

[0079] 2. The multimodal transport path optimization method based on deep reinforcement learning involved in the present invention takes the transport time, freight cost, carbon emissions, accident incidence rate and maximum load index as the quality evaluation indexes of the corresponding feasible paths, incorporates the transport risk and carbon emissions into the consideration of the path optimization scheme, can identify potential risks in advance so as to take preventive measures in time, reduce the occurrence probability and impact of accidents, reduce the carbon emissions and the emissions of other pollutants, promote green logistics and sustainable development, and achieve the purpose of reducing costs and improving service quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 is a flowchart of the multimodal transport path optimization method based on artificial intelligence involved in the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0081] To further understand the content of the present invention, the present invention will be described in detail in combination with the embodiments. The following embodiments are used to illustrate the present invention, but are not used to limit the scope of the present invention.

[0082] Referring to the appended Figure 1 As shown, the present invention relates to a multimodal transport path optimization method based on deep reinforcement learning, which includes the following steps:

[0083] S1. Construct a multimodal transport path network topology model, abstract the path network into a graph structure G=(V, E), where V={v1, v2,..., vm} represents the set of all routing nodes, and |V|=m represents the number of path nodes; E=(e12, e23,..., eij) represents the set of sub-paths between all adjacent nodes. Based on the starting point in the path network, multiple feasible paths from the starting point to the end point are formed as training samples. Among them, each feasible path should follow the following constraints:

[0084] Time window constraint: The total path time does not exceed the preset threshold ;

[0085] Load constraint: Cargo weight ;

[0086] Risk constraint: Accident rate ,

[0087] where h is the cargo load threshold, represents the accident incidence threshold.

[0088] After obtaining the feasible paths, calculate the transportation time, freight cost, carbon emissions, accident incidence, and maximum load index of each sub-path as the quality evaluation index (QE index) of the corresponding feasible path;

[0089] The index of the transportation time of each path is calculated by the following formula:

[0090] ,

[0091] where, d represents the transportation time, Psd represents the feasible path from the starting point s to the end point d , eij represents the sub-path between adjacent path nodes i and j in the feasible path;

[0092] The index of the freight cost of each path is calculated by the following formula:

[0093] ,

[0094] where, b represents the freight cost;

[0095] The index of the carbon emissions of each path is calculated by the following formula:

[0096] ,

[0097] where, l represents the carbon emissions index;

[0098] The index of the accident incidence of each path is calculated by the following formula:

[0099] ,

[0100] where, m represents the accident incidence;

[0101] The index of the maximum load of each path is calculated by the following formula:

[0102] ,

[0103] Among them, n represents the maximum load.

[0104] S2. Construct a deep reinforcement learning agent based on the state space, action space, and reward function. The specific steps are as follows:

[0105] S2.1. Establish the state space of the agent. The state space is a set of state information obtained by the agent from the path network. Each state contains two aspects of information, namely service request information (starting point - destination) and its corresponding feasible path state matrix.

[0106] The state space includes two items of information, service request information and feasible path state matrix, which are represented by the vector s t as follows:

[0107] ,

[0108] Among them, is a vector representing the state at the current moment t , D is service request information, including the starting point and the destination, TM is the feasible path state matrix corresponding to the service request information. The feasible path state matrix includes freight cost index, transportation time index, carbon emission index, accident incidence rate index, and maximum load index;

[0109] The said feasible path state matrix is represented as:

[0110] ,

[0111] where k is the number of feasible paths.

[0112] Since the element values of the state matrix vary greatly and cannot objectively reflect the influence of each network path state information, the intelligent path selection algorithm fluctuates too much during the training process and is difficult to converge. Therefore, the elements in the feasible path state matrix are normalized to the range of [0, 1] by the Min - Max method. The normalization formula is:

[0113] ,

[0114] where x i is an element in the feasible path state matrix, corresponding to a certain item of transportation time index, freight cost index, carbon emission index, accident incidence rate index, and maximum load index in the matrix, is the minimum value of the corresponding item index, is the maximum value of the corresponding item index, It is the index after normalization processing.

[0115] S2.2. Establish the action space of the agent. The action space is used to represent the set of feasible paths P = { P 1, P 2 , …, P k}, and it makes corresponding path decisions through ε - greed policy. The decision-making method is:

[0116] ,

[0117] where, is the path optimization strategy made under the current path network state , a represents the action, x is a random number within the range of [0, 1], ε is a variable that controls the random exploration probability with an initial value of 1, θ is the path network parameter of the common part; β and α are the unique parameters of the value function and the advantage function respectively, α is the learning rate;

[0118] The variable that controls the random exploration probability gradually decreases as the number of iterations increases. The update formula is:

[0119] ,

[0120] where, is the initial value of ε at the beginning of the iteration, represents the minimum value of ε, is the decay factor;

[0121] This method enables the transportation entity to conduct random exploration with a high probability in the initial stage of iteration, exploring all possible situations as much as possible. As the number of iterations increases, the transportation entity will choose the action with the maximum expected value with a higher probability.

[0122] S2.3. Establish the reward function of the agent. The reward function is the weighted QE index, which is expressed as:

[0123] ,

[0124] where, R is the reward function, , , , and respectively represent the weights corresponding to the freight cost index, transportation time index, carbon emission index, accident incidence rate index, and maximum load index, and the sum of the weights is 1;

[0125] S2.4. By constructing the state space, action space, and reward function, design the way for the agent to interact with the path set, that is, define the way for the agent to select the optimal path. The specific way is: utilize the network state , path optimization strategy and reward to iteratively update the action value function, aiming to maximize the expected reward value, and select the optimal path strategy under each path set state . The expression of the action value function is:

[0126] ,

[0127] where, Q represents the expected reward value, is the immediate reward value for selecting action under state , represents the maximum reward value for state to select each action, and γ is the discount factor, reflecting the importance of future rewards. In this embodiment, a deep reinforcement learning agent is constructed through the Dueling DQN algorithm. The calculation of the expected reward value Q is divided into a value function and an advantage function, and its expression is:

[0128] ,

[0129] where, s represents the state, a represents the action, θ is the path network parameter of the common part; β and α are the unique parameters of the value function and the advantage function respectively. V is the value function, representing the value that the state s itself has, and A is the advantage function, representing the outstanding value of selecting action a under state s, is an index variable that traverses all possible actions;

[0130] The improvement of Dueling DQN compared with the traditional DQN is that the calculation of the Q value is divided into a value function V(s; θ, β) and an advantage function A(s, a; θ, α), which respectively represent the value that the state s itself has and the outstanding value of selecting action a under state s, and the estimation bias of the Q value is reduced by centering the advantage function, improving the stability of the algorithm.

[0131] ​S3. The agent uses a combination of the Sum Tree priority sampling mechanism and the uniform sampling + reweighting sampling mechanism to select samples from the training samples and generate an optimal path. The specific method is as follows:

[0132] S3.1. The agent uses the Sum Tree priority sampling mechanism to select samples: Calculate the priority of each sample, and select a set number of samples in descending order of priority. The formula for calculating the priority is:

[0133] ,

[0134] where, i is the sample number, and each sample corresponds to a feasible path, p 1 is the priority, is the temporal difference error of the Sum Tree priority sampling mechanism;

[0135] The formula for calculating the temporal difference error is:

[0136] ,

[0137] where, is the reward value generated by the target neural network;

[0138] S3.2. The agent uses the uniform sampling + reweighting mechanism to select samples from the training samples: All feasible path samples are stored in a circular queue. Each time training, a set number of samples are randomly selected uniformly. The probability of each sample being selected is:

[0139] ,

[0140] where N is the total capacity of the sample pool, i is the sample number, p 2 is the probability of the sample being selected;

[0141] Then, for each selected sample, calculate its temporal difference error. The calculation formula is:

[0142] ,

[0143] where, is the temporal difference error of the uniform sampling + reweighting mechanism, is the path reward index value output by the target neural network, I is the current network estimated value, is the discount factor, is the sample i corresponding state, is the index variable for traversing all sample states, is an index variable for traversing all possible actions;

[0144] Then, re - weight the loss function. During the back - propagation process, dynamically adjust the loss weight according to the sample temporal difference error. The loss function is defined as:

[0145] ,

[0146] where, φ is the loss weight, M is the number of batch samples, is the smoothing constant, used to prevent zero - error samples from having no weight, is the priority strength coefficient, w represents the weight, j represents an index label that traverses all values;

[0147] S3.3. Combine the samples obtained in S3.1 and S3.2 to form a sample set.

[0148] During the intermodal transportation process, monitor the quality evaluation index of the path. Specifically, calculate the real - time reward value and compare it with the set reward value threshold. When the real - time reward value is less than the reward value threshold, it is determined that the quality evaluation index exceeds the threshold. When it exceeds the threshold, dynamically generate an optimal path according to the real - time path status data;

[0149] The expression for updating the optimal path is:

[0150] ,

[0151] where, represents function, p is the optimal path before update, P is the set of feasible paths, is the optimal path after update.

[0152] The above has described the present invention in detail in combination with the embodiments, but the content described is only the preferred embodiments of the present invention and cannot be considered as limiting the scope of implementation of the present invention. All equivalent changes and improvements made according to the scope of the present invention application should still fall within the scope covered by the patent of the present invention.

Claims

1. A multimodal transport path optimization method based on deep reinforcement learning, characterized in that: It includes the following steps: S1. Construct a multimodal transport path network topology model, abstract the path network into a graph structure, and generate multiple feasible paths from the starting point to the end point as training samples. Calculate the transportation time, freight cost, carbon emissions, accident rate, and maximum load index of each sub-path as the quality evaluation indicators of the corresponding feasible path; S2. Build a deep reinforcement learning agent based on the state space, action space, and reward function. The specific steps are as follows: S2.

1. Establish the state space of the agent. The state space includes two pieces of information: business request information and feasible path state matrix. The vector is represented as: ， in, Is a vector representing the current moment t The state of D is the business request information, including the starting point and the end point. TM The feasible path status matrix corresponding to the business request information includes freight cost index, transportation time index, carbon emission index, accident rate index and maximum load index; The feasible path state matrix is expressed as: ， in, d Indicates the transportation time, b Indicates the freight cost, l represents the carbon emission index, m represents the accident rate, n represents the maximum load, k is the number of feasible paths; S2.

2. Establish the action space of the agent. The action space is used to represent the set of feasible paths. ε - greed The strategy makes corresponding path decisions in the following ways: ， in, Current path network status The path optimization strategy made under Indicates action, x is a random number in the range [0,1], ε is a variable that controls the probability of random exploration with an initial value of 1, θ Path network parameters for the public part; β and α are the unique parameters of the value function and advantage function, α is the learning rate; S2.

3. Establish the agent's reward function, which is expressed as: ， Among them, R is the reward function, 、 、 、 and They represent the weights corresponding to the freight cost index, transportation time index, carbon emission index, accident rate index and maximum load index respectively, and the sum of the weights is 1; S2.

4. Define the way for the agent to select the optimal path, specifically: using the network status , path optimization strategy and rewards Iteratively update the action value function to maximize the expected reward value and select each path set state The optimal path strategy under , the expression of the action value function is: ， Among them, Q represents the expected reward value, In state Select Action The immediate reward value, Indicates status Select the maximum reward value for each action, γ is the discount factor, which reflects the importance of future rewards; S3. The agent selects samples from the training samples using a Sum Tree priority sampling mechanism combined with uniform sampling and reweighted sampling mechanisms. The specific method is as follows: S3.

1. The agent selects samples using the Sum Tree priority sampling mechanism: it calculates the priority of each sample and selects a set number of samples in descending order of priority. The priority calculation formula is: ， in, i is the sample number, each sample corresponds to a feasible path, For samples i Priority, For samples i The temporal difference error of the Sum Tree priority sampling mechanism; The calculation formula of the timing differential error is: ， in, The reward value generated by the target neural network; S3.

2. The agent selects samples from the training samples using a uniform sampling and reweighted sampling mechanism: all feasible path samples are stored in a circular queue, and a set number of samples are uniformly randomly selected during each training. The probability of each sample being selected is: ， in, N is the total capacity of the sample pool, i is the sample number, For samples i Probability of being selected; Then, for each selected sample, calculate its timing difference error. The calculation formula is: ， in, For samples i Timing difference error of uniform sampling and reweighted sampling mechanism, is the path reward estimate output by the target neural network, I is the current network estimate, is the discount factor, For samples i Corresponding status, is the index variable for traversing all sample states, is an index variable that traverses all possible actions; Then, the loss function is reweighted. During the back-propagation process, the loss weight is dynamically adjusted according to the sample time difference error. The loss function is defined as: ， in, is the loss weight, M is the number of batch samples, is a smoothing constant used to prevent zero error samples from having no weight, is the priority intensity coefficient, Representation sample i The weight of j Represents a traversal of all An index label for the value; S3.

3. Combine the samples obtained in S3.1 and S3.2 to form a sample set; During the intermodal transport process, the quality evaluation indicators of the routes are monitored. If the quality evaluation indicators exceed the threshold, the optimal route is dynamically updated based on the real-time route status data.

2. The multimodal transport path optimization method based on deep reinforcement learning according to claim 1, characterized in that: The transport time index of each path in S1 is calculated by the following formula: ， in, d Indicates the transportation time, Psd Indicates starting point s To the end d The feasible path, eij Represents adjacent path nodes in a feasible path i and j subpaths between; The freight cost index for each route is calculated using the following formula: ， in, b Indicates the freight cost; The carbon emission index of each path is calculated using the following formula: ， in, l represents the carbon emission index; The accident rate index of each path is calculated using the following formula: ， in, m represents the accident rate; The maximum load index of each path is calculated using the following formula: ， in, n Indicates the maximum load.

3. The multimodal transport route optimization method based on deep reinforcement learning according to claim 1, characterized in that: The elements in the feasible path state matrix are normalized by the Min-Max method, and the normalization formula is: ， in, x i is an element in the feasible path state matrix, corresponding to a certain transport time indicator, freight cost indicator, carbon emission indicator, accident rate indicator and maximum load indicator in the matrix. is the minimum value of the corresponding index, is the maximum value of the corresponding index, is the normalized index.

4. The multimodal transport route optimization method based on deep reinforcement learning according to claim 1, characterized in that: The variable controlling the random exploration probability gradually decreases as the number of iterations increases, and the update formula is: ， in, is the initial value of ε at the beginning of the iteration; represents the minimum value of ε; is the attenuation factor.

5. The multimodal transport route optimization method based on deep reinforcement learning according to claim 1, characterized in that: The deep reinforcement learning agent constructed in S2 adopts the Dueling DQN algorithm. The calculation of the expected reward value Q is divided into the value function and the advantage function, which are expressed as follows: ， Among them, s represents the state, Indicates action, θ Path network parameters for the public part; β and α They are the unique parameters of the value function and advantage function, V is the value function, which represents the value of the state s itself, and A is the advantage function, which represents the action selected under the state s. a The outstanding value of An index variable that iterates through all possible actions.

6. The multimodal transport route optimization method based on deep reinforcement learning according to claim 1, characterized in that: In S4, the real-time reward value is calculated and compared with the set reward value threshold. When the real-time reward value is less than the reward value threshold, it is determined that the quality evaluation index exceeds the threshold.

7. The multimodal transport route optimization method based on deep reinforcement learning according to claim 1, characterized in that: The expression for updating the optimal path in S4 is: ， in, express function, p is the optimal path before updating, P is the set of feasible paths, is the updated optimal path.

8. The multimodal transport route optimization method based on deep reinforcement learning according to claim 1, characterized in that: The feasible path in S1 includes the following constraints: Time window constraint: the total path time does not exceed the preset threshold ; Load constraint: cargo weight ; Risk Constraint: Accident Rate , in, eij Represents adjacent path nodes in a feasible path i and j The subpath between them, h is the cargo load threshold, Indicates the accident rate threshold.

Images