A port mixed traffic intersection automatic driving decision method and system based on deep reinforcement learning

Abstract

This invention discloses an autonomous driving decision-making method and system based on deep reinforcement learning at port mixed traffic intersections, relating to the field of autonomous driving technology. It includes: acquiring real-time state information of autonomous driving transportation equipment in a port mixed traffic intersection environment; heterogeneously fusing the real-time state information into a state vector for the autonomous driving transportation equipment; constructing a reinforcement learning decision-making model based on deep reinforcement learning (DDPG); building a port traffic environment simulation platform; training the DDPG-based reinforcement learning decision-making model; and embedding the trained DDPG-based reinforcement learning decision-making model into the onboard computing unit of the autonomous driving transportation equipment to control the operation of the autonomous driving transportation equipment. This invention can improve the traffic efficiency, stability, and safety of autonomous driving transportation equipment at port intersections, promote the unmanned transportation process in port mixed traffic scenarios, and reduce terminal renovation and operating costs.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to an autonomous driving decision-making method and system for mixed traffic intersections in ports based on deep reinforcement learning. Background Technology

[0002] Automated ports represent the forefront of modern logistics development, with automated guided vehicles (ARTs) playing a crucial role. However, internal port logistics hubs (such as intersections) are bottleneck areas for efficiency and safety, containing various traffic participants including unmanned automated guided vehicles (AGVs), manned internal trucks, manually driven forklifts, and pedestrians. This creates a typical mixed, unstructured, and highly dynamic complex environment, where traditional rule-based control methods lack flexibility and struggle to cope with uncertainty.

[0003] Currently, most commercial ports employ rule-based decision-making systems when ART (Automatic Tracking) passes through such mixed intersections. Its core logic involves pre-setting a series of "if-then" rules, such as "slow down and stop upon detecting a moving object ahead" or "wait for a fixed duration before the stop line." The advantages of this method are its simple logic, deterministic behavior, and ease of verification.

[0004] However, rule-based decision-making mechanisms have obvious limitations:

[0005] Low traffic efficiency: To ensure absolute safety, the rules are usually designed to be extremely conservative, resulting in frequent starts and stops or long waiting times for ARTs, making it impossible to take advantage of brief gaps in traffic flow to pass through intersections, thus becoming a bottleneck for the overall operational efficiency of the port.

[0006] Poor environmental adaptability and robustness: The rule system cannot effectively handle "long-tail scenarios" not covered by training data and highly dynamic interactions. For irrational behaviors of human-driven vehicles that do not follow the rules (such as cutting in front of others) or pedestrians suddenly entering the road, the system often has to adopt an emergency stop strategy, which may even lead to system "deadlock" and failure to complete the transportation task smoothly.

[0007] Insufficient intelligence: The rule system lacks the ability to learn and evolve, and cannot extract experience from massive amounts of actual operation data to optimize its own performance. The performance limit of the algorithm is fixed at the time of deployment and it is difficult to continuously improve.

[0008] Therefore, proposing an autonomous driving decision-making method and system based on deep reinforcement learning at port mixed traffic intersections to address the difficulties of existing technologies is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0009] In view of this, the present invention provides an autonomous driving decision-making method and system for port mixed traffic intersections based on deep reinforcement learning, which can improve the efficiency, stability and safety of autonomous driving transportation equipment at port intersections, promote the unmanned transportation process in port mixed traffic scenarios, and reduce terminal transformation and operation costs.

[0010] To achieve the above objectives, the present invention adopts the following technical solution:

[0011] An autonomous driving decision-making method for mixed traffic intersections in ports based on deep reinforcement learning includes:

[0012] S1. Obtain real-time status information of autonomous driving transportation equipment in the mixed traffic intersection environment of the port, and heterogeneously fuse the real-time status information into a state vector of autonomous driving transportation equipment.

[0013] S2. Using the DDPG algorithm as the reinforcement learning framework, construct a reinforcement learning decision model based on DDPG;

[0014] S3. Establish a port traffic environment simulation platform;

[0015] S4. In the port traffic environment simulation platform, the reinforcement learning decision model based on DDPG is trained to obtain the trained reinforcement learning decision model based on DDPG.

[0016] S5. The trained DDPG-based reinforcement learning decision model is embedded in the on-board computing unit of the autonomous driving transportation equipment. During real-time operation, the state vector of the autonomous driving transportation equipment is input into the trained DDPG-based reinforcement learning decision model to obtain action commands to control the driving of the autonomous driving transportation equipment.

[0017] Optionally, in the above method, in S1, the real-time state information of the autonomous transportation equipment in the mixed traffic intersection environment of the port is obtained, and the real-time state information is heterogeneously fused into a state vector of the autonomous transportation equipment, specifically as follows:

[0018] The traffic rules at the mixed traffic intersection of the port are encoded to form a rule semantic vector;

[0019] The relative position, relative speed, and heading angle difference between the obstacle and the autonomous driving transportation equipment are obtained, and the Euclidean distance between the obstacle and the autonomous driving transportation equipment is calculated. The relative position, relative speed, heading angle difference, and Euclidean distance between the obstacle and the autonomous driving transportation equipment are concatenated to form a dynamic environment feature vector.

[0020] The current speed, acceleration, position, heading angle, and position vector to the next target point of the autonomous driving transportation equipment are obtained to form the transportation equipment state vector;

[0021] Finally, the rule semantic vector, dynamic environment feature vector, and transportation equipment state vector are concatenated and standardized to form a complete state vector S. t .

[0022] Optionally, in S2 of the above method, a reinforcement learning decision model based on DDPG is constructed, specifically including:

[0023] S201, Actor Network Design:

[0024] The input to the Actor network is a state vector S formed by the rule semantic vector, the dynamic environment feature vector, and the transportation equipment state vector. t ;

[0025] The Actor network outputs a continuous action vector A. t The motion vector includes the desired acceleration and steering angular velocity;

[0026] The Actor network adopts a multi-layer fully connected structure, extracts features through 3 hidden layers, and uses the tanh function in the output layer to constrain the action value in the range of [-1, 1], and uses linear transformation to map to the desired acceleration and steering angular velocity range;

[0027] The Actor network is updated using gradient descent.

[0028]

[0029] in, For sampling batch size, Current state -Current Action The corresponding Q value;

[0030] Design of S202 and Critic networks:

[0031] The Critic network will use the state vector S t With the action vector A output by the Actor network t After concatenation, the input is used to extract the non-linear combination features of state and action through 3 fully connected hidden layers, and the output is the Q value of the current state-current action pair;

[0032] S203, Target Network Update:

[0033] The target Actor network is identical to the main Actor network, and the target Critic network is identical to the main Critic network. Both the target Actor network and the target Critic network employ a soft update strategy, as shown in the following formula:

[0034]

[0035] in, This is the soft update coefficient. For the target Actor network parameters, Main Actor network parameters, For the target Critic network parameters, The main Critic network parameters;

[0036] S204. Set up an experience replay mechanism:

[0037] Based on historical operational data, the sample (S) t A t ,r,S t+1 The samples are stored in the experience replay pool, and are assigned priority based on the absolute value of the temporal difference error, as shown in the following formula.

[0038]

[0039] in, For the first TD error for each sample For instant rewards, As a discount factor, Current state -Current Action The corresponding Q value, For the state at the next moment, For the next action, For the first The priority of each sample Prevent zero priority;

[0040] Combining the priority experience replay mechanism, the loss function of the Critic network is as follows:

[0041]

[0042] in, For sampling batch size, Assign weight parameters to the samples.

[0043] S205. Design reward functions, including: efficiency reward R_e, safety penalty R_s, progress reward R_p, comfort penalty R_c, and rule violation penalty R_r;

[0044] The reward function r(t) is a weighted sum of all reward components:

[0045] r(t=w1) R_e+w2 R_s+w3 R_p+w4 R_c+w5 R_r;

[0046] Among them, w1, w2, w3, w4, and w5 are the weight coefficients corresponding to each component, which are adjusted according to the design objectives before training.

[0047] Optionally, in S3, a port traffic environment simulation platform can be built using the methods described above. Specifically:

[0048] A digital twin environment for the port was built using the UE4 engine and the CARLA simulator.

[0049] Using SUMO, we can simulate the behavior of human drivers and other AGVs to generate complex interactive scenarios.

[0050] Optionally, in S4 of the above method, the reinforcement learning decision model based on DDPG is trained, specifically as follows:

[0051] S401. In the port traffic environment simulation platform, autonomous driving transportation equipment, as an intelligent agent, interacts with the environment in numerous rounds.

[0052] S402. Sample a batch of data from the experience replay pool according to priority, and use the Critic network to calculate the time-series difference error; minimize the loss function using gradient descent, clip the gradient, and update the Critic network parameters. ;

[0053] The Actor network is updated every 10 steps to optimize the policy function. The Critic network evaluates the Q-value in real time, calculates the policy gradient, and updates the Actor network parameters along the policy gradient direction. ;

[0054] S405. The target network adopts a soft update strategy, which stabilizes the training process by slowly and synchronously updating the target network. Training continues until the strategy performance converges.

[0055] An autonomous driving decision-making system for port mixed traffic intersections based on deep reinforcement learning, executing any one of the aforementioned autonomous driving decision-making methods for port mixed traffic intersections based on deep reinforcement learning, including:

[0056] The perception module, including LiDAR, camera, millimeter-wave radar, GPS / IMU, and V2X communication unit, transforms raw data into a state vector S that the decision-making module can understand through a multi-sensor fusion algorithm. t ;

[0057] The decision-making module deploys a pre-trained reinforcement learning decision-making model based on DDPG and receives the state vector S from the perception module. tOutput the current optimal decision action A. t This information is then transmitted to the downstream control module.

[0058] The control module includes a trajectory planner and a tracking controller. The trajectory planner refines the decision actions output by the decision module into a smooth, collision-free, and dynamically feasible trajectory. The tracking controller uses model predictive control or a PID controller to track the planned trajectory and generate throttle, brake, and steering control commands.

[0059] The simulation module, based on the UE4 engine and CARLA simulator, constructs a digital twin environment for the port; using SUMO, it simulates the behavior of human drivers and other AGVs to generate complex interactive scenarios.

[0060] The cloud platform completes the training process of the reinforcement learning decision model based on DDPG, and deploys the trained new model to the on-board computing unit of the autonomous driving transportation equipment wirelessly; through V2X communication, it realizes information interaction between vehicles, between vehicles and roads, and between vehicles and the cloud.

[0061] As can be seen from the above technical solution, compared with the prior art, the present invention provides an autonomous driving decision-making method and system for port mixed traffic intersections based on deep reinforcement learning, which has the following beneficial effects: Through learning, ART can actively seek safe gaps to pass through the intersection, avoiding unnecessary conservative waiting and significantly reducing the average passage time of the intersection; the deep reinforcement learning model can handle unseen dynamic scenarios and shows excellent prediction and decision-making capabilities for unstructured disturbances (such as mixed pedestrian and vehicle traffic); it enables ART to have the potential for continuous learning and performance optimization, and the decision-making process is based on a deep understanding of the scenario rather than rigid rules, significantly improving the level of intelligence; by embedding safety criteria into the learning objective through the safety and rule terms in the reward function, it ensures that the learned strategy strictly adheres to the safety bottom line while pursuing efficiency. Attached Figure Description

[0062] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0063] Figure 1 A flowchart of an autonomous driving decision-making method for mixed traffic intersections in ports based on deep reinforcement learning, provided by the present invention;

[0064] Figure 2 The diagram shows the structure of an autonomous driving decision-making system for a port mixed traffic intersection based on deep reinforcement learning, as provided by this invention. Detailed Implementation

[0065] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0066] In this application, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0067] Reference Figure 1 As shown, this invention discloses an autonomous driving decision-making method for mixed traffic intersections in ports based on deep reinforcement learning, comprising:

[0068] S1. Obtain real-time status information of autonomous driving transportation equipment in the mixed traffic intersection environment of the port, and heterogeneously fuse the real-time status information into a state vector of autonomous driving transportation equipment.

[0069] S2. Using the DDPG algorithm as the reinforcement learning framework, construct a reinforcement learning decision model based on DDPG;

[0070] S3. Establish a port traffic environment simulation platform;

[0071] S4. In the port traffic environment simulation platform, the reinforcement learning decision model based on DDPG is trained to obtain the trained reinforcement learning decision model based on DDPG.

[0072] S5. The trained DDPG-based reinforcement learning decision model is embedded in the on-board computing unit of the autonomous driving transportation equipment. During real-time operation, the state vector of the autonomous driving transportation equipment is input into the trained DDPG-based reinforcement learning decision model to obtain action commands to control the driving of the autonomous driving transportation equipment.

[0073] Furthermore, in S1, real-time status information of the autonomous transportation equipment in the mixed traffic intersection environment of the port is acquired, and the real-time status information is heterogeneously fused into a state vector of the autonomous transportation equipment, specifically as follows:

[0074] Traffic rules (such as stop signs, lane directional arrows, and speed limit signs) at mixed traffic intersections in ports are encoded using one-hot encoding or embedded encoding to form rule semantic vectors.

[0075] The vehicle-mounted sensors acquire the status of surrounding dynamic obstacles (vehicles, pedestrians), including their relative position, relative speed, and heading angle difference with the autonomous driving transportation equipment, and calculate their Euclidean distance with the autonomous driving transportation equipment. The status information of all obstacles is then stitched together into a dynamic environment feature vector.

[0076] The current speed, acceleration, position, heading angle, and position vector to the next target point of the autonomous driving transportation equipment are obtained to form the transportation equipment state vector;

[0077] Finally, the rule semantic vector, dynamic environment feature vector, and transportation equipment state vector are concatenated and standardized to form a complete state vector S. t .

[0078] Furthermore, in S2, a reinforcement learning decision model based on DDPG is constructed, specifically including:

[0079] S201, Actor Network Design:

[0080] The input to the Actor network is a state vector S formed by the rule semantic vector, the dynamic environment feature vector, and the transportation equipment state vector. t ;

[0081] The Actor network outputs a continuous action vector A. t The motion vector includes the desired acceleration and steering angular velocity;

[0082] The Actor network adopts a multi-layer fully connected structure, extracts features through 3 hidden layers, and uses the tanh function in the output layer to constrain the action value in the range of [-1, 1], and uses linear transformation to map to the desired acceleration and steering angular velocity range;

[0083] The Actor network is updated using gradient descent.

[0084]

[0085] in, For sampling batch size, Current state -Current Action The corresponding Q value;

[0086] Design of S202 and Critic networks:

[0087] The Critic network will use the state vector S t With the action vector A output by the Actor network t After concatenation, the input is used to extract the non-linear combination features of state and action through 3 fully connected hidden layers, and the output is the Q value of the current state-current action pair;

[0088] S203, Target Network Update:

[0089] The target network includes the target Actor network and the target Critic network, whose network parameters are periodically soft-updated from the online network to improve training stability;

[0090] The target Actor network is identical to the main Actor network, and the target Critic network is identical to the main Critic network. Both the target Actor network and the target Critic network employ a soft update strategy, as shown in the following formula:

[0091]

[0092] in, This is the soft update coefficient. For the target Actor network parameters, Main Actor network parameters, For the target Critic network parameters, The main Critic network parameters;

[0093] S204. Set up an experience replay mechanism:

[0094] Based on historical operational data, the empirical tuple samples (S) generated by the interaction between the agent and the environment are... t A t ,r,S t+1 The samples are stored in the experience replay pool, and are assigned priority based on the absolute value of the temporal difference error, as shown in the following formula.

[0095]

[0096] in, For the first TD error for each sample For instant rewards, As a discount factor, Current state -Current Action The corresponding Q value, For the state at the next moment, For the next action, For the first The priority of each sample Prevent zero priority;

[0097] Combining the priority experience replay mechanism, the loss function of the Critic network is as follows:

[0098]

[0099] in, For sampling batch size, Assign weight parameters to the samples.

[0100] S205. Design reward functions, including: efficiency reward R_e, safety penalty R_s, progress reward R_p, comfort penalty R_c, and rule violation penalty R_r;

[0101] Efficiency reward R_e: proportional to the distance the vehicle travels towards the target point per unit time;

[0102] Safety penalty R_s: When the distance to any obstacle is less than the safety threshold, a negative reward proportional to (safety threshold - distance to obstacle) is generated; if a collision occurs, a very large negative reward is generated and the current training round is terminated.

[0103] Progress reward R_p: A large positive reward is given when the ART successfully passes through an intersection or reaches a critical path point;

[0104] Comfort penalty R_c: Penalizes the absolute value of acceleration and jerk to encourage smooth driving;

[0105] Rule violation penalty R_r: When ART violates traffic rules (such as crossing the line or entering the wrong area), a negative reward is given;

[0106] The reward function is used to evaluate the quality of actions after each interaction, guiding the agent to learn.

[0107] The reward function r(t) is a weighted sum of all reward components:

[0108] r(t=w1) R_e+w2 R_s+w3 R_p+w4 R_c+w5 R_r;

[0109] Among them, w1, w2, w3, w4, and w5 are the weight coefficients corresponding to each component, which are adjusted according to the design objectives before training.

[0110] Furthermore, in S3, a port traffic environment simulation platform is built, specifically as follows:

[0111] Build a realistic virtual port environment based on UE4, Unity, or CARLA, including changes in weather and lighting;

[0112] Using SUMO or self-developed logic, simulate the behavior of human drivers and other AGVs (which can be rule-based or learning-based) to generate complex interactive scenarios;

[0113] Digital twins are used to model a real port at a 1:1 scale for testing and verification.

[0114] Furthermore, in S4, the reinforcement learning decision model based on DDPG is trained, specifically as follows:

[0115] S401. In the port traffic environment simulation platform, autonomous driving transportation equipment, as an intelligent agent, interacts with the environment in numerous rounds.

[0116] S402. Sample a batch of data from the experience replay pool according to priority, and use the Critic network to calculate the time-series difference error; minimize the loss function using gradient descent, clip the gradient, and update the Critic network parameters. ;

[0117] The Actor network is updated every 10 steps to optimize the policy function. The Critic network evaluates the Q-value in real time, calculates the policy gradient, and updates the Actor network parameters along the policy gradient direction. ;

[0118] S405. The target network adopts a soft update strategy, which stabilizes the training process by slowly and synchronously updating the target network. Training continues until the strategy performance converges.

[0119] Reference Figure 2 As shown, an autonomous driving decision-making system for port mixed traffic intersections based on deep reinforcement learning executes any one of the aforementioned autonomous driving decision-making methods for port mixed traffic intersections, including:

[0120] The perception module, including LiDAR, camera, millimeter-wave radar, GPS / IMU, and V2X communication unit, transforms raw data into a state vector S that the decision-making module can understand through a multi-sensor fusion algorithm. t ;

[0121] The decision-making module deploys a pre-trained reinforcement learning decision-making model based on DDPG and receives the state vector S from the perception module. t Output the current optimal decision action A. tThis information is then transmitted to the downstream control module.

[0122] The control module includes a trajectory planner and a tracking controller. The trajectory planner refines the decision actions output by the decision module into a smooth, collision-free, and dynamically feasible trajectory. The tracking controller uses model predictive control or a PID controller to track the planned trajectory and generate throttle, brake, and steering control commands.

[0123] The simulation module, based on the UE4 engine and CARLA simulator, constructs a digital twin environment for the port; using SUMO, it simulates the behavior of human drivers and other AGVs to generate complex interactive scenarios.

[0124] The cloud platform completes the training process of the reinforcement learning decision model based on DDPG, and deploys the trained new model to the on-board computing unit of the autonomous driving transportation equipment wirelessly; through V2X communication, it realizes information interaction between vehicles, between vehicles and roads, and between vehicles and the cloud, making up for perception blind spots and improving the accuracy and comprehensiveness of state St; the cloud-based intelligent agent can macroscopically optimize the traffic order at intersections and provide global information (such as the remaining time of the green light) to the vehicle-side DRL intelligent agent.

[0125] In one specific embodiment, firstly, a digital twin environment of the port is constructed based on the UE4 engine and CARLA simulator to simulate the real Rizhao Port container terminal intersection scenario; in the reward function, the safe distance is set to 5m, and w2 (safety weight) is much larger than other weights to ensure safety priority; during training, 16 parallel environment instances are used, and the SAC algorithm is used for approximately 5 million training steps; after training, the model is deployed to an onboard computing unit based on the NVIDIA DRIVE AGX Orin platform; when the vehicle is running at a real intersection, the perception module updates the state S at a frequency of 10Hz. t The decision module outputs action A at the same frequency. t Ultimately, the control module tracks and executes the commands.

[0126] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0127] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A decision-making method for autonomous driving at mixed traffic intersections in ports based on deep reinforcement learning, characterized in that, include: S1. Obtain real-time status information of autonomous transportation equipment in the mixed traffic intersection environment of the port, and heterogeneously fuse the real-time status information into a state vector of autonomous transportation equipment. S2. Using the DDPG algorithm as the reinforcement learning framework, construct a reinforcement learning decision model based on DDPG, specifically including: S201, Actor Network Design: The input to the Actor network is a state vector S formed by the rule semantic vector, the dynamic environment feature vector, and the transportation equipment state vector. t ; The Actor network outputs a continuous action vector A. t The motion vector includes the desired acceleration and steering angular velocity; The Actor network adopts a multi-layer fully connected structure, extracts features through 3 hidden layers, and uses the tanh function in the output layer to constrain the action value in the range of [-1, 1], and uses linear transformation to map to the desired acceleration and steering angular velocity range; The Actor network is updated using gradient descent. in, For sampling batch size, Current state -Current Action The corresponding Q value; Design of S202 and Critic networks: The Critic network will use the state vector S t With the action vector A output by the Actor network t After concatenation, the input is used to extract the non-linear combination features of state and action through 3 fully connected hidden layers, and the output is the Q value of the current state-current action pair; S203, Target Network Update: The target Actor network is identical to the main Actor network, and the target Critic network is identical to the main Critic network. Both the target Actor network and the target Critic network employ a soft update strategy, as shown in the following formula: in, This is the soft update coefficient. For the target Actor network parameters, Main Actor network parameters, For the target Critic network parameters, The main Critic network parameters; S204. Set up an experience replay mechanism: Based on historical operational data, the sample (S) t A t ,r,S t+1 The samples are stored in the experience replay pool, and are assigned priority based on the absolute value of the temporal difference error, as shown in the following formula. in, For the first TD error for each sample For instant rewards, As a discount factor, Current state -Current Action The corresponding Q value, For the state at the next moment, For the next action, For the first The priority of each sample Prevent zero priority; Combining the priority experience replay mechanism, the loss function of the Critic network is as follows: in, For sampling batch size, Assign weight parameters to the samples; S205. Design reward functions, including: efficiency reward R_e, safety penalty R_s, progress reward R_p, comfort penalty R_c, and rule violation penalty R_r; The reward function r(t) is a weighted sum of all reward components: r(t)=w1 R_e+w2 R_s+w3 R_p+w4 R_c+w5 R_r; Where w1, w2, w3, w4, and w5 are the weight coefficients corresponding to each component, which are adjusted according to the design objectives before training; S3. Establish a port traffic environment simulation platform, specifically: A digital twin environment for the port was built using the UE4 engine and the CARLA simulator. Using SUMO, we simulated the behavior of human drivers and other AGVs to generate complex interactive scenarios. S4. In the port traffic environment simulation platform, the reinforcement learning decision model based on DDPG is trained to obtain the trained reinforcement learning decision model based on DDPG. S5. The trained DDPG-based reinforcement learning decision model is embedded in the on-board computing unit of the autonomous driving transportation equipment. During real-time operation, the state vector of the autonomous driving transportation equipment is input into the trained DDPG-based reinforcement learning decision model to obtain action commands to control the driving of the autonomous driving transportation equipment.

2. The method for autonomous driving decision-making at port mixed traffic intersections based on deep reinforcement learning according to claim 1, characterized in that, In S1, real-time status information of the autonomous transportation equipment in the mixed traffic intersection environment of the port is obtained, and the real-time status information is heterogeneously fused into a state vector of the autonomous transportation equipment, specifically: The traffic rules at the port's mixed traffic intersections are encoded to form rule semantic vectors; The relative position, relative speed, and heading angle difference between the obstacle and the autonomous driving transportation equipment are obtained, and the Euclidean distance between the obstacle and the autonomous driving transportation equipment is calculated. The relative position, relative speed, heading angle difference, and Euclidean distance between the obstacle and the autonomous driving transportation equipment are concatenated to form a dynamic environment feature vector. The current speed, acceleration, position, heading angle, and position vector to the next target point of the autonomous driving transportation equipment are obtained to form the transportation equipment state vector; Finally, the rule semantic vector, dynamic environment feature vector, and transportation equipment state vector are concatenated and standardized to form a complete state vector S. t .

3. The method for autonomous driving decision-making at port mixed traffic intersections based on deep reinforcement learning according to claim 1, characterized in that, In S4, the reinforcement learning decision model based on DDPG is trained, specifically as follows: S401. In the port traffic environment simulation platform, autonomous driving transportation equipment, as an intelligent agent, interacts with the environment in numerous rounds. S402. Sample a batch of data from the experience replay pool according to priority, and use the Critic network to calculate the time difference error. The loss function is minimized using gradient descent, the gradient is clipped, and the Critic network parameters are updated. ; The Actor network is updated every 10 steps to optimize the policy function. The Critic network evaluates the Q-value in real time, calculates the policy gradient, and updates the Actor network parameters along the policy gradient direction. ; S405. The target network adopts a soft update strategy, which stabilizes the training process by slowly and synchronously updating the target network. Training continues until the strategy performance converges.

4. An autonomous driving decision-making system for port mixed traffic intersections based on deep reinforcement learning, executing the autonomous driving decision-making method for port mixed traffic intersections based on deep reinforcement learning as described in any one of claims 1-3, characterized in that, include: The perception module, including LiDAR, camera, millimeter-wave radar, GPS / IMU, and V2X communication unit, transforms raw data into a state vector S that the decision-making module can understand through a multi-sensor fusion algorithm. t ; The decision-making module deploys a pre-trained reinforcement learning decision-making model based on DDPG and receives the state vector S from the perception module. t Output the current optimal decision action A. t This information is then transmitted to the downstream control module. The control module includes a trajectory planner and a tracking controller. The trajectory planner refines the decision actions output by the decision module into a smooth, collision-free, and dynamically feasible trajectory. The tracking controller uses model predictive control or a PID controller to track the planned trajectory and generate throttle, brake, and steering control commands. The simulation module, based on the UE4 engine and CARLA simulator, constructs a digital twin environment for the port; using SUMO, it simulates the behavior of human drivers and other AGVs to generate complex interactive scenarios. The cloud platform completes the training process of the reinforcement learning decision model based on DDPG, and deploys the trained new model to the on-board computing unit of the autonomous driving transportation equipment wirelessly; through V2X communication, it realizes information interaction between vehicles, between vehicles and roads, and between vehicles and the cloud.

Images