A Method and System for Predicting the Sailing Time of Mud Transport Vessels Based on Reinforcement Learning and Multi-Modal Fusion

Abstract

This invention provides a method and system for predicting the sailing time of mud transport vessels based on reinforcement learning and multi-mode fusion. The method includes: integrating multi-source feature data using an attention mechanism and extracting the correlations between features to fully describe and construct a comprehensive state space encompassing internal and external factors; representing vessel features through vector embedding and calculating and representing tidal conditions by combining prior tidal-time physics principles; learning correlations among various data representation results using a self-attention mechanism, with the final output vector describing the current environment; and employing a deep deterministic policy gradient algorithm to output sailing behavior, including whether to wait, optimal speed, and route selection. This invention comprehensively considers the ease of use and theoretical completeness of the algorithm, using the negative value of the distance between the predicted and actual positions at the same time as the reward function to ensure that the model can learn stable sailing patterns over a relatively long period.

Description

Technical Field

[0001] This invention relates to the field of intelligent shipping technology for inland waterways, specifically to a method and system for predicting the sailing time of mud transport vessels based on reinforcement learning and multi-modal fusion. Background Technology

[0002] Inland waterways, as vital waterway infrastructure, serve multiple functions, including cargo transport and flood control. Dredging is a core means of improving the navigation capacity of inland waterways. However, the scheduling of dredging vessels directly impacts project efficiency and cost control, especially in complex inland waterway environments, where accurately predicting the sailing time of dredging vessels has become a pressing technical challenge.

[0003] Currently, research on ship navigation time prediction mainly relies on historical trajectory data analysis (such as AIS data) or statistical model-based methods. For example, by clustering analysis of a large number of historical navigation records, typical path patterns are extracted and time is estimated in combination with current environmental conditions. However, these methods have the following shortcomings: (1) Static modeling problem: Traditional methods usually assume that environmental conditions (such as flow velocity and channel status) remain unchanged in a short period of time, while inland waterways are greatly affected by factors such as tidal cycles and seasonal hydrological changes, and are highly dynamic. (2) Neglecting multi-source feature fusion: Existing studies rarely take into account multi-dimensional information such as tidal flow velocity, ship behavior characteristics and channel constraints, resulting in limited prediction accuracy. (3) Lack of real-time optimization capability: Traditional methods are mostly offline analysis, which makes it difficult to respond to environmental changes in real time (such as sudden changes in water flow direction or channel construction restrictions), and cannot meet the needs of dynamic scheduling in dredging operations.

[0004] Tidal current velocity is a significant external factor influencing inland waterway shipping, especially in channel areas near estuaries. The periodic changes in tides significantly alter local water flow speed and direction, thus changing the optimal navigation mode for dredgers. For example, sailing downstream during high tide can drastically shorten travel time. Furthermore, the navigation behavior of dredgers in dredging scenarios is highly complex and uncertain. On one hand, different ship operators' operating habits (such as acceleration / deceleration frequency and avoidance strategies) lead to differences in travel time; on the other hand, dredging operations require dredgers to frequently travel between dredging and unloading areas, resulting in certain regularities in ship behavior. However, existing research has not fully explored these behavioral characteristics to optimize travel time prediction.

[0005] In recent years, reinforcement learning, as an emerging artificial intelligence algorithm for solving complex planning problems, has shown broad application prospects in the field of intelligent shipping. Decision models trained based on reinforcement learning can infer navigation conditions based on real-time environmental feedback (such as tidal current speed and channel conditions), achieving accurate prediction of navigation time. However, the application of reinforcement learning in inland waterway dredging scenarios is still in the exploratory stage, especially in the lack of mature solutions for fusing multi-source features.

[0006] Patent application CN116307069A describes a method for predicting ship navigation time based on multi-task learning. The method includes: acquiring historical AIS data; processing the historical AIS data into historical traffic flow and navigation time data for each time period of each waterway according to the divided waterways, and collecting weather data for each waterway in the future; inputting the historical traffic flow, navigation time data, and weather data into the traffic flow prediction sub-model and the navigation time prediction sub-model for training; constructing an overall model framework; training the overall model framework and outputting the final navigation time for each waterway in the future; and calculating the total navigation time of the entire route using a rolling prediction method. However, this patent cannot completely solve the existing technical problems, nor can it meet the needs of this invention. Summary of the Invention

[0007] To address the shortcomings of existing technologies, the purpose of this invention is to provide a method and system for predicting the sailing time of mud transport vessels based on reinforcement learning and multi-modal fusion.

[0008] The method for predicting the sailing time of mud transport vessels based on reinforcement learning and multi-modal fusion provided by the present invention includes:

[0009] Step 1: Integrate environmental data, ship characteristics, and waterway constraints, extract the correlation between multi-source features through an attention mechanism, and construct a comprehensive state space;

[0010] The environmental data includes daily tide times and inland river environmental characteristics; the vessel characteristics include vessel ID and latitude / longitude location; the waterway constraints include navigation direction.

[0011] Step 2: The Deep Deterministic Policy Gradient (DDPG) algorithm is used to output navigation behavior decisions based on the comprehensive state space, including speed selection and path planning. The DDPG algorithm optimizes navigation decisions through a reward function to minimize the error between the projected position and the actual position, and uses an experience replay mechanism to improve training stability.

[0012] Preferably, step 1 includes: calculating the daily high and low tide times based on the lunar calendar date to generate a tidal state vector; decomposing the timestamp into four dimensions: month, day, week, and hour, embedding them into vectors respectively, and concatenating them into a time feature vector; representing the ship ID through vector embedding to generate a ship feature vector; mapping the latitude and longitude location information into a location feature vector through linear transformation; and using a self-attention mechanism to fuse the tidal state vector, time feature vector, ship feature vector, and location feature vector to generate a comprehensive state vector.

[0013] Preferably, the reward function is defined as the negative of the square of the channel distance between the projected position and the actual position, and the calculation formula is as follows:

[0014] r = -d 2 ((lat t ,lon t ),(lat t+1 ,lon t+1 ))

[0015] Where r is the reward function, d is the distance calculation function, and lat t lon t Latitude and longitude at the current time, lat t+1 lon t+1 To deduce the latitude and longitude of the location.

[0016] Preferably, the DDPG algorithm includes an Actor network and a Critic network:

[0017] The Actor network takes a composite state vector as input and outputs continuous speed values, where negative values ​​indicate waiting.

[0018] The Critic network evaluates the cumulative reward expectation of state-action pairs and updates parameters by minimizing the mean square error between the target Q-value and the predicted Q-value.

[0019] The target network parameters are gradually approximated to the main network parameters through a soft update method, and the update formula is:

[0020]

[0021] Where τ is the update rate, and These represent the parameters of the main network and the target network, respectively.

[0022] Preferably, the policy gradient optimization formula of the Actor network is:

[0023]

[0024] in, This represents the gradient of the Critic network Q with respect to action a. denoted as the gradient of the Actor network μ with respect to parameter θ; s is the state vector at a certain moment; v is the speed decision result of the corresponding Actor network; E is the expected cumulative reward of the Critic network output in the future.

[0025] The mud transport vessel sailing time prediction system based on reinforcement learning and multi-modal fusion provided by the present invention includes:

[0026] Module M1: Integrates environmental data, ship characteristics, and waterway constraints, extracts the correlation between multi-source features through an attention mechanism, and constructs a comprehensive state space;

[0027] The environmental data includes daily tide times and inland river environmental characteristics; the vessel characteristics include vessel ID and latitude / longitude location; the waterway constraints include navigation direction.

[0028] Module M2: Employs the Deep Deterministic Policy Gradient (DDPG) algorithm to output navigation behavior decisions based on the integrated state space, including speed selection and path planning. The DDPG algorithm optimizes navigation decisions through a reward function to minimize the error between the projected position and the actual position, and uses an experience replay mechanism to improve training stability.

[0029] Preferably, module M1 includes: calculating the daily high and low tide times based on the lunar calendar date to generate a tidal state vector; decomposing the timestamp into four dimensions: month, day, week, and hour, embedding them into vectors respectively, and concatenating them into a time feature vector; representing the ship ID through vector embedding to generate a ship feature vector; mapping the latitude and longitude location information into a location feature vector through linear transformation; and using a self-attention mechanism to fuse the tidal state vector, time feature vector, ship feature vector, and location feature vector to generate a comprehensive state vector.

[0030] Preferably, the reward function is defined as the negative of the square of the channel distance between the projected position and the actual position, and the calculation formula is as follows:

[0031] r = -d 2 ((lat t ,lon t ),(lat t+1 ,lon t+1 ))

[0032] Where r is the reward function, d is the distance calculation function, and lat t lon t Latitude and longitude at the current time, lat t+1 lon t+1 To deduce the latitude and longitude of the location.

[0033] Preferably, the DDPG algorithm includes an Actor network and a Critic network:

[0034] The Actor network takes a composite state vector as input and outputs continuous speed values, where negative values ​​indicate waiting.

[0035] The Critic network evaluates the cumulative reward expectation of state-action pairs and updates parameters by minimizing the mean square error between the target Q-value and the predicted Q-value.

[0036] The target network parameters are gradually approximated to the main network parameters through a soft update method, and the update formula is:

[0037]

[0038] Where τ is the update rate, and These represent the parameters of the main network and the target network, respectively.

[0039] Preferably, the policy gradient optimization formula of the Actor network is:

[0040]

[0041] in, This represents the gradient of the Critic network Q with respect to action a. denoted as the gradient of the Actor network μ with respect to parameter θ; s is the state vector at a certain moment; v is the speed decision result of the corresponding Actor network; E is the expected cumulative reward of the Critic network output in the future.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] (1) Higher prediction accuracy: By integrating multi-dimensional information such as tidal current speed, ship behavior patterns and channel constraints, the accuracy of navigation time prediction is significantly improved, and the prediction error of navigation time on multiple tracks can be reduced by more than 20%.

[0044] (2) Effective multi-source data representation capability: Features of various data types are extracted through prior knowledge embedding and vector embedding, and the correlation is learned based on the self-attention mechanism, enabling the model to fully perceive the state space of navigation and generate effective representation vectors.

[0045] (3) Flexible dynamic adaptability: By using the DDPG reinforcement learning algorithm, the reward function is designed to ensure the consistency between the agent's behavior and the actual navigation behavior. Ultimately, the model can perceive the navigation status, position, tides and other features in real time and make the correct navigation decisions accordingly, avoiding the limitations of traditional static modeling methods. Attached Figure Description

[0046] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0047] Figure 1 This forms the overall framework of the present invention;

[0048] Figure 2 This is a schematic diagram of a multi-source data fusion sensing module.

[0049] Figure 3 This is a schematic diagram of the navigation mode capture module. Detailed Implementation

[0050] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0051] Example

[0052] This invention proposes a navigation time prediction algorithm for inland waterway mud transport vessels based on reinforcement learning and multi-modal feature fusion. It aims to address the shortcomings of existing technologies in terms of dynamic environment interactivity, thereby improving the rationality of navigation process extrapolation and ultimately enhancing the accuracy of navigation time prediction. The framework of this invention is as follows: Figure 1 As shown, it mainly consists of two parts: a state-space representation based on multi-source input fusion and a reinforcement learning neural network for capturing and learning navigation patterns. Together, they construct a closed-loop planning and deduction framework of "environmental perception - navigation behavior decision-decision feedback", enabling the invention to perceive complex environments in real time and simulate navigation modes under real-world conditions, thereby achieving high-precision prediction of navigation time based on the deduction results.

[0053] Specifically Figure 1 The left half is the multi-modal feature fusion module. It uses an attention mechanism to integrate multi-source heterogeneous data (including environmental data, ship features, and waterway constraints) to fully explore the correlation between various features and form an effective state-space representation. Figure 1The right half is the reinforcement learning decision-making module, which uses the Deep Deterministic Policy Gradient (DDPG) algorithm to train the model. First, the state-space representation is input into the Actor network, and the model outputs the distribution of navigation behavior decision results. Then, the state-space representation vector and the Monte Carlo sampling-based navigation behavior decision results are input into the Critic network, which outputs a Q-value. Simultaneously, the model is trained to ensure the Actor network outputs decisions with the largest possible Q-values, and the Critic network outputs a consistent and stable Q-value. Finally, using the trained model, environmental information is input, and the navigation behavior output by the model is used to extrapolate the prediction of the navigation time.

[0054] Multi-source data fusion sensing module

[0055] like Figure 2 The multi-source feature fusion module of this invention aims to efficiently fuse multi-source features through embedded representations and self-attention mechanisms to generate a comprehensive vector representation that fully reflects the current environmental state. This module specifically includes the following steps: feature embedding and extraction based on prior knowledge, feature fusion based on attention mechanisms, and the construction of the final state space.

[0056] (1) Feature embedding and extraction:

[0057] To unify different types of features into the same dimensional space, different representation methods are used to process various types of input data and align them into vectors of the same dimension for subsequent construction of relationships.

[0058] First, daily tides are influenced by the gravitational pull of the Earth and the Moon. The tides are highest when the Moon, Earth, and Sun align in a straight line, and lowest when they form a right angle. To improve the model's usability, this invention abandons the method of estimating tides by consulting tide tables, and instead directly uses the lunar calendar date to estimate the tidal conditions for that day. The corresponding method is as follows:

[0059] t u = (Moon(Date)%15-3)×0.8

[0060] Among them, t u This indicates the high tide time, Date represents the current date, and Moon(·) converts the input date to the corresponding lunar calendar date. Additionally, the low tide time t... d This usually occurs 6 hours after high tide, that is:

[0061] t d =t u +6

[0062] Based on this method, by inputting the current timestamp t, the current high or low tide state can be calculated. The calculation method is as follows:

[0063]

[0064] Among them, s tide is a vector with all internal elements being learnable parameters. The above process represents the flood state with s tide and represents the ebb state with -s tide .

[0065] Since the navigation behavior is affected by the specific time at the same time, for example, whether it is working hours currently, the state space needs to include specific timestamp information. The implementation method is to split the current timestamp t into month t month , day t day , week t week , hour t hour in four dimensions. Embed the four time dimensions into four vectors respectively and splice them together, then there is:

[0066]

[0067] Among them, s time is the state representation vector generated according to the timestamp information.

[0068] In addition, since the drivers of different ships are different, resulting in different corresponding navigation habits, the model needs to distinguish different ships. The present invention uses the ship ID as the input and realizes the distinction of the ship's own characteristics through the vector embedding method:

[0069] s ship = Embedding(ID ship )

[0070] Among them, s ship represents the representation result of the ship's own characteristics.

[0071] Finally, for the latitude and longitude information, since it is a continuous value and the values of latitude and longitude can effectively reflect the distance from the current position to the ocean, it is necessary to directly map the two-dimensional position information to the target dimension through a linear transformation:

[0072] s pos = W pose (lat, lon) T + b pose

[0073] Among them, s pos represents the feature extraction result obtained according to the latitude and longitude position information; lat and lon are latitude and longitude; W pose is the weight matrix; b pose is the bias vector.

[0074] (2) Feature fusion based on attention mechanism

[0075] To capture the complex relationships between multi-source features, a self-attention mechanism is introduced for feature fusion. The objects of feature fusion include tidal states s. tide The concatenated timestamp vector s time The embedding vector s of the ship ID ship and the mapping vector s of location information pos First, the vectors are transformed linearly to obtain the query vector, key vector, and value vector. Then, the attention weight matrix is ​​calculated.

[0076]

[0077] Here, d represents the dimension of the key vector. Then, based on the output of the attention mechanism, a nonlinear mapping is achieved through a multilayer perceptron with residual connections, and layer normalization is used to stabilize the training process.

[0078] (3) State space construction

[0079] After fusion via a self-attention mechanism, the output vectors of the last layer at each position are concatenated and passed through a linear layer to generate a comprehensive state vector, which describes the overall state of the current environment.

[0080] s = W s concat(s tide ,s time ,s ship ,s pos )+b s

[0081] Among them, W s Let b be the weight matrix. s This is the bias vector. This vector contains three types of information: the current tidal state, the current timestamp, and the ship's own characteristics (ID, current position). Ultimately, this state vector will serve as input to a reinforcement learning algorithm to extrapolate navigation time and optimize decision-making.

[0082] The aforementioned multi-feature fusion module achieves the following technical effects: Feature association mining: It effectively captures the complex relationships between multi-source features by utilizing the self-attention mechanism, thereby improving the feature representation capability; State space integrity: It comprehensively considers features such as environment and ships, and constructs a vector representation that fully reflects the current environmental state; Dynamic adaptability: It combines prior knowledge to infer the tidal state, thereby enhancing the model's adaptability to dynamic environments.

[0083] Reinforcement Learning-Based Flight Pattern Extraction Module

[0084] Most existing flight time prediction algorithms are based on an end-to-end training model, taking various features as input and outputting an estimate of the flight time. These methods perform well in static environments; however, actual flight processes are dynamic, and the corresponding flight behavior decisions change dynamically with the current state. While this doesn't mean end-to-end models are ineffective in such scenarios, their lack of flexibility inevitably leads to unstable prediction results and a decline in long-term average accuracy.

[0085] (1) Problem background and basic settings

[0086] like Figure 3 First, in this scenario, as mentioned above, the state space S is the current navigation state and the surrounding environment, including ship characteristics, tidal conditions, position information and timestamp information.

[0087] In addition, the behavior space A in this scenario corresponds to the following two aspects: whether to wait and the sailing speed v. Since the ship's speed is necessarily 0 when it is in a waiting state, and the sailing direction is determined (going back and forth between the dredging point and the dumping point), in order to ensure the robustness of the model and that the output results can be parsed, this invention sets the behavior space only to the sailing speed, and defines that when the sailing speed is less than or equal to 0, it indicates that the ship is in a waiting behavior.

[0088] Meanwhile, the decision-making time interval for navigation behavior is one hour. Once the navigation behavior is determined at the previous moment, the mud-carrying vessel will travel at a constant speed along the channel at the output speed for the next hour. Therefore, based on the channel map information, the position of the vessel one hour later can be deduced according to the above principle. This deduction process is the state transition function:

[0089] s t+1 =f(s) t ,v t )

[0090] Where f() represents the state change pattern, that is, based on the current position and the sailing speed v t Predicting the position of the waterway at the next moment, s t Let s represent the state-space vector at time t. t+1 This represents the state space vector at the next moment.

[0091] Finally, unlike most planning problems, the ultimate goal of the reinforcement learning model established in this invention is not to obtain the optimal navigation behavior decision, but rather to obtain a navigation decision that most closely resembles human behavior. Therefore, the reward function is set to the negative square of the distance between the projected position and the actual position at the same time (i.e., the smaller the distance, the larger the reward), encouraging the model to output a strategy that closely approximates real navigation behavior, while penalizing projected results that deviate significantly from the actual position.

[0092] r = -d 2 ((lat t ,lon t ),(lat t+1 ,lon t+1 ))

[0093] Where r is the reward function and d is the distance calculation function, the present invention uses the waterway distance, that is, the total distance required to travel between two locations along the river.

[0094] (2) Model building process

[0095] Since the output navigation speed, as a behavior space, is a continuous value in the navigation scenario, this invention employs the Deep Deterministic Policy Gradient (DDPG) algorithm. DDPG is a reinforcement learning algorithm that combines value functions and policy optimization, suitable for navigation decision-making problems in continuous action spaces. The DDPG algorithm is an improvement on the Deep Q-Learning algorithm (DQN), specifically designed to solve planning problems in continuous state and behavior spaces. Therefore, the DDPG algorithm is still a special case of the basic reinforcement learning algorithm Q-Learning.

[0096] The core idea of ​​the DDPG algorithm is to combine the Actor-Critic framework, which consists of two models: an Actor network and a Critic network. The Actor network is used to define the policy function, while the Critic network is used to fit the value function, and the policy performance is improved through alternating optimization.

[0097] The policy function μ (in the Actor network) is a deterministic mapping that represents the state s given the state μ. t The optimal sailing speed v t Its mathematical expression is:

[0098] v t =μ(s) t |θ)

[0099] Here, θ represents the parameters of the Actor network. In this scenario, the policy function outputs a continuous value representing the sailing speed, where a negative value indicates no sailing and waiting in place.

[0100] The value function (Critic network) is used to evaluate the state-action pair (s) at a given time. t ,v t The quality of the Critic network is given by the input state vector s at a certain moment and the corresponding speed decision result v of the Actor network. The Critic network outputs the expected cumulative reward in the future, which is mathematically expressed as:

[0101]

[0102] Where γ∈[0,1] is the discount factor, r t It is the instant reward at time t.

[0103] The Critic network minimizes the difference between the target Q-value and the predicted Q-value using the mean squared error loss function.

[0104]

[0105] Where y represents the target Q value, it can be defined as:

[0106]

[0107] Here, s' and v' represent the state and behavior (sailing speed) at the next moment, respectively, and Q′ represents the target Critic network. This represents the parameters of the target network. The model architecture and initial parameters of Q′ are exactly the same as Q, but its parameters are gradually approximated to the main network parameters using a soft update method. That is, after training on one or more trajectories, the parameters of Q′ are set to the parameters of Q:

[0108]

[0109] Where τ is the update rate (in this invention, the value is 0.01). and These represent the parameters of the main network and the target network, respectively.

[0110] In this invention, the goal of optimizing the strategy function is to maximize the value function. Using the chain rule, the policy gradient formula can be expressed as:

[0111]

[0112] in, This represents the gradient of the Critic network with respect to action a. This represents the gradient of the Actor network with respect to the parameter θ. Using the above policy gradient formula, the Actor network can progressively optimize the policy parameters based on feedback from the Critic network.

[0113] To improve sample utilization and reduce data correlation, this algorithm employs an experience replay mechanism. The experience (s) generated in each interaction... t ,v t ,r t ,s t+1 The data is stored in the experience replay buffer and a batch of experiences is randomly sampled during training for batch updates.

[0114] Algorithm Overall Flow

[0115] Based on the above theoretical modeling process, the overall process of this invention is as follows. First, initialize the parameters θ and φ of the Actor network μ and the Critic network Q. Initialize the target network μ′ and Q′, and initialize the parameters of the target network to the parameters of the main network, clearing the experience buffer. Then, based on information such as timestamps, ship IDs, and locations, a representation of the state is generated using the multi-feature fusion module, and the optimal policy v, as perceived by the current model, is obtained through the Actor network. t =μ(s) t |θ)+ε, where ε represents the exploration noise. It follows a normal distribution with a mean of 0 and a standard deviation of 5. The overall quality of the "state-velocity" pair is evaluated using a Critic network, and the interaction experience (s) is recorded. t ,v t ,r t ,s t+1 ) and store it in a buffer. Next, randomly sample a batch of experience (s) from the buffer. t ,v t ,r t ,s t+1 By minimizing the loss function based on negative distance error, Update the parameters of the Critic network The Actor network parameters θ are updated using the policy gradient formula. Finally, the target network parameters are updated periodically using a soft update method, and the model parameters are periodically saved for subsequent inference and deployment.

[0116] During the inference phase, the simulation inputs state information into the model every hour, and the model outputs the current speed. Then, based on the current speed, it infers the ship's travel distance and position one hour later. The new position, along with the corresponding time and tidal information, is then input into the model again to make the next navigation decision. This process continues until the ship reaches the target location (dredging site, mud blowing site), and the total time spent by the ship during the simulation is returned as the predicted travel time.

[0117] Effect verification

[0118] This model is programmed in Python 3, utilizes the PyTorch interface to build a neural network structure, and runs on an NVIDIA A6000 GPU. The invention was validated on 14 real-world sampled ship trajectories. All trajectories were divided into two segments in a 7:3 ratio based on timestamps. The first segment was used to train the reinforcement learning model, and the second segment was used to test the accuracy of the voyage time prediction. Experimental results show that the invention can control the average error of voyage time prediction to approximately 2.3 hours. Compared to traditional prediction methods based on "distance-average speed," the prediction error is reduced by about 50%, demonstrating the invention's high adaptability to dynamically changing navigation environments and the stability of its prediction performance.

[0119] This invention achieves accurate prediction and dynamic optimization of the navigation time of inland waterway dredging vessels by organically combining multi-source feature fusion modeling with reinforcement learning algorithms. The multi-source feature fusion module fully utilizes environmental data, vessel characteristics, and waterway constraints to construct a comprehensive state space; the reinforcement learning module outputs the optimal navigation strategy through a deep deterministic policy gradient algorithm, ensuring the model's efficient operation in complex dynamic environments. This design concept and framework significantly improves the rationality and accuracy of navigation time prediction, providing intelligent technical support for inland waterway dredging projects.

[0120] Those skilled in the art will understand that, in addition to implementing the system, apparatus, and their modules provided by this invention in purely computer-readable program code, the same program can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system, apparatus, and their modules provided by this invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; alternatively, modules for implementing various functions can be considered both software programs implementing the method and structures within the hardware component.

[0121] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A method for predicting the sailing time of mud transport vessels based on reinforcement learning and multi-modal fusion, characterized in that, include: Step 1: Integrate environmental data, ship characteristics, and waterway constraints, extract the correlation between multi-source features through an attention mechanism, and construct a comprehensive state space; The environmental data includes daily tide times and inland river environmental characteristics; the vessel characteristics include vessel ID and latitude / longitude location; the waterway constraints include navigation direction. Step 2: The Deep Deterministic Policy Gradient (DDPG) algorithm is used to output navigation behavior decisions based on the comprehensive state space, including speed selection and path planning. The DDPG algorithm optimizes navigation decisions through a reward function to minimize the error between the projected position and the actual position, and uses an experience replay mechanism to improve training stability. Step 1 includes: calculating the daily high and low tide times based on the lunar calendar date to generate a tidal state vector; decomposing the timestamp into four dimensions: month, day, week, and hour, embedding them into vectors respectively, and concatenating them into a time feature vector; representing the ship ID through vector embedding to generate a ship feature vector; mapping the latitude and longitude location information into a location feature vector through linear transformation; and using a self-attention mechanism to fuse the tidal state vector, time feature vector, ship feature vector, and location feature vector to generate a comprehensive state vector. The reward function is defined as the negative of the square of the channel distance between the projected position and the actual position, and is calculated using the following formula: r=-d 2 ((years t lon t ),(years t+1 lon t+1 )) Where r is the reward function, d is the distance calculation function, and lat t lon t Latitude and longitude at the current time, lat t+1 lon t+1 To determine the location's latitude and longitude; The DDPG algorithm includes an Actor network and a Critic network: The Actor network takes a composite state vector as input and outputs continuous speed values, where negative values ​​indicate waiting. The Critic network evaluates the cumulative reward expectation of state-action pairs and updates parameters by minimizing the mean square error between the target Q-value and the predicted Q-value. The target network parameters are gradually approximated to the main network parameters through a soft update method, and the update formula is: Where τ is the update rate, and These represent the parameters of the main network and the target network, respectively. The policy gradient optimization formula for the Actor network is: in, This represents the gradient of the Critic network Q with respect to action a. denoted as the gradient of the Actor network μ with respect to parameter θ; s is the state vector at a certain moment; v is the speed decision result of the corresponding Actor network; E is the expected cumulative reward of the Critic network output in the future.

2. A system for predicting the sailing time of mud transport vessels based on reinforcement learning and multi-modal fusion, characterized in that, include: Module M1: Integrates environmental data, ship characteristics, and waterway constraints, extracts the correlation between multi-source features through an attention mechanism, and constructs a comprehensive state space; The environmental data includes daily tide times and inland river environmental characteristics; the vessel characteristics include vessel ID and latitude / longitude location; the waterway constraints include navigation direction. Module M2: Employs the Deep Deterministic Policy Gradient (DDPG) algorithm to output navigation behavior decisions based on the integrated state space, including speed selection and path planning. The DDPG algorithm optimizes navigation decisions through a reward function to minimize the error between the projected position and the actual position, and uses an experience replay mechanism to improve training stability. The module M1 includes: calculating the daily high and low tide times based on the lunar calendar date to generate a tidal state vector; decomposing the timestamp into four dimensions: month, day, week, and hour, embedding them into vectors respectively, and concatenating them into a time feature vector; representing the ship ID through vector embedding to generate a ship feature vector; mapping latitude and longitude location information into a position feature vector through linear transformation; and using a self-attention mechanism to fuse the tidal state vector, time feature vector, ship feature vector, and position feature vector to generate a comprehensive state vector. The reward function is defined as the negative of the square of the channel distance between the projected position and the actual position, and is calculated using the following formula: r=-d 2 ((years t lon t ),(years t+1 lon t+1 )) Where r is the reward function, d is the distance calculation function, and lat t lon t Latitude and longitude at the current time, lat t+1 lon t+1 To determine the location's latitude and longitude; The DDPG algorithm includes an Actor network and a Critic network: The Actor network takes a composite state vector as input and outputs continuous speed values, where negative values ​​indicate waiting. The Critic network evaluates the cumulative reward expectation of state-action pairs and updates parameters by minimizing the mean square error between the target Q-value and the predicted Q-value. The target network parameters are gradually approximated to the main network parameters through a soft update method, and the update formula is: Where τ is the update rate, and These represent the parameters of the main network and the target network, respectively. The policy gradient optimization formula for the Actor network is: in, This represents the gradient of the Critic network Q with respect to action a. denoted as the gradient of the Actor network μ with respect to parameter θ; s is the state vector at a certain moment; v is the speed decision result of the corresponding Actor network; E is the expected cumulative reward of the Critic network output in the future.

Images